Abstract

AI in medicine is a promising research field. However, translating the AI algorithms of softwares into medical practice face great obstacles. A significant reason is that the intricate interactions between human and AI still remain undiscovered, and thus their effectiveness under the collaborative decision-making scenario is unknown. Openly accessible datasets play a crucial role in exploring the intricate interactions between AI models and human clinicians and facilitating the development of AI in medicine. However, there still has no publicly available clinical data on the diagnosis effects and behaviors under the context of collaborative decision-making. This project addresses these challenges and provides the first database about the interaction of AI models and human clinicians on 7500 diagnosis records across 14 medical centers.

Background

The future of AI in medicine holds immense potential and importance, promising to revolutionize healthcare delivery and patient outcomes. AI technologies, including machine learning and deep learning, are poised to enhance diagnostic accuracy, predict disease outbreaks, personalize treatment plans, and streamline administrative tasks. However, relevant research indicates that current AI in medicine is challenging to directly apply in clinical settings. One of the most important reasons for this is the unknown interaction with clinicians. Human-in-the-loop is a new development trend, and an increasing number of studies recognize its importance. Real-world clinical data is the foundation that supports relevant researches. Unfortunately, there are currently almost no datasets available in this field. This database aims to provide the first attempt to promote the development and actual deployment of AI in medicine, comprising information related to the behavior variations of clinicians' diagnosis with or without the assistance of different AI models. Using Sepsis detection and prediction -- a leading cause of morbidity and mortality -- as the first step, the database contains all the 7500 diagnosis records and information of clinical trials, collaborated with 14 medical centers and 125 clinicians from December 2022 to May 2024.

Methods

Overview

This study has received approval from the Ethics Committee of Guilin Medical University, including the collection of clinician behavioral data, such as diagnostic decisions, time consumption, and all related activities. Informed consent has been obtained from all participating clinicians. The patient data utilized in this study are derived from the publicly accessible MIMIC databases. The authors have secured permission to use these databases, ensuring that no new ethical issues arise. In addition, all personally identifiable information in this database has either been removed or anonymized.

Data Collection and Processing

Collaborated with 14 medical centers and 125 clinicians from December 2022 to May 2024, we perform clinical trials and release the AI.vs.Clinician database as the first human-AI interaction data.

The creation of AI.vs.Clinician was carried out in three steps:
(1) creation of patient cohort based on MIMIC databases and criteria of human clinicians. The patient cohort will be collaboratively diagnosed by the human clinicians and AI models, and thus play pivotal roles in unraveling the interaction between the two entities.
(2) data deduplication for AI training. This step removes the patient cohort from the MIMIC databases and uses the remaining data for model training.
(3) AI model training to the state-of-the-art or state-of-the-practice quality.
(4) AI model inference on patient cohort.
(5) human clinicians' diagnosis on patient cohort with or without the assistance of AI models. The diagnosis behaviors and operations are recorded and collected with the consent of human clinicians.
(6) data validation.
(7) deidentification.

Technical Validation

The database construction process involved a comprehensive series of procedural and manual validations and assessments, meticulously ensuring aspects such as correctness, integrity, consistency, deidentification, and ethical compliance. Version control software was employed to manage the code, and all data processing and transformation operations were executed using reproducible scripts.

Data Description

AI.vs.Clinician provides the following tables:

  • Patient-related tables (14 CSV files: 1-14)
  • AI model-related tables (5 CSV files: 15-19)
  • Human clinician-related tables (3 CSV files: 20-22)

1. Patient_Case_Info.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • GROUP_ID: Group ID., selected patients divided into 5 groups.
  • SUBJECT_ID: corresponding SUBJECT_ID in MIMIC-IV.
  • HADM_ID: corresponding HADM_ID in MIMIC-IV.
  • ADMITTIME: corresponding ADMITTIME in MIMIC-IV.
  • GENDER: Patient's gender.
  • AGE: Patient's age.
  • MARITAL_STATUS: Patient's marital status.
  • RACE: Patient's race.
  • SEPSIS_ONSET_TIME: Onset time of sepsis patients.
  • CURRENT_TIME: Selecting the current time period for patients.
  • SEPSIS_LABEL: Whether the patient is a sepsis patient.If it's 1, then the patient is a sepsis patient; if it's 0, then the patient is not a sepsis patient.

2. Patient_Fundamental_ITEM.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • BASE_ITEM: Patient's base examination items.
  • VALUE: Value.
  • UNIT: Unit.
  • CURRENT_OR_HISTORICAL: Whether the examination item for this time falls within the current time period of the sample.

3. FLUID_INPUT.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • INPUTTIME: Fluid input time.
  • INPUT_AMOUNT: Fluid input amount.
  • MEDICATION_RATE: Fluid input rate.
  • FLUID_NAME: Fluid name.
  • FLUID_TYPE: Fluid type.

4. FLUID_OUTPUT.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • OUTPUTTIME: Fluid output time.
  • OUTPUT_AMOUNT: Fluid output amount.
  • FLUID_NAME: Fluid name.

5. Complete_Blood_Count.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • ITEM: Patient's examination items.
  • VALUE: Value.
  • UNIT: Unit.
  • CURRENT_OR_HISTORICAL: Whether the examination item for this time falls within the current time period of the sample.

6. Pathogen_Blood.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • ITEM: Patient's examination items.
  • BODY_PART: Body parts examined.
  • VALUE: Value.
  • UNIT: Unit.
  • CURRENT_OR_HISTORICAL: Whether the examination item for this time falls within the current time period of the sample.

7. Medical_Imaging.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • ITEM: Patient's examination items.
  • IMAGE_TYPE: Image Type. (The value of this field is for reference only and may not be completely accurate. In cases where some source databases do not provide this field, type extraction is carried out using a large model.)
  • IMAGE_BODYPART: Body Parts Imaged (Same as IMAGE_TYPE)
  • IMAGE_JPG: Imaging Pictures
  • BODY_PART: Body parts examined.
  • TEXT_REPORT: Image Text Report.
  • CURRENT_OR_HISTORICAL: Whether the examination item for this time falls within the current time period of the sample.

8. Arterial_Blood_Gas_Analysis.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • ITEM: Patient's examination items.
  • VALUE: Value.
  • UNIT: Unit.
  • CURRENT_OR_HISTORICAL: Whether the examination item for this time falls within the current time period of the sample.

9. Haemostatic_Function.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • ITEM: Patient's examination items.
  • VALUE: Value.
  • UNIT: Unit.
  • CURRENT_OR_HISTORICAL: Whether the examination item for this time falls within the current time period of the sample.

10. Culture_Smear.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • ITEM: Patient's examination items.
  • SPECIMEN_TYPE: Specimen type for inspection.
  • VALUE: Value.
  • UNIT: Unit.
  • CURRENT_OR_HISTORICAL: Whether the examination item for this time falls within the current time period of the sample.

11. Procalcitonin.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • ITEM: Patient's examination items.
  • VALUE: Value.
  • UNIT: Unit.

12. SOFA.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • CHARTTIME: Patient's examination time.
  • ITEM: Patient's examination items.
  • VALUE: Value.
  • UNIT: Unit.

13. Drug_History.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • STARTTIME: Starting time of drug administration.
  • DRUG: Name of drug.

14. MedicalHistory_ChiefComplaint.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • ITEM: Patient's examination items.
  • VALUE: Value.

15. Model_Property.csv
  • MODEL_ID: Unique identifier for each model.
  • MODEL_NAME: Model name.
  • TIME_PERIOD: The period before onset of illness.
  • DATASET: On the test or sample set.
  • SENSITIVITY: Sensitivity of the model on the dataset.
  • SPECIFICITY: Specificity of the model on the dataset.
  • PRECISION: Precision of the model on the dataset.
  • AUC: AUC of the model on the dataset.

16. Model_Dataset.csv
  • MODEL_DATA_ID: Unique identifier for each model data.
  • SPLIT_TYPE: Training set or validation set or test set.
  • HADM_ID: corresponding HADM_ID in MIMIC-IV.
  • MIMIC_SOURCE: Patient from MIMIC-III or MIMIC-IV.
  • SEPSIS_ONSET_TIME: Onset time of sepsis patients.

17. CoxPHM_Feature.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • Other features: Features used by the CoxPHM model.

18. LSTM_Feature.csv
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • Other features: Features used by the CoxPHM model.

19. Model_Cohort_Infer.csv
  • MODEL_INFER_ID: Unique identifier for model inference.
  • MODEL_ID: Unique identifier for each model.
  • PATIENT_CASE_ID: Patient unique identifier.
  • PREDICTION_RESULT: Results predicted by the model.
  • PROBABILITY_0H: Predicted probability of current onset.
  • PROBABILITY_3H: Predicted probability of onset in the next 3h.

20. Clinician_Info.csv
  • CLINICIAN_ID: Unique identifier for each clinician.
  • INSTITUTION_LEVEL: Clinician's institution level.
  • GENDER: Clinician's gender.
  • AGE: Clinician's age.
  • DEPARTMENT: Clinician's department.
  • YEARS_WORKED: Clinician's years worked.
  • CLASS_OF_POSITION: Clinicianr's class of position.
  • AREA_OF_EXPERTISE: Clinician's area of expertise.

21. Clinician_Click_Behavior.csv
  • INTERACTION_ID: Unique identifier for each clinician diagnose each sample patient.
  • CLICKED_ITEM: Clinician review the examination item.
  • CLICKED_TIME: The time when the clinician review the examination item.

22. Clinician_Diagnosis_Treatment.csv
  • CLINICIAN_ID: Unique identifier for each clinician.
  • PATIENT_CASE_ID: Unique identifier for each sample patient.
  • DIAGNOSED_ORDER: The order in which the clinician diagnosed the sample patient.
  • WITH_MODEL: With or without model diagnostic information.
  • MODEL_INFER_ID: Unique identifier for model inference.
  • MODEL_VISIBILITY: Visible or not for model property.
  • STARTTIME: The start time of the clinician's diagnosis of the sample patient.
  • ENDTIME: The end time of the clinician's diagnosis of the sample patient.
  • INTERACTION_ID: Unique identifier for each clinician diagnose each sample patient.
  • PRELIM_DIAGNOSIS: Clinicianr's preliminary diagnosis.
  • PRELIM_TREATMENT: Clinician's preliminary treatment plan.
  • PRELIM_TIMESTAMP: Clinician's preliminary diagnosis time.
  • FINAL_DIAGNOSIS: Clinician's final diagnosis.
  • FINAL_TREATMENT: Clinician's final treatment plan.
  • FINAL_TIMESTAMP: Clinician's final diagnosis time.
  • ACTION_TYPE: Clinician's operations, such as adding a diagnosis or modifying a diagnosis.
  • CLINICAL_TIME: The time required to assess similar patients in clinical diagnosis.

Usage Notes

The data presented here are collected during standard clinical procedures and may include anomalies due to archival processes. Researchers are encouraged to follow best practices when analyzing the data.
AI.vs.Clinician is a human-centered database that captures the diagnostic behaviors of clinicians when treating patients with and without the assistance of various AI models.
It contains 22 tables: 14 patient-related tables, 5 AI model-related tables, and 3 human clinician-related tables.
In the Patient_Case_Info.csv file, PATIENT_CASE_ID is the unique ID related to the patient.
In the Model_Property.csv file, MODEL_ID is the unique ID for each model.
In the Model_Cohort_Infer.csv file, MODEL_INFER_ID is the unique ID for a model's inference on a specific patient case.
In the Clinician_Info.csv file, CLINICIAN_ID is the unique ID for each clinician.
Therefore, you can use the mentioned IDs to access all behaviors and diagnoses performed by a clinician for a patient case with the support of a specific model.

Ethics

This research has been approved by the Ethics Committee of Guilin Medical University (Approval No:GLMC20221101).

Download

AI.vs.Clinician database comprises a set of comma-separated value (CSV) files and all related source code. Since the database contains not only the patients' information from MIMIC databases but also the clinicians' information from 14 medical centers, users must use the database with caution and respect. To access the database, the following steps need to be completed:

  • The researchers must complete the access steps required by MIMIC databases.
  • The researchers are required to sign a data use agreement, which delineates acceptable data usage and security protocols and prohibits attempts to identify individual clinicians and patients.
  • The researchers are required to provide a description of the research project.
Dataset download: The researchers are required to send an access request to the gaowanling@ict.ac.cn and provide a description of the research project, the CITI training certificate, and a data use statement.
Source code download: All the source code for creating patient cohort, AI training and inference, etc., is available from https://github.com/BenchCouncil/AI.vs.Clinician