Sepsis, a complex and life-threatening response to infection, is one of the primary causes of deaths in intensive care units (ICUs).^[1] This systemic inflammatory response, if not addressed promptly, can lead to tissue damage, organ failure, and eventually death. The World Health Organization (WHO) classifies sepsis as a global health priority. According to a 2020 report, an estimated 49 million people develop sepsis each year worldwide, and approximately 11 million sepsis-related deaths are reported annually.^[2] The rapid development and destructive effects of sepsis highlight the crucial importance of early detection and timely intervention.

Emergency departments (EDs) are the frontlines of the healthcare system, often being the first point of contact for patients seeking urgent medical attention.^[3] With the high influx of patients and the unpredictable nature of medical emergencies, EDs are the epicenter of swift medical decisions and timely interventions. As most sepsis patients manifest shortly after they arrive at EDs, EDs play a unique role in the early prediction and treatment of this deadly disease. Previous studies have consistently demonstrated the relationship between timely treatment and improved prognosis in sepsis patients.^[4,5] These findings emphasize the design of predictive methods specifically tailored for EDs. However, these methods are time-consuming and complex to perform, which limits their use in emergency settings.

Machine learning has been widely used in the field of medicine, including early warning, diagnosis, and treatments. Goh et al^[6] utilized unstructured data, including medical notes, to improve the prediction performance. Stewart et al^[7] performed a nightly prediction using sequential data from the previous day and alleviated class imbalance issues through self-supervised learning. However, most studies have focused on ICUs due to readily available data, with comparatively less focus on the early stages of ED triage. Related studies in EDs include those of Kim et al^[8] and Wardi et al,^[9] who used machine learning algorithms to predict septic shock in EDs. This study aims to develop a prediction system of sepsis within 24 h at the triage stage in EDs using machine learning.

We used the Medical Information Mart for Intensive Care (MIMIC-IV) (v2.2, published on January 6, 2023), which contains medical data of nearly 300,000 patients admitted to the EDs or ICUs of Beth Israel Deaconess Medical Center (BIDMC) between 2008 and 2019.^[10⇓-12]This database includes patient demographics, electronic health records (EHRs) from the ED or ICU, hospital laboratory events and more. Although the database is a recent update to MIMIC-III and has thus been less frequently utilized, it has been applied in prior studies.^[13]

The database implements the Sepsis-3 definition, which can be summarized using the Sequential Organ Failure Assessment (SOFA) ≥ 2, to release the table Sepsis-3.^[14] We considered these patients, 32,970 in all, to be sepsis patients and used the SOFA time in the same table as their sepsis onset time. To correlate the ICU and ED sections in the database, we used subject_id, which is unique to each patient, and hadm_id, which is unique to each hospital admission record. A comparison of patients’ sepsis onset time and their ED admission time revealed that 11,849 cases occurred within 24 h, confirming the validity of our study goal. The flow chart is shown in Figure 1.

For each patient, we used demographics and vital signs, mostly from the table triage from the ED section, as their features. Demographics included the time patients entered the ED, age, gender, and arrival transportation (among “walking in”, “ambulance”, “helicopter” and “other”). The strings were processed with label encoding. Basic vital signs included body temperature (degrees Fahrenheit), heart rate (beats per minute), respiratory rate (breaths per minute), saturation of pulse oximetry (SpO₂) (percentage), systolic blood pressure (SBP) (mmHg), and diastolic blood pressure (DBP) (mmHg). We also included acuity, an order of priority (1-5, with 1 as the highest priority), based on the Emergency Severity Index (ESI) five level triage system. These data were collected when patients entered the ED and are most of the applicable data in the MIMIC-IV at the ED triage stage. We selected them as: (1) they describe patients’ basic clinical status and can be easily accessed in practice; (2) features such as SBP and respiratory rate are applied in the quick SOFA (qSOFA) score, which may indicate sepsis risk; and (3) features such as SpO₂ or temperature are also used by other sepsis warning models.^[15] For feature engineering, we calculated the shock index (shock index=heart rate/SBP) and mean arterial pressure (MAP) (MAP=[SBP+2*DBP]/3) for each patient if their corresponding data existed. These features can provide additional information for the model and are frequently utilized in clinical practice. We excluded patients with more than 3 missing values for the 6 basic vital signs. These sepsis patients comprised the positive sample, while the negative sample included random ED patients who had not developed sepsis within a single complete hospitalization period.

Following sample selection, processing and preparation, we used an open-source Extreme Gradient Boosting (XGBoost) algorithm^[16] as our model to predict the occurrence of sepsis onset within 24 h after ED admission. Ten percent of the patients were randomly selected as the testing set, while the remaining patients were used for training with 10-fold cross-validation. The testing was performed on a random fold. We also used the sklearn implementation of XGBoost to carry out grid tuning. The model was trained for 150 rounds with the following parameters: max_depth 5, min_child_weight 3, eta 0.07, gamma 0.3, subsample 0.6, colsample_bytree 0.7, alpha 1, lambda 2, and objective “binary: logistic”.

For interpretation, we used the package SHapley Additive exPlanations (SHAP), a model interpretation based on game theory whose results have desirable properties.^[17,18] It can compute the importance of each feature, their effect on a certain model outcome, and the interaction between features. We will further demonstrate that this importance obtained by SHAP aligns with medical knowledge and common sense.

We collected a total of 8,219 positive patients and randomly selected 8,400 negative samples with their corresponding data.

We first performed a brief analysis of the collected data to assess feature distribution discrepancies between the two groups. The results are shown in Table 1. Statistical analysis revealed that sepsis patients were older, had a lower female ratio, SpO₂, and blood pressure, and had a higher heart rate and shock index.Additionally, they were more likely to arrive in an ambulance or have higher priority in the ESI system. The distribution of our data is listed in Table 1 and Figure 2. The P-value was calculated by the Mann-Whitney U test.

As a body temperature above 37.3 °C is generally used as one of the signs of infection, which is the cause of sepsis, we specifically analyzed the relationship between the distribution of sepsis onset time (time between ED admission and sepsis onset) and body temperature. The distribution for both high body temperature (> 37.3 °C) and normal body temperature (≤ 37.3 °C) is shown in Figure 3. Between patients with high body temperature (> 37.3 °C) and normal body temperature (≤37.3 °C), the distributions showed no significant discrepancy (P=0.387, calculated by the Mann-Whitney U test).

Among the 14,957 training samples, the average accuracy of the 10 folds reached 84.1%±0.3%, with an average sensitivity of 84.4%±0.3% and an average specificity of 83.7%±0.3%. Testing yielded a similar outcome, with a larger variance as the testing set was comparatively smaller, containing only 1,662 patients. We also plotted the receiver operating characteristic (ROC) curve (Figure 4), with an area under the curve of 0.92±0.02.

SHAP was utilized to obtain the summary plot of each feature’s importance, as shown in Figure 5, with the bar length indicating the average significance for the corresponding feature. The feature ED_Hour stood for the hour when the patient was admitted to the ED. The risk of sepsis occurrence within 24 h after ED admission was influenced the most by a lower acuity level (higher priority), arrival by an ambulance and older age. The risk of sepsis was also related to a higher shock index, respiratory rate, heart rate, and body temperature and lower SpO₂ or blood pressure (Figure 6). In addition, although not as significant as other features, males were more likely to develop sepsis than females were.

With SHAP, we also created dependency plots for each feature, illustrating how feature values could affect the prediction. The significant single-variate dependency plots are listed in the supplementary material (supplementary Figures 1-11), with obvious outliers excluded.

As acuity and arrival transportation might have varying criteria in practice, we excluded these two features and retrained the model. As these two features were among the most important ones, the new model slightly deteriorated in performance, reaching an accuracy of 81.0%±0.3%. The area under the ROC curve (AUC) was 0.90±0.02 (supplementary Figure 12). The sensitivity and specificity reached 78.8%±0.2% and 72.9%±0.2%, respectively. The SHAP summary plot yielded results similar to those of our first model (supplementary Figure 13).

Given the significant contribution of sepsis to mortality in ICUs and the crucial role of early treatment in reducing sepsis-related fatalities, the development of early prediction methods is necessary. As a large amount of sepsis (11,849 in 32,970) occurs soon after ED admission, it is preferable to have a prediction tool at the triage stage.

SOFA standard requires multiple laboratory examinations and is thus difficult to carry out in the ED. The qSOFA score was proposed for quick estimation of sepsis risk. It requires the measurement of respiratory rate, SBP, and Glasgow Coma Scale (GCS), which includes several manual tests to estimate the patient’s conscious state. High patient volume in the ED renders qSOFA assessment, especially the GCS, too burdensome to be applied on each patient. Even in the MIMIC-IV database itself, the GCS is not used for any ED patient.

In practice, medical workers might carry out sepsis checks on patients who are suspected to be infected. However, in the ED, there are few ways to detect infection. One common approach is to use body temperature, as those with a body temperature above 37.3 °C are more likely to be infected. However, body temperature is environmentally sensitive and changes with the time of the day, making it a less reliable criterion. Due to the small difference in quartiles of body temperature between sepsis patients (within 24 h) and non-patients upon ED arrival (Table 1), it is difficult to use body temperature as a criterion for identifying sepsis risk in clinical practice. Furthermore, among the sepsis patients, only approximately 25% arrived at the ED with a body temperature above 37.3 °C. In Figure 3, we plotted the distribution of sepsis onset hours (time from ED admission to sepsis onset).

Another approach to detect infection is to carry out examinations, such as C-reactive protein (CRP). However, potential patients might still be missing. Among the enrolled 8,219 sepsis patients, only approximately 55% (4,519) either exhibited a body temperature > 37.3 °C or had undergone a CRP test. Furthermore, issuing and finishing examinations take time. For example, in the MIMIC-IV, most CRP examinations require 5-6 h. This could cause a severe delay in detecting sepsis and thus a rise in mortality.

Therefore, we propose an XGBoost network, which can be used early in the triage stage in the ED. Using just the patient’s demographics and a few vital signs, we achieved 84% accuracy in predicting sepsis within 24 h. We reached similar accuracy in the testing set, which proved our model to be stable. As the required data can be automatically recorded and the XGBoost model is computationally efficient, this system imposes minimal additional workload, allowing the prediction to be applied in the ED, and extra care and attention can be given to those at risk of sepsis. This approach might assist medical workers in identifying sepsis early, enabling effective early treatment. Such a system will be particularly beneficial for rural or country hospitals, where both medical resources and personnel are usually limited.

The second model’s performance decrease indicated that in addition to basic features such as age, sex or vital signs, additional features could also enhance the prediction. These features include assessment results such as acuity and related information such as arrival transportation. Conversely, the model maintained a comparatively good performance, which has two implications. First, the system demonstrates versatility across various settings, including differences in triage systems or patient transportation. Furthermore, despite excluding two of the most discriminative features, the remaining features preserved considerable classification ability, underscoring the robustness of our method. Consistent performance during grid tuning can lead to a similar conclusion, suggesting that the features have probably exploited most of the network’s discrimination ability, reaching stable performance.

The basic analysis of our raw data (Table 1) is consistent with the traditional understanding of sepsis, which is more likely to occur in elderly people and males. It is also related to changes in heart rate, SpO₂ and blood pressure. The overall feature distribution of sepsis and non-sepsis patients is consistent with the conclusions of previous studies or epidemiological studies.^[19] The SHAP values further helped us explain the prediction, and the results supported the conclusions of medical knowledge and common sense. The most important feature was acuity. Those triaged to higher priority already had serious health issues, causing them to be prone to infection and multiple system failure, leading to sepsis. This approach worked similarly for older people and those who arrived by ambulance. A higher respiratory rate, higher heart rate, lower SpO₂, and lower blood pressure are associated with medical knowledge of sepsis. A higher respiratory rate and lower SBP are used in qSOFA to estimate sepsis. The additional feature shock index and MAP were also crucial, especially the shock index, as it supplemented the model with a division relationship, which XGBoost alone struggled to capture. Although gender did not exhibit much importance, the finding that males are more likely to suffer from sepsis is in line with the existing conclusions.

Our SHAP dependency plot, presented in the supplementary files, validated the summary plot’s findings, with more details worthy of discussion. We observed an increase in the SHAP value for both low and high body temperatures or heart rates (supplementary Figures 8, 9). Abnormally low values for these crucial vital signs, apart from high values, also implied malfunction in related body systems. For both SBP and DBP, there was a slight increase in the SHAP value when the blood pressure was high. This could be a result of both the previously discussed reason and the interference of extreme values. In contrast, MAP exhibited more stable SHAP values, demonstrating the effectiveness of this crafted feature (supplementary Figure 7).

During the research process, we also considered using other features. Race was excluded for having no identifiable influence. The ED section of the MIMIC-IV database also contains data on pain and main complaints. Pain was supposed to fall on a scale of 0-10, but on the table, there are numbers above 10 and strings such as “unable to answer” or other descriptive phrases, making the feature difficult to use. The main complaint consisted of variable phrases or sentences. More complex models were needed to process these texts, potentially complicating the model’s application.

By taking extra time, the issuing and completing of examinations can still offer valuable information. In the future, there could be a three-stage method. In the first stage, when the patients entered the ED, the current model was applied. In the second stage, after a period (e.g., 6 h), some examinations were issued, and this information could act as an extra feature to train another model. In the third stage, after the examinations are completed (e.g., 12 h after ED triage), the laboratory results can provide important information for the training of a third model. The three prediction outcomes can progressively assist medical workers in deciding the risk of sepsis and level of care.

The MIMIC-IV is among one of the preferable databases for prediction in the ED. Its ED section offers detailed information about ED patients, including demographics, vital signs, current medication, and diagnosis. Its HOSP section also contains data from various laboratory events. For sepsis, the derived table contains patients’ SOFA score sequences and corresponding scores for different systems, and Sepsis-3 offers accurate labels for sepsis. With the help of the unique labels subject_id and hadm_id, we could establish the correlation between these sepsis labels and the same patients’ ED records, which enabled us to conveniently create our desired dataset.

To our knowledge, the MIMIC-IV is the only available open database for researchers to obtain both ED and ICU data. Although the definition used in the MIMIC-IV Sepsis-3 table can be limited,^[20] it is the most suitable for our research. As the MIMIC-IV is based on data from the American Medical Center (BIDMC), the model might be biased to the corresponding conditions. Retraining is needed to use the model in other hospitals. In some cases, patients who arrive by ambulance might be transferred from other hospitals and might not be at urgent risk. Such conditions will change the importance of corresponding features and can cause performance deterioration without re-training. False alarms are another potential problem. At the current threshold (0.5), the accuracies of the two classes are comparable. In a scenario where sepsis patients are rare, this can cause large numbers of false positives, leading to alarm fatigue and waste of resources. Adjusting the threshold can reduce the false positive rate but risks missing potential sepsis patients. A better model or data processing approach might mitigate this trade-off.

Our model demonstrated that machine learning can be applied to effectively predict sepsis, as few demographic and vital sign data were collected in the ED. This automatic forecast system offers a viable substitute for the current high-workload methods, especially in situations where those methods are too costly or in rural hospitals lacking high-skilled personnel. Furthermore, the idea of a light-weight classifier can be expanded to forecasting more diseases or health conditions.

Funding: This study was supported by the National Key Research and Development Program of China (2021YFC2500803) and the CAMS Innovation Fund for Medical Sciences (2021-I2M-1-056).

Ethical approval: Not applicable.

Conflicts of interest: The authors declare that they have no competing interests.

Contributions: JW supervised the research and was responsible for its content, including the data analysis, review, and writing. JYX performed the data analysis and drafted the manuscript. TZ reviewed the results and drafted the manuscript. JDG designed the study and reviewed the results. YCL and HDZ interpreted the results. YTC, MTY, ZTL, QMM, and ZML participated in the discussion. All authors have read, edited, and approved the manuscript.

All the supplementary files in this paper are available at http://wjem.com.cn.