AI for pRedicting Exacerbations in KIDs with aSthma (AIRE-KIDS)

Recurrent exacerbations remain a common yet preventable outcome for many children with asthma. Machine learning (ML) algorithms using electronic medical records (EMR) could allow accurate identification of children at risk for exacerbations and facilitate referral for preventative comprehensive care to avoid this morbidity. The authors developed ML algorithms to predict repeat severe exacerbations (i.e., asthma-related ED visits or future hospital admissions) for children with a prior asthma ED visit at a tertiary care children's hospital.

Up to 25% of children with an asthma ED visit will have a repeat visit in the following year, despite numerous evidence-based treatments available to optimize asthma control. With current shortages in primary care, preventative intervention is frequently missed in the ambulatory setting. The authors note that there is a lack of guidance or agreed consensus on how to identify this vulnerable group of children who are most at risk for repeat severe asthma exacerbations.

The study used retrospective Epic EMR data to train ML models for predicting future severe asthma exacerbations among patients treated for an asthma exacerbation in the ED at the Children's Hospital of Eastern Ontario (CHEO). A future severe asthma exacerbation was defined as an ED re-visit or hospitalization with asthma as the most responsible diagnosis within one year of the initial asthma ED visit.

Two retrospective datasets were used: a training dataset (pre-COVID-19) accrued patients over 2 years between February 2017 to March 2019, with outcomes observed up to March 2020, and a post-COVID validation dataset with an accrual period from July 2022 to April 2023, with outcomes observed up to April 2024. Patients with an index visit for asthma during the peak COVID19 period (March 2020-June 2022) were excluded due to significantly lower frequency of asthma exacerbations observed following provincial infection control measures.

A total of 68 clinical features were extracted from the EMR and further linked to air pollution and neighborhood marginalization data. Features included:

• Prior health care utilization: previous asthma-related ED visits or admissions, prior non-asthma respiratory ED visits or admissions within one year prior to the index visit

• Operational variables: day and month of the index visit, time from arrival to triage or disposition, hourly average number of patients present in the ED, average wait time during each patient's visit

• Demographic features: age during the index visit, sex, whether the patient has a primary care physician, distance of the hospital from the postal code of the patient's home, province of residence

• Severity features: first oxygen saturation recording, timing of order and administration of oral corticosteroids, Pediatric Respiratory Assessment Measure (PRAM) score at triage, Canadian Triage Acuity Score (CTAS)

• Clinical variables: presence of food and environmental allergens, lab test results such as serum eosinophils count and immunoglobulin E levels, medical complexity as reflected in prior visits to subspecialty clinics, presence of specific comorbidities

• Environmental and social features (for Ontario residents): Air Quality Health Index (AQHI), Nitrogen Dioxide, Ozone, and PM2.5 readings at different intervals (1 day, 2 days, 3 days prior to index visit), and neighborhood marginalization quintile using the Ontario marginalization (ON-Marg) index

Patients were included if they had an asthma diagnosis (ICD code J45, J46) at the index encounter during the accrual period. Patients were excluded if they had attended a respirology appointment or received comprehensive asthma education prior to the index visit, or if they were admitted to hospital during the same encounter as the index asthma ED visit.

Two types of boosted decision trees (LGBM and XGBoost) and three open-source LLMs with fine-tuning (DistilGPT2, Llama 3.2 1B, and Llama-8b-UltraMedical) were evaluated. The hyperparameters of boosted tree models were tuned and optimized using Bayesian optimization with 5-fold cross-validation. Predicted values were calibrated using beta calibration. For LLMs, a textual encoder transformed each tabular record into text, and predictions were generated five times for each patient and averaged. Due to privacy rules, commercial off-site LLMs could not be utilized.

The F1 score was deemed the most appropriate evaluation metric for the clinical context, being the harmonic mean of precision and recall. The authors note that "capacity within the CHEO asthma program is resource-limited and may not be available to all patients with the predicted outcome. Therefore, we wanted a model that maximizes the proportion of patients that were predicted as high risk that were actually high risk (precision), hence ensuring that the resources are used most effectively." Cutoffs were selected to maximize the F1 score. Three baselines were established: (a) the 'CHEO best practice' rule currently used in the ED, (b) a naïve prediction rule predicting everyone is high risk, and (c) a random prediction of risk status based on a 0.5 probability. AUC was also reported to enable comparison with previous studies.

There were 2,716 patients in the training cohort and 1,237 in the validation cohort. The majority were preschoolers (mean age 4.4-4.5 years, median 3 years), 34.3-36.5% were female, and the majority (72.4% - 81.8%) resided in Ontario. Outcome distributions were similar between cohorts: 29.3% repeat asthma ED visits in training and 28.5% in validation cohorts; 11.9% and 17.8% for admissions respectively.

The current CHEO best practice rule for alerting ED clinicians about which patients to refer to the CHEO asthma program achieved F1 scores of 0.270 (ED visit) and 0.248 (admission) on training data, and 0.334 (ED visit) and 0.313 (admission) on validation data.

Internal validation on the training dataset using nested cross-validation showed AUC values ranging from 0.530 to 0.666 for predicting ED visits and 0.5 to 0.74 for predicting admissions, with lower AUCs observed for the LLMs. The best performance was for the LGBM model.

When applied to the validation dataset, F1 scores for predicting ED visits ranged from 0.292 to 0.503, and from 0.140 to 0.342 for predicting admissions. In all cases, F1 scores were highest for the LGBM models and lowest for the LLMs. The LGBM model achieved an AUC of 0.702 with F1 of 0.503 for ED visits, and AUC of 0.673 with F1 of 0.342 for admissions.

SHAP analysis revealed the most predictive features for repeat asthma ED visits included: previous asthma ED visit, CTAS, medical complexity, food allergy, ED visit for other respiratory diagnoses, and age. For predicting future asthma hospitalization, the top features included: medical complexity, previous asthma ED visit, average wait time, PRAM score during the index visit, and food allergy.

Based on SHAP feature importance, parsimonious models were created with reduced feature sets:

• AIRE-KIDS ED model: features included prior asthma ED visit, CTAS, complexity, food allergy, prior ED visits for non-asthma respiratory diagnoses, and age. This achieved an AUC of 0.712 and F1 of 0.51 at a cutoff threshold of 0.250 on the validation dataset.

• AIRE-KIDS HOSP model: features included complexity, prior asthma ED visit, average wait time in the ED, PRAM at triage, and food allergy. This achieved an AUC of 0.65 and F1 score of 0.375 at a cutoff threshold of 0.170.

The authors note that this is a nontrivial improvement over the current decision rule which has F1=0.334. The best naïve decision rule that predicts all patients having high risk has an F1 score of 0.442 and 0.302 respectively for ED visits and admissions.

The authors conclude that "AIRE-KIDS ED and AIRE-KIDS HOSP are new ML models for accurately predicting repeat asthma related ED visits or future asthma-related hospitalizations in children, and are superior to current decision-making processes for triaging referrals for comprehensive preventative care from the ED. They further state that LGBM remains a superior ML approach for modeling this type of clinical prediction, outperforming newer LLMs, despite their popularity in other applications, though ongoing research in this area is needed as LLM's continue to evolve."

The next steps involve local deployment and prospective evaluation at our centre in the form of a clinical decision support system and subsequent validation at other pediatric centres. Successful implementation "could redefine how we triage and manage pediatric asthma in the acute care setting, supporting more equitable access to comprehensive or specialist care while reducing the morbidity associated with repeat severe exacerbations."

The authors acknowledge several limitations: potential lack of generalizability to other centres requiring additional validation; restriction to local healthcare visit data potentially missing outcomes at other centres (though more than 94% of children within 50km attend CHEO); and missing variables difficult to obtain from EMR data including adherence to asthma controller medications or exposure to cigarette smoke in the home. Including patients who received asthma education or attended a respirology clinic visit in the outcome variables introduces error, though the authors argue results are arguably better than presented here because of this.

Based on the AIRE-KIDS paper, here are the specific improvements I can make to AI systems, and what the improved systems can do:

Improvement: The paper demonstrates that LGBM (Light Gradient-Boosting Machine) significantly outperforms three fine-tuned open-source LLMs (DistilGPT2, Llama 3.2 1B, Llama-8b-UltraMedical) for tabular clinical prediction tasks. The AUC for ED visit prediction was 0.666 (LGBM) vs. 0.530–0.587 (LLMs). For hospital admission, LGBM achieved 0.742 AUC vs. 0.500–0.650 for LLMs.

What the improved AI system can do: Automatically default to gradient-boosted tree models (LGBM/XGBoost) for structured EMR data, reserving LLMs only for unstructured text. This avoids the common pitfall of deploying LLMs on tabular data where they underperform, saving computational cost and improving accuracy.

Improvement: The paper used SHAP values to reduce from 68 features to just 6 for ED visit prediction (prior asthma ED visit, CTAS, medical complexity, food allergy, prior non-asthma respiratory ED visits, age) and 5 for hospital admission (medical complexity, prior asthma ED visit, average ED wait time, PRAM score, food allergy). This reduction maintained performance (AUC 0.712 for ED visits) while simplifying deployment.

Improvement: Instead of relying solely on AUC, the paper optimized the decision threshold to maximize the F1 score (harmonic mean of precision and recall). The AIRE-KIDSED model achieved F1 = 0.510 vs. the current best-practice alert's F1 = 0.334, and vs. a naïve predict all high-risk baseline of F1 = 0.442.

Improvement: The paper applied beta calibration to convert LGBM pseudo-probabilities into true probabilities. This is essential because boosted trees produce uncalibrated outputs that can mislead clinicians about actual risk.

Improvement: The models were trained on pre-COVID data (Feb 2017–Feb 2019) and validated on post-COVID data (Jul 2022–Apr 2023), demonstrating robustness despite significant changes in healthcare utilization patterns during the pandemic.

Improvement: The paper accounted for patients who received asthma education or specialist care (which reduces exacerbation risk) by including these as positive outcomes. This prevents underestimating model performance.

Improvement: The model incorporated operational ED features such as average wait time and patient volume, which were predictive of hospital admission. This goes beyond traditional clinical features.

Improvement: The paper emphasized that models trained at one center may require local recalibration due to differences in case mix and practice patterns. They prioritized parsimony and transparency to facilitate transfer.

Improvement: SHAP analysis revealed clinically plausible predictors (e.g., prior ED visits, triage acuity, food allergy), which builds clinician trust and supports adoption.

Improvement: The paper developed separate models for ED revisit (AIRE-KIDSED) and hospital admission (AIRE-KIDSHOSP), recognizing these as distinct outcomes with different clinical implications.

In summary, the improved AI system can:

• Accurately predict repeat asthma ED visits (AUC 0.712) and hospitalizations (AUC 0.65) in children

• Outperform current clinical decision rules by 53% (F1: 0.510 vs. 0.334)

• Operate in real-time at the ED point of care with only 5–6 easily obtainable features

• Provide calibrated, explainable risk scores that support clinical decision-making

• Maintain performance across temporal shifts and be adaptable to other centers

• Optimize resource allocation for limited-capacity preventative care programs

Recurrent exacerbations remain a common yet preventable outcome for many children with asthma. Machine learning (ML) algorithms using electronic medical records (EMR) could allow accurate identification of children at risk for exacerbations and facilitate referral for preventative comprehensive care to avoid this morbidity. We developed ML algorithms to predict repeat severe exacerbations (i.e. asthma-related emergency department (ED) visits or future hospital admissions) for children with a prior asthma ED visit at a tertiary care children's hospital. Retrospective pre-COVID19 (Feb 2017 - Feb 2019, N=2716) Epic EMR data from the Children's Hospital of Eastern Ontario (CHEO) linked with environmental pollutant exposure and neighbourhood marginalization information was used to train various ML models. We used boosted trees (LGBM, XGB) and 3 open-source large language model (LLM) approaches (DistilGPT2, Llama 3.2 1B and Llama-8b-UltraMedical). Models were tuned and calibrated then validated in a second retrospective post-COVID19 dataset (Jul 2022 - Apr 2023, N=1237) from CHEO. Models were compared using the area under the curve (AUC) and F1 scores, with SHAP values used to determine the most predictive features. The LGBM ML model performed best with the most predictive features in the final AIRE-KIDS ED model including prior asthma ED visit, the Canadian triage acuity scale, medical complexity, food allergy, prior ED visits for non-asthma respiratory diagnoses, and age for an AUC of 0.712, and F1 score of 0.51. This is a nontrivial improvement over the current decision rule which has F1=0.334. While the most predictive features in the AIRE-KIDS HOSP model included medical complexity, prior asthma ED visit, average wait time in the ED, the pediatric respiratory assessment measure score at triage and food allergy.