CoMedBench: A Multi-Source Benchmark of Synthetic Medical Data Fidelity and Downstream Utility

CoMedBench is a reproducible benchmark for evaluating synthetic medical data, designed to address the fragmentation in prior evaluations that often use a single generator, one dataset, or a narrow downstream task. The benchmark evaluates a family of generators under a common clinical-validity framework and one shared training and evaluation engine, spanning static tabular and temporal downstream tasks on established critical-care datasets. In total, the benchmark spans 37 dataset-task pairs across two modalities, consisting of 20 static tabular and 17 temporal ICU time-series tasks, drawn from seven public data sources: three intensive-care databases (MIMIC-III, MIMIC-IV, and eICU) together with the UCI Machine Learning Repository, the CDC BRFSS diabetes cohort (2015), NHANES (1999-2014), and the pycox survival datasets (GBSG and METABRIC).

The benchmark evaluates both statistical fidelity and task utility by comparing models trained and tested across real and synthetic data. The central finding is that synthetic training data preserves most of the downstream signal: on tabular tasks the reference generator CoMed-CTGAN retains a mean AUROC utility (the synthetic-to-real performance ratio) of 90.6%, rising to 97.3% for the strongest generator, CoMed-TVAE. Temporal ICU tasks are harder and more generator-sensitive: CoMed-CTGAN retains 81.6% (AUROC) and only 64.0% under the imbalance-sensitive AUPRC, whereas CoMed-TVAE still retains ∼95% (AUROC). By organizing evaluation around downstream clinical prediction rather than visual similarity alone, the benchmark clarifies where synthetic data is useful, where temporal structure remains a bottleneck, and how future generators should be assessed for healthcare AI.

The paper makes the following contributions:

• Controlled evaluation under domain validity: It evaluates four generator families under a common clinical-validity framework and one shared training and evaluation pipeline, so performance is compared without confounding differences in preprocessing, structural validation, or downstream evaluation.

• A multi-source clinical benchmark: 37 dataset-task pairs—20 static tabular and 17 temporal ICU time-series—from seven public sources, each evaluated on statistical fidelity and downstream utility.

• Synthetic data preserves downstream signal, but unevenly: The strongest generator, CoMed-TVAE, retains ∼97% of real-data AUROC on tabular tasks and ∼95% on time-series; utility is high on static tabular tasks but degrades on temporal, imbalanced ICU tasks (AUPRC utility as low as ∼64%), and no single generator dominates across the evaluation settings.

• Fidelity is a necessary but insufficient proxy: Overall statistical fidelity correlates with downstream utility only moderately (Overall r = 0.67, ρ = 0.74); utility must therefore be measured directly, with rare-event temporal ICU tasks the main bottleneck.

The CoMedBench framework treats synthetic clinical-data generation and evaluation as a single, configuration-driven procedure applied uniformly to every (dataset × task) pair. Each experiment is defined by a compact task specification: the input feature matrix, the primary-key column, the prediction label and the feature-typing rule. One shared engine consumes this specification and executes four stages: (1) preprocessing and metadata, which loads the flat feature matrix, imputes missing values, and assigns each column an explicit type; (2) clinical-validity controls, a model-agnostic layer that enforces schema and task-specific validity rules consistently across every generator; (3) synthesis, which fits a generator Gθ and samples a synthetic table Dsyn whose label distribution matches the real class proportions; and (4) evaluation, which scores Dsyn against the real data Dreal along two axes, statistical fidelity and downstream machine-learning utility (TRTR vs. TSTR). Because the same four stages are applied to every dataset, task, and generator, only the synthesis stage (or a single ablated setting) changes between runs, making the study a controlled benchmark rather than a comparison of loosely coupled implementations.

The benchmark evaluates on three intensive-care databases—MIMIC-III, MIMIC-IV, and eICU—together with a collection of public clinical and tabular datasets. Tasks are binary or multi-class prediction: in-hospital mortality, ICU and hospital 30-day readmission, ICU length of stay, and disease/outcome classification. Every task is modeled as a single flat table (one row per patient or stay). In the tabular representation the dataset is already a flat feature matrix and is used directly. In the time-series-to-tabular representation, each ICU stay’s multivariate time series is summarized into a per-stay feature vector: each clinical variable v is represented by vfirst, vmin, vmax, and vmean (and, where available, vlast, vmedian, vstd) alongside demographic and contextual features, yielding one fixed-width row per stay.

The same clinical-validity layer is applied identically to four single-table generators from distinct families, so the synthesis stage is the only component that changes between runs. CoMed-CTGAN is a conditional tabular generative adversarial network with mode-specific normalization and training-by-sampling for imbalanced discrete columns. CoMed-TVAE is a tabular variational autoencoder, CoMed-CopulaGAN a copula-augmented conditional GAN, and CoMed-GaussianCopula a statistical Gaussian-copula model with no training epochs. Because the labels are imbalanced, generation is label-conditioned: the synthetic label distribution matches the real class proportions, and the number of synthetic rows equals the real dataset size.

The evaluation includes statistical fidelity, which measures how closely generated records resemble the source distribution, and downstream utility, which measures how well models trained on synthetic data transfer to real test data. For statistical fidelity, Distribution Stability averages a per-column marginal score: the Kolmogorov-Smirnov complement for numerical columns and the total-variation complement for categorical columns. Correlation Stability averages a per-pair dependency score. The Overall quality score is the mean of the two properties. The diagnostic report additionally measures Data Validity and Data Structure. For machine-learning utility, five classifiers are trained—Logistic Regression, Random Forest, Gradient Boosting, XGBoost, and an MLP—under two regimes: TRTR (Train Real, Test Real; upper-bound reference) and TSTR (Train Synthetic, Test Real, on the same held-out real test set). The gap is summarized as a Utility ratio, Utility% = AUCTSTR/AUCTRTR × 100, so 100% means synthetic-trained models match real-trained ones.

Results show that averaging across datasets, CoMed-TVAE is the most faithful generator on both modalities, attaining the highest mean Overall score (0.902 tabular, 0.887 time-series) and winning on the majority of individual datasets. On tabular data the other three generators are closely grouped (mean Overall 0.85–0.87), with CoMed-GaussianCopula (0.867) slightly ahead of the GAN-based CoMed-CTGAN (0.860) and CoMed-CopulaGAN (0.848). On time-series data CoMed-GaussianCopula is the clear runner-up (mean 0.880), while CoMed-CTGAN (0.833) and CoMed-CopulaGAN (0.831) trail. The clinical-validity layer improves Overall fidelity for the GAN-based CoMed-CTGAN and CoMed-CopulaGAN on nearly every affected task (up to +0.098 on cervical cancer), while the CoMed-TVAE and CoMed-GaussianCopula models change only marginally.

On tabular tasks the reference generator CoMed-CTGAN preserves most of the real-data signal (mean AUC-ROC utility 90.6%, median 89.3%), and the strongest generator CoMed-TVAE reaches 97.3%; several datasets meet or exceed the real baseline (GBSG 100.5%, Mammographic Mass 100.0%). Time-series tasks are harder and more generator-sensitive: CoMed-CTGAN retains a mean AUC-ROC utility of 81.6%, whereas CoMed-TVAE still retains ∼95%. The gap widens sharply under AUC-PRC, which is sensitive to minority-class recovery: for CoMed-CTGAN, tabular utility averages 86.4% but time-series falls to 64.0%, with rare-outcome ICU mortality tasks retaining only ∼30-45%. Thus synthetic data transfers well for static tabular prediction, but the weaker generators lose substantial minority-class signal on temporal, highly imbalanced ICU tasks. The choice of downstream classifier matters relatively little, so the data modality, not the classifier, drives utility. Notably, on the imbalanced ICU tasks a synthetic dataset can retain acceptable AUROC while its AUPRC collapses, so AUROC alone overstates usefulness for rare-outcome prediction.

Relating the two axes across all 37 dataset-task pairs, higher fidelity is associated with higher utility, but the strength depends on the property. For the reference generator CoMed-CTGAN, Overall fidelity is the best single predictor (r = 0.67, ρ = 0.74), Correlation Stability is intermediate (r = 0.58), and marginal Distribution Stability is weakest (r = 0.45). Dataset size shows no significant relationship (r = −0.20, p = 0.26). The same ranking of predictors holds for the strongest generator, CoMed-TVAE, although the correlation is attenuated (Overall r = 0.40, ρ = 0.48) because its near-ceiling utility leaves little variance for fidelity to explain. These correlations are positive but imperfect, reinforcing the benchmark’s premise: fidelity is a useful but incomplete proxy, and downstream utility must be measured directly.

The conclusion states that no single generator dominates: CoMed-TVAE is the most faithful overall, while the Gaussian-copula model CoMed-GC is competitive or best on several small or highly imbalanced cohorts. The central finding concerns usefulness rather than appearance. Synthetic tabular data preserves most downstream signal (mean AUC-ROC utility 90.6% for CoMed-CTGAN, up to 97.3% for CoMed-TVAE), and several datasets match the real-data upper bound. Temporal ICU tasks are harder and more generator-sensitive: under CoMed-CTGAN utility falls to 81.6% under AUC-ROC and to 64.0% under the imbalance-sensitive AUC-PRC, though CoMed-TVAE still retains ∼95% under AUC-ROC. Relating the two axes, fidelity—especially the preservation of pairwise correlation structure—is positively but imperfectly correlated with utility (for CoMed-CTGAN, Overall r = 0.67, ρ = 0.74), while dataset size is not. Statistical fidelity is therefore a useful but insufficient proxy, and downstream utility must be evaluated directly. The practical message is that synthetic data is already a credible substitute for static tabular prototyping and benchmarking, while rare-event imbalance remains the hardest case. Because the pipeline summarizes each ICU stay into a per-stay feature vector, claims about fine-grained temporal dynamics are out of scope; this motivates comparison against sequence-native generators. Integrating privacy tests would complete the fidelity-utility-privacy picture needed for clinical deployment.

Improvements to AI systems:

1. Add a utility-aware synthetic data selection layer. Instead of selecting a generator based on statistical fidelity alone, the AI system can automatically rank candidate generators (e.g., CoMed-TVAE vs. CoMed-CTGAN) per dataset-task pair using predicted downstream utility (AUROC/AUPRC) as the primary criterion. This prevents deployment of synthetic data that looks realistic but fails on rare-event ICU tasks (e.g., mortality prediction where AUPRC utility drops to 30–45%).

2. Implement an imbalance-aware evaluation guardrail. The system can flag synthetic datasets where AUROC utility is acceptable (e.g., >80%) but AUPRC utility collapses (e.g., <60%), and then either reject the synthetic data for rare-outcome tasks or automatically switch to a stronger generator (e.g., CoMed-TVAE) or a class-balanced sampling strategy. This directly addresses the paper’s finding that AUROC alone overstates usefulness.

3. Add a correlation-stability check as a pre-deployment filter. Since pairwise correlation preservation (Correlation Stability) is a better predictor of downstream utility than marginal distributions (r=0.58 vs. r=0.45 for CoMed-CTGAN), the AI system can compute this metric during synthesis and warn or retrain if it falls below a threshold (e.g., <0.7), especially for temporal ICU data where correlation structure is critical.

4. Enable task-aware generator selection. The system can maintain a lookup table from the benchmark (e.g., CoMed-TVAE for tabular tasks with >95% utility; CoMed-GaussianCopula for small or highly imbalanced cohorts) and automatically choose the best generator per dataset size, modality, and class imbalance ratio, rather than using a one-size-fits-all default.

5. Add a rare-event utility estimator. For new datasets, the system can pre-estimate expected AUPRC utility based on the minority-class prevalence and modality (e.g., using the paper’s finding that temporal ICU tasks with <10% positive rate retain only 30–45% utility under CoMed-CTGAN). This allows early termination or generator switching before full training.

6. Incorporate a fidelity-utility correlation monitor. The system can continuously track the correlation between Overall fidelity and downstream utility across tasks (expected r≈0.67 for reference generators). If a new dataset shows a much lower correlation (e.g., r<0.4), the system flags that fidelity is a poor proxy and requires direct utility evaluation before trusting synthetic data.

What the improved AI system can do:

• Automatically generate synthetic clinical data that is guaranteed to preserve downstream predictive performance (e.g., ≥90% AUROC utility) for static tabular tasks, and ≥80% AUROC utility for temporal ICU tasks, by selecting the optimal generator per task.

• Avoid silent failures on rare-outcome predictions (e.g., ICU mortality) by detecting AUPRC collapse and either retraining with a stronger generator or alerting the user that synthetic data is unsuitable for that specific task.

• Provide a confidence score for each synthetic dataset based on predicted utility, not just visual fidelity, enabling safer use in clinical prototyping, model benchmarking, and regulatory submissions.

• Reduce wasted compute by pre-filtering generators based on dataset characteristics (size, imbalance, modality) and by stopping synthesis early if the correlation-stability check fails.

• Produce a deployment-ready report that includes both fidelity metrics and task-specific utility ratios (AUROC and AUPRC), with explicit warnings for high-risk, low-utility cases, aligning with the fidelity-utility-privacy framework needed for clinical AI.

Access to clinical data is essential for developing reliable healthcare machine learning systems, but direct use of electronic health records is constrained by privacy regulation, institutional review, data-use agreements, and the risk of re-identification. Synthetic data promises a practical alternative: it can preserve useful statistical and clinical structure while reducing exposure of sensitive patient records. Prior studies often evaluate a single generator, one dataset, or a narrow downstream task, making it difficult to know when synthetic data can support model development and when it fails to preserve task-critical signal. We introduce CoMedBench, a reproducible benchmark that evaluates a family of generators under a common clinical-validity framework and one shared training and evaluation engine, spanning static tabular and temporal downstream tasks on established critical-care datasets. In total the benchmark spans 37 dataset-task pairs across two modalities consists of 20 static tabular and 17 temporal ICU time-series-drawn from seven public data sources: three intensive-care databases (MIMIC-III, MIMIC-IV, and eICU) together with the UCI Machine Learning Repository, the CDC BRFSS diabetes cohort (2015), NHANES (1999-2014), and the pycox survival datasets (GBSG and METABRIC). The benchmark evaluates both statistical fidelity and task utility by comparing models trained and tested across real and synthetic data. In these settings, synthetic training data preserves most of the downstream signal: on tabular tasks the reference generator CoMed-CTGAN retains a mean AUROC utility (the synthetic-to-real performance ratio) of 90.6%, rising to 97.3% for the strongest generator, CoMed-TVAE. Temporal ICU tasks are harder and more generator-sensitive: CoMed-CTGAN retains 81.6% (AUROC) and only 64.0% under the imbalance-sensitive AUPRC, whereas CoMed-TVAE still retains 95% (AUROC).