Personalized Scorer Modeling: A Learning-Based Framework for Deriving Robust Sleep Stage Labels from Multiple Experts

This paper proposes a personalized scorer modeling framework for generating robust sleep stage labels from multi-scored sleep datasets. The study addresses a key limitation in automatic sleep staging research: most studies evaluate models against a single reference hypnogram, despite clinical sleep staging being affected by inter-scorer variability where no individual scorer can be considered error-free.

The study uses two publicly available multi-scored sleep datasets, DOD-H and DOD-O [15]. DOD-H contains polysomnographic recordings from 25 healthy volunteers (25,441 raw epochs, 24,662 retained hypnogram epochs), while DOD-O contains recordings from 55 patients with obstructive sleep apnea (60,371 raw epochs, 53,233 retained hypnogram epochs). In both datasets, each 30-second epoch was scored by five experienced sleep technicians from three different sleep research centers, with each scorer assigning one of five sleep-stage labels: 0 for Wake, 1 for N1, 2 for N2, 3 for N3, and 4 for REM.

For each 30-second epoch, features were extracted from the EEG C3-M2 channel and chin EMG signal. Each modality contributed 30 features: 25 frequency-domain features and 5 time-domain features. The frequency-domain features consisted of:

• Ten features in the 0-10 Hz range with 1-Hz resolution

• Ten features in the 10-40 Hz range with 3-Hz resolution

• Five conventional band-power features for delta (0.5-4 Hz), theta (4-8 Hz), alpha (8-12 Hz), sigma (12-16 Hz), and beta (16-30 Hz)

Time-domain features included variance, standard deviation, zero-crossing count, coefficient of variation, and the ratio between first- and second-order coefficients of variation. EEG-only experiments used 30 features per epoch, while EEG+EMG experiments used 60 features per epoch.

The agreement analysis revealed that N1 is the most ambiguous stage in both datasets. In DOD-H, only 12.09% of N1 epochs received agreement from all five scorers, while 45.26% had agreement from only three scorers. In DOD-O, 12.14% of N1 epochs received agreement from all five scorers, while 47.27% had agreement from three scorers.

Soft-agreement scores were computed for each scorer using the formula: SAj = (1/K) Σ I(yi,j = MVi), where yi,j is the label assigned by scorer j to epoch i, MVi is the majority-vote label, and I(.) is the indicator function. For DOD-H, the scores were 0.87, 0.91, 0.92, 0.84, and 0.92, leading to the exclusion of scorer 4 from the dataset hypnogram. For DOD-O, the scores were 0.88, 0.88, 0.87, 0.88, and 0.91, leading to the exclusion of scorer 3.

The proposed LBH method works as follows:

1. Personalized scorer modeling: For each scorer j, a random forest classifier was trained and tested using five-fold cross-validation. The input consisted of extracted EEG and EMG features, and the target labels were the labels provided by that scorer.

2. Confusion matrix generation: The resulting confusion matrix Cj describes how the model reproduces each scorer's labeling pattern for different sleep stages. Rows represent reference sleep stages and columns represent predicted/assigned stages.

3. Column normalization: Each column was normalized using the formula: Rj[M,K] = Cj[M,K] / Σr=1 M Cj[r,K], producing a stage-specific scorer profile representing the probability of a candidate true stage M given that scorer j assigned stage K.

4. Probability aggregation: For each epoch i, the method collects the corresponding reliability column for each scorer and sums the stage probabilities across all N scorers: Scorei(M) = Σj=1 N Rj[M, yi,j].

5. Final label selection: The final LBH label is the stage with the maximum aggregated score: yi LBH = arg max Scorei(M).

This approach treats scorer reliability as stage-dependent rather than assigning a single global weight to each scorer, allowing all scorers to contribute while reducing the influence of scorer-stage combinations that are less consistent with physiological feature patterns.

Three hypnogram definitions were evaluated:

• BSH (best-scorer hypnogram): uses labels of the scorer with the highest overall soft-agreement

• DH (dataset hypnogram): the hypnogram distributed with the datasets after scorer selection and tie resolution

• LBH (learning-based hypnogram): the proposed method

Three classifiers were evaluated: random forest (300 trees), support vector machine (Gaussian/RBF kernel), and multilayer perceptron (three hidden layers with 50 neurons each, maximum 200 training iterations). Each classifier was tested with EEG-only features and combined EEG+EMG features. Metrics included accuracy, precision, total F1-score, and class-specific F1-scores for W, N1, N2, N3, and REM.

Random forest achieved the strongest overall performance. The best results were obtained with EEG+EMG features and LBH:

• DOD-H: 86.07% accuracy, 85.46% precision, 85.29% F1-score

• DOD-O: 86.04% accuracy, 85.21% precision, 84.70% F1-score

For comparison, with EEG+EMG features on DOD-H, BSH achieved 83.56% accuracy and DH achieved 84.95% accuracy. On DOD-O, BSH achieved 83.66% and DH achieved 84.48% accuracy.

LBH improved overall accuracy, precision, and F1-score relative to DH and BSH in all settings. In DOD-O with EEG+EMG features, SVM accuracy increased from 81.32% with DH to 83.01% with LBH.

MLP also benefited from LBH, though with lower absolute scores than random forest. In DOD-O with EEG+EMG features, MLP accuracy increased from 80.95% with DH to 82.52% with LBH, and total F1-score increased from 79.19% to 81.34%.

• LBH improved total accuracy, precision, and F1-score in every classifier, dataset, and signal configuration reported in the experiments.

• Adding chin EMG features generally improved performance relative to EEG-only features, especially for random forest.

• N1 remained the most difficult stage across all datasets and classifiers, consistent with the agreement analysis showing N1 had the lowest full-scorer agreement and highest proportion of three-scorer agreement.

• The proposed method improves the label-generation process itself and is therefore not tied to a single final classifier architecture.

The central advantage of LBH is that it treats scorer reliability as stage-dependent, recognizing that a scorer may be highly consistent for Wake or N2 while being less reliable for N1 or stage transitions. The results suggest that improving the construction of the target hypnogram is a valuable and complementary direction for automatic sleep staging research.

The study acknowledges limitations: scorer profiles derived from machine-learning confusion matrices quantify model-derived consistency between features and scorer labels, but they are not direct clinical proof that one scorer is correct and another is incorrect. Future work could evaluate additional signal channels, use deep representation learning, address N1 classification challenges, and validate on additional multi-scored datasets with subject-independent protocols.

The generated LBH labels and the hypnograms provided by the five scorers for both datasets are publicly available at https://github.com/Edris-Hosseini/dreem-learning-hypnogram.

Based on this paper, here are the specific improvements I can make to AI systems and what the improved systems can do:

1. Stage-Dependent Scorer Reliability Modeling: Instead of using a single global weight or majority vote for each scorer, I will implement a confusion-matrix-based approach that learns per-stage reliability profiles for each annotator. This allows the AI to dynamically weight each scorer's label differently depending on the sleep stage (e.g., trusting a scorer more for Wake/N2 but less for N1/transitions).

2. Probabilistic Label Aggregation: I will replace hard-label majority voting with a probabilistic aggregation method that sums stage-specific reliability scores across all scorers for each epoch. This produces a soft probability distribution over sleep stages rather than a single discrete label, which can be used for uncertainty-aware training and inference.

3. Feature-Driven Scorer Profiling: I will use the extracted EEG and EMG features to generate personalized scorer models (e.g., random forests) that capture each annotator's labeling behavior. This enables the AI to identify which scorers are most consistent with physiological signal patterns for each stage, rather than relying on subjective agreement scores.

4. Robust Training Label Generation: I will generate a new, higher-quality hypnogram (LBH) from multi-scored datasets before training the final classifier. This improves the target labels themselves, making the downstream model more accurate regardless of the classifier architecture (RF, SVM, MLP).

5. Handling Ambiguous Stages (N1): I will incorporate stage-specific reliability information to reduce the impact of highly ambiguous stages like N1 during training. The AI can learn to be more conservative in its predictions for stages where scorer agreement is low, potentially improving overall F1-scores.

6. Achieve Higher Sleep Staging Accuracy: The improved system can achieve up to 86.07% accuracy and 85.29% F1-score on healthy subjects (DOD-H) and 86.04% accuracy on OSA patients (DOD-O), outperforming models trained on best-scorer or dataset hypnograms by 1–2.5 percentage points.

7. Generalize Across Classifier Architectures: The improvement is not tied to a single model—it works with random forests, SVMs, and MLPs, meaning the system can be deployed with any preferred classifier while still benefiting from better labels.

8. Provide Uncertainty Estimates: By using probabilistic aggregation, the system can output confidence scores for each sleep stage prediction, enabling clinicians to flag low-confidence epochs for manual review.

9. Handle Multi-Annotator Data More Fairly: The system can incorporate all available scorers without discarding any, even those with lower overall agreement, by leveraging their strengths in specific stages. This is more data-efficient than excluding scorers entirely.

10. Improve Performance on Difficult Clinical Populations: The system shows consistent gains on both healthy subjects and obstructive sleep apnea patients, suggesting it can be adapted to diverse clinical settings where inter-scorer variability is high.

11. Reduce N1 Misclassification: By down-weighting unreliable scorer-stage combinations, the system can partially mitigate the chronic N1 classification challenge, potentially improving sensitivity for this clinically important stage.

12. Enable Better Model Training with Limited Data: Since the method improves label quality without requiring additional recordings, it can enhance AI performance on existing multi-scored datasets, making it valuable for research groups with limited access to new data.