TransNRank: Towards Accurate Neoantigen Ranking with Transformer

Personalized neoantigen prediction is a significant challenge due to the scarcity of positive samples, the noise of the experimental data, the severe class imbalance trait and the complex of immunogenicity features. Traditional methods, such as linear regression and XGBoost, fail to model long-range dependencies and contextual relationships within peptide features, which limits the performance of the neoantigen positive recall rate. To address these issues, this paper presents TransNRank, a novel deep learning framework based on the Transformer architecture. By leveraging the self-attention mechanism, our model captures both local and global feature contexts, enabling more accurate recognition of immunogenic neoantigens. To handle the class imbalance problem, the framework utilizes a positive-aware training objective... assigning more weights to those few positive samples.

Extensive experiments were performed on the NCI, TESLA and HiTIDE datasets. The results demonstrate that TransNRank can push the upper bound top 20 recall rate of neoantigen prediction from 46.9% (45 from 96) to 53.1% (51 from 96), while reducing the training epochs from 200 epochs to 20 epochs. When compared against all 25 teams participating in the Tumor Neoantigen Selection Alliance (TESLA), the trained Transformer model achieved the best performance among all teams in terms of both FR [fraction ranked] and TTIF [top-20 immunogenic fraction]. While the model showed superior recall for immunogenic peptides within the top-20 and top-50 lists, its recall within the top-100 list was inferior to that of LR.

Through feature ablation experiments, the researchers found that removing insignificant features to reduce the input dimensionality of peptides does not drastically impair the overall performance of the model. The study identified seven core features that are critical for prediction: NetMHCpan Rank, MixMHCpred Rank, PRIME Rank, Mutation at Anchor, NetStab Rank, MixMHCpred Score DAI, and TCGA Cancer Expression. Notably, Mutation at Anchor emerged as an unexpectedly influential factor in neoantigen ranking, and TCGA-derived cancer expression level exhibited greater importance than expression values calculated from tumor RNA‑seq data, underscoring the value of publicly available reference expression resources in this context.

The paper also evaluated the impact of intersection filtering as an ensemble strategy. The findings indicate that intersection filtering failed to improve the Top-N recall of any single base classifier, because strict intersection filtering forms a consensus bottleneck and discards true positives with inconsistent predictions across models, raising false negative rates. However, intersection filtering did increase the proportion of immunopeptides among the retained candidates, most likely by reducing false negatives. Consequently, the authors suggest that intersection filtering [is] suitable for scenarios with limited experimental validation capacity as a negative screening method to eliminate low-confidence peptides, rather than a positive selection strategy.

The study acknowledges several limitations: the limited training set (57 patients, 82 positive peptides) restricts the construction of deeper Transformer structures; the model only targets 8–12 amino acid HLA-I peptides and is not applicable to HLA-II neoantigens; all validation labels originate from in vitro assays instead of real clinical vaccine responses; and TCR-peptide binding features are not integrated into the current framework.

1. Self-Supervised Pre-training for Deep Architectures

By pre-training the Transformer on massive, unlabeled datasets of peptide-MHC binding sequences, the system can overcome the scarcity of labeled positive neoantigen samples. This allows for the construction of much deeper and more complex Transformer layers that can capture higher-order biological patterns without being restricted by the small size of clinical training sets.

2. Multi-Modal TCR-Peptide-MHC Cross-Attention

By integrating T-cell receptor (TCR) sequence embeddings into a multi-modal Transformer using cross-attention mechanisms, the system can move beyond predicting simple peptide-MHC binding. The improved system will be able to model the ternary complex interaction, significantly increasing the accuracy of predicting actual T-cell activation and immunogenicity.

3. Dual-Target HLA-I and HLA-II Prediction Framework

By implementing a multi-task learning architecture with variable positional embeddings to accommodate different peptide lengths (8–12 amino acids for HLA-I and 13–25 for HLA-II), the system can expand its scope. This allows the AI to identify a much broader range of neoantigens, including those presented via the HLA-II pathway, which is critical for CD4+ T-cell responses.

4. Hybrid Transformer-Linear Ensemble

By utilizing a stacked ensemble approach that combines the Transformer’s high-precision top-20 ranking with the robust top-100 recall of linear models (LR), the system can eliminate the current recall gap. This improved system will provide highly accurate top-tier candidates while ensuring that potential neoantigens are not missed in broader screening lists.

5. Clinical-Response Transfer Learning

By employing a transfer learning strategy that fine-tunes models trained on in vitro binding assays with smaller, high-quality datasets of real-world clinical vaccine responses, the system can bridge the gap between laboratory affinity and therapeutic efficacy. This allows the AI to predict which neoantigens are most likely to trigger a successful immune response in actual patients.

6. Soft-Voting Probabilistic Ensemble

By replacing strict intersection filtering with a weighted, probabilistic consensus mechanism, the system can avoid the consensus bottleneck. This allows the AI to retain true positive neoantigens that might have slightly inconsistent scores across different models, thereby maintaining high recall while still effectively filtering out low-confidence peptides.

7. Structural Graph Neural Network (GNN) Feature Integration

By incorporating GNNs to model the 3D spatial relationship between the mutation site and the MHC binding pocket, the system can provide a more sophisticated representation of Mutation at Anchor. This enables the model to understand the structural impact of a mutation on binding stability and TCR recognition more deeply than current categorical or rank-based features.