Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training

The gist: Self-supervised pre-training, combining masked autoencoder reconstruction with relational voxel-level objectives, enables a sparse Vision Transformer framework to learn reusable representations from heterogeneous detector data for energy-frontier neutrino detectors.

Introduction and Motivation

Accelerator-based neutrino physics is entering an energy frontier regime in which interactions reach the TeV scale and produce exceptionally dense, overlapping detector signatures Event interpretation is impractical for conventional reconstruction and challenging for supervised models trained from scratch, particularly when labelled data are scarce and analyses span diverse objectives The challenge is not simply whether learned models outperform existing pipelines, but whether any practical analysis of these events is feasible without them This is precisely the setting in which representation learning becomes compelling

The Framework and Architecture

The proposed framework utilizes a sparse Vision Transformer (ViT)-like architecture that combines masked autoencoder (MAE) pre-training with relational voxel-level objectives available from simulation The encoder is later fine-tuned on flavour identification, charmed-quark identification, event kinematics and vertex reconstruction The framework introduces a sparse encoder for heterogeneous detector data that combines sparse convolutional patch embeddings, module-aware self-attention, and Perceiver-IO fusion across calorimetric and tracking streams

The input representation involves converting native detector representations into detector-specific tokens The 3DCal is tokenised in patches of 12 × 12 × 10 voxels, yielding a 4×4×20 patch grid The AHCAL is tokenised in patches of 6 × 6 × 5 voxels (3 × 3 × 8 grid, up to 72 tokens) The ECAL is encoded as a compact token derived from its energy matrix The muon spectrometer input is variable length: track or hit-plane summaries are embedded and pooled into a compact spectrometer token A hierarchical encoder performs module-level self-attention before Perceiver-IO fusion of the heterogeneous detector streams

Self-Supervised Pre-training Strategy

The pre-training strategy is a two-phase schedule combining masked reconstruction with relational voxel-level targets In the first phase, the model is trained for 400 epochs using masked reconstruction alone To map a single decoder token back to the many voxels within its patch, a multi-rank separable basis (initialised from discrete cosine transform (DCT) coefficients) is used In the second phase, training continues from this checkpoint for 100 additional epochs while introducing, with probability 0.5 per batch, a relational forward pass in which the encoder predicts voxel-level ghost labels, hierarchy labels and particle-category labels on kept 3DCal patches (with a lower mask ratio of 0.25) The ghost target is binary, but the two semantic targets are not hard one-hot labels: each reconstructed voxel inherits all matched truth contributions, weighted by their fractional contribution to that voxel and normalised across classes

Downstream Fine-Tuning and Evaluation

The pre-trained encoder is then fine-tuned jointly on event-level classification and regression tasks This joint fine-tuning tests whether the pre-trained representation is useful as a shared basis for multiple measurements rather than for a single optimised classifier Downstream targets include neutrino flavour, charmedquark category, visible momentum, jet momentum, and the primary interaction vertex Performance is evaluated with one-vs-rest ROC curves and a figure of merit FOM = S/√S + B

Key Findings

The results demonstrate that self-supervised pre-training becomes most valuable precisely where energy-frontier neutrino events are hardest to interpret The MAE+Rel objective provides the largest gains for the less abundant and more topologically complex signatures For the dominant channels, the gains are consistent, with MAE+Rel showing improvements in areas like ντ CC → had where jet reconstruction is especially challenging The relational objectives disproportionately improve the lower-yield channels that are most critical for the detector’s physics reach

The learned representation improves performance and data efficiency across a multi-task fine-tuning suite, and transfers beyond the source domain to publicly available benchmarks spanning different detector technologies and energy regimes The transfer results place the work in a broader context by showing that the pre-training captures structure that is useful well beyond the original detector, task and energy range

The study concludes that for dense neutrino events, reconstruction-style objectives are a strong starting point but not the whole answer The results show several ingredients usually discussed separately can coexist in one detector-aware sparse encoder The present work does not claim that a general detector foundation model has been achieved

Conclusion

The study establishes the representation-learning basis on which such studies could be built for energy-frontier neutrino physics It outlines a concrete path towards more general detector encoders and motivates further work on hybrid pre-training objectives and domain adaptation

--- Page 22 ---

TABLE 2: Model size and active computational cost. The downstream count refers to the model used for fine-tuning and inference in the headline FASERCal experiments. The pre-training count includes the masked-autoencoder decoder and relational heads, which are discarded before downstream fine-tuning

--- Page 20 ---

FIG. 3: Event display of a simulated charged-current muon antineutrino interaction in the FASERCal detector. The event is a high-energy ¯νµ CC interaction with a true visible energy of Evis ≃ 1265 GeV and a reconstructed primary-lepton momentum of pµ ≃ 745 GeV

--- Page 21 ---

FIG. 2: Supplementary class-resolved regression boxplots. Regression-error boxplots for the selected event sample, resolved by true flavour category and target observable

--- Page 10 ---

FIG. 4: Data-efficiency study for the three initialisation strategies. Performance of Scratch, MAE and MAE+Rel across labelled training budgets from 100 to 100,000 events

--- Page 8 ---

TABLE I: Generator robustness and cross-domain transfer benchmarks. A single table collects the three external-validation summaries

--- Page 13 ---

TABLE 1: Training hyperparameters. Pre-training proceeds in two phases: Phase 1 trains with masked reconstruction only (MAE), and Phase 2 continues from the Phase 1 checkpoint with an additional relational pass (MAE+Rel)

--- Page 5 ---

FIG. 1: Example events from the test set illustrating masked reconstruction and relational tasks in the 3DCal detector. For each event, the top panel displays the ground-truth detector readout

--- Page 6 ---

FIG. 3: Attribution and representation diagnostics for the learned representation. A Two example patch-level 3DCal saliency maps for test charged-current events from the fine-tuned MAE+Rel

--- Page 20 ---

FIG. 5: Overview of the framework. A FASERCal case study, with the 3DCal as the primary interaction volume followed by the ECAL, AHCAL and muon spectrometer

--- Page 14 ---

Appendix B: Supplementary architectural details

1. Tokenisation and latent structure

In the configuration used for the main experiments, the 3DCal is tokenised into 12 × 12 × 10 voxel patches, yielding a 4×4×20 patch grid

--- Page 19 ---

Appendix D: Supplementary target and loss definitions

1. Pre-training targets

Masked reconstruction predicts voxel occupancy and charge for masked 3DCal and AHCAL patches, together with masked ECAL and muon-spectrometer summaries when those inputs are dropped

--- Page 20 ---

Appendix C: Supplementary training details

1. Improve event interpretation in high-energy neutrino physics by employing a sparse Vision Transformer (ViT)-like framework that combines masked autoencoder (MAE) pre-training with relational voxel-level objectives. This system can perform flavour identification, charmed-quark identification, momentum regression and vertex reconstruction on simulated FASERCal events, achieving performance comparable to scratch training with an order of magnitude more data when only a few hundred to a few thousand labelled events are available.

2. Develop reusable detector representations by utilizing a multimodal pre-training strategy that augments masked reconstruction with relational voxel-level targets (ghost identification, interaction hierarchy and particle-category labels). This allows the system to learn meaningful spatial and semantic correlations before any downstream fine-tuning is applied, which is particularly effective in channels where dense overlap, secondary activity and partial containment complicate classification.

3. Enhance data efficiency by leveraging the pre-trained encoder, as it demonstrates that MAE+Rel gives the strongest overall performance across the study even at small label budgets. This means a system can achieve flavour-classification performance exceeding Scratch trained on roughly 104 events with only about 103 labelled events, making rare event studies more practical.

4. Enable cross-domain transfer by utilizing the learned representation to perform well on public datasets outside the source domain, such as plastic scintillator benchmarks and PILArNet (LArTPC). This allows the system to adapt its knowledge from one detector technology or energy scale to another, suggesting that the pre-training captures structure that is useful well beyond the original detector.

5. Increase robustness against systematic uncertainties by implementing a coherent global energy-scale stress test on the learned representation, which shows no acute brittleness to an O(10%) coherent calorimeter-scale bias. This means the system can maintain stable performance even when faced with detector miscalibration or scale shifts during inference.

6. Establish physically motivated latent spaces by using attribution and representation diagnostics, which show that 3DCal provides the backbone of the event interpretation while auxiliary branches contribute in physically plausible, channel-dependent ways. This allows for a diagnostic understanding of model reliance, identifying which detector inputs are used for specific physics tasks.

7. Create flexible deployment strategies by using detector-subsystem ablations to understand how information is utilized, confirming that suppressing a branch is not the same as modelling a biased or inefficient branch. This informs the design of future systems by suggesting that a realistic deployment programme should include subsystem-specific perturbations to test robustness against real-world inefficiencies.

Accelerator-based neutrino physics is entering an energy-frontier regime in which interactions reach the TeV scale and produce exceptionally dense, overlapping detector signatures. In this regime, event interpretation becomes impractical for conventional reconstruction approaches, particularly when labelled data are scarce and the analysis spans diverse downstream objectives. We present a sparse ViT framework for learning reusable representations from heterogeneous detector data. Self-supervised pre-training combines masked autoencoder reconstruction with relational voxel-level objectives for hierarchy, ghost and particle identification, and the resulting shared encoder is then jointly fine-tuned across classification and regression tasks. Evaluated on simulated events from the proposed FASERCal concept at the LHC, we find that pre-training consistently improves neutrino flavour and charm-quark identification, momentum regression, and vertex reconstruction over training from scratch, with the addition of relational objectives yielding further gains in the most topologically complex channels. Interpretability analyses further show that pre-training yields a more structured latent space, while detector-subsystem ablations recover physically plausible channel-dependent roles for the heterogeneous inputs. A data-efficiency study shows that, with roughly 10 cubed labelled events, the pre-trained encoder already matches the flavour-classification performance of a randomly initialised model trained on an order of magnitude more data. The learned representations also transfer effectively to publicly available benchmarks spanning different detector technologies and energy scales, matching or exceeding published baselines. These results support self-supervised pre-training on multimodal detector data as a scalable route towards reusable representations for neutrino and particle-detector analysis.