CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics".

Jane: The gist The CAPT model introduces a continuous autoregressive transformer for calcium population dynamics that enables cross-dataset and cross-species transfer for calcium population dynamics How it works CAPT models continuous…

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So we just covered the high-level idea of CAPT: this Continuous Autoregressive Population Transformer for calcium population dynamics. To summarize what they are actually proposing, they’re tackling the problem that current models are too tailored to single datasets and want to see if we can build foundation-style models for neural population dynamics.

Tom: They claim that by pretraining on a large-scale mouse dataset, you can learn general representations applicable to diverse tasks like forecasting and decoding behavior. But the real challenge they identify is generalization beyond the data distribution they were trained on because datasets vary a lot in species, brain regions, imaging conditions, sampling rates, neuron counts and experimental paradigms.

Lu: They propose CAPT to solve this by modeling continuous calcium traces directly through a continuous patch tokenization strategy and training it autoregressively with a mean squared error objective. This sets up the end-to-end training process that allows for adaptation to diverse downstream tasks.

Meng: So, the core mechanism is this continuous patching and the autoregressive training loop, right? Can you tell me more about how they handle making sure the predictions are accurate during that autoregressive prediction phase?

Tom: They define their autoregressive objective as a mean squared error over predicted future patches from past activity. To make it more robust to imperfect histories, they apply two lightweight corruptions to the input history while keeping the prediction targets unchanged.

Jane: That adds a layer of robustness, which is important when you’re dealing with continuous data where the history might be noisy or incomplete in a real lab setting. So, it's not just learning patterns; it's learning how to predict sequences reliably even when the input sequence isn't perfect.

Lu: They also looked at related work, specifically classical methods like latent dynamical systems and state-space models for neural forecasting, and then they brought in Transformer architectures to leverage foundation-style modeling of invasive neural recordings.

Tom: And their evaluation was quite broad; they tested this transferability across eight independent datasets: three mouse datasets, three larval zebrafish datasets, and two C. elegans datasets. This is a pretty wide range of biological systems for them to test against.

Jane: It’s impressive that they didn't just test the transfer on a single new dataset, but across different species and labs to prove the model isn't just memorizing mouse-specific features or lab-specific noise. What kind of performance gains did they see when moving between these groups?

Tom: In all transfer experiments, CAPT consistently outperformed specialized and general-purpose baselines like POYO+, POCO, and CalM. And across both the pretraining and transfer datasets, CAPT shows significantly better performance in most cases.

Lu: The two-step protocol they used separates neural population adaptation from task adaptation by only updating new embeddings during forecasting transfer or only updating the decoder during behavior decoding. This directly tests if a single continuous autoregressive backbone can provide those reusable representations across datasets, experimental paradigms, and species.

Meng: If it works like that, it means the backbone is really doing the heavy lifting of learning the general representation for everyone else. But what about deployment? How much of this complex transformer architecture are we talking about needing to run in a real-time system?

Tom: It's an autoregressive model, so inference involves sequentially predicting patches. The complexity is managed by the Transformer structure itself, but they are focused on the adaptation modules being lightweight enough to update efficiently.

Jane: So, if you’re listening and you’re thinking about applying this to a new species or lab setup, what’s the main promise of CAPT that makes it worth paying attention to?

Lu: The promise is that continuous autoregressive modeling provides an effective framework for learning calcium population representations that transfer across datasets and experimental paradigms, which is something we haven't seen consistently before.

Meng: It sounds like a powerful tool, but I still wonder about the training data requirements. They pretrained it on a large-scale mouse dataset; how much more data do you think they need to train it on if they wanted to generalize even further across different phyla?

Tom: The current finding is that freezing the backbone and only updating embeddings works well, suggesting we don't need massive retraining for every new target task. But as Lu mentioned, the limitation is that CAPT is currently pretrained on a single large-scale mouse calcium imaging dataset.

Jane: So, it’s a strong proof of concept for transferability across species and labs, but it’s still rooted in that initial mouse training. That sets us up perfectly for talking about how this impacts the broader field of AI modeling for biological data.

Conclusion: Tom: So wrapping up the discussion on CAPT: its full title is A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics. It’s about moving away from models that are stuck in one experimental setting.

Jane: Essentially, the authors are showing that a single pretrained model can learn a set of neural population representations that transfer across different species, different labs, and even different experimental paradigms when adapted correctly.

Lu: The main implication is that we can build foundation-style models for neural population dynamics by leveraging large-scale recordings to learn general representations applicable to diverse tasks. This moves us past just solving one specific forecasting problem in one specific animal.

Meng: For deployment, this means we have a starting point—a pretrained backbone—that we can fine-tune with minimal data for new biological targets, which is a huge win for researchers who need to apply these models quickly to novel systems.

Tom: I think the thing that stands out is the success of separating neural population adaptation from task adaptation, which directly validates this idea of learning reusable representations. It proves that the continuous autoregressive backbone isn't just fitting noise; it’s learning structure.

Jane: And alongside those predictive wins, they found that these CAPT embeddings capture shared functional space across datasets and actually encode cell-identity related structure in C. elegans data, which is a deep biological finding for us.

Lu: That shared functional space idea means the model is learning something fundamental about how neurons interact or are organized regardless of the specific species it was trained on.

Tom: So, this work suggests that continuous autoregressive modeling is an effective way to learn representations that generalize across datasets and species for calcium imaging data. It’s a simple framework to start with.

Jane: It gives us a solid foundation for how we can approach building AI models that can actually handle the massive heterogeneity of biological data out there.