JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures".

Jane: The gist The JEPA-DNA framework introduces a novel continual training framework for genomic foundation models that shifts the learning objective from literal token reconstruction to latent feature alignment via a…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're talking about this paper called "JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures." It’s written by a team including Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman, Guy Leib, Omri Nayshool, Dan Ofer, Tal Zinger, Dan Dominissini, Gideon Rechavi and Nicole Bussola. The title itself tells you the main goal is grounding those genomic foundation models.

Jane: Grounding what? That sounds very technical for a first listen. What does that mean in simple terms when we talk about these big genomic models?

Tom: Well, the paper says that these models usually learn by doing something called Masked Language Modeling or Next-Token Prediction to learn the "Laws of Nature." But Tom and Jane are saying that this token-level reconstruction isn't enough. It focuses too much on just the literal sequence.

Lu: Exactly. They point out that these generative paradigms prioritize token-level reconstruction over high-level functional context, which means they might ignore the big picture regulatory logic or long-range interactions in a genome (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures #pg2).

Meng: So if a model only learns to reconstruct the exact sequence tokens, what kind of stuff is it missing? Is it just the subtle biological meaning?

Jane: It’s more than that. The paper suggests that this localized supervision can lead models to over-allocate capacity to high-frequency sequence "noise," like repetitive elements or neutral polymorphisms, at the expense of understanding the actual functional context (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures #pg2).

Tom: Right. So they introduce JEPA-DNA, which integrates a Joint-Embedding Predictive Architecture with traditional generative objectives to shift the learning signal from token recovery to semantic alignment. That’s the core idea.

Lu: They introduce JEPA-DNA as a model-agnostic continual training framework that uses this predictive architecture to supervise global sequence embeddings in a latent space (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗ Abstract Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP). While effective at capturing local syntax, these generative paradigms prioritize tokenlevel reconstruction over high-level functional context. We introduce JEPA-DNA, a model-agnostic continual training framework that integrates a Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. By supervising global sequence embeddings in a latent space, JEPA-DNA forces models to predict the functional representations of masked genomic segments, shifting the learning signal from token recovery to semantic alignment. We evaluate JEPADNA on seventeen diverse genomic benchmark tasks, demonstrating consistent gains in linear probing and zero-shot performance regardless of <ref:2602.17162#pg1,We evaluate JEPADNA on 17 diverse genomic benchmark tasks, demonstrating consistent gains>

Title and authors: Jane: So how does this latent prediction actually work? It sounds like a lot of math happening behind the scenes. Can you break down what the Joint-Embedding Predictive Architecture does?

Tom: Okay, so there are two main encoders involved. You’ve got a Context-encoder, Eθ, which takes the masked input sequence x and spits out a contextual latent sequence hctx = Eθ(x). Then you have a Target-encoder, E¯θbar whose weights are updated using an exponential moving average of θ to give you a latent target sequence z = E¯θbar(y).

Lu: And then the Predictor head, Pϕ, maps those context representations into that target latent space to output predicted target latents zˆ = Pϕ(hctx). The whole point is learning to align those context-derived representations with the target encoder’s latent sequence. It’s coupling token-level recovery with latent-space prediction (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Meng: So they use a dual-masking process too? That sounds like they are trying to make sure the model doesn't just memorize things easily.

Tom: Yeah, it’s a two-stage continual training procedure. Stage one keeps the backbone encoder frozen and only optimizes the attached predictor branch. Stage two updates both jointly using a composite loss function that balances language modeling, latent prediction, variance, and covariance losses (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Jane: That sounds like they are trying to keep the model stable while it learns these new semantic features. What are the actual losses they use to keep things balanced?

Lu: The total objective function minimizes Ltotal = λ1Lllm + λ2Ljepa + λ3Lvar + λ4Lcov, which balances language modeling loss, latent prediction loss using cosine similarity to ground it by forcing hglob to capture functional semantics, and variance and covariance regularization losses to prevent representation collapse (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Title and authors: Tom: And those variance and covariance regularization losses, Lvar and Lcov, are important because they prevent the latent vectors from collapsing into one another. Meng would know about that stuff when he's deploying models at scale.

Meng: Yeah, if you don't have those constraints like variance loss and covariance loss, the latent space can just become a blob where everything gets mapped to the same thing, which means you lose all that fine-grained biological detail. That would be a deployment nightmare if we were trying to use this in production.

Jane: So what do these improvements actually show in practice? They mention evaluation on seventeen diverse genomic benchmark tasks and zero-shot protocols <ref:2602.17162#pg1,on 17 diverse genomic benchmark tasks>. What kind of results did they see?

Lu: They found consistent gains in linear probing and zero-shot performance across different underlying GFM architectures or generative objectives (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Tom: They specifically showed that the optimal configuration for incremental utility involves combining regularized latent prediction with Scheduled Masking. And they found that a four-layer attention-based predictor head showed peak performance across tasks. That's concrete engineering data.

Meng: If you’re looking at deployment, having a specific architecture like that attention-based predictor head makes sense because it means you know exactly what kind of computation you’re adding to the pipeline. It gives us something measurable to optimize beyond just the overall accuracy number.

Jane: And for people who don't read papers, what does this mean for their daily life or their work? What specific biological tasks do they see this framework helping with?

Lu: They consistently improve performance on promoter recognition, splice-site classification, meQTL prediction, and causal eQTL tasks. These are all things that rely heavily on a broader regulatory context rather than just local sequence syntax (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Title and authors: Tom: And in zero-shot evaluation using the DNABERT-two backbone, they improved performance in six out of eight tasks with a mean AUROC gain of plus four point one percent. That’s a solid metric showing it works without retraining for every new thing.

Jane: So if you're just listening, this means that instead of training a whole new model for every specific genomic task, you can use the JEPA-DNA framework to improve an existing foundation model by aligning its learned representations into a semantic world model.

Lu: It paves the way toward AI understanding Nature through biologically meaningful representation learning (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Meng: From a deployment standpoint, if we can use this to ground models across disparate architectures and objectives, that’s useful. It means we can apply this latent grounding as a corrective layer to existing GFMs regardless of whether they started with MLM or NTP. That makes it more versatile for production systems.

Jane: And what about the limitations? What doesn't JEPA-DNA do well? Does it have any blind spots?

Lu: The authors mention that while they introduced this predictive latent objective, alternative self-supervised strategies like GPN-MSA leverage multisequence alignment to learn from cross-species conservation patterns, identifying functional constraints through substitution probabilities (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Tom: So the alternative methods like MSA-based ones require computationally expensive preprocessing and are limited by the availability of high-quality alignments. JEPA-DNA bypasses that need for negative samples or complex alignment pipelines by capturing functional invariants directly from the raw sequence (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Jane: That’s a key point for me to remember. So the main thing is that JEPA-DNA focuses on learning features directly from the raw sequence using a predictive objective instead of relying on external alignment or complex augmentations.

Title and authors: Lu: Exactly. It bypasses the need for negative samples or complex alignment pipelines, offering a more scalable path toward grounding genomic foundation models in biological semantics (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Tom: Alright, so to wrap up on this paper called "JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures," the main idea is shifting the learning objective from literal token reconstruction to latent feature alignment.

Jane: It’s about using a Joint-Embedding Predictive Architecture to force models to predict the functional representations of masked genomic segments, which helps them learn features that are invariant to low-level sequence noise.

Lu: And this framework is model-agnostic, allowing it to serve as a continual pre-training phase to ground existing GFMs across disparate architectures and objectives (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Meng: And the results show consistent performance gains on tasks like promoter recognition and splice-site classification across various genomic regimes (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Tom: It shows that this approach helps the models internalize the high-level regulatory logic and long-range interactions that govern genomic function (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

Jane: So, fundamentally, JEPA-DNA provides a method that captures high-level biological context more effectively than traditional generative objectives alone. We're looking at how this changes the way we train models to understand Nature through biologically meaningful representation learning (JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architecture Ariel Larey1 Elay Dahan2 Amit Bleiweiss3 Raizy Kellerman4 Guy Leib4 Omri Nayshool4 Dan Ofer4 Tal Zinger4 Dan Dominissini4 Gideon Rechavi4 Nicole Bussola5 Simon Lee5 Shane O’Connell5 Dung Hoang5 Marissa Wirth5 Alexander W. Charney5 Nati Daniel1,∗ Yoli Shavit1,∗).

The paper's summary: Tom: So, to recap, JEPA-DNA is taking these big genomic foundation models that are trained just on predicting the next letter or token and they're trying to change their goal to actually learn the underlying biological functions instead of just memorizing the sequence.

Jane: That’s a good way to put it. So it’s not about getting better at spelling DNA anymore, but about understanding what that DNA actually *does* in a cell or an organism.

Lu: Exactly. The paper says they use this Joint-Embedding Predictive Architecture to force the model to predict what the missing parts of the sequence should look like in a meaningful sense, which is different from just predicting the next nucleotide based on local context alone.

Meng: I’m curious about that "meaningful sense" part. How are they making sure that prediction isn't just some random guess? That sounds like a big challenge when you’re trying to teach an AI biology without actually having the answers written down in a perfect textbook.

Tom: They tackle that by adding these regularization terms. They balance the standard language modeling loss, which keeps it grounded in the actual sequence data, with this new latent prediction loss, which forces that latent representation to align with what they expect it to be.

Jane: So they’re essentially saying: "Be a good sequence predictor *and* try really hard to make your internal representation match the biological reality we're trying to capture." That sounds like a balancing act.

Lu: Right. And they add variance and covariance regularization losses, which is important because if you just force the latent space to align, it can collapse into a single point where everything looks the same. These losses keep the vectors spread out so they actually represent different biological concepts.

Meng: So what does this actually mean for deploying these models? If I’m running something in production, how does this framework make it more robust against those kinds of collapses you mentioned?

Tom: It means if the model starts to lose its ability to distinguish between similar but functionally different genomic regions, these regularization terms should keep the latent space healthy and diverse enough for it to maintain accuracy on downstream tasks.

Jane: And when they look at the results, they show that this isn't just theoretical math. They test it on seventeen different kinds of genomic problems—things like recognizing where a gene starts or where a splice site is—and they see solid improvements across the board.

Lu: They even showed gains in zero-shot performance using a backbone called DNABERT-two meaning the model can pick up new tasks without needing specific training for every single one of those seventeen benchmarks.

Tom: And that zero-shot gain, they report a mean AUROC increase of plus four point one percent on six out of eight tasks. That’s concrete evidence that this latent alignment is actually building something useful, not just adding noise.

Jane: It really changes things for researchers because it suggests that we don't always need to throw away the existing foundation models and start from scratch to get better at complex biology. We can just add this layer of latent semantic understanding on top.

Meng: So if I’m thinking about scaling this up, what’s the main thing I need to worry about? Is it computationally expensive compared to just using a standard transformer for inference?

Lu: The predictor head architecture matters here. They found that a four-layer attention-based predictor head gives them the best performance across all those tasks. So, when you deploy, you’re not just adding a simple feed-forward layer; you need to choose an architecture like that to get the most benefit from the latent prediction loss.

Tom: I like that focus on a specific architecture because it gives us something tangible to tune and measure beyond just the overall accuracy number. It makes sense for engineering, you know?

Jane: So, fundamentally, this paper shows a way to move beyond just token-level reconstruction and toward learning representations that are actually aligned with the functional biology of genomic data. This opens up avenues for AI systems to understand nature through representations that make biological sense.

The paper's improvements: Tom: So, to summarize what we just heard, JEPA-DNA is using that Joint-Embedding Predictive Architecture to force genomic models to align their internal representations with actual biological functions rather than just memorizing the sequence text.

Jane: Right, so it’s moving the goalposts from "can this model predict the next base" to "can this model represent a promoter region in a way that makes sense biologically."

Lu: The authors suggest that by using that latent prediction loss combined with those variance and covariance regularization terms, you get representations that are not just aligned but also stable. They are actively preventing the latent space from collapsing into a single point where all genomic features look identical.

Meng: I’m thinking about what this means for the actual training pipeline. If we use this continual training strategy, we're still doing two stages, right? Stage one freezing the encoder and just optimizing that predictor branch before you start updating both together.

Tom: That’s right. The framework is designed for incremental utility, so you can start by just fine-tuning the prediction head while keeping the main backbone frozen to stabilize things early on before you introduce all those complex joint updates.

Jane: So they’re giving us a step-by-step way to introduce this new semantic understanding without completely retraining the massive genomic model from scratch every time we want to test it on a new task.

Lu: Precisely. And they show that when you combine that latent prediction loss with scheduled masking—where you gradually increase how much you mask the sequence during training—you get the best results across different benchmark tasks. It’s about timing those learning signals correctly.

Tom: The result they got on tasks like meQTL and causal eQTL predictions is pretty strong, showing that this isn't just theoretical alignment; it actually boosts performance on real biological inference tasks.

Jane: And for someone just listening, what this changes is the way we view foundation models in science. It suggests that the next generation of AI biology won't just be about pattern matching syntax; it’s going to be about learning a latent world model that captures the regulatory logic governing how DNA actually works.

Lu: That’s the big picture. They are paving the way toward AI understanding nature through representation learning that is biologically meaningful, which is a shift from just looking at sequence patterns to capturing functional context directly in the latent space.

Conclusion: Tom: So we’ve gone through the mechanics of JEPA-DNA, seeing how they shift the learning objective from just token reconstruction to aligning latent features for genomic function, and they showed solid gains on tasks like promoter recognition and splice-site classification.

Jane: That means we can start thinking about foundation models not just as text predictors but as systems that actually have some grasp of biological context.

Lu: The implication is that we’re moving toward a way to train models where the representation learned is invariant to low-level sequence noise, meaning they capture the high-level regulatory logic instead of getting stuck on repetitive sequence elements.

Meng: From a production standpoint, this makes sense because it means if we fine-tune an existing genomic model using this latent alignment technique, we can expect better stability and more meaningful results on downstream tasks without having to overhaul the whole training pipeline.

Lalam: I'm confident in my assessment that JEPA-DNA successfully grounds these models by forcing them to learn functional semantics through that predictive objective.

Tom: It’s a clean way to introduce biological knowledge into the AI loop without requiring us to manually engineer every single functional feature into the loss function.

Jane: It really opens up avenues for AI understanding nature through representation learning that is actually grounded in biology, which is something we’ve been chasing for a long time.

Lu: Future work will look at taking this JEPA objective and applying it to training models from scratch on massive, multi-species datasets, which would test its limits in a much broader context.

Meng: I'm just wondering about the efficiency of that latent prediction step; if it adds too much overhead to inference time, we’ll struggle with deployment at scale.

Tom: That’s a fair concern, Meng, but the authors found an optimal configuration with a four-layer attention-based predictor head which suggests there is a specific architectural sweet spot for efficiency.

Jane: So it seems like the next step is to see if we can use these latent models to predict things outside of genomic data, maybe even connecting sequence features to clinical outcomes.