RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

summary

The gist

The gist The first retrieval-based gLM that scales pretraining to longer contexts (100× longer than existing MSA-based gLMs) allows it to capture both across-species evolutionary relationships and

In short

RAGenome is a new retrieval-based genomic language model that scales pretraining to much longer contexts (100 times longer than existing models) by efficiently using whole-genome alignments. It improves gene finding performance significantly while maintaining strong performance on variant effect prediction, unifying evolutionary modeling and long-range genomic understanding.

Key concepts

Retrieval-based gLM: This is a type of language model for genomic data that uses a retrieval mechanism. Instead of reading every piece of input data linearly, it searches a large database (like a whole-genome alignment) to find relevant sequences to help predict or understand the query sequence.
Whole-Genome Alignment (WGA): A WGA is a collection of aligned sequences from multiple species, like 100 vertebrates. The paper uses this as its knowledge base. Crucially, RAGenome ignores unaligned 'gap tokens' in the WGA to make the retrieval process faster and more efficient.
Taxonomy Embeddings: These are numerical representations of a species' place in the tree of life, from superkingdom down to species level. By combining these embeddings, RAGenome allows phylogenetically related species to share information about their evolutionary history within the model.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts · Read on arXiv

Frederikke Isa Marin, Panagiotis Antoniadis, Dionysia Danai Brilli, Andreas Bjerregaard, Rachael DeVries, Yan Li, Ole Winther, Wouter Boomsma

University of Copenhagen · Novo Nordisk A/S · Technical University of Denmark

The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data. However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks. Recently, MSA-based pretraining has been proposed as an efficient alternative, but existing models are limited to short input contexts, restricting their use to short-range tasks, such as variant effect prediction. In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100 times longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions. Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants. RAGenome provides competitive gLM performance at a fraction of the training cost, unifying evolutionary modeling and long-range capabilities within a single, flexible, scalable framework. Code is available at https://github.com/PanosAntoniadis/RAGenome.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts".

Jane: The gist The first retrieval-based gLM that scales pretraining to longer contexts (100× longer than existing MSA-based gLMs) allows it to capture both across-species evolutionary relationships and within-species longer-range interactions.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we’ve looked at "RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts," and it’s clear this is a new approach to making genomic language models handle much longer sequences than what was possible with older MSA pretraining methods > We saw the authors, Panos Antoniadis and his team, managed this by using a retrieval mechanism that smartly throws out those useless gap tokens from whole-genome alignments >

Jane: Right, so the key thing is scaling those models up to thirteen thousand three hundred twelve nucleotides which is one hundred times longer than existing ones > The paper shows that they can capture both across species evolutionary relationships and interactions within a single species over much longer distances >

Lu: It’s clever because it keeps only the aligned nucleotides in the model training, which makes it way more efficient than just processing everything naively > And the taxonomy embeddings help organize those relationships phylogenetically >

Meng: From an engineering standpoint, that efficiency is huge because you can train on a much bigger dataset without needing a super massive compute budget right away > But they do have to be careful about what they’re doing when it comes to inference; there are still alignment requirements involved >

Tom: That’s fair. They show it works well on things like gene finding and variant effect prediction, which proves that the model maintains short-range performance even with this long-range understanding > It really shows how important it is to have models that can handle both fine-grained local details and those massive long-range connections simultaneously >

Jane: It suggests we could finally start making predictions about complex gene structures or variant effects that depend on looking at a much wider genomic neighborhood > This framework unifies the evolutionary modeling aspect, which is the deep history of life, with the long-range structural understanding of the genome in one flexible architecture >

Lu: The architecture itself is really interesting too; they use local attention for within-species stuff and cross-attention for those big evolutionary jumps > That combination seems to be what allows it to work so well on different scales >

Meng: I wonder how practical this is for real labs, though they do flag that the retrieval mechanism requires a whole-genome alignment at inference time, and they need to worry about the model size needed for the best results >

Tom: That’s a fair point. But overall, RAGenome is presented as the first gLM to unify explicit evolutionary modeling with long-range genomic understanding within a single, efficient architecture > The future work involves further scaling of RAGenome in model size and training data to close that gap with even larger models >

Conclusion: Tom: So, RAGenome is this paper's name: "RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts," and it was written by Panos Antoniadis and his team.

Jane: Right, it’s about how they took existing genomic language models and made them way bigger by using a retrieval system that smartly ignores the gap tokens in the alignments.

Tom: That’s right. It shows they can scale those models up to thirteen thousand three hundred twelve nucleotides, which is one hundred times longer than what we used to see from standard MSA pretraining methods.

Lu: What’s interesting is that this mechanism lets the model capture both across species evolutionary relationships and interactions within a single species over much longer distances.

Meng: From an engineering standpoint, that efficiency is huge because you can train on a much bigger dataset without needing a super massive compute budget right away.

Lalam: I see this as unlocking a whole new level of genomic modeling where we aren't limited by how much sequence context we can cram into the system.

Tom: Exactly. So, what does this mean for us listening? It means models that can actually learn those long-range evolutionary relationships across different species in a way that was previously impossible with these standard methods.

Jane: It suggests we could finally start making predictions about complex gene structures or variant effects that depend on looking at a much wider genomic neighborhood.

Tom: It’s about unifying the evolutionary modeling aspect—the deep history of life—with the long-range structural understanding of the genome in one flexible framework.

Lu: The architecture itself is really interesting too, they use local attention for within-species stuff and cross-attention for those big evolutionary jumps.

Meng: I wonder how practical this is for real labs, though we still have to worry about those alignment requirements at inference time and what kind of model size you actually need to get the best results.

Lalam: The vision here is that a model that understands both the local structure and the deep evolutionary history could help us predict complex disease mechanisms with much more context.

Jane: It really does show how important it is to have models that can handle both the fine-grained local details and those massive long-range connections simultaneously.

Tom: Next up, we're going to look at exactly how they did this scaling, starting with the retrieval mechanism itself.

More episodes

← Home