Scaling RNA Foundation Models through Iterative Refinement

summary

The gist

The gist: Eco-RNA introduces an 80Mparameter RNA foundation model that makes inference depth adjustable through learned iterative refinement, establishing internal inference computation as an

In short

Eco-RNA is an 80M parameter RNA foundation model that makes inference depth adjustable through learned iterative refinement. It uses a shared Transformer core repeatedly to refine sequence representations. This method demonstrates that internal inference computation, controlled by iteration count, is a key scaling dimension for RNA models, outperforming larger models with fewer parameters.

Key concepts

Iterative Refinement: Eco-RNA refines sequence representations by repeatedly applying a shared core transformation. The number of times this refinement happens (the iteration count) is learned during training and controls the effective depth of the model without increasing the total parameter count. This allows for adjustable inference depth.
Recurrent Depth Framework: The architecture reuses a single shared Transformer core multiple times, decoupling the executed inference depth from the fixed number of model parameters. A recurrent update equation dictates how information is retained and updated across these repeated applications, enabling internal scaling.
One-Sided Reconstruction Anchor: A training criterion that anchors deeper reconstructions to a shallower reference point within the same trajectory. This loss function encourages the model to learn meaningful, progressively refined representations during the iterative process, guiding the model toward better deep outputs.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 18 episodes
Federated Learning — explained in 17 episodes
Flow Matching — explained in 16 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes

This episode discusses

The paper

Scaling RNA Foundation Models through Iterative Refinement · Read on arXiv

Pengan Chen, Liang Hong, Xinyuan Liu, Han Huang Ziqian Lin, Jiayang Chen, Yimin Fan, Jiyue Jiang Yu Li

Department of Computer Science and Engineering, The Chinese University of Hong Kong

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Scaling RNA Foundation Models through Iterative Refinement".

Jane: The gist: Eco-RNA introduces an 80Mparameter RNA foundation model that makes inference depth adjustable through learned iterative refinement,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, what does the actual mechanism look like? They are using a shared, input-conditioned Transformer core and applying it repeatedly based on a learned iteration count.

Jane: That sounds like they’re reusing the same core but feeding it different information at each step, which is interesting because it suggests that the refinement isn't just adding more layers.

Lu: Exactly. They use a recurrent-depth framework where one shared CORE is reused K times, and they have a prelude representation that gets injected at every iteration to supply sequence information directly into the update process.

Tom: So, the math involves this recurrence relation s(k) equals N r alpha⊙s(k-one) plus delta⊙Be, k equal one through K <ref:2610.13106#pg3>. It's a learned way of deciding how much of the previous state to keep versus how much new input is added at each step.

Meng: That sounds like a complex update rule to manage for training, but if it works well, it means we can tune the depth without retraining the entire model structure from scratch for every use case.

Lalam: The paper also mentions their training objective involves masked reconstruction loss at sampled terminal depths, along with relative displacement penalty and a one-sided reconstruction anchor.

The paper's summary: Jane: That leads into the training objectives, right? They aren't just doing simple prediction tasks; they are optimizing the model to have these specific properties.

Tom: They use this one-sided reconstruction anchor loss, which is pretty novel. It basically trains the model to favor deeper endpoints whose reconstruction loss is higher than a shallower reference from the same path.

Lu: That anchor loss reinforces those deeper reconstruction points, effectively training iterative computation within a self-supervised backbone. It's like they are guiding the model to learn better representations at different levels of refinement simultaneously.

Meng: From an engineering standpoint, that sounds like it helps stabilize training because you’re giving the model a targeted way to improve its performance at different scales without just relying on one single loss signal.

Lalam: And they show results on the RNAGym fitness benchmark, where increasing depth from one iteration up to four iterations raises the aggregate absolute Spearman correlation from zero point two four two to zero point two eight six, and at four iterations, Eco-RNA surpasses Evo two 40B on all three aggregate fitness metrics despite being five hundred times fewer parameters <ref:2610.13106#pg2>.

The paper's improvements: Tom: So, to wrap this up, the main point from "Scaling RNA Foundation Models through Iterative Refinement" is that internal inference computation can be treated as a scaling dimension for these RNA foundation models.

Jane: It seems like they've established that a single recurrent checkpoint can give you several different operating points for compute and performance. They are showing how to use this iterative refinement to tune the depth based on what you need for a specific task.

Lu: The results connect this internal inference capability to both the way the model learns its representations and how well it performs downstream predictions, specifically showing gains in splicing and aptamer assays where deeper predictions matter most.

Meng: For deployment, that means we can have a fixed-size model that we can reconfigure dynamically for different tasks just by changing the iteration count at inference time, which is a huge win for efficiency.

Lalam: The paper suggests that this depth adjustment reduces held-out reconstruction loss at multiple stages of pretraining and those depth-dependent gains persist even after adapting to sequence labels or chemical reactivity.

Conclusion: ---: Conclusion ---

Tom: So, to wrap up "Scaling RNA Foundation Models through Iterative Refinement," the main point is that internal inference computation can be treated as a scaling dimension for these RNA foundation models.

Tom: Yeah, it’s about linking the recurrent state dynamics directly to a criterion in prediction space. It connects representation learning and prediction training in a way that was previously separate.

Jane: That's really interesting because it shows how you can use the model's own internal computation to guide its learning process during pretraining.

Lu: The separation of the task-level criterion for training additional recurrent computation within a shared sequence model is something that opens up a lot of possibilities for future architectures.

Meng: If we can tune the budget this way, it means we can optimize compute per prediction much more effectively on real traffic, not just on some idealized benchmark.

Lalam: I'm sure these findings about Eco-RNA will keep researchers looking at how to scale these foundation models for more complex biological sequences.

Tom: We'll be talking about that next week when we look at some of those agents and alignment papers from arXiv, but for now, this is how we use internal computation to tune the model’s depth.

More episodes

← Home