PUMA: Learning a Mutation-Aware Vocabulary of Protein Units

arXiv:2503.08838 · cs.CL, q-bio.QM · Submitted 2025-03-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "PUMA: Learning a Mutation-Aware Vocabulary of Protein Units".

Jane: PUMA introduces a method for discovering evolutionarily meaningful protein sequence units by explicitly incorporating mutational relationships into an iterative merging process,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So Jane, we've been looking at the paper "PUMA: Learning a Mutation-Aware Vocabulary of Protein Units," and it seems like the main idea is this method for finding these fundamental protein units by directly incorporating how mutations happen into the process.

Jane: Right, Tom? It’s about building a vocabulary where each unit isn't just some random sequence pattern, but something that also shows its evolutionary relatives through their changes.

Lu: I think what PUMA is doing really excites me because it’s bridging that gap between statistical frequency and the actual history of evolution in a way we haven't fully explored before. They are moving beyond just finding domains or motifs and aiming for these truly fundamental, contiguous blocks of information.

Meng: From an engineering standpoint, I’m curious how they manage to make that connection between frequency patterns and the substitution matrices computationally feasible without it just exploding in complexity? We need a method that scales well for real-world protein sequences.

Lalam: I see this as incredibly powerful because if we can build a vocabulary that explicitly tracks parent and child relationships through mutations, the culture of how we model biological systems could shift toward more biologically coherent structures.

Tom: Exactly, Lalam. The core thesis of this PUMA method is that it constructs a vocabulary by integrating evolutionary substitution patterns into an iterative merging process, giving us this interpretable way to see how units relate genealogically. Jane, can you tell us what they claim about the overall goal of this research?

Jane: Absolutely. The paper explains that PUMA aims to identify a set of protein units where each unit is defined as a contiguous segment of the amino acid sequence discovered through mutation-aware frequency analysis. The authors emphasize that these units are not only statistically recurrent patterns but also evolutionary units, explicitly linked to related variants via amino acid substitutions guided by matrices like BLOSUM or PAM.

Lu: That linkage through substitution matrices is the key here; it’s not just about co-occurrence in a sequence, it’s about tracing the diversification of those patterns over time. It provides a hierarchical organization that connects parent units and their mutational variants directly into the vocabulary structure.

Meng: So they're creating a structure where the "parent-child and sibling relationships are preserved in the vocabulary structure," which is how this PUMA genealogy organizes things, right? But what about those constraints they put on the process, like when it comes to vocabulary size or alignment scores?

Tom: That's a great point about the constraints. The paper details parameters like the vocabulary size ranging from eight hundred up to fifty-one thousand two hundred and how they adjust substitution matrices based on whether they are using BLOSUM62 or PAM250. They even mention cutting off alignment scores at zero point seven, zero point eight, or zero point nine to keep things computationally manageable.

Paper summary: Jane: It sounds like they are being very deliberate about balancing the depth of the evolutionary tracing with the practical limits of what a computer can handle in terms of search complexity. The authors also mention filtering patterns using frequency cut-offs, which they applied at values like zero zero point zero zero five, and zero point one to filter out coincidental sequences.

Lalam: That deliberate filtering shows a mature approach to this kind of discovery; it’s not just throwing every pattern at the wall and hoping something meaningful emerges. It suggests that the definition of a "meaningful fragment" is being rigorously refined through this iterative process.

Lu: And what's particularly fascinating is how they validate this structure using metrics like the SAME sibling rate to show that mutations within a PUMA family actually correlate with favorable biological outcomes, specifically showing they are more likely to be clinically benign. That connects the sequence discovery directly to observable biological reality.

Meng: I'm interested in how they connect this vocabulary back to external knowledge, because having a list of units isn't useful unless we can map them to known functions, and they used graph-aware topic modeling for that. That mapping process sounds complex to implement reliably in practice.

Tom: It is certainly intricate, but the results suggest it works well; the paper notes that their approach outperformed all baselines when mapping these discovered units to Gene Ontology terms. This suggests that PUMA isn't just generating arbitrary strings, but rather discovering units with inherent functional coherence.

Jane: So when we look at the broader implications, the paper points out that the ESM-two protein language model actually contextually prefers substitutions that stay within these PUMA families. This means PUMA is capturing an implicit evolutionary structure that large models have already learned, which is really interesting for understanding how those models work.

Lu: The potential here for AI culture is huge; if we can build a vocabulary that mirrors this deep evolutionary history, it could fundamentally improve how generative AI systems predict and design new proteins, moving them away from purely statistical patterns toward evolutionarily informed designs.

Meng: From a practical standpoint, if this method can reliably generate these units, it means we could start designing novel protein sequences that are not just functional on paper but are also structurally sound and evolutionarily plausible. That moves us closer to creating new therapeutic targets or industrial enzymes based on this new vocabulary.

Lalam: I think the most profound impact is in how we conceptualize life itself; if we can segment the language of life into these units that explicitly track their evolutionary lineage, it gives us a richer framework for understanding biological complexity. This kind of organized knowledge could help AI systems develop more nuanced and contextually aware biological intelligence.

Paper summary: Tom: So to wrap up this section on the "PUMA: Learning a Mutation-Aware Vocabulary of Protein Units," we've seen how they define units as both statistically recurrent and evolutionarily connected segments, validated by their correlation with benign mutations in assays. The authors show this vocabulary aligns well with external knowledge from language models and functional annotations.

Jane: And the conclusion they draw is that this hierarchical organization provides a new understanding framework for the compositional nature of proteins. It moves beyond simple pattern matching into a genealogical view of protein structure and function.

Lu: I think what's really important is that this research allows us to leverage data-driven models to discover these units directly from the one-dimensional sequence, which challenges some of the traditional reliance on three dee structural data for unit identification.

Meng: My concern remains scalability; they've mentioned generating mutations for units shorter than three or longer than twelve residues is computationally constrained, which means we have to deal with some limitations in the diversity of variants we can actually explore.

Lalam: Even with those constraints, the ability to generate a vocabulary that respects evolutionary history is significant because it builds a foundation for AI that understands deep biological context rather than just superficial sequence similarities.

Tom: It sounds like we've covered the main claims of "PUMA: Learning a Mutation-Aware Vocabulary of Protein Units," from its focus on mutation-aware merging to its validation against biological outcomes and functional alignment with language models. We’ve also touched on the practical considerations regarding computational constraints and the broader vision for AI in biology.

Jane: Right, and as we wrap up this part of our discussion, the authors conclude that their PUMA method successfully establishes these units as discrete building blocks that collectively span entire protein sequences without gaps. It really solidifies the idea of a structure built on both frequency and history.

Lu: That structural foundation is what makes this research so compelling because it’s not just about finding more patterns; it’s about organizing those patterns into a meaningful, biologically relevant hierarchy. It opens up new avenues for how we model protein evolution and function using sequence data alone.

Meng: So, to think about the impact on the world, if this vocabulary discovery method becomes standard, it could drastically speed up the design of novel enzymes or diagnostic tools by ensuring the sequences we generate are inherently evolutionarily sound. That practical application is where I see the immediate value.

Lalam: And on a cultural level, this work pushes us to think about what it means for AI to truly understand biological systems—not just mimic them, but structure their underlying evolutionary grammar. This kind of deep structural understanding could inspire entirely new ways we approach complex problem-solving in any domain.

Conclusion: Tom: So, we've seen how PUMA builds a vocabulary of protein units by merging statistical frequency with evolutionary history, and now Jane, what do you think about that title?

Jane: I think the title perfectly captures the core idea; it shows this new method connects sequence patterns directly to their evolutionary path through mutations. It’s a really clear way to describe something complex.

Lu: From an AI perspective, the real power is that this isn't just about finding sequences; it’s about creating a genealogical structure for proteins that large language models can actually understand and use for design.

Meng: I'm thinking practically, though; if this vocabulary gives us a better way to build protein units, does that translate into faster or more reliable ways to engineer useful enzymes?

Lalam: I see it as a cultural shift; instead of just seeing sequences as static strings, we start viewing them through an evolutionary lens that helps us structure and predict biological complexity in a much deeper way.

Tom: Exactly! The authors, they’re really showing how this iterative process yields units that are both statistically common and biologically connected through mutation.

Jane: And the authors are emphasizing that these units aren't just arbitrary fragments; they serve as discrete building blocks for entire protein sequences, which is a big deal for coverage.

Lu: They prove that by explicitly incorporating substitution matrices like BLOSUM or PAM into the merging process, we can generate a vocabulary where parent-child relationships are actually preserved in the structure itself.

Meng: That level of explicit connection between units and their mutational variants sounds incredibly useful for building predictive models, though I wonder how much computational power is needed to run that iterative merging process effectively.

Tom: Well, we’re going to look at the conclusion of the PUMA paper now; Tom and Jane will break down exactly what this means for the future of protein sequence discovery.

Department of Computer Engineering, Boaziçi University · C. Eugene Bennett Department of Chemistry, West Virginia University · Department of Pharmaceutical Chemistry, Istanbul Medipol University · Istanbul Medipol University Research Institute for Health Sciences and Technologies (SABITA)

cs.CL, q-bio.QM

Submitted: 2025-03-11

Updated: 2026-10-01

Comments: 23 pages, 10 figures, 9 tables, 1 algorithm

Code: https://github.com/boun-tabi-lifelu/PUMA

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: PUMA introduces a method for discovering evolutionarily meaningful protein sequence units by explicitly incorporating mutational relationships into an iterative merging process, offering an

Key concepts

PUMA Unit
A PUMA unit is a specific, biologically meaningful segment of a protein sequence. It is defined as a fragment that is both statistically recurrent (frequently appearing) and evolutionarily related to other variants through observed amino acid substitutions. These units form the core vocabulary.
Frequency-Based Merging
This initial step identifies sequence patterns that appear frequently in the data. It works like tokenization, grouping similar sequence fragments together based on statistical recurrence before moving on to evolutionary analysis. This establishes the basic building blocks of the vocabulary.
Mutation-Aware Expansion
Once initial units are found, this step uses substitution matrices (like BLOSUM) to predict and identify variants related by amino acid changes. This iteratively expands the vocabulary by connecting parent units to their specific mutational variants, creating a genealogical structure.

Terminology

Summary

PUMA introduces a method for discovering evolutionarily meaningful protein sequence units by explicitly incorporating mutational relationships into an iterative merging process, offering an interpretable vocabulary that bridges statistical frequency and evolutionary history.

How it works

The PUMA method constructs a vocabulary of protein units and their genealogical structure by integrating evolutionary substitution patterns into an iterative merging process. The process begins with Step 1: Frequency-Based Merging, where statistically recurrent patterns are identified. This is followed by Step 2: Mutation-Aware Expansion, where an iterative algorithm guided by substitution matrices identifies evolutionarily related variants. This results in a PUMA Genealogy that organizes units into distinct families, explicitly connecting parent units and their mutational variants.

A PUMA unit is formally defined as a contiguous segment of the amino acid sequence that represents a biologically meaningful fragment discovered through mutation-aware frequency analysis. This definition rests on several integrated principles:

  1. It is a statistically recurrent pattern, identified through iterative merging of frequently co-occurring sequence fragments, following the tradition of data-driven tokenization [15].

  2. It is simultaneously an evolutionary unit, explicitly connected to related variants through amino acid substitutions guided by established substitution matrices (BLOSUM [29], PAM [30]), which encode observed patterns of evolutionary change.

  3. It belongs to a genealogical family, a set of related units connected through plausible mutations, where parent-child and sibling relationships are preserved in the vocabulary structure.

  4. It serves as a discrete building block for complete sequence coverage, ensuring that PUMA units collectively span entire protein sequences without gaps.

Key Components and Constraints

The PUMA process is governed by specific parameters that control the discovery of units:

((

Vocabulary Size: Ranging from 800 to 51,200, doubling at each step.

Substitution Matrices: BLOSUM62, BLOSUM45, PAM70, and PAM250 were tested. As a parameter approaches 1.0 (PAM250), PUMA converges toward standard BPE behavior.

Alignment Score Cut-off (a): Tested at 0.7, 0.8, and 0.9; values below 0.7 were excluded due to exponential search complexity, while PAM250 was restricted to 0.8 and 0.9 for computational feasibility as it approaches the limit of feasibility as it approaches 1 (Page 5).

Frequency Cut-off (f): Cut-off values of 0, 0.005, 0.05, and higher were applied to filter coincidental patterns.

Mutations for units shorter than 3 and longer than 12 residues are not generated due to computational complexity constraints (Page 4). The process repeats until the vocabulary size target is met (Algorithm 1).

Validation of Biological Relevance

The biological relevance of PUMA units is validated through multiple lines of evidence. The validation strategy demonstrates that mutations within a PUMA family correlate with favorable biological outcomes: they are more likely to be clinically benign variants and to yield higher fitness in experimental assays.

(Page 3) This is quantified using the SAME sibling rate metric, measuring the proportion of cases where the protein unit at a specific site before and after a mutation are siblings within the PUMA genealogy. For clinical substitutions, PUMA families consistently favor benign mutations, with a mutation being nearly three times more likely to be benign than pathogenic on average at vocabulary size 51,200.

(Page 3) In experimental fitness assays (DMS), the results mirror these findings: Across models at vocabulary size 51,200, the Positive bin showed a mean SAME sibling rate of 0.067 versus 0.042 for the Negative bin.

Contextual and Functional Alignment

PUMA units are shown to align with external biological knowledge through several contextual analyses:

  1. ESM-2 Contextual Preference: The ESM-2 protein language model contextually prefers substitutions that remain within PUMA families, demonstrating that PUMA captures evolutionary relationships implicitly learned by large models.

  2. Topic Modeling: A graph-aware topic modeling approach, which incorporates the genealogical structure via a weighted adjacency matrix A, was used to establish strong associations between discovered units and Gene Ontology terms. This method outperformed all baselines in mapping units to functional annotations (Figure 9a).

  3. Functional Specificity: Analysis of PUMA families rooted in polyalanine units confirmed their biological significance, as these parent units ranked highly for nucleic acid-related functions (DNA binding, RNA binding), suggesting that the genealogical grouping captures functionally coherent motifs.

Comparison with Existing Methods

PUMA's contribution is highlighted by comparing its output against frequency-based methods like Byte-Pair Encoding (BPE).

Improvements for AI systems

Based on the provided scientific paper, here are specific improvements that could be made to existing AI systems, such as Protein Language Models (pLMs) and sequence tokenization methods like Byte-Pair Encoding (BPE), using the PUMA framework:

  1. Enhance Protein Sequence Tokenization for pLMs:

  2. Improve Interpretability of Black-Box Models:

  3. Develop Evolutionarily Grounded Protein Language Representations:

  4. Create Biologically Informed Feature Engineering Pipelines:

  5. Enhance Protein Sequence Tokenization for pLMs:

The current tokenization methods (like BPE) are frequency-based and fail to capture the explicit evolutionary relationships between sequence variants, treating them as distinct entities rather than related units.

The improved system would use the PUMA framework to generate a vocabulary based on Mutation-Aware Merging. This means instead of just merging the most frequent adjacent amino acid pairs, it would iteratively merge pairs that are frequently co-occurring AND whose resulting mutations maintain high alignment scores (guided by BLOSUM/PAM matrices) and high frequency cut-offs.

The improved system will produce a vocabulary where units are explicitly linked genealogically—parent, child, and sibling relationships are encoded. This transforms the input sequence from a flat string into a structured language of protein units that respects evolutionary history.

  1. Improve Interpretability of Black-Box Models:

Current methods for interpreting pLMs often focus on identifying specific learned features (e.g., binding sites) rather than providing a general, human-understandable vocabulary for the language itself.

The improved system would integrate PUMA's genealogical structure directly into the model's representation layer. Instead of relying solely on attention maps to pinpoint residues, the system would use PUMA units as words to represent segments in a structured manner.

This allows for:

  • Identifying which specific protein units (PUMA families) are driving predictions or high-scoring outputs within the model’s internal representations.

  • Providing a clear, hierarchical map of how abstract model features correspond to biologically meaningful, evolutionarily related sequence fragments.

  1. Develop Evolutionarily Grounded Protein Language Representations:

The current representation in pLMs is implicit and learned from massive data, which may not explicitly reflect known evolutionary constraints or functional motifs at the unit level.

The improved system would utilize PUMA's output—the vocabulary and its genealogical structure—to create an explicit, evolutionarily grounded language model. This means the model is trained not just on sequences, but on a structured graph of units and their mutational relationships.

This allows for:

  • Training models that are inherently biased toward biologically plausible substitutions (those within PUMA families).

  • Creating representations where functional information is explicitly linked to evolutionary context, moving beyond simple statistical co-occurrence.

  1. Create Biologically Informed Feature Engineering Pipelines:

When using pLMs for downstream tasks (e.g., function prediction or variant classification), current methods often rely on external annotations or simple sequence features.

The improved system would use PUMA's functional alignment capabilities to create a feature engineering pipeline that explicitly incorporates the discovered units and their known functional associations (via GO term mapping).

This allows for:

  • Developing unit-specific features where the presence of a specific PUMA family (e.g., polyalanine tracts) is treated as a high-level biological signal, rather than just a collection of individual amino acids.

  • Performing targeted searches or classification tasks that are intrinsically linked to known evolutionary and functional motifs discovered by PUMA, leading to higher sensitivity and specificity in predicting clinically relevant outcomes (benign vs. pathogenic variants).

Abstract

Modeling protein sequences as a language has made language models a powerful tool in computational biology, yet the language itself remains poorly understood. A key step toward understanding it is identifying its constituent units. In natural languages, morphemes can occur in multiple forms; similarly, in proteins, mutations can give rise to variations of a unit that persist through evolution, forming families of related units. We introduce PUMA (Protein Units via Mutation-Aware Merging), an algorithm that learns protein units from sequence and explores their mutational variants using substitution matrices, forming a genealogy of unit families. Our results show that mutations remaining within a PUMA family are more often benign than the substitution matrix alone predicts, and that PUMA genealogy improves molecular function representations compared to treating units independently. A case study of a unit family demonstrates relatedness beyond homology. PUMA achieves competitive performance on downstream tasks when used as a protein language model tokenizer. Moreover, collapsing units into families results in a smaller embedding table and faster training. Together, these results support PUMA as a biologically grounded protein vocabulary that organizes protein units into plausible families of mutational variants. The source code is available at https://github.com/boun-tabi-lifelu/PUMA.

Sources

Related papers