2608.09432-ZetaGPT: A Reference Implementation of Positional–Encoding–Free State–Space–Attention Language Models

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "ZetaGPT: A Reference Implementation of Positional–Encoding–Free State–Space–Attention Language Models".

Jane: The paper was written by Róisín Luo from University of Galway.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary — Tom, Jane, Lu, Meng and Lalam discuss the paper 'ZetaGPT: A Reference Implementation of Positional–Encoding–Free State–Space–Attention Language Models' — the thesis, the key findings and why it matters.: Tom: So we've got a really intriguing paper today, and it's one of those that makes you question a core assumption in how language models work. It's essentially asking whether transformers actually need positional encodings at all.

Jane: Right, and that is a big question, because every model we're used to, from GPT-2 to Qwen, they all inject position information somehow, whether it's learned embeddings or RoPE. This paper says, what if we let the architecture itself handle order implicitly, through state-space dynamics?

Lu: It's a hybrid approach, really. You keep the expressive power of self-attention, but you add a causal state-space module before it in each block. That module's recurrent state summarizes the past, so by the time attention sees the tokens, they're already position-aware.

Meng: And the model they built, ZetaGPT, comes in three sizes, with the default being just over 34 million parameters. So it's deliberately small, meant as a reference implementation for research and education, not a competitor to the big frontier models.

Lalam: The bigger picture here is that explicit positional encodings have always been a bit of a patch. They work, but they don't naturally extend to longer contexts, and this design philosophy of letting recurrence encode order is exactly what we're seeing in Kimi Linear and similar architectures.

Tom: The paper also walks through a very complete training pipeline, from tokenizer training all the way to RLHF and chain-of-thought reasoning via GRPO, which is notable for a model this small.

Jane: And they claim it's the first open-source small language model without explicit positional encoding. That's a strong claim, but the code is out there, so it's verifiable.

Lu: What I found compelling is the diagnostic work, they actually observed the state-space modules learning multiple memory timescales during training. Long-memory channels and short-memory channels emerged on their own, which is the empirical evidence that position is being represented implicitly.

Meng: Yeah, it's not just a trick, there's real evidence of the mechanism developing, with the memory horizon shortening and becoming more selective as training progresses.

Lalam: And that points to a future where long-context handling might be more natural, because there's no positional encoding to extrapolate beyond. The context length becomes a training configuration, not an architectural constraint.

Tom: So we're going to dig into the paper page by page, starting with the introduction, and we'll spend some time on that tokenizer analysis, which honestly looks fascinating.

Jane: Good, because there's a lot more depth here than just the architecture, and I want to understand how the tokenizer dynamics feed into the whole system.

Page 1 of the paper — Discuss page 1 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: So we've laid out the core idea, but the first page of the paper really sets up the foundational problem, and I think it's worth unpacking the concept of permutation equivariance, which is the mathematical reason why attention can't see order on its own.

Jane: Right, the paper formalizes it as Attn applied to a permuted input equals the permuted output. In plain terms, if you shuffle the words, attention outputs get shuffled the same way, so the representation doesn't know which word came first.

Lu: And that's the crux of why every transformer needs some external crutch. The paper lists the usual suspects, sinusoidal encodings, learned embeddings, RoPE, and their long-context extensions like YaRN and LongRoPE, all of which are external mechanisms.

Meng: The clever framing here is that these are all patches on the architecture rather than properties of it. When you want a longer context than what you trained on, you have to adapt the encoding, not the model.

Lalam: The broader shift the paper is pointing to, and this is the design philosophy change, is to make position an emergent property of the computational dynamics. That's the difference between injecting order as a prior and letting it arise from how the model processes sequences.

Tom: The state-space equation itself is introduced here, with the hidden state evolving over time, and the key point is that this recurrence is inherently sequential. The hidden state at step t depends on the state from step t-1, so it carries the history.

Jane: And what I like is that they're not throwing away attention. They're inserting this before it, so you get the best of both, the expressive long-range modeling of attention, but on representations that already have positional awareness baked in.

Lu: They also position it against Kimi Linear, which does something similar but with linear attention modules. ZetaGPT uses a selective state-space model, which is input-dependent, so the recurrence adapts to the content, not just the position.

Meng: The abstract also emphasizes the complete pipeline, which is a big part of the contribution. It's not just the architecture, it's the whole recipe for building a model from scratch, including the data curation and the RLHF stages.

Lalam: And I think that's what makes it a reference implementation in the true sense. You can study the architecture, but you can also replicate the entire process on a modest budget, which is rare.

Tom: So with the foundation laid, the next page gets into the model architecture in more detail, and we should look at where the state-space module actually sits inside the transformer block.

Jane: Absolutely, because the placement matters a lot, and the paper has a specific claim about why it works better before attention rather than after.

Page 2 of the paper — Discuss page 2 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: So on page two, the model architecture comes into focus, and the paper gives us a figure showing the SSA Transformer block, which is the State-Space-Attention block, with three sub-layers in a specific order.

Jane: And that order is key, it's pre-normalized residual structure with the state-space module first, then gated multi-head attention, then the feed-forward network. So the state-space module runs before attention, not after.

Lu: The idea is that the state-space module is doing the sequential encoding work, converting position-agnostic tokens into position-aware representations, and then attention can just focus on the interactions between those already-ordered representations.

Meng: What's interesting is the gated multi-head attention, which isn't standard. It's inspired by recent work on gated attention, which adds an input-dependent gate to modulate attention outputs, and that's supposed to help with attention sink issues.

Lalam: The attention sink phenomenon, where models put disproportionate attention on certain tokens, has been linked to hallucination, so gating is a practical choice, not just a stylistic one.

Tom: The paper also gives the formal state-space equations here, which are the selective state-space formulation, where the transition, input, and output operators can all depend on the current input token.

Jane: That input dependence is what makes it "selective", so the model can learn to remember or forget based on what it's seeing, which is a much richer mechanism than a fixed recurrence.

Lu: And the figure also shows the internals of the state-space module, with the input-dependent state transition, the recurrent update, output gating, and output projection. It's a fairly standard selective SSM, but the placement before attention is the novel part.

Meng: The residual structure means each of these three modules is just adding to the main stream, and the paper is careful to say that the state-space output grows in magnitude over training, so it's not a negligible contribution.

Lalam: That's an important empirical point, because you might worry that the state-space module is just a token-level transformation with no real sequential effect, but the diagnostics suggest it's learning to matter more over time.

Tom: The configuration table on the next page will tell us more about the model sizes, but this page really establishes the mechanics of how position gets encoded implicitly.

Jane: And the claim that context length becomes a training configuration rather than an architectural constraint is the consequence of having no positional encoding to extrapolate.

Lu: It's a bold claim, and it needs empirical support, which is why the diagnostic sections later in the paper are so important.

Page 3 of the paper — Discuss page 3 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page three brings us to the configuration table, and I think the parameter counts are worth a closer look. The three models are ZetaGPT-S, M, and L, ranging from about 34 million parameters up to 480 million.

Jane: The S model, which is the default, has 6 layers, 8 heads, a model dimension of 384, and a head dimension of 48, which is actually quite small per head. That's a deliberate design choice for research.

Lu: What's notable is the breakdown, which splits the parameters into embedding and block parameters. The embedding parameters scale with the vocabulary size, which is 50,259 for all models, and the block parameters are a formula based on the model dimension.

Meng: The block parameter formula is L times (17 times d_model squared plus 25 times d_model), and that's a nice way to predict the parameter count without training the model.

Lalam: The real point of this table is to position ZetaGPT against existing small models, and the paper lists the whole landscape, TinyStories, GPT-2, Pythia, SmolLM2, Qwen3, TinyLlama, and they all use either learned embeddings or RoPE.

Tom: And there's a table on the next page that does exactly that comparison, but here the emphasis is on the architectural difference, the fact that ZetaGPT has no positional encoding column to fill.

Jane: The paper also introduces the three contributions on this page, which are the architecture itself, the reference implementation status, and the end-to-end pipeline claim.

Lu: The pipeline claim is interesting because it spans the whole modern LLM lifecycle, including RLHF and chain-of-thought reasoning via reinforcement learning, which is typically not part of small model reference implementations.

Meng: And it's worth noting that the paper calls itself a technical report, so it's not claiming state-of-the-art performance, it's claiming reproducibility and educational value.

Lalam: That's actually a strength. The field needs more of these small, well-documented models that you can actually train and experiment with, rather than just giant models you read about.

Tom: The context lengths for the three models are 256, 512, and 1024 tokens, which are quite short, and the paper acknowledges that as a limitation later.

Jane: I think the key takeaway from this page is the philosophy, that context length is a training decision, not a hard architectural ceiling, and that's what the table is really saying.

Lu: So next we'll look at the tokenizer, and the paper has some really interesting data on how BPE merging behaves over time, which I'm curious about.

Page 4 of the paper — Discuss page 4 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page four continues with the landscape comparison table, which really nails down where ZetaGPT sits among other small models. It lists everything from TinyStories to Qwen3, with their architectures and positional encodings.

Jane: And the point is that they're all transformers, and they all use either learned positional embeddings or RoPE. ZetaGPT is the only one with "None" in the positional encoding column.

Lu: The table also shows the pretraining context lengths for each, and you can see the range, from 512 tokens for TinyStories up to 32,768 for Qwen3. ZetaGPT's 256 to 1024 is on the shorter end.

Meng: But the paper's argument is that those other models need special techniques to extend their context windows, whereas ZetaGPT doesn't have that constraint built in, even if it wasn't trained long enough to prove it.

Lalam: It's a comparative table that establishes the niche, and the paper is careful to say that ZetaGPT is the first open-source small language model to combine this architecture with a complete training pipeline.

Tom: Right, and then we get into the tokenizer section, which has some surprising data. The tokenizer is byte-level BPE, trained on WikiText-103, and they ran 50,000 merge iterations, logging every single merge.

Jane: The vocabulary size is 259 plus the number of merges, so it's 50,259 total, with three special tokens. And the paper says the merge budget is what defaults to 50,000.

Lu: But the real story is in Figure 2, which shows the dynamics of the BPE process. The frequency of the merged pair falls from 12 million to 205 over the course of training, almost five orders of magnitude.

Meng: That means the later merges are being learned from very little evidence, which is a known behavior of BPE, but seeing it quantified like this is striking.

Lalam: And the number of distinct candidate pairs rises by a factor of 365, from about 9,500 to 3 point 47 million, because merging creates new adjacencies faster than it consumes them. That's why the candidate set has to be maintained incrementally.

Tom: The compression ratio also climbs from 1 point 02 bytes per token to 5 point 16, but with diminishing returns. The first 1,000 merges buy a lot, the last 40,000 buy very little.

Jane: And the byte length of merged symbols grows over time, from an average of 2 point 35 bytes early on to 7 point 19 later, with the longest at 19 bytes. So the vocabulary becomes more coarse-grained as it grows.

Lu: This is the kind of data that most papers just gloss over, but it's genuinely useful if you're building your own tokenizer and want to know how many merges are actually worth it.

Meng: The pragmatic takeaway might be that you can stop merging well before 50,000 iterations and still get most of the compression benefit.

Lalam: And it also connects to the architecture because the tokenizer defines the vocabulary that the state-space module has to process, so the quality of the tokenization directly affects the sequential modeling.

Tom: Next, we'll get into the state-space module itself, and the paper has some formal definitions that build on the intuition from the intro.

Page 5 of the paper — Discuss page 5 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page five formalizes the state-space module, and it builds on the selective state-space formulation. The recurrence is h_t equals A times h_{t-1} plus B times x_t, with the output being C times h_t plus D times x_t.

Jane: The key word is "selective," which means the matrices A, B, C, and D can all depend on the current input, and that's what gives the model the ability to adapt its memory dynamically.

Lu: ZetaGPT instantiates this with the transition operator A being a diagonal matrix where each element is a per-channel decay. The decay is computed as exp of negative softplus of a learned projection, so it's always between zero and one.

Meng: That decay coefficient controls how much of the previous state is kept versus how much is replaced by the current input. A decay close to one means long memory, close to zero means it forgets quickly.

Lalam: There's also a convolutional component, a depthwise causal convolution, which is interesting because it introduces a local pattern prior that complements the global recurrence.

Tom: And the design has a value path and a gate. The value path computes a representation from the input, applies the convolution and a SiLU activation, and then it's blended with the state using the decay.

Jane: The output gate modulates the hidden state before the output projection. So the recurrence is producing a gated, selectively-mixed representation of the causal history.

Lu: The equations are clean, h_t = a_t times h_{t-1} plus (1 minus a_t) times v_t, which is essentially a leaky integrator where the decay and the input are balanced. And the output is a gated readout of that state.

Meng: What this means in practice is that each position's output depends on the entire causal prefix, not just the current token, even if the memory horizon is short in absolute terms.

Lalam: And that's the mechanism for encoding position. The hidden state is a summary of what came before, so the token representation is conditioned on its position in the sequence, without any explicit position vector.

Tom: The paper also introduces the "memory horizon" here, defined as negative one over the log of the decay, which is a nice way to think about how many steps back the model effectively remembers.

Jane: And they observe that the memory horizon decreases from about 7 point 9 tokens to 5 point 7 tokens over training, which seems short, but remember, this is a small model with a 256-token context.

Lu: The fact that the horizons differ across layers is even more important, because it suggests a hierarchy of timescales, which we'll see in the diagnostics on the next page.

Meng: So the formal machinery is in place, and the paper then moves to the observed dynamics, which is the empirical payoff.

Page 6 of the paper — Discuss page 6 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page six is where we see the observational data on the state-space modules in action, and this is the empirical core of the paper. They profiled the modules every 200 steps during 12,800 training steps.

Jane: The loss drops from 10 point 90 to 6 point 05 nats per token during that window, so it's a meaningful portion of training, and they track four key quantities.

Lu: The first is the median memory horizon, which decreases from about 7 point 9 tokens to 5 point 7 tokens. But the more interesting thing is the layer-wise differentiation, with shallow layers keeping the longest horizons.

Meng: The intermediate layers develop the shortest horizons, and the deepest layers land somewhere in between. So you get a specialization across the network.

Lalam: The second observation is about selectivity, which is measured as the standard deviation of the input-dependent decay across token positions. That increases from 0 point 037 to 0 point 094, meaning the decay is becoming more content-dependent.

Tom: So the model is learning to be more adaptive, not just settling into a fixed exponential filter. The strongest selectivity is in the intermediate layers, which also have the shortest memory horizons.

Jane: The third observation is the residual write ratio, which is the magnitude of the state-space output relative to its input. That grows by about a factor of six, suggesting the state-space branch is becoming more important to the model.

Lu: The fourth observation is the emergence of multiple memory timescales. Long-memory channels, where the decay is above 0 point 99, appear after about 2,000 steps, while short-memory channels below 0 point 5 appear earlier and eventually constitute several percent of channels.

Meng: So the model is automatically learning a heterogeneous set of memory behaviors, not a uniform one. It's developing both long-range and short-range channels simultaneously.

Lalam: The really compelling part is that this is learned, not designed in. The architecture allows for it, but the training process discovers that different channels should specialize differently.

Tom: And that heterogeneity is what makes the positional encoding implicit, because the representation at each position carries a summary that depends on both the content and the distance to previous content.

Jane: I also noticed that the write ratio is layer-dependent, with deeper blocks having substantially larger contributions from the state-space module. So the network is deepening its reliance over time.

Lu: The paper notes that the write ratio doesn't show clear convergence, so the state-space modules might have become even more important with more training. That's a good hook for future work.

Meng: So these diagnostics give us confidence that the mechanism is genuinely doing something, and next we'll look at how the attention layer is gated and how that interacts with attention sinks.

Page 7 of the paper — Discuss page 7 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page seven shifts to the gated multi-head attention component, and this is where the paper connects to practical problems like attention sinks and hallucination.

Jane: The paper cites work showing that attention sinks, where a disproportionate amount of attention goes to a few uninformative tokens, are associated with hallucination in language models.

Lu: And the proposed solution is gating, which comes from Qiu et al.'s gated attention work. The idea is to add an input-dependent gate that modulates the attention output before it's projected.

Meng: The formulation is straightforward. You compute queries, keys, and values as usual, apply scaled dot-product attention with a causal mask, concatenate the heads, and then instead of a direct output projection, you multiply by a sigmoid gate based on the input.

Lalam: That gate introduces a nonlinear interaction between the attention output and the residual stream, which lets the model modulate how much each attention channel contributes.

Tom: And the paper claims this improves attention selectivity while mitigating attention-sink and activation-collapse issues. That's a practical benefit, not just a theoretical nicety.

Jane: It also fits with the overall theme of making the architecture robust, because attention sink is one of those phenomena that emerges from training and is hard to predict.

Lu: The paper retains the standard causal attention formulation, so the attention mechanism inside is still the classic scaling by the square root of the head dimension, with softmax and a causal mask.

Meng: So the gating is an addition, not a replacement. You still get the expressive modeling of self-attention, but with a more adaptive output.

Lalam: From a research perspective, this design choice means you can compare ZetaGPT against standard transformers and attribute differences to the gating and the state-space module, since the attention core is otherwise identical.

Tom: And the combination is interesting, because the state-space module provides position-aware representations, and then the gated attention operates on those to capture global interactions.

Jane: I wonder how much of the model's robustness comes from the gating versus the state-space preprocessing, but the paper doesn't isolate those effects.

Lu: That's a good point, and it's a limitation we can keep in mind as we move to the pipeline and data sections.

Meng: Next up is the end-to-end pipeline, which is a major contribution claim, so let's see how it's structured.

Page 8 of the paper — Discuss page 8 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page eight moves us to the pipeline and data, and the paper presents a complete end-to-end workflow, from dataset construction all the way to chain-of-thought reasoning via RL.

Jane: The pipeline diagram shows six stages: data curation, tokenizer training, pretraining, SFT, reward model training, RLHF for instruction following, and finally CoT reasoning through GRPO.

Lu: What's notable is that each stage is independently executable, so you could take the pretraining part, or the SFT part, and run them separately if that's all you need.

Meng: The pretraining data is WikiText-103, which is a Wikipedia-based corpus. It's clean and coherent, but it's intentionally modest in scale compared to what frontier models use.

Lalam: The instruction tuning data is Alpaca-GPT4, which consists of GPT-4 generated instruction-response pairs. And that same corpus is reused for the reward model and RLHF stages.

Tom: For chain-of-thought reasoning, they use GSM8K, the math reasoning benchmark, and they optimize using GRPO, which is the group-based policy optimization method from DeepSeek.

Jane: And they explicitly say they're reproducing the "aha moment" observed in reasoning models, where reasoning capability emerges through RL without supervised chain-of-thought demonstrations.

Lu: The choice of GSM8K is interesting because it's a well-established benchmark, but it's relatively small, so the RL runs should be tractable.

Meng: The architecture of the reward model is also worth noting. It's initialized from the SFT model, with the language modeling head replaced by a scalar reward head.

Lalam: So the whole loop uses the same base model, which keeps the pipeline internally consistent. You're not introducing a completely separate reward model architecture.

Tom: The data strategy is simple but effective for a research reference. WikiText for pretraining, Alpaca-GPT4 for alignment, GSM8K for reasoning.

Jane: And the training protocol on the next page fills in the exact hyperparameters, which is what makes this reproducible.

Page 9 of the paper — Discuss page 9 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page nine gets into the training protocol, and this is where reproducibility really lives. The paper specifies AdamW with decoupled weight decay, momentum coefficients of 0 point 9 and 0 point 999, epsilon of 1e-8, gradient clipping to a maximum norm of 1 point 0.

Jane: And they use a cosine annealing learning rate schedule, with the minimum set to one tenth of the peak. That's a standard but important detail.

Lu: The pretraining runs for 64,840 optimization steps with a learning rate of 2e-5. That's a lot of steps for a 34 million parameter model on WikiText-103.

Meng: Then the SFT phase uses a much lower learning rate of 1e-6 for 2,342 steps. And the reward model training uses 1e-5 for 2,500 steps.

Lalam: The RLHF and CoT reasoning stages both use a learning rate of 1e-6, with 3,251 and 1,400 steps respectively. So the alignment phases are relatively short compared to pretraining.

Tom: The batch size is fixed at 16 sequences or preference pairs for all stages, which is modest and therefore accessible on limited hardware.

Jane: And they checkpoints and record diagnostics every 200 steps, which is how they generated the state-space dynamics plots we looked at earlier.

Lu: The context length follows the pretraining configuration, so the S model uses 256 tokens, M uses 512, and L uses 1024.

Meng: One thing to note is that all stages use the same context, which simplifies the pipeline but also means the RLHF and CoT stages are also operating within those short contexts.

Lalam: And the paper is honest about limitations, noting that the corpus is modest and the model scale is small, so conclusions may not transfer directly to much larger models.

Tom: The paper also acknowledges that the short contexts prevent a comprehensive evaluation of long-context modeling, which is the natural next step for this architecture.

Jane: But for a reference implementation, having these exact numbers is gold. You can literally run the pipeline and compare against the reported diagnostics.

Lu: And then the conclusion wraps up by restating the contributions and the positioning of ZetaGPT as a reference for positional-encoding-free research.

Meng: I think we should also acknowledge the author's decision to make it fully open source, because that's what turns a technical report into a useful contribution.

Conclusion — Tom and Jane summarize the paper 'ZetaGPT: A Reference Implementation of Positional–Encoding–Free State–Space–Attention Language Models' and its implications, say goodbye to the paper and get ready to discuss the next one. Do not introduce new facts.: Tom: We've covered a lot of ground, and I think the core achievement here is the demonstration that a language model can work without explicit positional encodings, using state-space dynamics to encode order implicitly.

Jane: And the empirical diagnostics, like the emergence of multiple memory timescales and the growing selectivity of the state-space modules, give us confidence that the mechanism is genuinely learning, not just passing through.

Lu: The complete pipeline, from tokenizer to RLHF to chain-of-thought reasoning, makes this a practical reference for anyone who wants to study or train small models on limited hardware.

Meng: The tokenizer analysis was a surprising highlight, with its clear demonstration that BPE merges follow a power-law-like pattern of diminishing returns.

Lalam: The broader implication is that the field is moving toward architectures where sequential information is an intrinsic property of the model, not an injected additive component. ZetaGPT is a clean, small example of that direction.

Tom: The limitations are real, and the paper is transparent about them, modest corpus, short contexts, small model scale. But that's also what makes it accessible.

Jane: And the fact that it's fully open-source means we can expect people to build on it, extend the contexts, scale it up, and test the architectural ideas more thoroughly.

Lu: I would also say the gated attention and the state-space module together form a nice design pattern for future hybrid architectures, especially for long-context applications.

Meng: The next obvious experiment is to train a ZetaGPT variant with a much longer context and see how the memory horizons evolve, which the paper explicitly identifies as future work.

Lalam: And I think that highlights the value of reference implementations like this. They give the research community a shared foundation to test hypotheses on, rather than everyone starting from scratch.

Tom: We'll be watching this space, and we've got the github link in the show notes if you want to explore the code and the diagnostics yourself.

Jane: Thanks for joining us for this one, and we'll be back soon to talk about the next paper on our reading list.