Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

arXiv:2607.01774 · cs.AI, cs.CL · Submitted 2026-07-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Subliminal Clocks: Latent Time Modelling in Diffusion Language Models".

Jane: Diffusion Language Models (DLMs) are being investigated to determine if they internally represent denoising progress,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Alright everyone, let's kick things off by talking about the paper's title and who put it out there. This paper is called "Subliminal Clocks: Latent Time Modelling in Diffusion Language Models," and it immediately tells us we are looking at a hidden mechanism within these diffusion models.

Jane: It’s a very evocative title, Tom; "Subliminal Clocks" suggests something hidden and internal, hinting that the model is keeping track of time without us ever explicitly telling it what step it's on.

Lu: I think the authors, Rulli et al., are tackling a fundamental question about how DLMs function internally when they lack explicit timestep conditioning, which is a core challenge in this area of research.

Meng: They’re trying to figure out if there's an implicit mechanism for tracking the diffusion process that we can tap into, which is exactly what engineers need to understand for implementation.

Lalam: For Lalam, the authors are tackling the mystery of how these complex models manage temporal information when they aren't given explicit time steps during training or inference.

Tom: Exactly; they’re asking if there's a way to decode that latent representation of denoising progress using internal activations.

Jane: The implication is that we might be able to gain much better insight into the decision-making process of these models by looking at their internal state variables.

Lu: If this holds up, it means we can start treating the model't not as a black box but as something with an understandable internal temporal flow.

Meng: That understanding is crucial because right now, when things go wrong in generation, it’s often hard to pinpoint whether the error came from a poor prompt or an internal misalignment of time tracking.

Lalam: For Lalam, this research offers a pathway to build AI systems that can self-diagnose their own progression and adjust their behavior accordingly.

The paper's summary: Tom: Now, let's get into the core summary of "Subliminal Clocks: Latent Time Modelling in Diffusion Language Models." The main finding is that DLMs do encode a latent representation related to the diffusion timestep within their residual streams.

Jane: In simple terms, this means these models are internally tracking how much they’ve denoised, which is the continuous time variable of the diffusion process, even though they aren't explicitly conditioned on it.

Lu: They found that this signal can be reliably extracted using probes across layers and that it’s consistently represented throughout the network depth, with a high R2 coefficient indicating strong information presence.

Meng: This is a major point because it means we can actually measure progress directly inside the network structure instead of just looking at input and output metrics.

Lalam: For Lalam, this means we have a verifiable metric for how "advanced" the model is in its generation process that isn't just an external observation.

Tom: And they went further to demonstrate that you can explicitly steer the model by using these mean activation vectors grouped by their denoising step t to push it toward a target progress bin.

Jane: So, they showed that this extracted signal isn't just observational; it’s an actionable control signal for influencing the model's behavior during generation.

Lu: The methodology involves computing mean activation vectors (mu t,l) and defining a perturbation h t,l based on a target using Equation four to achieve steering <ref:2607.01774#pg2>.

Meng: So, the paper proves the mechanism works through this specific algebraic manipulation of these hidden states based on the estimated progress.

Lalam: This level of detail is what makes it so compelling; it moves beyond just stating that something *might* be there to showing exactly how to use it.

The paper's improvements: Tom: Moving on, let's talk about the specific improvements and suggestions this paper proposes for future work. They focus on extracting this signal and then using it effectively to steer the model.

Jane: The key improvement they suggest is implementing a probe network across every layer to continuously estimate that empirical denoising progress metric, tau t, which should be bounded in the range of zero to one.

Lu: This probe needs to output a scalar value for every layer, and it’s crucial that it accurately reflects the progress at step t according to Equation eleven <ref:2607.01774#pg2>.

Meng: From an engineering viewpoint, this means we need a robust way to train these probes so they are reliable indicators of the true denoising time variable.

Lalam: And once you have that estimate, the next step is using latent subspace steering by projecting the target change onto a low-dimensional subspace defined by Equation seven <ref:2607.01774#pg2>.

Tom: That subspace steering is important because it ensures that we only perturb the directions most relevant to the denoising progress signal, suppressing any orthogonal noise that might cause unwanted artifacts.

Jane: And they also suggest an adaptive control strategy where you apply the steering intervention dynamically based on the target denoising step bin to modulate downstream metrics like entropy and confidence precisely.

Lu: That dynamic modulation allows us to intentionally manipulate model uncertainty, for example, forcing higher confidence in later denoising stages or inducing specific levels of uncertainty at certain points in the generation process.

Meng: This adaptive control strategy is what separates a simple steering attempt from a sophisticated intervention; it lets us tune the model's risk profile on the fly.

Lalam: I think this adaptive approach is what unlocks truly controllable generation trajectories, allowing us to guide the model through a path of changing uncertainty.

Conclusion: Tom: So, wrapping up our discussion on "Subliminal Clocks: Latent Time Modelling in Diffusion Language Models," we’ve covered how this research successfully showed that DLMs possess an internal latent representation of denoising progress and demonstrated its decodability and steerability.

Jane: It really boils down to the fact that we have a measurable signal, tau t, and a mechanism to use it to systematically modulate downstream metrics like confidence and entropy in predictable ways.

Lu: The geometric characterization—the low-dimensional structure of the mean vectors suggesting a shared trajectory across layers is perhaps the most profound architectural insight they've provided for understanding internal organization.

Meng: From an engineering standpoint, the ability to use subspace steering to focus our control on the relevant directions is what gives us a practical tool that’s robust against stochastic artifacts.

Lalam: The ultimate implication is that we can start building AI systems that exhibit controllable cognitive states, allowing us to guide their development through intentional temporal progression during generation.

Tom: It’s a significant piece of research because it validates the idea that these models have a latent clock running inside them, and this paper opens up avenues for much finer control over their outputs than we thought possible.

Jane: We’ve learned how to read the internal dynamics of DLMs and use that knowledge to sculpt their generation process in a way that is systematic rather than random noise.

Lu: This work provides a strong foundation for future work where we can build on these geometric representations to design more sophisticated temporal control mechanisms into the core architecture.

Meng: For practical application, it means we can start designing models where uncertainty itself is an intentional feature, which is a significant step toward deploying AI in high-stakes scenarios.

Lalam: And for Lalam, this work confirms that we are moving toward building AI systems that exhibit controllable cognitive states, allowing us to guide their development through intentional temporal progression during generation.

Sapienza University of Rome

cs.AI, cs.CL

Submitted: 2026-07-02

Updated: 2026-10-03

Importance score: 92/100

The gist: Diffusion Language Models (DLMs) are being investigated to determine if they internally represent denoising progress, and this work shows that DLMs do encode a latent representation related to the

Key concepts

Empirical Denoising Progress (τt)
This measures the actual denoising progress at a specific step 't' by calculating the ratio of unmasked tokens to the total length. It quantifies how much noise has been removed from the diffusion process up to that point.
Mean Activation Vectors (µt,l)
These are calculated by averaging hidden states within a specific layer, grouped according to their denoising step 't'. These vectors are used to approximate the internal representation of the denoising progress signal across different layers.
Subspace Steering
This technique involves restricting steering perturbations to a chosen set of principal directions (subspaces) within the model's latent space. This method proved effective because steering within this low-dimensional subspace produced coherent effects, while orthogonal perturbations had minimal impact.

Terminology

Summary

Diffusion Language Models (DLMs) are being investigated to determine if they internally represent denoising progress, and this work shows that DLMs do encode a latent representation related to the diffusion timestep within their residual streams.

The gist: DLMs do in fact encode a latent representation related to the diffusion timestep within their residual streams.

How it works

The study investigates whether DLMs internally model a signal related to the denoising step, specifically relating this signal to the fraction of unmasked tokens, denoted as the empirical denoising progress at step t, defined as τt:= UR(xt) / L (Equation 11). The researchers test this by training MLP probes for each layer l in models like LLaDA and Dream to predict the current τt using a residual-stream hidden state as input. The success of these probes is quantified by the R2 coefficient, which is found to be high across all layers, indicating that information about τ is consistently represented throughout the network depth.

The researchers also test whether this signal can be explicitly extracted and used to steer the model. They approximate the internal representation of τ by computing mean activation vectors (µt,l) over all hidden states in a layer, grouping them according to their denoising step t (Equation 3). By using these mean vectors to define a perturbation, they steer the model toward a target denoising progress bin tˆ. This steering mechanism is defined as h˜j t,l:= h j t,l − µt,l + µt,l ˆ (Equation 4), where the shared component cancels out.

Assessing the Signal’s Importance

The importance of the identified signal is assessed by comparing downstream computations before and after steering. The researchers measure three complementary quantities: the variation in entropy, the variation in confidence, and the KL-divergence between clean and steered distributions (Equation 6). They find that steering along the found τ directions produces systematic and interpretable effects. Specifically, when steering towards larger values of tˆ relative to the current step t (i.e., tˆ− t > 0), the model becomes more confident and its entropy decreases. Conversely, when tˆ−t < 0, confidence decreases and entropy increases. Furthermore, the KL divergence grows approximately proportionally to the distance tˆ−t, indicating progressively larger deviations from the original distribution as the steering target moves further away from the current denoising step.

Characterising the Signal’s Geometry

The study characterizes how this signal is represented internally by analyzing its structural properties. The mean vectors exhibit a low-dimensional structure, and projections into two- and three-dimensional PCA spaces reveal a shape that is closely related to structures that models have been shown to exhibit when operating on counting or time-dependent and sequentially ordered tasks. A non-parametric 2D trajectory f(t) is constructed by averaging the standardized 2D projections of the mean vectors across all layers, showing that all points closely follow the proposed shared trajectory, implying a general model-level 2D representation of the mean vector components exists.

Cross-layer τ Representations

The analysis examines how these discovered mean vectors are connected across different layers. The results show a structural difference between LLaDA and Dream in this regard: In LLaDA, most layers exhibit strong alignment in their representations, with the exception of layer 32, which appears nearly orthogonal to the others. In contrast, Dream shows a more heterogeneous organisation: representations are only consistently aligned within specific blocks of layers. This suggests that there may be a model-level basis for encoding τ across layers, as high correlation implies similar downstream effects. Furthermore, the analysis of self-attention and MLP components reveals that self-attention and MLP vectors are anticorrelated, sharing a common direction but pointing in opposite ways.

Low-dimensional Steering

The causal importance of the low-dimensional subspaces is explored through subspace steering. By restricting the steering vector to a chosen set of principal directions (Equation 7), researchers find that steering within the subspace closely resembles the unrestricted one, while perturbations orthogonal to it produce incoherent effects, implying those directions carry little of the τ representation. The analysis further confirms this structure by showing that increasing the subspace dimensions from one to two yields results similar to using unprojected mean vectors, and that orthogonal perturbations have low impact.

Depth Correction

The study investigates how the model corrects an injected perturbation across its depth. Figure 16 shows that Early denoising steps are markedly more sensitive to the injected perturbation, but at later steps, it is suppressed within the first few layers above the injection site. This indicates that the model’s ability to correct an injected tˆ depends strongly on the denoising step at which the intervention occurs. Moreover, a residual drift decreases monotonically as the steered step becomes more compatible with tˆ= 100, i.

Improvements for AI systems

As a fastidious and diligent researcher, I have analyzed the core findings of this paper, Subliminal Clocks: Latent Time Modelling in Diffusion Language Models. The central discovery is that Diffusion Language Models (DLMs), despite lacking explicit timestep conditioning, internally encode a latent representation of denoising progress (related to the continuous diffusion time variable). This signal is decodable via probing and can be causally manipulated to steer model behavior.

Here are the specific, actionable improvements for AI systems derived from this research:

)

Diffusion Language Models (DLMs) can be transformed into time-aware or progress-aware models by extracting and injecting a latent denoising signal. This allows for fine-grained control over the model's internal state during generation, leading to more predictable and controllable outputs.

Specifically, you can implement the following improvements:

  1. [Time/Progress Steering Module]: Implement a probe network (as described in Section 3) to continuously estimate the empirical denoising progress metric, denoted as a proxy for the continuous diffusion time variable, across every layer of a DLM. This probe should output a scalar value bounded in the range (0, 1).

  2. [Latent Subspace Steering]: Instead of using random perturbations (as seen in Equation 5), steer the model's hidden states using mean activation vectors derived from the estimated progress metric at each layer. Specifically, apply a perturbation vector derived from:

  3. [Subspace-Aware Perturbation]: The steering vector should be constructed by projecting the target change onto a low-dimensional subspace (e.g., using Equation 7). This ensures that only the directions most relevant to the denoising progress signal are perturbed, while orthogonal noise is suppressed, leading to more focused control.

  4. [Adaptive Control Strategy]: The steering intervention should be applied dynamically based on the target denoising step bin (i.e., target value of τ) to modulate downstream metrics like entropy and confidence precisely. This allows for intentional manipulation of model uncertainty—for example, forcing higher confidence in later denoising stages or inducing specific levels of uncertainty at certain points in the generation process.

  5. [Token-Distribution Control]: By observing the effect of steering on token distributions (Section 4.3), you can design interventions that specifically target sequence-level statistics, such as influencing the probability of generating specific control tokens (like End-Of-Sequence tokens, EOS) based on the desired denoising progress trajectory.

The improved AI system will gain the following capabilities:

  1. [Controllable Generation Trajectories]: The system will be able to generate text that follows a pre-defined denoising path (e.g., gradually increasing confidence or entropy) rather than following its default, stochastic trajectory. This is crucial for tasks requiring nuanced control over output style or complexity during generation (e.g., creative writing, code generation with specific error rates).

  2. [Predictable Uncertainty Modulation]: The system can be explicitly tuned to exhibit desired uncertainty profiles throughout the generation process. For instance, it could be forced to be more certain about a specific factual claim at an early stage of denoising and become more exploratory later, which is vital for safety-critical applications or complex reasoning chains.

  3. [Enhanced Interpretability of Latent Dynamics]: By analyzing how the low-dimensional geometry (the 2D parabola described in Section 5.2) evolves across layers, researchers can gain a deeper understanding of how DLMs organize and represent time/progress information internally, potentially leading to better architectural designs for future models.

  4. [Robustness Against Stochastic Artifacts]: Because the steering is based on a learned, structured signal rather than random noise, the system's behavior under controlled perturbations becomes more robust and predictable compared to standard adversarial or random steering methods.

Sources

Related papers