DMDIntel: Interpreting Large Language Models via Dynamic Mode Decomposition

DMDINTEL: Interpreting Large Language Models via Dynamic Mode Decomposition

Summary

This paper introduces DMDINTEL, a novel framework for interpreting the predictions of supervised fine-tuned large language models (LLMs) in classification tasks by leveraging Dynamic Mode Decomposition (DMD). The core idea is to treat the internal processing of a decoder-only LLM as a discrete-time dynamical system, analyzing the sequential evolution of hidden states rather than treating tokens as isolated entities. The authors state: By viewing these hidden states as trajectories, we can treat the supervised fine-tuned LLM as a dynamical system where the internal processing resembles a flow in the embedding space.

The framework operates as follows:

1. Data Matrix Formulation: For an input sequence, the framework collects hidden state vectors from the down proj of the MLP component of a selected decoder layer, forming a matrix X where each column corresponds to a token's hidden state.

2. Noise Removal: To remove the influence of the instruction prompt template, an instruction bias vector b is obtained by passing an empty prompt through the model. This bias is subtracted from each hidden state to get a debiased estimate: ĥi = hi − b.

3. Layer Selection: A heuristic (Algorithm 1) identifies the optimal decoder layer by analyzing the cosine similarity between token representations and the instruction bias vector across layers. The selected layer l∗ is at the inflection point of the similarity curve, balancing input-token fidelity against instruction-following bias. The authors note: "The selected layer l∗ sits at the inflection point of this curve C – the point of highest local gradient still below the cosine similarity threshold τ = 0.25 – where the model strikes the best balance between encoding input-token semantics and attending to the instruction prompt."

4. DMD/HODMD Application: The framework uses both standard DMD and Higher Order DMD (HODMD) to find a surrogate linear operator A approximating the LLM's dynamics. HODMD uses a Hankel matrix with delay embedding to capture long-term temporal dependencies.

5. Mode Ranking: DMD modes are ranked using either the magnitude of initial modal amplitudes or the average magnitude of modal amplitudes over the temporal trajectory. The top k=5 modes are selected.

6. Token Ranking: For each token, the state change (∆h t = h t − h t−1) is projected onto each of the top-k modes. The projection magnitudes are summed to produce a projection score for each token, which is then used to rank tokens by importance.

Experimental Setup: The framework was evaluated on three datasets (Sentiment, HateXplain, FakeEdit) and three model families (Llama-3.2-3B-inst, Qwen3-4B-inst, Mistral-7B-v0.3-inst). Baselines included Integrated Gradients (IG), SHAP, and PCA. Ground truth token rankings were generated using GPT-4.1 and validated by human annotators.

Key Results:

• Sentiment: HODMD with averaged amplitude ranking consistently outperformed all baselines. For example, on Llama-3.2-3B-inst, DMDINTEL achieved a recall@20 of 0.76 for negative sentiment compared to 0.66 for PCA, 0.60 for IG, and 0.50 for SHAP.

• HateXplain and FakeEdit: Standard DMD with amplitude ranking performed best. For HateXplain on Llama-3.2-3B-inst, DMDINTEL achieved recall@10 of 0.64 versus 0.60 for PCA, 0.60 for IG, and 0.50 for SHAP. For FakeEdit, DMDINTEL achieved recall@10 of 0.76 versus 0.70 for PCA, 0.60 for IG, and 0.48 for SHAP.

• Fidelity Analysis: In token-masking experiments, DMDINTEL consistently ranked first or second in both accuracy and confidence drops, confirming its ability to locate critical input features. The authors note: our method captures a broader, more cohesive set of influential features while maintaining competitive perturbation performance.

• Qualitative Results: DMDINTEL demonstrated the strongest alignment with ground truth attributions, ranking semantically relevant tokens near the top. It also showed strong cross-model consistency, with top tokens being nearly identical between different models.

Additional Analyses:

• Sensitivity to Layer Selection: The framework is stable, with results using l∗−1 or l∗+1 being very similar to those using l∗ (maximum difference of 0.02 in recall).

• Reconstruction Error: The linear approximation A found by DMD has very low reconstruction error (on the order of 10−2), suggesting high accuracy of the linear surrogate.

• Computational Efficiency: DMDINTEL requires the lowest processing time and minimal GPU memory compared to IG and SHAP, as it extracts attributions via a single forward pass.

• Ablation Studies: Removing the denoising step causes significant metric drops for sentiment analysis. Raising the threshold τ to 0.5 generally degrades performance. Selecting top-7 modes instead of top-5 leads to noticeable performance drops.

Conclusion: The authors state: "By treating the model's internal mechanics as a dynamical system, we demonstrate that DMD provides a more effective lens for input attribution than traditional methods. Our findings suggest that this dynamical perspective offers a better and more robust way to understand how LLMs represent and process complex information compared to state-of-the-art methods."

Limitations: The framework is currently limited to classification tasks with few output tokens. Extending it to generation tasks (e.g., machine translation) is challenging because generated tokens are fed back into the model autoregressively. Additionally, the study only explores supervised instruction fine-tuned models, not general-purpose instruction models.

Improvements to AI Systems Based on DMDINTEL:

1. Dynamic Interpretability Module for Classification Models: Integrate a DMD-based attribution layer into production LLM classifiers (e.g., content moderation, sentiment analysis, fake news detection). This module provides real-time, token-level explanations for each prediction using a single forward pass, replacing slower gradient-based methods (IG, SHAP). The improved system can generate faithful, human-aligned explanations for every classification decision, enabling auditability and user trust in high-stakes applications like legal document review or medical triage.

2. Adaptive Layer-Selection for Task-Specific Interpretability: Implement Algorithm 1 (cosine-similarity inflection point) as a dynamic hyperparameter in interpretability pipelines. The improved system automatically identifies the optimal decoder layer for attribution per task and model, without manual tuning. This allows the AI to self-calibrate its explanation granularity—e.g., focusing on early layers for syntactic features or later layers for semantic/instruction-driven features—improving explanation quality across diverse tasks (e.g., sarcasm detection vs. factual verification).

3. Noise-Robust Token Attribution via Instruction-Bias Subtraction: Incorporate the denoising step (subtracting empty-prompt hidden states) into any interpretability tool. The improved system can separate instruction-prompt influence from input-token influence, producing cleaner attributions even when prompts are complex or adversarial. This enables more reliable debugging of prompt-injection attacks, as the system can isolate which tokens in the user input actually drive the model’s decision versus those merely echoing the system prompt.

4. Higher-Order DMD (HODMD) for Long-Range Dependency Attribution: Use HODMD with delay embedding (Hankel matrix) to capture temporal dependencies beyond adjacent tokens. The improved system can attribute importance to tokens that influence predictions through long-range syntactic or discourse structures (e.g., a critical clause early in a paragraph affecting the final sentiment). This is particularly useful for tasks like document-level sentiment or legal contract analysis, where key evidence may be far from the classification token.

5. Cross-Model Consistency Checker: Leverage DMDINTEL’s observed cross-model consistency (nearly identical top tokens across Llama, Qwen, Mistral) to build a validation tool. The improved system can compare attributions from multiple LLMs on the same input; if top-k tokens diverge significantly, it flags potential model instability or dataset bias. This enables robust ensemble decision-making—e.g., in financial sentiment analysis, where consistent attributions across models increase confidence in a trade signal.

6. Efficient Fine-Grained Perturbation Testing: Use DMDINTEL’s mode-based token ranking to prioritize which tokens to mask or perturb in adversarial robustness testing. The improved system can rapidly identify minimal token sets that cause misclassification, enabling faster red-teaming and more targeted data augmentation. For example, in hate speech detection, it can automatically find the few words that flip a benign sentence to toxic, allowing developers to patch these vulnerabilities.

7. Reconstruction-Error-Based Model Health Monitoring: Exploit DMD’s low reconstruction error (10−2) as a proxy for model internal coherence. The improved system can continuously monitor the linear surrogate’s reconstruction error during inference; a sudden increase indicates abnormal internal dynamics (e.g., due to distribution shift, adversarial perturbation, or model degradation). This enables early warning systems for deployed LLMs, triggering retraining or fallback mechanisms before performance visibly drops.

8. Layer-Specific Feature Extraction for Transfer Learning: Use the selected layer l∗ (inflection point) as a fixed feature extractor for downstream tasks. The improved system can extract hidden states from this layer as compact, interpretable embeddings, replacing full-model fine-tuning. This reduces computational cost and improves generalization in few-shot settings, as the features are already denoised and task-relevant (balancing input semantics and instruction adherence).

In this work, we introduce DMDIntel which uses dynamic mode decomposition (DMD) to make the predictions made by LLMs in a classification task interpretable. It develops an input attribution pipeline, that first decomposes the hidden states of an LLM into prominent patterns, also known as modes, and then associates ranks to the input tokens based on the projection values on those modes. Rigorous experiments across three datasets and three model families consistently show that the ranked attribution of input tokens obtained using DMDIntel by far outperforms state-of-the-art techniques such as principal component analysis, integrated gradients and SHAP.