Continual Learning in Transition

This survey paper, authored by Zhiyan Hou, Dan Zhang, Tao Feng, and colleagues, addresses a fundamental shift underway in the field of continual learning (CL). The authors argue that classical CL—which has primarily focused on enabling models to update and retain knowledge through parameter-centric mechanisms, e.g., training strategies, architectural designs, and weight adaptation—is being reshaped by emerging paradigms in the era of large language models (LLMs) and agentic AI. They identify that "on-policy learning broadens the space of update mechanisms; test-time training extends CL from the training phase to inference; and external harness components such as memory, skill libraries, and interaction protocols extend the evolutionary boundaries of model capabilities far beyond the static parameter space. Collectively, these developments indicate a transition from parameter-centric learning toward system-level adaptation."

The paper's central contribution is a reformulation: "we reformulate continual learning in the era of large language models and agentic AI as continual capability evolution across mechanism, timing, and locus, and propose a three-axis taxonomy that uniformly characterizes this process." The three axes are: When (the temporal stage of capability evolution), Where (the carrier of capability), and How (the update mechanism).

The paper first revisits the classical formulation, describing it as a model that learns under a non-stationary data distribution. Training data arrive as a sequence of tasks T1,...,T K, each associated with its own distribution D k:= p(X k, Y k) and loss L k. The central obstacle is catastrophic forgetting: updates that improve performance on T k may perturb the representations or decision boundaries that support earlier tasks, sharply degrading their performance.

Classical methods are grouped into four families: replay-based, gradient-based, architecture-based, and regularization-based methods. Crucially, the authors show that these families share three implicit assumptions:

• (i) capability acquisition and retention primarily occur during a dedicated training stage before deployment;

• (ii) accumulated capability is primarily carried by model parameters;

• (iii) updates are typically implemented through gradient-based optimization over externally supplied or previously collected data.

These assumptions, the paper argues, are not intrinsic to the broader goal of continual learning; rather, they reflect the design choices of the setting in which the field originally developed. Each assumption corresponds to a fixed answer to the When, Where, and How questions, and the developments of the LLM era turn each answer into a genuine variable: learning now happens at several stages of the model lifecycle, capability accumulates on carriers beyond parameters, and updates are driven by mechanisms beyond off-policy gradients.

The When axis asks at which stage of the model lifecycle continual capability evolution is realized. The paper notes that learning is no longer confined to a single sequential training process and identifies three stages.

Pre-training: "Continual pretraining turns pretraining itself from a one-shot procedure into a multi-stage process over evolving corpora, so that a model already trained on a base distribution can be further adapted to new domains, languages, or temporal slices without restarting from scratch. Representative work includes the simple continual pretraining recipe of Ibrahim et al. combining learning-rate rewarming with a small replay buffer of base-distribution tokens, and continual knowledge learning (CKL), which formulates keeping a language model's world knowledge current as renewing outdated facts while retaining time-invariant ones."

Post-training: "post-training has evolved from a single instruction-tuning step into a longitudinal pipeline: supervised fine-tuning is typically followed by one or more preference- or reward-alignment stages (e.g., RLHF, RLVR), and deployed models continue to be revised in successive versions. Each round of updating risks eroding what earlier rounds established, which makes this stage a continual-learning problem in its own right. The paper cites evidence that sequential training on new tasks erodes not only general ability but also instruction following and safety alignment, and discusses the concept of spurious forgetting, a recoverable loss of task alignment rather than a loss of underlying knowledge."

Inference-time: The inference-time position on the When axis refers to capability evolution that occurs after deployment. The key distinction is persistence. The paper contrasts general inference-time computation such as search, sampling, self-consistency which usually leaves the system unchanged once the query is completed with inference-time continual learning, which writes information obtained during inference back into the system, through writable states or parameter updates, so that earlier inputs, feedback signals, or self-generated supervision can influence later behavior. Methods such as TTT-LM update trainable hidden-state modules within a test sequence, allowing contextual information from earlier tokens to be compressed into fast weights, and TTRL introduces reward-driven test-time learning, deriving supervision from the model's own sampled outputs on unlabeled test inputs.

The Where axis asks where acquired knowledge and capabilities are stored and updated. The paper observes that the more fundamental shift in the LLM era... is that capability accumulation increasingly extends beyond the parameter space. Agent systems externalize part of their evolving capability into the harness layer surrounding the core model, including memory, skills, and protocols.

Parameters: Parameters remain the canonical carrier of continual learning: new capabilities are consolidated by modifying trainable weights, and forgetting is often studied as interference within the shared parameter space. The key change in the LLM era is in the granularity of parametric updates, with methods like O-LoRA constraining the LoRA directions of different tasks to be mutually orthogonal during continual fine-tuning, and Progressive Prompts learning a separate soft prompt for each task. These methods refine the parametric carrier from a monolithic parameter space into a set of modular, locally updated components.

The Harness Layer: Following the externalization view, the harness "divides, by the nature of what it carries, into three classes: memory is information storage that is readable and writable across inference calls (such as dialogue history, long-term memory banks, and retrieved document fragments); skills are executable units that the model actively invokes (such as external tools, function modules, and callable sub-agents); and protocols are rules and formats that govern how the model interacts with its environment (such as system prompts, rule files, and message-format schemas)."

• Memory: "provides LLM systems with a pathway for capability accumulation that does not depend on gradient updates: the model writes facts, experience, and intermediate states into external storage and retrieves them in subsequent inference. Methods like MemoryLLM equips the model with a fixed-size learnable memory pool into which the model itself integrates new knowledge during inference, while MemoryBank consolidates long-term memory for dialogue agents through an Ebbinghaus-style forgetting curve, and A-MEM takes atomic propositions as the smallest unit of reading, writing, and forgetting."

• Skills: skills encapsulate the model's reusable execution flows as externally callable units. By adding, refining, and composing skill units, an agent extends its action repertoire without updating backbone weights. Voyager lets an LLM agent write code as reusable skills while interacting with its environment and store them in a persistent skill library, and SkillRL jointly optimizes the skill library and the policy via reinforcement learning.

• Protocols: protocols externalize the behavioral rules and communication conventions between the model and its environment as editable artifacts. Promptbreeder uses an LLM to evolve prompts themselves through self-referential mutation, making the system prompt a carrier that task feedback continually rewrites. The paper notes that the research landscape of protocols as a continual-learning carrier remains at an early stage.

The How axis is extended in two directions. First, within gradient-based learning, it broadens the data source from off-policy data decoupled from the current policy... to on-policy rollouts generated by the policy that is currently being updated under reward signals. Second, it moves beyond backpropagated gradients altogether, accumulating capability through mechanisms that never compute an analytic gradient of a loss.

Off-policy vs. on-policy: The paper explains that "the genuinely new mechanism that the LLM era brings to the How axis is on-policy learning, where training data is generated by the very policy being updated, conditioned on reward signals; the reward-driven alignment paradigms of post-training (e.g., RLHF, RLVR) are its principal instances. Its significance for CL is described as structural rather than incidental: citing Shenfeld et al., the paper notes that on-policy RL implicitly prefers, among the many solutions to a new task, the one with smallest KL, so its drift away from previously acquired capabilities stays small—whereas off-policy supervised fine-tuning can converge to distributions arbitrarily far from the reference model."

Beyond gradients: Beyond the gradient-based regimes lies a family of mechanisms that accumulate capability without ever backpropagating an analytic gradient of a loss. This includes model merging (the most established member), which composes capability directly in weight space among already-trained tensors; forward-only or zeroth-order optimization, where ZeroFlow "showed that such forward-pass optimization is an effective continual learner: across forgetting benchmarks the implicit smoothing of zeroth-order updates matches or exceeds first-order fine-tuning at mitigating catastrophic forgetting; and heuristic learning, in which an agent revises an external system from feedback rather than gradient signals."

The paper emphasizes that methods should be read through combined labels rather than single categories: A method may be associated with more than one applicable label, and its characterization may change over time when capability moves between lifecycle stages or carriers. Examples include Reflexion, which combines inference-time evolution, a harness-based memory carrier, and learning beyond gradients through reflective text generation and direct memory writing; TTRL, which combines inference-time parameter adaptation with on-policy learning; and AgentEvolver, which spans parameters and the harness: on-policy reinforcement learning updates the policy parameters, while selected trajectories are accumulated in experience memory. SKILL0 is described as a Harness (Skills)-to-Parameters trajectory along the Where dimension rather than as a method with one fixed carrier, since it first accumulates reusable capability in a harness-based skill library and subsequently internalizes selected skills into model parameters through a curriculum.

The paper's central observation is that "the field is shifting from an isolated algorithmic problem, namely the suppression of catastrophic forgetting in a single model, toward a broader question of how capability is organized across a whole system: where it should live, when it should be updated, and how it should be combined, retained, and released." The discussion is organized around five themes.

1. From Longer Context to Persistent Capability: The paper describes a three-layer hierarchy of carriers: The innermost layer is the parameters... The middle layer is external memory... The outermost layer is the context window. It argues that across the three layers a clear trend emerges: as capability moves outward, write cost falls and interpretability rises, while recall stability and capacity ceilings fall in step. The context-only route faces a hard ceiling on length and a soft ceiling on effective attention due to the needle-in-a-haystack phenomenon. The paper also identifies retrieval decay as a problem: external memory likewise requires an active forgetting mechanism.

2. Why Harness-Level Accumulation Is Not Enough: The paper addresses the question of whether engineering progress on the harness already constitutes continual learning. Its position is that the engineering progress on the harness already attains part of the goal of continual learning at the functional level, but does not replace it at the mechanistic level, for three reasons: the harness offers externally editable capability rather than self-accumulated capability; methods that do close the loop remain far from mature in evaluation, stability, and reproducibility; and capability on the harness is strongly coupled to context and transfers less readily than capability in parameters.

3. Coordinating Memory, Skills, Protocols, and Parameters: The paper proposes bidirectional transfer between parameters and harness, scheduling capability across carriers, and recognizing that forgetting becomes multi-faceted and hard to localize. It argues that it may be more useful to recast forgetting, from a catastrophe at the parameter layer into a process of active compression and release across composite carriers.

4. Long-Horizon Agents as the Real Testbed: The paper argues that "Future continual learning should be judged not by short tasks or static benchmarks, but by whether an agent can hold its goal, track state, correct errors, and accumulate experience across long, multi-step interactions in open environments. It identifies error accumulation as the core difficulty": when a task spans tens to hundreds of steps, small per-step errors accumulate and amplify along the call chain.

5. Continual Learning as a Priority on the Path to AGI: The paper states a clear judgment: continual learning for the frontier model is the first priority, whereas domain-specific continual learning is not. It offers an honest estimate of the distance: "the memory-and-harness route is rapidly approaching the capability ceiling of a general assistant at the engineering level, but that genuine AGI, a system capable of autonomous capability expansion, cross-domain transfer, and robust long-horizon decision making, remains a considerable distance away."

The survey concludes: "This survey reframes continual learning in the LLM era as a boundary extension along three independent dimensions: When, Where, and How. Classical continual learning is situated as a single specific point in the three-axis coordinate space, while LLM-era methods extend the boundary independently along each axis... The framework not only organizes the distribution of existing methods but, through the contrast between dense and empty regions, identifies unfilled coordinates that offer concrete directions for future work." The authors acknowledge limitations including boundary crossings in the taxonomy, coverage limited to work available before the submission deadline, and the rapidly evolving state of evaluation protocols for harness-layer methods.

1. Add a writable memory pool with self-supervised consolidation during inference. The model writes compressed facts, states, and intermediate decisions into a fixed-size external store via forward passes only (no backprop). Separate write-on-read and consolidate-on-idle cycles so the pool doesn't churn. This enables the system to accumulate task-relevant knowledge across long conversations and multi-session interactions without finetuning, and to keep working while the consolidation happens in background.

2. Implement reward-driven test-time learning with fast-weight modules. Instantiate small trainable hidden-state layers (e.g., LoRA-like adapters in attention) that update during inference using the model's own sampled outputs as pseudo-labels, with reward signals derived from environment feedback. This enables the system to improve on unlabeled test inputs, correct errors mid-episode, and adapt its behavior to a new user or domain within a single session rather than waiting for the next training run.

3. Split the context window into a three-layer capability hierarchy with explicit write-cost scheduling. Treat parameters as the slow, stable layer; external memory as the medium layer; and in-context as the fast, ephemeral layer. Before writing to context, check whether the information already exists in memory or parameters; promote frequently retrieved context fragments into memory, and frequently promoted memories into parameter-level finetuning. This enables the system to avoid context-window flooding, reduce retrieval decay, and decide autonomously where each capability should live over time.

4. Build an active forgetting/compression mechanism into external memory. Implement an Ebbinghaus-style decay curve over memory items, plus a compression pass that merges redundant entries into summaries and a deletion pass for superseded facts (e.g., outdated world knowledge). This enables the system to keep memory bounded and recall precision high, preventing stale or contradictory information from degrading decisions in long-horizon deployments.

5. Add a persistent skill library with skill-vs-backbone credit assignment. Let the agent write reusable execution flows (code, tool-call sequences, sub-agent definitions) into an external library during interaction, tag each skill with a success-rate estimate, and periodically promote high-performing skills into the backbone via curriculum finetuning. This enables the system to extend its action repertoire without weight updates, then gradually internalize reliable skills, achieving capability growth that survives restarts and transfers to new environments.

6. Implement protocol-level self-modification with rollback. Allow the system prompt and rule files to be edited by the system itself based on task feedback (e.g., via self-referential prompt mutation), but keep every protocol versioned and associate each edit with a measured performance delta; auto-revert on degradation. This enables the system to discover better interaction formats and behavioral rules over time while protecting against prompt drift that erodes alignment or instruction following.

7. Use on-policy RL (not SFT) for sequential capability acquisition. When learning a new skill or domain after alignment, train with policy-gradient methods on the model's own rollouts with KL regularization toward the reference policy, instead of supervised finetuning on static data. This enables the system to acquire new capabilities with minimal drift away from previously learned skills, safety alignment, and instruction-following behavior, because on-policy updates implicitly prefer nearby solutions.

8. Add spurious-forgetting detection and recovery. Track a small probe set of instruction-following and safety behaviors during any finetuning or alignment stage; if scores drop without a corresponding knowledge loss (indicating spurious forgetting), trigger targeted realignment on a small replay set rather than full retraining. This enables the system to distinguish recoverable misalignment from true knowledge erasure, avoiding costly retraining runs and keeping deployed models aligned across successive update rounds.

9. Implement error-accumulation guards for long-horizon agents. Add a self-check step after each sub-action that scores the likelihood of the current trajectory against a learned model of successful trajectories, plus an external trace buffer that lets the agent rewind to the last high-confidence state. Combine this with inference-time learning that records the correction as a memory skill. This enables the system to complete tasks spanning tens to hundreds of steps without compounding per-step errors, and to improve its own error-recovery behavior after each long episode.

10. Add bidirectional parameter–harness transfer with a coordinating scheduler. Maintain a capability map that tracks which capabilities live in parameters, memory, skills, or protocols, and schedule transfers: demote rarely used parametric knowledge into memory to free capacity; promote frequently used memory/skills into parameters during idle computation windows. This enables the system to behave as a unified learning system rather than a set of disconnected mechanisms, with capability retention, recall stability, and capacity ceilings managed as one coherent resource-allocation problem.

11. Include a retrieval-decay guard on the harness. Before every retrieval from memory, compute a recency–utility score; if retrieval quality drops because old items are being over-weighted, trigger consolidation instead of adding more items. This enables the system to maintain stable recall over very long deployments, where naive external memory otherwise degrades into a haystack that no longer supports reliable retrieval.

12. Add an evaluation harness that measures capability evolution along all three axes. Track per-carrier capability curves (parameter, memory, skill, protocol), transfer scores between carriers, forgetting rates per carrier, and error-accumulation curves over episode length, rather than a single accuracy number. This enables the system to be monitored and improved in deployment, because you can see exactly which carrier is failing, when drift happens, and whether harness-to-parameter promotion is actually working.

An integrated system with these improvements can: persist and grow capability across sessions, adapt within a single interaction, learn new skills without eroding old ones, self-correct over long horizons, forget actively and deliberately, transfer capabilities between fast and slow storage, and be verifiably monitored on all of it—moving from a static model to a continually evolving agentic system.

Classical continual learning (CL) has primarily focused on enabling models to update and retain knowledge through parameter-centric mechanisms, e.g., training strategies, architectural designs, and weight adaptation. However, emerging paradigms are reshaping the scope of CL beyond this traditional model adaptation view. For instance, on-policy learning broadens the space of update mechanisms; test-time training extends CL from the training phase to inference; and external harness components such as memory, skill libraries, and interaction protocols extend the evolutionary boundaries of model capabilities far beyond the static parameter space. Collectively, these developments indicate a transition from parameter-centric learning toward system-level adaptation. To characterize this transition, we examine the evolution of continual learning through three dimensions: When, How, and Where learning occurs. The How dimension encompasses off-policy, on-policy, and beyond-gradient optimization mechanics. The When dimension captures evolution across pre-training, post-training, and inference-time stages. The Where dimension delineates updates occurring within internal parameters versus external structural constraints. Anchored by this tri-axial framework, we systematically survey representative methods, trace the ongoing transition of continual learning, and discuss the key challenges, broader implications, and future directions arising from this paradigm shift.