AgentGarten: Code Worlds for Evolving Agents

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "AgentGarten: Code Worlds for Evolving Agents".

Tom: The gist The framework introduces AgentGarten,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at AgentGarten today. This paper tackles that big problem of making interactive virtual worlds that are both faithful to the rules and realistic visually for agents. It claims they can achieve this by coupling simulators or game engines with a shared neural renderer one <ref:2610.12374#pg1,game engines with a shared neural renderer>.

Jane: Exactly. The core idea is that agents learn best through exploration and interaction, but what they learn is totally limited by the environments they practice in, which have to be consistent in state and rules, and realistic in how observations look compared to the real world one <ref:2610.12374#pg1,by the environments they practice in, which>.

Lu: AgentGarten addresses that bottleneck by coupling simulators or game engines with a shared neural renderer one <ref:2610.12374#pg1,game engines with a shared neural renderer>. The simulation backends keep track of a persistent world state and run program-defined interaction rules, while the renderer creates visual observations from structured conditions exported through a common interface.

Meng: So it separates the control—the state and the rules—from the visuals. That way, you get rigorous control over what's happening in the simulation without having to build a custom visual pipeline for every single scene one <ref:2610.12374#pg1>.

Lalam: And then they build this neural renderer by adapting a pretrained video model to geometry conditions, distilling it using something called Adversarial Forcing, and optimizing it for real-time interaction one <ref:2610.12374#pg1,a pretrained video model to geometry conditions>.

Tom: Adversarial Forcing is the trick there. It makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and it adds real-data adversarial supervision to improve its visual quality one <ref:2610.12374#pg1,makes history prefilling differentiable through exact replay, so that losses on later>.

Jane: It sounds complicated, but what it means is that as the agent interacts and sees more stuff, the visual representation gets better because it’s being trained against real data during that process.

Lu: The paper defines a code world as combining a scene program p, an engine that runs it, and this neural renderer R theta one <ref:2610.12374#pg1>. The engine updates the state and captures a structured condition ct from the camera, which is something like surface normals or depth or reference text one <ref:2610.12374#pg1>.

Tom: So you have this scene program defining the rules, and then the engine executes those rules to get a new state plus that structured condition. That's how they capture what's happening in the world at any given moment one <ref:2610.12374#pg1>.

Meng: And then the neural renderer R theta takes that structured condition along with some cached visual history to generate new observations, which are fed back to the agent one <ref:2610.12374#pg1>.

Jane: It’s a loop: actions lead to state updates, which lead to new observations, and those observations inform the next action. This whole setup is designed for real-time interaction where agents get immediate feedback one <ref:2610.12374#pg1>.

Tom: The empirical study shows that pretrained foundation model agents observing only through this neural rendering can rapidly learn semantic tool knowledge and improve outcomes across rounds by using written playbooks in things like hide-and-seek one <ref:2610.12374#pg1>.

Lu: In those examples, the agents perceive the world strictly through rendered observations, diagnose failures, and distill those insights into written playbooks that subsequent agents inherit and refine one <ref:2610.12374#pg1>.

Jane: It shows that these evolving agents don't just learn from trial and error; they learn to write down their strategies based on what they see visually one <ref:2610.12374#pg1>.

Tom: The paper evaluates code worlds across different appearances, like Halloween or Medieval Minecraft, and tasks like navigation or tool use one <ref:2610.12374#pg1>. The agents develop increasingly sophisticated strategies over rounds of interaction, such as building shelters or crossing ramps in those games one <ref:2610.12374#pg1>.

Meng: From an engineering standpoint, the ability to write new worlds as code and render them through the same interface means you can scale up both the number of environments and the difficulty alongside your agents one <ref:2610.12374#pg1>.

Jane: It suggests a future where agents keep evolving continuously through this loop of interaction and refinement rather than just learning in isolation one <ref:2610.12374#pg1>.

Tom: The title, "AgentGarten: Code Worlds for Evolving Agents," really captures what's going on here. It points to the idea that environments are no longer static; they are dynamic things that agents can interact with and change over time one <ref:2610.12374#pg1>.

Lu: And the authors point out that what an agent can learn is strictly bounded by the environment it practices in, which has to be faithful, with consistent state, rules, and dynamics one <ref:2610.12374#pg1,be faithful, with consistent state, rules, and dynamics>.

Jane: So AgentGarten gives us a way to build environments that are both faithful and realistic by dividing the work between a simulator or game engine and that shared neural renderer one <ref:2610.12374#pg1>.

Tom: It's about moving toward systems where new worlds can be written as code and rendered through the same interface, which allows those environments to grow in number and difficulty alongside the agents practicing in them one <ref:2610.12374#pg1>.

Conclusion: Tom: So we've been looking at AgentGarten today, and now we’re hitting the conclusion to see what all this actually means for agents moving forward.

Jane: It boils down to a framework that couples simulators or game engines with a shared neural renderer.

Lu: They’re trying to solve that problem of making interactive virtual worlds both faithful to the rules and realistic visually.

Meng: The paper shows they did this by separating the simulation backends—which keep track of state and rules—from the renderer, which generates visual observations from a common interface.

Lalam: And they built this neural renderer by adapting a video model, using something called Adversarial Forcing to distill it into a real-time system.

Tom: So the main point is that agents can now evolve through experience because you can write new worlds as code and render them through the same interface.

Jane: That lets environments grow in number and difficulty right along with the agents that practice in them.

Lu: It really suggests a future where we don't just have static environments, but these things that keep evolving through interaction.

Meng: From an engineering standpoint, it’s about moving past building custom visual assets for every single scene.

Lalam: The core contribution is this framework for code worlds that links persistent state to a shared real-time neural renderer through structured conditions.

Tom: So what does this mean if we think about the agents themselves? What they can learn is bounded by the environments they practice in, which must be consistent and realistic.

Jane: AgentGarten builds those environments so they are both faithful and realistic by dividing the work between a simulator or game engine and that shared neural renderer.

Lu: It's about moving toward systems where new worlds can be written as code and rendered through the same interface, which allows those environments to grow in number and difficulty alongside the agents practicing in them.

Meng: We’re seeing agents improve by distilling each round of experience into playbooks that subsequent agents inherit and refine.

Lalam: That loop—where they perceive, interact, diagnose failures, and distill insights—is what drives the evolution we're talking about here.

Tom: It’s a step toward agents that keep evolving through interactive experience instead of just learning in isolation.

Jane: And the authors are pointing toward this idea that environments can scale in both number and difficulty alongside their agents.

Lu: That’s a really interesting direction, connecting programmable dynamics with a shared real-time neural renderer.

Meng: It makes the whole interaction loop much more robust when you can code the world itself.

Lalam: We're expecting agents that keep evolving through interaction to come from that loop, and it's not just about playing a game anymore.