Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning".

Jane: The gist The Prorsa framework introduces an agentic RL framework for prediction-guided prefill-decode resource allocation that achieves end-to-end rollout speedups ranging from 1.11× to 4.05× over prefill-decode colocation.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning. It sounds like they’re trying to speed up how these complex AI agents run during their training rollouts, and the results are showing some pretty big gains.

Jane: Exactly. So, what's the core idea here? It seems like they’re looking at a specific problem in agentic RL where prefill and decode workloads end up mixing together in a way that slows things down compared to just keeping them together on one system.

Lu: The main thing Prorsa tackles is that agentic RL has a workload pattern totally different from traditional reinforcement learning because you get prefill requests constantly popping up during the rollout phase, which means both prefill and decode tasks are happening at the same time throughout the whole process.

Meng: So, like, if I'm training an agent, I've got these multi-turn interactions where every step needs a new prefill request because you keep adding context to it while it's running. That means prefill and decode are fighting for resources at once during rollout.

Lalam: That coexistence is what makes separating them, or PD disaggregation, look promising for speeding things up in agentic RL rollout <ref:2610.16592#pg2>.

Tom: Right, so the paper shows that separating prefill and decode can help by reducing interference between the data-parallel attention modules, enabling parallelism specific to each stage, and ultimately cutting down how long it takes to finish a whole trajectory.

Jane: That sounds like a real efficiency boost. But they also point out that just having a fixed way of splitting those resources might not work well because the workload changes depending on what task you're doing or over time.

Lu: That’s the caveat they highlight, and it’s important because workloads aren't constant; they shift as you progress through the rollout <ref:2610.16592#pg3>.

Meng: I worry about that variability at scale. If we set up a static split and then the agent starts hitting a phase where it demands way more prefill, that whole setup probably breaks down quickly.

Lalam: Prorsa addresses this by using an adaptive framework, combining PD disaggregation with worker allocation that changes based on workload forecasts <ref:2610.16592#pg5>. The insight is that the RL framework actually gives us signals about what inference demand is coming up later in the trajectory's life.

Tom: So, instead of guessing how much prefill or decode you need, Prorsa tries to predict it by monitoring things like sample dispatch and tool execution to forecast short-horizon demand <ref:2610.16592#pg5>.

Jane: What kind of prediction are we talking about here? How does this forecasting actually translate into changing how the resources are allocated between prefill and decode roles?

Title and authors: Lu: They have a Workload Predictor that formalizes the demand estimation by tracing each trajectory through a state machine <ref:2610.16592#pg5>. This predictor estimates both prefill demand and decode-admission token volume from four different sources, including data sampler, sandbox startup, the ready queue backlog, and tool execution <ref:two thousand six hundred ten point one six five nine two#pg3.

Meng: Four sources for prediction sounds comprehensive. So they're not just looking at the immediate request but also how many other trajectories are waiting in line or when the next tool call is expected. That’s useful for knowing what to allocate right now, right?

Tom: It is useful, and that’s where the mechanics come in. Based on those forecasts, Prorsa adjusts the PD ratio by switching rollout workers between prefill and decode roles to keep instance allocation matching those changing demands <ref:two thousand six hundred ten point one six five nine two#pg5. The role switcher uses decode KV-cache requirements combined with prefill demand to set the PD instance ratio while trying to keep a safety margin under GPU budget constraints <ref:two thousand six hundred ten point one six five nine two#pg8.

Jane: So, it’s not just about prediction; it’s about actively reallocating resources dynamically based on what the system expects to need next. That sounds like a lot of real-time coordination for an RL rollout <ref:two thousand six hundred ten point one six five nine two#pg8.

Lu: The practical implementation involves role switching, where the executor picks the least-loaded eligible instance for a required change <ref:two thousand six hundred ten point one six five nine two#pg8. This instance is taken out of the routing pool, and any requests on it get cancelled and retried elsewhere using saved prefixes or persistent KV cache state <ref:two thousand six hundred ten point one six five nine two#pg8. Then the engine rebuilds the communication and cache resources for the target role after all in-flight work has settled <ref:two thousand six hundred ten point one six five nine two#pg8.

Meng: That sounds a bit heavy operationally, canceling and retrying requests on other instances. What about the cost of that constant switching? Can we afford that overhead at scale?

Tom: The goal is to maximize safety under the GPU budget constraints, which means they are trying to keep utilization high while managing the allocation shift <ref:two thousand six hundred ten point one six five nine two#pg8. The prediction accuracy itself is also measured; they see prefill and decode-admission forecasts ranging from five point four zero percent to fourteen point three seven percent for prefill, and about five point one nine percent to thirteen point six nine percent for decode <ref:two thousand six hundred ten point one six five nine two#pg9.

Jane: So, the prediction accuracy isn't perfect—it’s in that range—but the system is designed to handle that uncertainty by being adaptive rather than static <ref:two thousand six hundred ten point one six five nine two#pg8. That’s a realistic approach for something this complex.

Lu: The results are pretty compelling. Across various agentic RL scenarios, Prorsa gets rollout speedups ranging from one point one one times up to four point zero five times over prefill-decode colocation in both synchronous and asynchronous settings <ref:two thousand six hundred ten point one six five nine two#pg3.

Tom: And for a specific model like DeepSeek-V3 point 2, Prorsa shows efficiency gains of two point five four times up to four point zero five times on the coding agent task and two point three five times up to two point nine nine times on the general agent task <ref:two thousand six hundred ten point one six five nine two#pg11. That’s a solid number showing real speedup in practice against the baseline <ref:two thousand six hundred ten point one six five nine two#pg7.

Jane: It really shows that when you manage the prefill and decode separation this way, you see much better GPU utilization. They found an average GPU idle ratio of three point seven six percent to fifty-nine point nine eight percent under PD colocation because of that interference <ref:two thousand six hundred ten point one six five nine two#pg7. Prorsa fixes that by making sure the resources are used more effectively <ref:two thousand six hundred ten point one six five nine two#pg3.

Title and authors: Lu: Furthermore, they achieved reductions in end-to-end trajectory latency by one point one seven times to seven point three three times compared to PD colocation on the general agent task <ref:two thousand six hundred ten point one six five nine two#pg11. That speaks directly to mitigating that long-tail problem that plagued synchronous rollout systems <ref:two thousand six hundred ten point one six five nine two#pg3.

Meng: So, for someone like me who’s worried about deployment costs and bottlenecks, a factor of four speedup is significant because it means we can process way more training data in the same amount of time <ref:two thousand six hundred ten point one six five nine two#pg3.

Tom: The role switching strategy itself gives us speedups between one point zero times and one point four four times across different settings <ref:two thousand six hundred ten point one six five nine two#pg8. It’s not just the prediction, it’s the execution of that switch that adds value <ref:two thousand six hundred ten point one six five nine two#pg8.

Jane: So, to put it simply, Prorsa is using a forecast system to dynamically manage prefill and decode resources so they aren't interfering with each other while the agent is training <ref:two thousand six hundred ten point one six five nine two#pg3. This adaptive approach seems crucial because a fixed allocation just doesn't hold up when things get busy <ref:two thousand six hundred ten point one six five nine two#pg3.

Lu: The prediction accuracy itself is important context; the forecasts for prefill and decode-admission volumes are in the five point four zero percent to fourteen point three seven percent range for prefill, and five point one nine percent to thirteen point six nine percent for decode <ref:two thousand six hundred ten point one six five nine two#pg9. That tells us how much confidence we have in the input to the allocation decisions <ref:two thousand six hundred ten point one six five nine two#pg8.

Meng: I'm still skeptical about those forecast numbers when you look at real traffic, but if it’s guiding the system toward better utilization, it might be worth testing on actual production loads <ref:two thousand six hundred ten point one six five nine two#pg7.

Tom: The final point is that Prorsa uses this adaptive PD roleswitching strategy guided by those workload forecasts to handle fluctuating demand <ref:two thousand six hundred ten point one six five nine two#pg8. The evaluation confirms it effectively unlocks the performance of PD disaggregation during rollout, reaching up to four point zero five times the throughput of PD colocation <ref:two thousand six hundred ten point one six five nine two#pg13.

Jane: It’s a very neat system because it connects the trajectory's lifecycle signals directly to resource allocation decisions through that prediction layer <ref:two thousand six hundred ten point one six five nine two#pg5. This whole Prorsa framework is basically about making sure the hardware usage matches the actual demand of the agentic RL process <ref:two thousand six hundred ten point one six five nine two#pg3.

Lu: And it addresses that initial challenge from page one, which was that a fixed prefill-decode allocation can fail when tasks or workloads change over time <ref:two thousand six hundred ten point one six five nine two#pg3. It gives you flexibility in a changing environment <ref:two thousand six hundred ten point one six five nine two#pg3.

Meng: So, for the listener who is just listening, this means that if you are running complex AI training that involves many back-and-forth steps, this method could potentially make the process much faster and more efficient by being smarter about how it uses its GPUs <ref:two thousand six hundred ten point one six five nine two#pg3.

Tom: That’s the core message of Prorsa, showing a way to get significant speedups—up to four point zero five times—over the standard prefill-decode colocation setup in agentic RL rollout <ref:two thousand six hundred ten point one six five nine two#pg3. We'll be diving into how they actually built that prediction next.

The paper's summary: Tom: So, we're looking at Prorsa again because they've built this whole agentic RL framework that predicts resource allocation based on how the agents move through their training steps to get speedups up to four times over the standard way of running prefill and decode together.

Jane: Exactly. The core idea is this system doesn't just use a fixed split for prefill and decode resources, which usually gets messy when you have these multi-turn interactions going on all the time during rollout.

Tom: Right. They introduce this agentic RL framework that uses prediction to guide how those prefill and decode resources are allocated dynamically based on the workload forecasts they generate from monitoring trajectory lifecycles.

Jane: So, what's the practical mechanism behind that? How does it actually change things compared to just having two separate systems running side by side?

Tom: The framework monitors things like sample dispatch, sandbox startup times, tool execution, and reward completion to forecast how much prefill demand and decode KV-cache requirements will be needed in the short term.

Jane: And based on those forecasts, Prorsa adjusts the PD ratio by switching workers between prefill and decode roles to keep the allocation aligned with what's coming next.

Tom: It uses a role switcher that sets the number of instances for prefill based on throughput and for decode based purely on KV-cache capacity, all while trying to keep a safety margin under GPU budget constraints.

Jane: So it’s an adaptive system that tries to anticipate demand rather than just reacting to it, which sounds like it could really handle those fluctuating workloads you talked about earlier.

Tom: It does. And the evaluation is pretty strong. They show speedups ranging from one point eleven times up to four point five times over the prefill-decode colocation setup in both synchronous and asynchronous settings.

Jane: That's a big range, and it shows that this isn't just a nice theoretical idea; it’s actually delivering significant performance improvements on concrete agentic RL tasks like coding agents.

Tom: On DeepSeek-V3 point two they see efficiency gains of two point five four times up to four point five times on the coding agent task and two point three five times up to two point nine nine times on the general agent task <ref:2610.16592#pg1>. That's a real number showing how much faster things run with this method compared to just sticking with the standard setup.

Jane: And they also found that Prorsa really helps fix that problem where you get a lot of GPU idleness because prefill and decode are interfering across those data-parallel attention modules in the colocation setup.

Tom: They did. Under PD colocation, they saw an average GPU idle ratio between three point seven six percent and fifty-nine point nine eight percent because of that interference. Prorsa cuts that down significantly by making the resources work better together.

Jane: It seems like this adaptive approach addresses the problem of a fixed allocation failing over time, which was a major challenge they set out to solve in agentic RL rollout.

Tom: Exactly. The paper shows how predicting those demand signals and switching roles can unlock the potential of prefill-decode disaggregation during rollout, getting you up to four point five times the throughput of colocation.

Jane: It's a neat way to connect the actual lifecycle signals of a trajectory—how long it takes, what tools it uses—directly into making smarter decisions about where your compute power should go.

Tom: And while they report prediction accuracies in that five point four zero percent to fourteen point three seven percent range for prefill and around five point one nine percent to thirteen point six nine percent for decode, the results show the system still performs well enough to be practical.

Jane: So, it’s a feasible method that uses prediction and dynamic allocation to manage the complexity of agentic RL rollout better than just using a static split between prefill and decode systems.

Tom: The implication is that for any complex AI training involving many back-and-forth steps, this adaptive resource management could make the whole process much faster and more efficient by being smarter about how it uses its GPUs.

The paper's improvements: Tom: So, we're talking about Prorsa's suggested improvements now because the system isn't just about prediction; it's about how it actually manages those resource switches and what that means for deployment.

Jane: Right, so we talked about the prediction layer, but what does that translate to in terms of actual system behavior when things get busy? What specific changes are they proposing?

Tom: They suggest a dynamic role switching mechanism driven by those short-horizon workload forecasts we discussed earlier. This is not just a static split; it’s an active reallocation of instances.

Jane: So, if the prediction says we're about to hit a prefill spike, what exactly happens to the workers? How does that role switch actually execute in practice?

Tom: The executor selects the least-loaded eligible instance for each required role change. That instance gets removed from the routing pool and stops taking new requests, and any existing work on it gets cancelled and retried elsewhere using saved prefixes or persistent KV cache state.

Jane: It sounds like a bit of a heavy operation, but that’s how they keep things synchronized during the transition so the overall system doesn't just get bogged down by switching overhead.

Tom: Exactly. Once all the in-flight work has settled, the engine rebuilds role-specific communication and cache resources for the target role. It’s about ensuring a clean handover between stages without losing progress on requests.

Jane: That makes sense, it’s about making sure that when you switch roles, you don't just lose time waiting for the new configuration to load up. And that directly tackles the long-tail problem we mentioned earlier in synchronous rollout.

Tom: It addresses that latency issue by isolating decode from prefill interference. For synchronous rollout on a coding agent task, they found a reduction in end-to-end trajectory latency between one point one seven times and seven point three three times compared to just using prefill-decode colocation.

Jane: That's a huge range of improvement, and it directly impacts how quickly an agent can actually finish its job during training. It shows that managing the resource contention at the attention module level is really paying off for real-world task completion time.

Tom: And looking at the role switching strategy alone, they found speedups between one point zero times and one point four four times across different settings. That means even just optimizing how you move instances around provides a measurable gain without needing a perfect prediction every single time.

Jane: So, this isn't just about having a better predictor; it’s about having a smarter way to actually move the compute power around based on what the trajectory needs at that exact moment. It’s an improvement in the operational layer of training, not just the model architecture itself.

Tom: Right. The implication is that if you're running large-scale agentic RL, you can’t rely on a fixed allocation to keep things snappy; you need this kind of dynamic, prediction-guided management to sustain high throughput over long rollouts.

Conclusion: Tom: So we're wrapping up Prorsa because to sum it all up, they’ve shown an agentic RL framework that uses prediction to guide how prefill and decode resources are allocated dynamically, achieving rollouts of one point one one times to four point five times over the standard way of running them together.

Jane: That's a lot of speedup, and it really shows how much better things can get when you stop treating those two workloads as just being on the same system.

Tom: Exactly. The main implication is that if you have complex AI training involving lots of back-and-forth steps, this adaptive management could make the whole process much faster by being smarter about how it uses its GPUs.

Jane: It shifts the focus from a fixed hardware setup to a dynamic resource strategy guided by real-time workload signals, which is something many deployment engineers struggle with.

Tom: I agree. The prediction accuracy itself is in that five point four zero percent to fourteen point three seven percent range for prefill and around five point one nine percent to thirteen point six nine percent for decode, which means the system still performs well enough to be practical under those conditions.

Jane: So it's a feasible method that uses prediction and dynamic allocation to manage the complexity of agentic RL rollout better than just using a static split between prefill and decode systems.

Tom: The final thing is that Prorsa uses this adaptive PD roleswitching strategy guided by workload forecasts to handle fluctuating demand, proving it effectively unlocks the potential of prefill-decode disaggregation during rollout.

Jane: It's a very neat system because it connects the trajectory's lifecycle signals directly into resource allocation decisions through that prediction layer.

Tom: And that’s what really matters for us, because it shows how to get significant speedups—up to four point five times—over the prefill-decode colocation setup in agentic RL rollout.

Lu: If this works reliably on real traffic and across different tasks, it means we can design training infrastructure that scales much more effectively without manually tuning every single resource setting for every new environment.

Meng: I’m still cautious about those prediction accuracy numbers when you look at production loads, but if this method gives us a way to keep utilization high while managing the allocation shift, it's worth testing on actual traffic.

Lalam: I'm sure of my answer here; the literature supports this direction because the workload patterns in agentic RL are inherently dynamic, and Prorsa is built to handle that exact variability.

Tom: So to wrap up, Prorsa shows a way to get massive speedups by being adaptive instead of static when it comes to resource allocation during agentic RL rollout.

Jane: It’s a very neat system because it connects the trajectory's lifecycle signals directly into resource allocation decisions through that prediction layer.

Lu: If this works reliably on real traffic and across different tasks, it means we can design training infrastructure that scales much more effectively without manually tuning every single resource setting for every new environment.

Meng: I’m still cautious about those prediction accuracy numbers when you look at production loads, but if this method gives us a way to keep utilization high while managing the allocation shift, it's worth testing on actual traffic.

Lalam: I'm sure of my answer here; the literature supports this direction because the workload patterns in agentic RL are inherently dynamic, and Prorsa is built to handle that exact variability.