Towards Certificate-Driven Software Porting: A Self-Improving Agentic Harness for Scientific Program Optimization

arXiv:2609.34069 · cs.AI, cs.PL, cs.SE · Submitted 2026-09-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Towards Certificate-Driven Software Porting".

Jane: The gist The certificate-driven evolutionary search (CDES) extends evolutionary search with enforceable restrictions derived from failed candidates, recorded as certificates of assumptions, checker evidence, and justified restrictions.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Now that we’ve seen how they use certificates to guide the search, let's look at what this whole approach is actually called and who wrote it. The paper is titled "Towards Certificate-Driven Software Porting: A Self-Improving Agentic Harness for Scientific Program Optimization."

Jane: That title tells us a lot about the goal. It’s not just about porting code; it’s about creating a self-improving system using an agentic harness to optimize scientific programs, and that's what this paper is all about.

Lu: The authors are Piyush Jha, Aishik Ghosh, and Vijay Ganesh from the Georgia Institute of Technology and Lawrence Berkeley National Laboratory. They are combining expertise from computer science with physics research.

Meng: That mix of backgrounds is important because they need to understand both the software engineering challenges of porting complex code and the underlying physical simulations that the code represents.

Tom: It seems like their main contribution is moving beyond simple prompt-based feedback, which we just talked about, by creating this CDES system where failed candidates become enforceable restrictions.

Jane: They are essentially giving the AI a structured way to learn from what went wrong, turning failures into rules that block bad choices and guide repairs in the next round.

Lu: The authors focus on how information from those failed candidates can be converted into these enforceable restrictions, which is the core mechanism of CDES <ref:2609.34069#pg2>.

Meng: So they aren't relying on just a better prompt; they’re building an internal control logic that actively rejects choices and mandates repairs based on what the system has learned.

Tom: That's right, and it’s a big step because it means the search process itself is self-improving because accumulated certificates change which future modifications are permitted without needing to retrain the underlying LLM.

Jane: It means the agent iteratively refines its search space based on previously validated errors, leading to increasingly accurate and efficient code generation over time.

Lu: This is a fundamental shift in how we think about using LLMs for complex tasks like scientific code translation <ref:2609.34069#pg2>.

Meng: From an engineering perspective, that autonomy is what makes it practical; it allows the system to manage the complexity of generating and repairing large pieces of code by itself.

Tom: So, we’re moving from just hoping for good results to building a structured feedback mechanism where errors actually shape the future search space.

Jane: And this moves us closer to having AI systems that can handle tasks that require deep, sustained reasoning about complex software structures.

Lu: It's about using symbolic feedback in areas like RLSF and constrained reinforcement learning to drive these kinds of improvements <ref:2609.34069#pg2>.

Meng: The authors are drawing on existing methods, like those from FunSearch or AlphaEvolve, but they’re making it smarter by adding this certificate-driven control layer.

Tom: So they’ve taken state-of-the-art evolutionary program search and overlaid this new layer of constraint enforcement to make it much more robust for porting code.

Jane: This whole setup is designed specifically to handle the messy, error-prone nature of translating legacy scientific code into something runnable on modern hardware.

The paper's summary: Tom: So what is the actual high-level summary of what they’re proposing here? Essentially, they are introducing this Certificate-Driven Evolutionary Search system to enhance evolutionary search methods with enforceable restrictions derived from failed candidates.

Jane: They take the information from those failed attempts—the certificates of assumptions, checker evidence, and justified restrictions—and convert them into rules that control the subsequent search.

Lu: The key mechanism is this certificate database which supplies feedback to the mutator LLM and evidence to a conflict checker and analyzer <ref:2609.34069#pg2>.

Meng: This database acts as a repository of knowledge about what kind of assumptions lead to failures, allowing the system to learn from those specific errors.

Tom: The control logic then actively enforces these validated restrictions during generation and repair by rejecting conflicting choices, backtracking when needed, and performing targeted repairs while trying to preserve compatible edits.

Jane: It’s not just feedback; it’s active enforcement. The system doesn't just read the certificate; its logic uses it to reject choices and guide the repair process directly.

Lu: This means CDES doesn't rely on prompt compliance alone, which is a crucial distinction they make when comparing it to previous methods <ref:2609.34069#pg2>.

Meng: So the paper is arguing that this structured control logic is what separates this method from older evolutionary search approaches where the LLM might just be guessing based on a good prompt.

Tom: It’s essentially saying that the system can learn to reject bad choices and actively guide itself toward better solutions through these enforced restrictions.

Jane: This makes the entire process of porting legacy code much more systematic and less reliant on chance or perfect prompting from the LLM.

The paper's improvements: Tom: The paper highlights several specific improvements they’ve made to this system, focusing on how this certificate feedback actually helps in real-world scenarios.

Jane: One major improvement is the reliability boost; they found that the fraction of candidates passing required correctness checks increased significantly from fifty-five percent up to ninety percent.

Lu: That’s a major win for scientific software porting, because it means we're much more confident in the code being generated for complex tasks <ref:2609.34069#pg2>.

Meng: And they also improved the efficiency side; they showed that proposals that actually improve throughput by more than ten percent increased from fifty percent up to eighty percent with this certificate feedback loop.

Tom: Plus, they quantified the cost savings too; the cost per passing proposal dropped by about ten point seven percent, which shows it’s not just improving quality but also doing it more economically.

Jane: That combination of higher quality and better efficiency for a lower cost is what makes this method very compelling when you look at practical applications in porting large codebases.

Lu: They are also showing that the system can achieve optimization patterns that improve on existing expert implementations, like reusing earlier calculations, which is a specific kind of optimization pattern.

Meng: And they showed a hybrid benefit too; combining their generated code with Celeritas’s components for particle energy selection and direction calculation increased GPU throughput by sixteen point one percent over just using Celeritas <ref:2609.34069#pg3>.

Tom: So the improvements aren't just theoretical; they are backed up by these quantifiable metrics showing concrete gains in speed and correctness across different benchmarks.

Jane: It’s a very strong demonstration that this certificate-driven approach delivers tangible benefits when applied to scientific optimization problems.

Conclusion: Tom: So, to wrap up, the authors of "Towards Certificate-Driven Software Porting: A Self-Improving Agentic Harness for Scientific Program Optimization" have provided a proof-of-concept for reliably and scalably porting large legacy scientific codebases.

Jane: They’ve demonstrated that this approach connects conflict detection with certificate validation and enforces search control to create a system that can handle complex optimization tasks effectively.

Lu: It’s a proof of concept for using certificate feedback to create an agentic harness that learns from failures and guides itself toward better solutions, which is really powerful for complex scientific software.

Meng: The practical application here is showing how we can build systems where the AI gets smarter simply by accumulating validated errors without needing continuous retraining.

Tom: They showed that this method generates code with optimization patterns that are actually better than what expert-written implementations achieve on their own, and combining it with other tools yields even more gains.

Jane: It’s a very solid demonstration of how to enforce a flexible range of correctness requirements while still achieving high performance targets on demanding simulations.

Lu: Future work will focus on testing how the cost and correctness benefits of these certificates change across different types of functions and optimization difficulties, which is where the next research directions lie.

Meng: And they’ll be looking at those trade-offs to see if this method stays effective when applied to a much wider variety of scientific problems.

Tom: So, the CDES system is a powerful way to connect conflict detection with certificate validation and enforced search control for scientific optimization, offering a reliable path toward porting large codebases.

Jane: It’s an interesting development because it shows how we can build AI systems that learn to self-regulate based on concrete evidence of past failures.

Lu: Thanks for joining us today. We've been really interested in this work on the CDES system and its potential applications in scientific computing.

Meng: It’s a very exciting area for practical engineering because it shows a way to build better tools that can handle the complexity of legacy code generation.

Tom: We’ll keep an eye on those future studies to see where this research takes us next.

School of Computer Science, Georgia Institute of Technology, USA · School of Physics, Georgia Institute of Technology, USA · Lawrence Berkeley National Laboratory, USA

cs.AI, cs.PL, cs.SE

Submitted: 2026-09-28

Updated: 2026-09-28

Comments: Submitted to ML4PS 2026

Code: https://github.com/leanprover-community/physlib

Project page: https://diffblue.github.io/cbmc

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The certificate-driven evolutionary search (CDES) extends evolutionary search with enforceable restrictions derived from failed candidates, recorded as certificates of assumptions, checker

Key concepts

Certificate-Driven Evolutionary Search (CDES): CDES extends evolutionary search by turning failed candidate results into 'certificates.' These certificates act as enforceable rules or restrictions that guide the next generation of candidate code. This feedback loop allows the system to learn from mistakes and improve subsequent searches, ensuring better optimization.
Evaluation Harness: This harness is a testing system that translates CPU programs into GPU code (CUDA) for scientific simulations. It performs multiple checks, including formal analysis and numerical comparisons against reference solutions. If a candidate fails any mandatory check, it generates a certificate of failure, which feeds back into the search process.
Certificate Database: The certificate database stores the records of failed candidates and the restrictions derived from them. These certificates are used to enforce constraints on future code generation attempts. This mechanism moves beyond simple prompt compliance by creating a separate control logic that validates candidate choices.
Geant4 Functions: These are specific scientific functions, like the Compton function or Fermi function, used as test cases in the study. They represent complex calculations common in physics simulations. The paper uses these functions to benchmark how well CDES-generated GPU code performs compared to CPU versions and expert implementations.

Terminology

Certificate-Driven Evolutionary Search — 7× in this paper
Ablation — 6× in this paper · explained in 1 episode
Benchmark — 6× in this paper · explained in 2 episodes
Analyzer — 4× in this paper · explained in 1 episode
Core — 4× in this paper · explained in 2 episodes
Reinforcement Learning — 4× in this paper · explained in 4 episodes
Abstraction — 2× in this paper · explained in 1 episode
Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
Verification — 2× in this paper · explained in 2 episodes
Ablation study — 1× in this paper · explained in 1 episode
Adam — 1× in this paper · explained in 2 episodes
Alignment — 1× in this paper · explained in 1 episode
Conference on Programming Language Design and Implementation — 1× in this paper
Conservation Laws — 1× in this paper · explained in 3 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 2 episodes
Nadia — 1× in this paper · explained in 2 episodes

Summary

The gist The certificate-driven evolutionary search (CDES) extends evolutionary search with enforceable restrictions derived from failed candidates, recorded as certificates of assumptions, checker evidence, and justified restrictions.

Certificate-Driven Evolutionary Search (CDES)

CDES extends an AlphaEvolve-style core implemented with ShinkaEvolve where information from failed candidates can be converted into enforceable restrictions on subsequent search. The certificate database provides feedback to the mutator LLM, but CDES does not rely on prompt compliance alone, as its control logic separately enforces validated restrictions on candidate choices and guides the repair process. This design works with frontier LLMs through API calls and requires neither access to model weights nor additional training, drawing on symbolic feedback in RLSF and constrained reinforcement learning in CDRL.

Evaluation Harness

The evaluation harness integrates CDES with tool selection and correctness checks to translate CPU programs into CUDA, NVIDIA’s GPU programming platform. This harness goes beyond unit tests to combine formal checks, numerical comparisons, physics checks, and GPU safety tests. The harness records a hash identifying the exact candidate source code tested; failed or unresolved mandatory checks block acceptance and enter the certificate database. Formal analysis includes Lean package verification of selected requirements called contracts rather than whole-program equivalence, while execution-based validation involves differential tests comparing reference and candidate outputs on matching inputs within fixed numerical tolerances.

Experimental Design

The study evaluates two complementary Geant4 functions as case studies: the Compton function and the Fermi function. The Compton function is chosen because Celeritas has an expert GPU implementation, while the Fermi function is chosen because, to our knowledge, it has no matching GPU implementation already available. The benchmark compares CPU reference, CDES-generated GPU code, and Celeritas where available using approximately 250,000 inputs on a 10 GB partition of an NVIDIA H100 GPU. Speedup is defined as reference time divided by generated-code time, and throughput is inputs processed per second.

Performance Results

CDES-generated GPU code achieves 13.78× and 23.54× speedups over the CPU versions under the standalone timing protocol, with GPU throughput exceeding an expert implementation by 14.9%. The optimization patterns observed include reusing earlier calculations and simplifying repeated work; for instance, in Compton, CDES-generated code reuses a value computed inside a helper to avoid repeated work. Furthermore, combining Celeritas’s code for selecting particle energies with the generated code for calculating directions increases GPU throughput by 16.1% over Celeritas (Table 2).

Ablation Study

An ablation over execution settings shows that certificate feedback increases the fraction of candidates passing required correctness checks from 55% to 90%. Similarly, the fraction of proposals that also improve throughput by over 10% increases from 50% to 80%. The cost per passing proposal is lowered by 10.7%, demonstrating that certificate feedback improves the passing rate and lowers cost per passing proposal.

Conclusion

CDES connects conflict detection, certificate validation, and enforced search control for scientific optimization, providing a proof-of-concept approach toward reliably and scalably porting large legacy scientific codebases. Future work will test how the cost and correctness benefits of certificates vary across functions and optimization difficulty. The generated code exhibits optimization patterns that improve on expert-written implementations, and combining it with Celeritas components shows complementary benefits from human–AI collaboration. The paper presents a proof-of-concept approach toward reliably and scalably porting large legacy scientific codebases, demonstrated on two Geant4 functions while enforcing a flexible range of correctness requirements.

Page 1</ref:2609.34069

Page 2</ref:2609.34069

Page 3</ref:2609.34069

Page 4</ref:2609.34069

Page 5</ref:2609.34069

Page 7</ref:2609.

Improvements for AI systems

The CDES mechanism enforces restrictions through specific control logic: CDES control logic rejects conflicting choices, adds required components, and repairs choices while preserving compatible edits. This allows the system to learn from failures by converting them into enforceable restrictions that guide subsequent searches, rather than relying on potentially flawed prompt compliance alone.
The system can perform self-improvement in optimization without retraining the LLM: The search is self-improving because accumulated certificates change which future modifications are permitted, without retraining the LLM. This means the agent iteratively refines its search space based on previously validated errors, leading to increasingly accurate and efficient code generation.
The system can achieve higher reliability in complex scientific tasks: Certificate feedback increases the fraction of candidates passing required correctness checks from 55% to 90%. This quantifiable improvement demonstrates that enforcing constraints derived from past failures significantly enhances the quality and correctness of generated code, which is critical for porting sensitive scientific software.
The system can leverage complementary expert knowledge for superior performance: Combining Celeritas’s code for selecting particle energies with our code for calculating directions and writing outputs. This hybrid increases GPU throughput by 16.1% over Celeritas. This suggests the agent can identify and integrate optimizations from multiple sources, facilitating human–AI collaboration to surpass existing expert implementations.
The system can generate optimized code that reuses prior computational effort: In Compton, it keeps a value that Celeritas computes inside a helper (Figure 2). CDES-generated code reuses an earlier calculation. This capability enables the AI to implement sophisticated optimization patterns, such as reusing intermediate results and simplifying repeated work within the generated code.

Abstract

The upgrade and rewriting of large scientific codebases has traditionally been a major challenge. While evolutionary search with large language models (LLMs) can port and accelerate legacy code, repair feedback in prompts alone does not prevent subsequent candidates from repeating the same errors. We introduce Certificate-Driven Evolutionary Search (CDES), which extends evolutionary search with enforceable restrictions derived from failed candidates, recorded as certificates of assumptions, checker evidence, and justified restrictions. Its control logic enforces these restrictions through rejection, backtracking, and targeted repair while preserving compatible edits. We apply CDES to CPU-to-GPU translation of two particle-simulation functions from the Geant4 toolkit, evaluated with a harness that goes beyond unit tests to combine formal checks, numerical comparisons, physics checks, and GPU safety tests. Generated implementations achieve 13.78x and 23.54x function-level speedups over CPU code, including data conversion and transfers; for one function, GPU throughput exceeds an expert implementation by 14.9%, reaching 16.1% when complementary components are combined. In an ablation over execution settings, certificate feedback increases the fraction of candidates passing required correctness checks from 55% to 90%.