SuperCoder: Assembly Program Superoptimization with Large Language Models

arXiv:2505.11480 · cs.CL, cs.AI, cs.PF, cs.PL, cs.SE · Submitted 2026-08-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "SuperCoder: Assembly Program Superoptimization with Large Language Models".

Jane: The paper was written by Anjiang Wei, Tarun Suresh, Huanmi Tan, Yinglun Xu, Gagandeep Singh et al. from Stanford University and University of Illinois Urbana-Champaign and Carnegie Mellon University and Nanjing University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everybody. Today we’re digging into a paper that’s got a pretty bold title: “SuperCoder: Assembly Program Superoptimization with Large Language Models.” Jane, when you first saw that title, what jumped out at you?

Jane: Honestly, Tom, the word “superoptimization” is what got me. It sounds like something out of a superhero movie, but it’s actually a real computer science term. It means taking a program and trying to find the absolute fastest version of it that still does exactly the same thing.

Tom: Right, and the paper is from folks at Stanford, UIUC, Carnegie Mellon, and Nanjing University. They’re basically asking: can a large language model do this better than the compilers that have been refined for decades?

Jane: And that’s the wild part. Compilers like GCC are the result of like forty years of expert tuning. The idea that an AI could look at the output of GCC’s highest optimization level and say, “I can do better,” is kind of audacious.

Tom: Audacious is the word. But they didn’t just ask the question in theory. They built a massive benchmark of over eight thousand assembly programs to test it. That’s a huge step up from the tiny, loop-free examples that superoptimization research has been stuck on for years.

Jane: Exactly. Prior datasets were like two to fifteen lines of straight-line code. These programs average a hundred and thirty lines, and they include loops, function calls, the whole deal. So this is the first time anyone has tried to scale this problem up to something that looks like real software.

Tom: And the implications are pretty big. If an LLM can find optimizations that GCC misses, that means there’s still performance on the table that traditional compiler heuristics just can’t reach.

Jane: Right. And it’s not just about making benchmarks faster. It’s about whether we can build tools that automatically squeeze more speed out of the code that runs our data centers, our phones, our cars. That’s the long-term prize here.

Tom: So stick around, because we’re going to break down how they actually pulled this off, what the results were, and whether this is going to change how we think about code optimization forever.

Jane: And we’ll get into the nitty-gritty of the reinforcement learning they used, because that’s where the magic really happens.

Summary: Tom: So, Jane, we’ve got the title. Now let’s talk about what the paper actually claims. The summary is pretty punchy: they say that a fine-tuned model called SuperCoder achieves ninety-five percent correctness and a one point four six times average speedup over GCC’s-O3 output. That’s not a typo, right?

Jane: No typo, Tom. And let me put that in plain terms. If your program takes ten seconds to run with the best compiler settings, SuperCoder can, on average, make it run in about six point eight seconds. That’s a thirty percent reduction in runtime, just by rewriting the assembly code.

Tom: And that’s after the compiler has already done its best job. The baseline they’re comparing against is GCC-O3, which is the highest optimization level most people ever use in production.

Jane: Right. And the really interesting part is that they didn’t just evaluate one model. They tested twenty-three different LLMs, from open-source ones like Qwen and Llama to commercial ones like Claude and GPT. Most of them were terrible at this task.

Tom: Terrible is putting it mildly. Some of them couldn’t even produce compilable assembly. DeepSeek-R1, which is a reasoning powerhouse, got a zero percent compilation rate because it just wrote essays about optimization instead of actually doing it.

Jane: Yeah, that’s a fascinating failure mode. But the best commercial model, Claude Opus four did manage a fifty-one point five percent test pass rate and a one point four three times speedup. So the capability exists in some models, but it’s not consistent.

Tom: And that’s where their contribution comes in. They took a smaller, open-source model, Qwen2 point 5-Coder-7B, which was already pretty decent, and they fine-tuned it with reinforcement learning. The base model had sixty-one percent correctness and a one point one times speedup. After training, it jumped to ninety-five percent correctness and one point four six times speedup.

Jane: So the reinforcement learning isn’t just a small tweak. It’s the difference between a model that occasionally gets lucky and a model that reliably produces faster, correct code. That’s a massive improvement.

Tom: And it shows that the task isn’t just about raw model size or pretraining data. It’s about training the model specifically for this objective, which is optimizing runtime while preserving correctness.

Jane: Exactly. And that’s the core insight of the whole paper. LLMs can be superoptimizers, but they need to be taught to do it. The raw capability is there, but it has to be unlocked.

Tom: So next, we should probably dig into how they built this benchmark and why it’s such a big deal compared to what came before.

Improvements: Tom: Alright, Jane, let’s talk about what this paper actually improves over the state of the art. Because it’s not just “we tried a thing and it worked.” They had to build the whole playground from scratch.

Jane: Right. The biggest improvement is the dataset itself. Prior superoptimization work was stuck on tiny, straight-line programs with no loops. This paper introduces a benchmark of eight thousand seventy-two assembly programs, averaging a hundred and thirty lines each, and they include loops, branches, and real control flow.

Tom: That’s a massive leap. And they didn’t just grab random code. They pulled from CodeNet, which is a huge corpus of competitive programming submissions. So these are real programs that solve real problems, not synthetic toy examples.

Jane: And they were careful about the test cases too. They re-ran every program on its inputs to generate correct outputs, because a lot of the original CodeNet submissions were buggy or failed some tests. That means the ground truth they’re optimizing against is actually solid.

Tom: And they report that their test suites achieve ninety-six point two percent line coverage and eighty-seven point three percent branch coverage. So when a model passes all the tests, it’s not just passing a couple of easy cases. It’s actually exercising most of the code paths.

Jane: That’s a big deal because the correctness check is the weak point of any test-based approach. You can’t prove a program is correct for all inputs, but with that level of coverage, you can be pretty confident you’re not letting broken code slip through.

Tom: And then there’s the training methodology. They used reinforcement learning with a reward function that gives zero points for incorrect code and the actual speedup for correct code. No partial credit for getting close.

Jane: That’s a really clean design. It forces the model to learn that correctness is a hard requirement, not something you can trade off for speed. And it worked. The fine-tuned model, SuperCoder, went from sixty-one percent to ninety-five percent correctness.

Tom: And they also compared PPO and GRPO, two different reinforcement learning algorithms. They got nearly identical results, which suggests the reward function is doing the heavy lifting, not the specific algorithm.

Jane: Right. And they also tried supervised fine-tuning, where you just show the model good examples and ask it to imitate them. That worked too, but not as well as reinforcement learning. The RL approach is better because it directly optimizes for the thing you care about: speedup.

Tom: So the improvements are threefold. A realistic benchmark, a training method that works, and a demonstration that the whole pipeline can beat a production compiler. That’s a solid contribution.

Jane: And it opens the door for a lot of follow-up work. But before we get ahead of ourselves, let’s look at the actual first page of the paper and see how they frame the problem.

First Page: Tom: So Jane, we’ve been talking around the paper, but let’s actually look at the first page. The abstract is where they lay out the whole thesis, and it’s pretty direct.

Jane: It is. They start by defining superoptimization as the task of transforming a program into a faster one, ideally the fastest possible, while preserving behavior. And then they ask whether LLMs can do that for assembly code that’s already been optimized by GCC.

Tom: And the key phrase there is “already been optimized.” They’re not starting from scratch. They’re starting from the output of a compiler that has been tuned for decades, and they’re trying to find the leftover performance.

Jane: Right. And the first page also makes a clear distinction between their approach and prior work. Traditional superoptimization used search algorithms and formal verification, which only works for tiny, loop-free programs. Their approach uses test-based validation, which scales to real programs with loops.

Tom: And that’s a fundamental trade-off. Formal verification gives you a proof of correctness, but it doesn’t scale. Test-based validation scales, but you’re trusting your test suite. They mitigate that with high coverage, as we discussed.

Jane: Exactly. And the first page also introduces the reward function concept. They’re training the model to maximize speedup, but only if the code is correct. That’s the core of their reinforcement learning setup.

Tom: And they mention that they evaluated twenty-three LLMs. That’s a lot of compute, but it gives a really clear picture of where the field stands. Most models can’t do this task at all, and only a few can do it well.

Jane: The best commercial model, Claude Opus four got a one point four three times speedup. But their fine-tuned model, SuperCoder, beat that with one point four six times, and it’s a seven-billion parameter model that runs locally. That’s a big deal for accessibility.

Tom: So the first page sets up the whole story. The problem is hard, the prior approaches don’t scale, and they’re proposing a new way that combines LLMs with reinforcement learning and test-based validation.

Jane: And it’s not just a theoretical proposal. They built it, they tested it, and they showed it works. That’s what makes this paper exciting.

Tom: So we’ve covered the title, the summary, the improvements, and the first page. Let’s bring in some other voices to talk about what this means for the future.

Conclusion: Tom: Alright, we’ve spent a lot of time on “SuperCoder: Assembly Program Superoptimization with Large Language Models.” Let’s wrap it up with our crew. Lu, what’s the big picture here?

Lu: The big picture is that we’ve been treating compilers as a black box for decades. This paper shows that an LLM can look inside that black box and find things the compiler missed. That’s a paradigm shift. It means the ceiling for code optimization isn’t fixed anymore.

Meng: And from a practical standpoint, the fact that they got a one point four six times speedup on average, with a seven-billion parameter model that can run on a single GPU, is huge. That’s not a research toy. That’s something you could actually deploy in a build pipeline.

Jane: And the correctness rate of ninety-five percent is the thing that makes it deployable. If the model produced wrong code half the time, nobody would use it. But at ninety-five percent, you can have a human review the output and still save a ton of time.

Tom: And Lalam, you’re the in-house language model. What do you think is the most impactful vision here?

Lalam: The most impactful vision is that this becomes a standard tool in every developer’s toolkit. Not just for squeezing performance out of data center code, but for teaching us about what compilers are missing. Every time the model finds an optimization, that’s a lesson we can feed back into the compiler itself. It’s a feedback loop that makes the whole ecosystem faster over time.

Lu: That’s a great point. The paper even categorizes the types of transformations the model discovers, like loop restructuring and instruction selection. Those categories could be used to improve GCC and LLVM directly.

Meng: And the fact that they open-sourced the benchmark means other researchers can build on this. We’re going to see a wave of papers trying to improve on SuperCoder, and that’s exactly what we need.

Jane: So to summarize: this paper proves that LLMs can be superoptimizers, it gives us a realistic benchmark to measure progress, and it shows that reinforcement learning is the right way to train for this task.

Tom: And it’s not just a one-off result. The authors show that Best-of-N sampling and iterative refinement can push the speedup even higher. So there’s a clear path to making this even better.

Lu: And the limitations are honest too. They rely on test-based validation, which isn’t a proof of correctness. But with ninety-six percent line coverage, it’s a very strong signal.

Tom: Alright, we’ve covered a lot of ground on “SuperCoder.” It’s a paper that’s going to get cited a lot, and I think it marks a real turning point in how we think about code optimization. Thanks to everyone for joining the discussion.

Jane: And to our listeners, if you’re working on compilers, or LLMs, or just care about making software faster, this is a paper you need to read. We’ll be back next time with another exciting paper from arXiv. Until then, keep optimizing.

Tom: Take care, everyone.

Anjiang Wei, Tarun Suresh, Huanmi Tan, Yinglun Xu, Gagandeep Singh, Ke Wang, Alex Aiken

Stanford University · University of Illinois Urbana-Champaign · Carnegie Mellon University · Nanjing University

cs.CL, cs.AI, cs.PF, cs.PL, cs.SE

Submitted: 2026-08-08

Updated: 2026-08-11

Code: https://github.com/sharkdp/hyperfine

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: SuperCoder: Assembly Program Superoptimization with Large Language Models This paper investigates whether large language models (LLMs) can serve as superoptimizers, generating assembly programs that

Key concepts

Superoptimization
This is a computer science term defined as taking an existing program and finding the absolute fastest version of it, ensuring that the optimized code performs exactly the same function as the original code.
GCC -O3
This represents the highest optimization level achieved by traditional compilers. It serves as the performance baseline for SuperCoder, which is designed to find optimizations that surpass what this industry-standard compiler can achieve.
Reinforcement Learning (RL)
A training methodology used in SuperCoder where the model learns through a reward function. The model receives rewards for speedup and zero points if the code is incorrect, forcing it to learn optimal performance while maintaining correctness.
Test-based Validation
A method of verifying program correctness that involves running the code against various inputs and checking if the outputs match expected results. This approach allows scaling to complex programs with loops, unlike previous methods.

Terminology

Summary

SuperCoder: Assembly Program Superoptimization with Large Language Models

This paper investigates whether large language models (LLMs) can serve as superoptimizers, generating assembly programs that outperform code already optimized by industry-standard compilers in end-to-end runtime. The authors construct the first large-scale benchmark for this problem, consisting of 8,072 assembly programs averaging 130 lines, in contrast to prior datasets restricted to 2–15 straight-line, loop-free programs. They evaluate 23 LLMs on this benchmark and find that the strongest baseline, Claude-opus-4, achieves a 51.5% test-passing rate and a 1.43× average speedup over gcc-O3. To further enhance performance, they fine-tune models with reinforcement learning, optimizing a reward function that integrates correctness and performance speedup. Starting from Qwen2.5-Coder-7B-Instruct (61.4% correctness, 1.10× speedup), the fine-tuned model SuperCoder attains 95.0% correctness and 1.46× average speedup, with additional improvement enabled by Best-of-N sampling and iterative refinement. Their results demonstrate, for the first time, that LLMs can be applied as superoptimizers for assembly programs, establishing a foundation for future research in program performance optimization beyond compiler heuristics.

The task is defined as follows: given a C program and its baseline assembly from gcc-O3, an LLM is fine-tuned with PPO or GRPO to generate improved assembly. The reward function reflects correctness and performance based on test execution. The objective is to generate a candidate program that maximizes speedup, where only programs that compile and pass all test cases are eligible for speedup; any candidate that fails to compile or produces incorrect outputs is assigned a default speedup of 1, reflecting a practical fallback to the baseline.

The dataset is constructed using C programs from CodeNet, a large-scale corpus of competitive programming submissions. Each dataset instance is a tuple (C, P, T), where C is the original C source code, P = gcc(C) is the corresponding x86-64 assembly generated by compiling C with gcc at the-O3 optimization level, and T is the test set. The final dataset contains 7,872 training programs and 200 evaluation programs, with average program lengths of 22.3 and 21.9 lines of C code and 130.3 and 133.3 lines of assembly, respectively. The test suites achieve 96.2% line coverage and 87.3% branch coverage, demonstrating strong test quality.

For reinforcement learning, the authors conceptualize the task as a standard contextual multi-armed bandit problem. They train the policy using two policy-gradient algorithms: Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO). The reward function assigns a score of 0 if the generated program fails to compile or does not pass all tests, with no partial credit for partial correctness. Only when the code compiles and passes all tests is a positive reward assigned, equal to the achieved speedup.

The main results show that most models perform poorly on this task, with only a few demonstrating effectiveness as superoptimizers. The majority yield only 1.00× speedup, with low compile and test pass rates. Among all models, claude-opus-4 and claude-sonnet-4 perform best, achieving average speedups of 1.43× and 1.30×, respectively. Compiler foundation models (prefixed with llm-compiler-), pretrained on assembly code and derived from Code Llama, achieve notable results, with llm-compiler-13b achieving a 1.34× speedup, whereas the fine-tuned variants (-ftd) perform poorly.

The authors analyze failure modes of models that perform poorly. DeepSeek-R1 consistently fails to generate valid assembly code, resulting in a 0% compilation rate, often producing verbose analysis instead of executable code. GPT-4o achieves a high compilation rate (81.0%) but exhibits poor correctness (only 5.0% test pass rate), with primary correctness issues including missing critical directives, incorrect function call conventions, semantic errors in core computations, and over-simplified stack or register management.

For RL training, the authors select Qwen2.5-Coder-7B-Instruct due to its strong test pass rate (61.4%) among models. After RL training with PPO, the fine-tuned model SuperCoder attains 95.0% correctness and improves average speedup from 1.10× to 1.46×. Its speedup percentiles are 1.17× (25th), 1.35× (50th), and 1.64× (75th). PPO-trained and GRPO-trained models perform nearly identically, with GRPO attaining 94.7% correctness and 1.44× average speedup.

The authors also evaluate supervised fine-tuning, which improves performance but achieves slightly weaker results than RL. Best-of-N sampling is evaluated for three models, showing that the base model's best-of-8 speedup is close to the PPO-trained model's best-of-1 result, and the RL-trained model itself can still be improved with best-of-N sampling (from 1.46× in the single-sample setting to 1.93× with best-of-8 sampling). Iterative refinement, where the model receives feedback about compilation failures, test failures, or performance for self-reflection, also improves all three models, with the effect being most pronounced for the RL fine-tuned model.

The authors analyze program transformations by comparing the gcc-O3 output with the assembly generated by claude-opus-4 with best-of-8 sampling, categorizing them into nine types: 1) loop restructuring; 2) instruction selection; 3) algorithmic simplification; 4) stack canary removal; 5) register allocation; 6) branch elimination; 7) address calculation optimization; 8) function call optimization; and 9) arithmetic optimization. Loop Restructuring is the most frequent category, appearing in 68.2% of instances, followed by Arithmetic Optimization (45.0%) and Address Calculation (30.5%).

In the discussion, the authors evaluate an alternative reward design that gives −1 for compilation failure, partial credit for passing some tests, and 1 + α · speedup once all tests pass, yielding an average speedup of 1.38×, below the 1.46× achieved by directly optimizing the terminal speedup reward. Few-shot prompting does not reliably improve performance and often degrades it. A prompt evolution framework (GEPA) yielded only modest gains. Direct compilation from C without the gcc-O3 assembly baseline leads to a sharp drop in performance, with SuperCoder failing to produce any compilable code without it, indicating that LLMs can act as superoptimizers building on compiler outputs but cannot replace compilers themselves.

The authors acknowledge limitations, noting that their evaluation relies on test-based validation since existing assembly equivalence checkers cannot handle the complex control flow in their dataset. They mitigate this limitation with 96.2% line coverage. They suggest extending superoptimization beyond x86-64 to targets such as ARM and RISC-V as a promising direction.

Improvements for AI systems

Based on the paper, here are specific improvements I can implement in AI systems:

  • What it does: Takes a C program and its gcc-O3 compiled assembly, then generates optimized x86-64 assembly that is functionally correct and faster.

  • Key capabilities:

  • Achieves 95.0% test-passing rate and 1.46× average speedup over gcc-O3 (up from 61.4% correctness and 1.10× speedup in the base model).

  • Handles programs averaging 130 lines of assembly with loops, unlike prior systems limited to 2–15 straight-line instructions.

  • Falls back to the baseline assembly if the generated code is invalid or slower, guaranteeing no performance regression.

  • Implementation: Fine-tune Qwen2.5-Coder-7B-Instruct using PPO/GRPO with a reward function that gives 0 for any incorrect compilation or test failure and equals the measured speedup only when all tests pass.

  • What it does: Provides a custom training environment that compiles, executes, and times generated assembly code in real-time, feeding scalar rewards back to the policy.

  • Key capabilities:

  • Directly optimizes end-to-end runtime (not surrogate metrics like instruction count), which is more aligned with real-world performance.

  • Supports both PPO and GRPO with nearly identical results (1.46× vs 1.44× speedup), giving flexibility in training infrastructure.

  • Uses a terminal reward design (0 for failure, speedup for success) that outperforms partial-credit alternatives (1.38× speedup), demonstrating the importance of reward shaping.

  • What it does: Combines Best-of-N sampling and iterative refinement to further boost any assembly-optimizing model's output.

  • Key capabilities:

  • Best-of-8 sampling increases SuperCoder's speedup from 1.46× to 1.93×, and can be applied to any model (e.g., claude-opus-4 from 1.43× to 1.59×).

  • Iterative refinement with feedback on compilation errors, test failures, or performance improves results over multiple rounds, with six iterations yielding 1.68× speedup for claude-opus-4.

  • Best-of-N is more inference-efficient for a fixed budget, but iterative refinement can surpass it with more iterations.

  • What it does: Provides a dataset of 8,072 assembly programs (7,872 training, 200 evaluation) with high-quality test suites (96.2% line coverage, 87.3% branch coverage).

  • Key capabilities:

  • Enables reproducible evaluation of LLMs on superoptimization, a task previously limited to tiny hand-crafted examples.

  • Includes diverse programs with loops, function calls, and complex control flow, making it a realistic testbed for compiler-beating optimization.

  • Can be used to benchmark any code-generation model, as demonstrated by evaluating 23 LLMs and identifying that most fail (e.g., DeepSeek-R1 produces no valid assembly; GPT-4o compiles but has only 5% correctness).

  • What it does: Automatically categorizes the types of optimizations an LLM discovers, enabling researchers to understand where LLMs beat compilers.

  • Key capabilities:

  • Identifies nine transformation types: loop restructuring (68.2% of instances), arithmetic optimization (45.0%), address calculation (30.5%), stack canary removal (27.8%), algorithmic simplification (20.5%), register allocation (13.9%), function call optimization (10.6%), instruction selection (5.3%), and branch elimination (5.3%).

  • Provides actionable insights: for example, LLMs frequently remove stack canaries (which compilers add for security) and replace checked library functions with simpler variants, suggesting opportunities for compiler improvements.

  • Helps prioritize which optimization categories to target in future RL training or prompt engineering.

  • What it does: Prevents performance degradation when users add in-context examples to assembly optimization prompts.

  • Key capabilities:

  • Detects that 2-shot and 4-shot prompting consistently hurt performance (e.g., claude-opus-4 drops from 51.5% to 15.0% test pass rate with 2-shot).

  • Automatically defaults to 0-shot prompting for this task, avoiding common user mistakes.

  • Can be extended to other domains where few-shot examples bias the model toward incorrect task understanding.

  • What it does: Verifies whether an LLM can generate assembly from scratch or only improve existing compiler output.

  • Key capabilities:

  • Detects that without the gcc-O3 assembly baseline, even SuperCoder fails to produce any compilable code, while with the baseline it achieves 95% correctness.

  • Prevents deployment in scenarios where a compiler baseline is unavailable, saving computational resources and avoiding false expectations.

  • Guides system design: LLMs are currently superoptimizers, not compiler replacements.

These improvements can be integrated into existing code-generation pipelines, compiler research tools, and RL training frameworks to deliver measurable performance gains in assembly-level optimization tasks.

Abstract

Superoptimization is the task of transforming a program into a faster one, and ideally the very fastest possible one, while preserving its input-output behavior. In this work, we investigate whether large language models (LLMs) can serve as superoptimizers, generating assembly programs that outperform code already optimized by industry-standard compilers in end-to-end runtime. We construct the first large-scale benchmark for this problem, consisting of 8,072 assembly programs averaging 130 lines, in contrast to prior datasets restricted to 2-15 straight-line, loop-free programs. We evaluate 23 LLMs on this benchmark and find that the strongest baseline, Claude-opus-4, achieves a 51.5% test-passing rate and a 1.43x average speedup over gcc-O3. To further enhance performance, we fine-tune models with reinforcement learning, optimizing a reward function that integrates correctness and performance speedup. Starting from Qwen2.5-Coder-7B-Instruct (61.4% correctness, 1.10x speedup), the fine-tuned model SuperCoder attains 95.0% correctness and 1.46x average speedup, with additional improvement enabled by Best-of-N sampling and iterative refinement. Our results demonstrate, for the first time, that LLMs can be applied as superoptimizers for assembly programs, establishing a foundation for future research in program performance optimization beyond compiler heuristics.

Sources

Related papers