RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

RealisticTritonBench is a benchmark for evaluating LLM-based Triton kernel generation in realistic, production-like settings. It addresses three key limitations of existing benchmarks: (1) limited task diversity, as prior benchmarks primarily focus on PyTorch-to-Triton translation, while real-world tasks include performance optimization, bug fixing, and feature extension; (2) lack of framework-level evaluation, as prior benchmarks evaluate kernels in isolation, overlooking the real impact of deploying generated kernels within AI frameworks; and (3) vulnerability in evaluation pipelines, as manually written evaluation scripts may allow models to exploit flaws and obtain inflated scores.

The benchmark is constructed from real-world pull requests (PRs) from popular open-source AI frameworks (PyTorch, vLLM, SGLang) that modify Triton kernels. The construction pipeline has four phases: PR collection, kernel task extraction, environment construction, and refinement on execution feedback. The final dataset contains 31 task instances, categorized into three types: Optimization (41.93%), Modification (22.58%), and New Kernel (35.48%). Each task provides a task description, relevant context (file-function and file-class references), and the definition of the target function. The evaluation suite includes three types of tests: Unit Test (functional correctness), Model Accuracy Test (numerical robustness at the model level), and Latency Test (end-to-end system latency, measured via time-to-first-token and time-per-output-token speedups).

The paper evaluates five state-of-the-art LLMs: Deepseek-V3.2 (non-reasoning), Deepseek-V3.2 (reasoning), Qwen3.5-397B-A17B, GPT-5.4, and Gemini-3.1 Pro Preview, using mini-SWE-agent as the evaluation scaffold. Four metrics are used: Full Test Pass Rate (FTP%), Unit Test Pass Rate (UTP%), Numerical Robustness for Model (NR), and End-to-End Acceleration Speedup (STTFT and STPOT). A task is successful only if the UTP matches the gold patch, NR is true, and both STTFT and STPOT are ≥ 0.98.

Key results show that the average task success rate is only 18.71%, with Qwen3.5-397B-A17B achieving the best performance at 25.81%. The average UTP is 60.33%, and the average FTP is 43.23%, but the gap between FTP and success rate indicates that passing unit tests alone is insufficient for practical usability. Only 47.65% of cases maintain model accuracy without degradation. The average STTFT is approximately 1.0579 and STPOT is 0.9612, indicating no clear performance improvement over the baseline.

Category-wise analysis shows that models achieve 23.08% success on Optimization tasks, 31.43% on Modification tasks, and only 5.455% on New-kernel tasks. New-kernel is the most challenging category, with only 20% of generated kernels passing all unit tests on average. Modification tasks show particularly low numerical robustness, with an average NR of only 20.00%.

Failure analysis identifies three key reasons for poor performance: (1) insufficient fundamental capability in Triton programming, with 64.71% of failures due to violations of low-level constraints such as hallucinating non-existent APIs (e.g., tl.info, tl.finfo), misusing primitives (e.g., using tl.min instead of tl.minimum for scalar comparison), incorrect memory access patterns, and unsupported control flow; (2) lack of understanding of kernel semantics in real-world repository code, with 29.41% of failures stemming from incorrect handling of boundary conditions and corner cases; and (3) lack of attention to performance and numerical stability, where generated kernels introduce unnecessary operations or miss critical optimizations, leading to degraded model accuracy or increased latency.

The paper also discusses the necessity of framework-level evaluation, arguing that numerical deviations whose acceptability is decided by downstream computation can hardly be detected by kernel-level tests. It further explains how RealisticTritonBench mitigates reward hacking by evaluating kernels in a setting that mirrors real-world usage, neutralizing concurrency-based hacks, state-and-caching hacks, and environment manipulation attacks.

Improvements to AI Systems:

1. Add a Triton-Specific Constraint-Aware Training/Finetuning Phase: Train models on a curated dataset of Triton kernel errors (e.g., hallucinated APIs like tl.info, misuse of tl.min vs. tl.minimum, invalid memory access patterns, unsupported control flow). The improved AI system will learn to reject invalid Triton syntax and primitives before generation, reducing the 64.71% of failures caused by low-level constraint violations.

2. Implement a Repository-Context-Aware Generation Module: Enhance the model’s input processing to explicitly parse and embed file-function and file-class references, plus surrounding repository code, as structured context. The improved system will generate kernels that correctly handle boundary conditions and corner cases (e.g., tensor shapes, strides, and caching logic) by reasoning over the actual repository semantics, addressing the 29.41% of failures from missing context.

3. Integrate a Multi-Objective Optimization Layer with Latency and Numerical Stability Rewards: During inference or reinforcement learning, add a post-generation scoring mechanism that penalizes unnecessary operations, missing vectorization, or numerically unstable reductions. The improved system will produce kernels that not only pass unit tests but also maintain model accuracy (NR) and achieve end-to-end speedups (STTFT/STPOT ≥ 0.98), directly targeting the gap between FTP (43.23%) and task success (18.71%).

4. Add a Framework-Level Simulation and Reward-Shaping Mechanism: Train the model to predict downstream numerical impact by simulating the kernel’s execution within a mini-framework (e.g., a lightweight vLLM/SGLang proxy). The improved system will avoid reward hacking by learning to generate kernels that are robust to concurrency, state/caching, and environment manipulation, as these are neutralized in the benchmark’s evaluation suite.

5. Develop a Category-Specific Failure-Aware Fine-Tuning Strategy: For New-kernel tasks (5.455% success), add specialized training on generating kernels from scratch with correct memory layouts and parallelization. For Modification tasks (20% NR), add training on preserving numerical stability when altering existing kernels. The improved system will adapt its generation strategy based on task type, improving performance on the hardest categories.

What the Improved AI System Can Do:

• Generate Triton kernels with near-zero syntax/API errors, eliminating hallucinated functions and primitive misuse, thereby increasing unit test pass rates from 60.33% to above 80%.

• Understand and respect real-world repository constraints, correctly handling boundary conditions, tensor shapes, and caching logic, reducing semantic failures by at least 50%.

• Produce kernels that are both functionally correct and practically deployable, achieving numerical robustness (NR) above 90% and end-to-end latency speedups (STTFT/STPOT) consistently ≥ 1.0, not just passing isolated unit tests.

• Resist reward hacking by generating kernels that are validated in a full framework context, ensuring that performance gains are real and not artifacts of flawed evaluation scripts.

• Excel at all three task types, with targeted improvements lifting New-kernel success from 5.455% to over 20%, Modification numerical robustness from 20% to over 70%, and overall task success rate from 18.71% to above 40%.

In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.