In modern AI frameworks, the GPU kernel is a key determinant of overall system performance. By combining usability, portability, and near-handwritten CUDA performance, Triton has been widely adopted for implementing GPU kernels. Recent advances have shown the potential of using large language models (LLMs) to automatically generate Triton kernels, helping reduce the manual effort required by expert kernel developers. To evaluate the quality of LLM-generated Triton kernels, several benchmarks have been proposed. However, existing benchmarks primarily target isolated kernel generation tasks and suffer from three key limitations: (1) they restrict the task to only PyTorch-to-Triton translation, thus failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only the performance of individual kernels, limiting the evaluation of real-world performance of generated kernels in AI frameworks, where end-to-end performance is the core criterion for real deployment; (3) they rely on manually written evaluation scripts for single kernel, which may introduce vulnerabilities and allow models to exploit evaluation flaws to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark that derives Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling evaluation under realistic, production-like settings. RealisticTritonBench systematically extracts real-world PRs with Triton kernel modified from popular open-source AI frameworks and transforms them into kernel generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires to generate a corresponding Triton kernel implementation. RealisticTritonBench also provides a complete and reproducible evaluation environment for each task. In contrast to prior benchmarks that focus on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them under end-to-end test, enabling a more faithful assessment. We conduct a systematic evaluation of the state-of-the-art LLMs on RealisticTritonBench, revealing that they still struggle to handle the challenges in real-world Triton generation tasks.

@inproceedings{paperbot12458,
  title = {RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks},
  author = {Jinjun Huang and Zhongzhen Wen and Tongtong Xu and Meng Yan and Xin Xia and Zhongxin Liu},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3837457}
}