一句话:提出两阶段 LLM 编译器测试框架 LegoFuzz:离线生成小型代码片段,在线组合复用,在 GCC/LLVM 发现 66 个 bug。
用大语言模型测试编译器已有前景,但现有方法存在两个关键问题:生成的测试程序往往过于简单,且反复调用 LLM 进行测试的计算开销很高。本文提出一种新的编译器测试框架,将测试过程解耦为离线与在线两个阶段:离线阶段用 LLM 生成一批规模小但特性丰富的代码片段;在线阶段则以策略性方式组合复用这些片段,构造出高质量且合法的测试程序,再用于测试编译器。作者实现了面向 C 编译器的工具 LegoFuzz,并在最广泛使用的 C 编译器 GCC 与 LLVM 上评估。结果显示,该工具共发现 66 个 bug,其中近一半是误编译(miscompilation)bug——这类 bug 严重且难以发现,现有基于 LLM 的工具均未能找到。作者认为这一高效设计为 AI 模型在 C 编译器之外的软件测试中的应用开辟了新可能。
Testing compilers with AI models, especially large language models (LLMs), has shown great promise. However, current approaches struggle with two key problems: The generated programs for testing compilers are often too simple, and extensive testing with the LLMs is computationally expensive. In this paper, we propose a novel compiler testing framework that decouples the testing process into two distinct phases: an offline phase and an online phase. In the offline phase, we use LLMs to generate a collection of small but feature-rich code pieces. In the online phase, we reuse these code pieces by strategically combining them to build high-quality and valid test programs, which are then used to test compilers. We implement this idea in a tool, LegoFuzz , for testing C compilers. The results are striking: we found 66 bugs in GCC and LLVM, the most widely used C compilers. Almost half of the bugs are miscompilation bugs, which are serious and hard-to-find bugs that none of the existing LLM-based tools could find. We believe this efficient design opens up new possibilities for using AI models in software testing beyond just C compilers.