一句话：面向无参考编译器的新生语言编译器，适配并工业评估 SynFuzz 与 MetaFuzz 两种测试技术。
现有编译器测试技术多面向成熟编译器，难以直接用于新生语言的新编译器：缺少参考编译器做差分测试、缺少程序分析工具支撑蜕变测试，且适配语言特性成本高。作者开展工业案例研究，分别适配出 SynFuzz（基于代码片段合成测试程序，以编译崩溃为预言）和 MetaFuzz（构造任意输入下等价的测试程序，减少对程序分析工具的依赖）。在某全球 IT 公司新生语言新编译器的两个版本上，与测试团队实践及 AFL 对比，SynFuzz 在相同时长内发现更多缺陷，两类技术均可与现有实践互补；共发现 11 个此前未知缺陷，均已被修复，并总结了经验与建议。
Due to the critical role of compilers, many compiler testing techniques have been proposed, two most notable categories among which are grammar-based and metamorphic-based techniques. All of them have been extensively studied for testing mature compilers. However, it is typical to develop a new compiler for a new-born programming language in practice. In this scenario, the existing techniques are hardly applicable due to some major reasons: (1) no reference compilers to support differential testing, (2) lack of program analysis tools to support most of metamorphic-based compiler testing, (3) substantial implementation effort incurred by different programming language features. Hence, it is unknown how the existing techniques perform in this new scenario. In this work, we conduct the first exploration (i.e., an industrial case study) to investigate the performance of the existing techniques in this new scenario with substantial adaptations. We adapted grammar-based compiler testing to this scenario by synthesizing new test programs based on code snippets and using compilation crash as test oracle due to the lack of reference compilers for differential testing. We also adapted metamorphic-based compiler testing to this scenario by constructing equivalent test programs under any inputs to relieve the dependence on program analysis tools. We call the adapted techniques SynFuzz and MetaFuzz, respectively. We evaluated both SynFuzz and MetaFuzz on two versions of a new compiler for a new-born programming language in a global IT company. By comparing with the testing practice adopted by the testing team and the general fuzzer (AFL), SynFuzz can detect more bugs during the same testing time, and both SynFuzz and MetaFuzz can complement the other two techniques. In particular, SynFuzz and MetaFuzz have detected 11 previously unknown bugs, all of which have been fixed by the developers. From the industrial case study, we summarized a series of lessons and suggestions for practical use and future research.
@inproceedings{ZhaoCFYW23,
title = {Testing the Compiler for a New-Born Programming Language: An Industrial Case Study (Experience Paper)},
author = {Yingquan Zhao and Junjie Chen and Ruifeng Fu and Haojie Ye and Zan Wang},
booktitle = {ISSTA},
year = {2023},
doi = {10.1145/3597926.3598077}
}