一句话：提出 N+1 版本差分测试，通过自动合成并注入断言的程序检测和定位 JavaScript 引擎与 ECMAScript 规范缺陷。
现代语言采用 CI/CD 方式开发，规范与实现共同演化，传统差分测试中规范作为预言机的假设不再成立，二者都可能存在缺陷。本文提出 N+1 版本差分测试：先依据语言规范中的语法和语义自动合成程序，再向合成程序注入断言以检查最终程序状态并生成一致性测试，随后在多个实现上执行以检测引擎与规范中的错误，并利用统计信息定位规范缺陷。作者实现 JEST，针对 JavaScript 引擎与 ECMAScript 规范进行测试。在支持 ES11 的四个引擎上，JEST 合成 1700 个程序，覆盖 ES11 的 97.78% 语法和 87.70% 语义，检测出 44 个引擎错误和 27 个规范错误。
Modern programming follows the continuous integration (CI) and continuous deployment (CD) approach rather than the traditional waterfall model. Even the development of modern programming languages uses the CI/CD approach to swiftly provide new language features and to adapt to new development environments. Unlike in the conventional approach, in the modern CI/CD approach, a language specification is no more the oracle of the language semantics because both the specification and its implementations (interpreters or compilers) can co-evolve. In this setting, both the specification and implementations may have bugs, and guaranteeing their correctness is non-trivial. In this paper, we propose a novel N+1-version differential testing to resolve the problem. Unlike the traditional differential testing, our approach consists of three steps: 1) to automatically synthesize programs guided by the syntax and semantics from a given language specification, 2) to generate conformance tests by injecting assertions to the synthesized programs to check their final program states, 3) to detect bugs in the specification and implementations via executing the conformance tests on multiple implementations, and 4) to localize bugs on the specification using statistical information. We actualize our approach for the JavaScript programming language via JEST, which performs N+1-version differential testing for modern JavaScript engines and ECMAScript, the language specification describing the syntax and semantics of JavaScript in a natural language. We evaluated JEST with four JavaScript engines that support all modern JavaScript language features and the latest version of ECMAScript (ES11, 2020). JEST automatically synthesized 1,700 programs that covered 97.78% of syntax and 87.70% of semantics from ES11. Using the assertion-injected JavaScript programs, it detected 44 engine bugs in four different engines and 27 specification bugs in ES11.
@inproceedings{paperbot10890,
title = {JEST: N+1-Version Differential Testing of Both JavaScript Engines and Specification},
author = {Jihyeok Park and Seungmin An and Dongjun Youn and Gyeongwon Kim and Sukyoung Ryu},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00015}
}