一句话：提出特征敏感与特征调用路径敏感覆盖准则，提升语言实现一致性测试生成质量并发现大量 JS 实现缺陷。
语言实现一致性测试对保证正确执行环境很重要，但人工维护测试成本高。传统图覆盖从机械化语言规范生成测试时，无法区分使用相同语义函数的语言特征及其内部部分，导致测试质量下降。本文提出特征敏感（FS）覆盖及变体特征调用路径敏感（FCPS）覆盖，并用 k-limiting 扩展，细化测试需求。作者将其集成到基于覆盖率引导变异模糊测试的 JEST，针对 JavaScript 规范生成测试。对 ES13（2022），50 小时内合成 237,981 个一致性测试；评测 8 个主流 JavaScript 实现（4 引擎、4 转译器），发现 143 个不同缺陷，85 个获确认，83 个为新缺陷。
The conformance testing of programming language implementations is crucial to support correct and consistent execution environments. Because manually maintaining conformance tests for real-world programming languages is cumbersome and labor-intensive, researchers have presented various ways to make conformance tests effective and efficient. One such approach is to use graph coverage, one of the most widely-used coverage criteria, to generate tests that reach different parts of a mechanized language specification. Since mechanized specifications use functions or inductive definitions to describe the semantics of language features, traditional graph coverage criteria for software work as they are. However, they may not produce high-quality conformance tests because language implementations often have specialized execution paths for different features, even when their semantics descriptions use the same functions. Traditional graph coverage may not distinguish test requirements of such language features, which degrades the quality of conformance testing. Similarly, it may not distinguish test requirements of different parts of the same language feature when their semantics descriptions use the same functions. We present feature-sensitive (FS) coverage as a novel coverage criterion to generate high-quality conformance tests for language implementations. It is a general extension of graph coverage, refining conventional test requirements using the innermost enclosing language features. We also introduce feature-call-path-sensitive (FCPS) coverage, a variant of FS coverage, and extend both coverage criteria using the 𝑘-limiting approach. To evaluate the effectiveness of the new coverage criteria for language implementations, we apply them to a mechanized specification of JavaScript. We extend JEST, the state-of-the-art JavaScript conformance test synthesizer using coverage-guided mutational fuzzing, with various FS and FCPS coverage criteria. For the latest JavaScript language specification (ES13, 2022), our tool automatically synthesizes 237,981 conformance tests in 50 hours with five coverage criteria. We evaluated the conformance of eight mainstream JavaScript implementations (four engines and four transpilers) with the synthesized conformance tests and discovered bugs in all of them. The tool detected 143 distinct conformance bugs (42 in engines and 101 in transpilers), 85 of which were confirmed by the developers and 83 of which were newly discovered bugs.
@article{paperbot1984,
title = {Feature-Sensitive Coverage for Conformance Testing of Programming Language Implementations},
author = {Jihyeok Park and Dongjun Youn and Kanguk Lee and Sukyoung Ryu},
journal = {Proceedings of the ACM on Programming Languages},
volume = {7},
number = {PLDI},
year = {2023},
doi = {10.1145/3591240}
}