一句话:提出 Swami,从结构化自然语言规范中提取测试预言并生成可执行测试,聚焦异常行为与边界条件。
软件规范常用自然语言描述期望行为,但这类规范难以被自动验证。本文提出 Swami,一种从结构化自然语言规范中提取测试预言(oracle)并生成可执行测试的自动化技术。Swami 重点关注异常行为与边界条件——这些情况常导致现场故障,却往往被开发者忽略而未手动编写测试。作者在官方 JavaScript 规范 ECMA-262 上评估 Swami,生成的测试有 98.4% 与规范精确一致。用 Swami 增强开发者编写的测试套件后,覆盖率得到提升,并在 Rhino 中发现 1 个此前未知缺陷和 15 个缺失的 JavaScript 特性,在 Node.js 中发现 1 个此前未知缺陷,还在 ECMA-262 规范中发现 18 处语义歧义。
Software specifications often use natural language to describe the desired behavior, but such specifications are difficult to verify automatically. We present Swami, an automated technique that extracts test oracles and generates executable tests from structured natural language specifications. Swami focuses on exceptional behavior and boundary conditions that often cause field failures but that developers often fail to manually write tests for. Evaluated on the official JavaScript specification (ECMA-262), 98.4% of the tests Swami generated were precise to the specification. Using Swami to augment developer-written test suites improved coverage and identified 1 previously unknown defect and 15 missing JavaScript features in Rhino, 1 previously unknown defect in Node.js, and 18 semantic ambiguities in the ECMA-262 specification.