一句话：提出 Scratch 程序测试的形式框架并实现 Whisker，支持自动化与基于属性的测试。
Scratch 等积木式编程环境让初学者无需处理语法错误，但功能错误仍可能导致程序行为不正确，目前识别和理解这些错误完全依赖人工。本文提出一个形式化测试框架，详细刻画 Scratch 程序测试问题，并实例化为 Whisker 工具，为 Scratch 程序提供自动化、基于属性的测试功能。在真实学生和教师程序上的实证评估表明，Whisker 能成功测试 Scratch 程序，平均达到 95.25% 代码覆盖率；虽然测试不稳定等问题在 Scratch 场景同样存在，但自动化与基于属性的测试能准确复现并替代教师繁琐的人工评分工作，为支持编程学习者提供新途径。
Block-based programming environments like Scratch foster engagement with computer programming and are used by millions of young learners. Scratch allows learners to quickly create entertaining programs and games, while eliminating syntactical program errors that could interfere with progress. However, functional programming errors may still lead to incorrect programs, and learners and their teachers need to identify and understand these errors. This is currently an entirely manual process. In this paper, we introduce a formal testing framework that describes the problem of Scratch testing in detail. We instantiate this formal framework with the Whisker tool, which provides automated and property-based testing functionality for Scratch programs. Empirical evaluation on real student and teacher programs demonstrates that Whisker can successfully test Scratch programs, and automatically achieves an average of 95.25 % code coverage. Although well-known testing problems such as test flakiness also exist in the scenario of Scratch testing, we show that automated and property-based testing can accurately reproduce and replace the manually and laboriously produced grading efforts of a teacher, and opens up new possibilities to support learners of programming in their struggles.
DOI 原文 ·
@inproceedings{paperbot5129,
title = {Testing scratch programs automatically},
author = {Andreas Stahlbauer and Marvin Kreis and Gordon Fraser},
booktitle = {FSE},
year = {2019},
doi = {10.1145/3338906.3338910}
}