一句话:研究变异分析自动生成的变异体能否替代人工植入缺陷与“全配对”评分,用于评估学生测试套件的质量。
在计算机科学课程中,评估学生编写的测试套件通常有两种做法:对照教师手工植入的缺陷,或与所有其他学生的实现进行“全配对”比较。前者耗时且易出错,后者在公平准确应用上需要大量人工与计算开销。变异分析能够自动在实现中植入潜在缺陷,是一种可能的替代方案。已有文献表明,在大规模开源软件项目中变异体可以替代真实缺陷,但变异体是否代表学生常犯的错误尚不清楚。本文考察变异体能否有效替代学生代码中的缺陷,以及变异体的检测是否与手工植入缺陷检测、错误学生实现检测相关;若成立,教师便可使用开源变异分析工具生成的变异体来评估学生测试套件。
A common practice in computer science courses is to evaluate student-written test suites against either a set of manually-seeded faults (handwritten by an instructor) or against all other student-written implementations (“all-pairs” grading). However, manually seeding faults is a time consuming and potentially error-prone process, and the all-pairs approach requires significant manual and computational effort to apply fairly and accurately. Mutation analysis, which automatically seeds potential faults in an implementation, is a possible alternative to these test suite evaluation approaches. Although there is evidence in the literature that mutants are a valid substitute for real faults in large open-source software projects, it is unclear whether mutants are representative of the kinds of faults that students make. If mutants are a valid substitute for faults found in student-written code, and if mutant detection is correlated with manually-seeded fault detection and faulty student implementation detection, then instructors can instead evaluate student test suites using mutants generated by open-source mutation analysis tools.