一句话：通过大规模实验比较人工与自动反馈，发现自动反馈主要提升新手排错效率而非概念理解。
论文研究自动自适应编译错误反馈能否帮助新手程序员更有效地学习编程。作者开展大规模实验（N=480，超1万人时），在七周内比较人工辅导与两种自动反馈策略：直接给出正确解法，或生成示例/类比引导修正。利用生存分析对学生错误率建模，发现自动反馈能让学生更快解决代码错误，优于人工反馈；但该优势主要是流程/后勤性的，考试中撤回反馈后即消失。随着问题复杂度增加，自动反馈相对人工辅导的优势扩大；示例引导与具体修复反馈在不同编程错误类别上各有独立优势。结果为自动反馈在新手编程教学中的收益划定了清晰边界。
Can automated adaptive feedback for correcting erroneous programs help novice programmers learn to code better? In a large-scale experiment, we compare student performance when tutored by human tutors, and when receiving automated adaptive feedback. The automated feedback was designed using one of two well-known instructional principles: (i) presenting the correct solution for the immediate problem, or (ii) presenting generated examples or analogies that guide towards the correct solution. We report empirical results from a large-scale (N = 480, 10,000 + person hour) experiment assessing the efficacy of these automated compilation-error feedback tools. Using the survival analysis on error rates of students measured over seven weeks, we found that automated feedback allows students to resolve errors in their code more efficiently than students receiving manual feedback. However, we also found that this advantage is primarily logistical and not conceptual; the performance benefit seen during lab assignments disappeared during exams wherein feedback of any kind was withdrawn. We further found that the performance advantage of automated feedback over human tutors increases with problem complexity, and that feedback via example and specific repair have distinct, non-overlapping relative advantages for different categories of programming errors. Our results offer a clear and granular delimitation of the pedagogical benefits of automated feedback in teaching programming to novices.
@inproceedings{paperbot5680,
title = {Characterizing the pedagogical benefits of adaptive feedback for compilation errors by novice programmers},
author = {Umair Z. Ahmed and Nisheeth Srivastava and Renuka Sindhgatta and Amey Karkare},
booktitle = {ICSE},
year = {2020},
doi = {10.1145/3377814.3381703}
}