一句话：研究发现代码编写会低估学生对编程模式的知识，结合编辑、修订与识别任务能更准确评估。
论文探讨代码编写任务在多大程度上能衡量学生对编程模式与反模式的知识，以及如何更准确地评估这类知识。作者调查了328名中级计算机科学学生，比较其完成写代码、编辑他人代码、修订自己不同结构代码等任务的表现，任务聚焦于返回布尔表达式以及在 if-else 中使用唯一代码。结果显示，代码编写有时会低估学生知识：在返回布尔表达式的任务中，超过55%最初采用非专家结构的学生在未被提示如何改进的情况下成功修订为专家结构；超过25%最初写出非专家代码的学生能正确地将他人的非专家代码编辑为专家结构。这表明非专家代码并不一定是深层误解的可靠指标。此外，代码编写与代码编辑虽相关但关系较弱，仅以写代码作为预测因子的模型解释力不足15%方差；加入识别专家代码、判断专家代码比非专家代码更可读等预测因子后，模型准确度提升。论文认为，结合编写、修订、编辑与识别任务可更准确评估学生对编程模式的知识。
How well do code-writing tasks measure students’ knowledge of programming patterns and anti-patterns? How can we assess this knowledge more accurately? To explore these questions, we surveyed 328 intermediate CS students and measured their performance on different types of tasks, including writing code, editing someone else’s code, and, if applicable, revising their own alternatively-structured code. Our tasks targeted returning a Boolean expression and using unique code within an if and else.We found that code writing sometimes under-estimated student knowledge. For tasks targeting returning a Boolean expression, over 55% of students who initially wrote with non-expert structure successfully revised to expert structure when prompted - even though the prompt did not include guidance on how to improve their code. Further, over 25% of students who initially wrote non-expert code could properly edit someone else’s non-expert code to expert structure. These results show that non-expert code is not a reliable indicator of deep misconceptions about the structure of expert code. Finally, although code writing is correlated with code editing, the relationship is weak: a model with code writing as the sole predictor of code editing explains less than 15% of the variance. Model accuracy improves when we include additional predictors that reflect other facets of knowledge, namely the identification of expert code and selection of expert code as more readable than non-expert code. Together, these results indicate that a combination of code writing, revising, editing, and identification tasks can provide a more accurate assessment of student knowledge of programming patterns than code writing alone.
@inproceedings{NurollahianRW23,
title = {Improving Assessment of Programming Pattern Knowledge through Code Editing and Revision},
author = {Sara Nurollahian and Anna N. Rafferty and Eliane Wiese},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse-seet58685.2023.00012}
}