一句话：提出 REFERENT，通过迁移学习与作业信息让 Transformer 无需参考程序即可为错误编程提交生成反馈。
编程课程学生需要反馈来提升编程能力，但现有自动反馈生成方法依赖正确提交作为参考；在缺少参考程序时，它们转用变异算子，容易导致搜索空间爆炸。为此，作者提出 REFERENT：对预训练 Transformer 进行迁移学习，并使用过往作业的提交历史，以及标题、标签、作业描述和测试用例等作业信息，从而在有限资源下无需参考程序生成反馈。初步研究表明，REFERENT 能为 32.7% 的错误提交生成反馈；当有参考程序时，性能提升至 50.7%。研究还考察了提交历史、作业信息以及开源软件修复知识对反馈生成的作用。
Students require feedback on programming assignments to improve their programming skills. An Automated feedback generation (AFG) technique proposes to provide feedback-corrected submissions for incorrect student programming submissions in programming courses. However, these techniques are limited as they rely on the availability of correct submissions as a reference to generate feedback. In situations where correct submissions are not available, they resort to using mutation operators, which can lead to a search space explosion problem. In this work, we propose REFERENT, Transformer-based feedback generation using assignment information. REFERENT uses transfer learning on a pre-trained model with data from students’ submission history from the past assignment. To generate assignment-related feedback, we use a title, tag, assignment description, and test case as assignment information. REFERENT can generate feedback without a reference program in limited resources. We conducted a preliminary study to confirm the effectiveness of REFERENT and the feasibility of using assignment information. REFERENT generated feedback for 32.7% of incorrect submissions without reference programs and that its performance increased up to 50.7% when reference programs were used. We also check whether the submission history, assignment information, and repair knowledge of open-source software help generate feedback.
@inproceedings{HeoJCL23,
title = {REFERENT: Transformer-based Feedback Generation using Assignment Information for Programming Course},
author = {Jinseok Heo and Hohyeon Jeong and Dongwook Choi and Eunseok Lee},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse-seet58685.2023.00035}
}