一句话：基于 AtCoder 提交历史实证分析在线评测中错误解决率与耗时随题目难度和用户熟练度的变化。
在线评测系统（OJS）能自动编译、验证并执行用户提交的程序，为自主学习和算法训练提供反馈，但针对用户错误解决行为与支持需求的研究仍不充分。本文分析 AtCoder（被 CodeNet 收录）的提交历史，聚焦错误解决率、解决耗时，并考察这些指标如何随题目难度与用户解题熟练度变化。研究发现：超时错误（TLE）最难解决，且不受用户编程熟练度影响；在特定难度前后，编译错误与 TLE 的比例发生反转；错误解决耗时随用户熟练度提高而减少，且与错误类型无关。这些发现有助于识别需要支持的错误类型，并帮助竞赛组织者评估参与者能力。
An online judge system (OJS) is a system that automatically compiles, verifies, and executes programs submitted by users for specific problems, and provide feedback. OJSs offer a vast array of programming-related problems, enabling users to choose relevant problems and engage in self-directed programming and algorithms learning. However, OJS research and learning support using OJS remain limited and insufficient. We analyzed the submission histories of AtCoder, an OJS platform included in CodeNet, focusing on error resolution rates, the time required for resolution, and the variation of these metrics based on the experience of users in solving problems of varying difficulty levels. We made the following findings: (1) Time limit exceeded error is the most difficult to solve regardless of the programming proficiency of users. (2) Reversal in the proportions of compilation errors and time limit exceeded errors before and after problems of specific difficulty level. (3) Error resolution time decreases with the programming proficiency of users regardless of error types. These findings are crucial in identifying errors in which users require support and enabling competition organizers to assess the programming abilities of participants.
DOI 原文 ·
@inproceedings{paperbot8984,
title = {An Empirical Study of the Error Characteristics in an Online Judge System},
author = {Shota Shimizu and Erina Makihara and Norihiro Yoshida},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3727228}
}