一句话：实证揭示在线评测平台测试套件普遍存在假阳性，43.4%的题目含错误解被判正确。
在线评测平台广泛用于教育、竞赛、招聘、职业训练和大语言模型训练，其判题可靠性取决于预定义测试套件。然而，错误解可能因测试不足而被判为正确，形成假阳性。本文对939道题、541,552个被平台判为正确的解进行实证研究，识别假阳性解并评估测试套件有效性。结果显示，43.4%的题目包含假阳性解，共发现3,440个bug。尽管如此，这些测试套件在常用指标上仍表现良好：88.2%的假阳性解达到100%行覆盖，78.9%达到100%分支覆盖，32.5%的变异得分满分。作者公开了检测到的假阳性解和生成的测试输入，表明需要进一步剔除假阳性并提升测试套件有效性。
Online Judge platforms play a pivotal role in education, competitive programming, recruitment, career training, and large language model training. They rely on predefined test suites to judge the correctness of submitted solutions. It is therefore important that the solution judgement is reliable and free from potentially misleading false positives (i.e., incorrect solutions that are judged as correct). In this paper, we conduct an empirical study of 939 coding problems with 541,552 solutions, all of which are judged to be correct according to the test suites used by the platform, finding that 43.4% of the problems include false positive solutions (3,440 bugs are revealed in total). We also find that test suites are, nevertheless, of high quality according to widely-studied test effectiveness measurements: 88.2% of false positives have perfect (100%) line coverage, 78.9% have perfect branch coverage, and 32.5% have a perfect mutation score. Our findings indicate that more work is required to weed out false positive solutions and to further improve test suite effectiveness. We have released the detected false positive solutions and the generated test inputs to facilitate future research.
@inproceedings{paperbot8058,
title = {Who Judges the Judge: An Empirical Study on Online Judge Tests},
author = {Kaibo Liu and Yudong Han and Jie M. Zhang and Zhenpeng Chen and Federica Sarro and Mark Harman and Gang Huang and Yun Ma},
booktitle = {ISSTA},
year = {2023},
doi = {10.1145/3597926.3598060}
}