一句话：实证揭示基于 Codeforces Elo 的大模型竞技编程评测受提交顺序、赛题选择与运行随机性影响，分数可差逾千分。
基于 Codeforces 的 Elo 评分已成为业界评估大语言模型竞技编程能力的重要指标，但报告中常缺乏关键实验细节，导致同一模型版本的分数出现近 500 分的巨大波动。本文对 Elo 评测中的隐藏偏差因素进行系统性实证研究，考察三类因素：（1）提交结果的时序顺序，（2）比赛难度与赛题的选择，（3）大模型运行间的随机波动。作者构建了一个受控基准，涵盖 37 场近期 Codeforces 比赛与 13,691 个生成测试用例，并以此量化各因素的影响。结果显示，改变提交顺序可使 Elo 分数偏移 394 分，更换所选比赛对同一模型造成的差异最高达 1,122 分；而在相同赛题上重复评测时，平均分最大波动达 349 分。作者据此指出，缺乏严格标准化与透明实验设置时，直接的 Elo 分数比较不可靠且可能产生误导。
As Large Language Models (LLMs) achieve breakthroughs in complex reasoning, Codeforces-based Elo ratings have emerged as a prominent metric for evaluating competitive programming capabilities in industry. However, these ratings are often reported without critical experimental details, leading to significant discrepancies-illustrated by recent reports where the same model version’s score fluctuated by nearly 500 points (out of 2,029). This paper presents a systematic empirical study on the hidden factors biasing Elo evaluations: (1) the temporal ordering of submissions, (2) contest difficulty selection, and (3) run-to-run stochastic variability of LLMs. Utilizing a controlled benchmark of 37 recent Codeforces contests and 13,691 generated test cases, we demonstrate that Elo scores are highly sensitive to these parameters. Our findings reveal that varying submission orders can shift scores by 394 points, while contest selection can cause differences of up to 1,122 points for the same model. Run-to-run performance exhibits substantial instability, with a maximum difference of 349 points in mean scores observed when evaluating identical contests. We conclude that direct Elo comparisons are unreliable and potentially misleading without strict standardization and transparent reporting of experimental settings.
@inproceedings{abs-2602-05891,
title = {When Elo Lies: Hidden Biases in Codeforces-Based Evaluation of Large Language Models in Practice},
author = {Shenyu Zheng and Ximing Dong and Xiaoshuang Liu and Chun Yong Chong and Gustavo Oliva and Dayi Lin and Boyuan Chen and Shaowei Wang and Ahmed E. Hassan},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834458}
}