一句话：提出 CES 基准，评估 LLM 模拟程序执行时的正确性、连贯性与一致性，揭示推理捷径与数据泄漏。
本文提出 CES（C Execution Simulation），一个评估 LLM 模拟程序执行并运用该推理完成编程任务能力的任务。除衡量执行模拟中变量预测的正确性外，CES 引入“连贯性”概念，判断模拟是否符合常识执行逻辑，即使预测值不正确，从而排除因推理捷径、幻觉或潜在数据泄漏导致的可疑正确输出。CES 还提出一种新指标，在强、弱、随机三种 prime path coverage 谱下衡量同一或不同测试间推理的一致性。使用 CES 评估 16 个 LLM（含 3 个推理 LLM）发现，在 HumanEval 上 81.42% 的执行模拟是连贯的，其中 46.92% 导致正确输出预测，53.08% 导致错误预测。GPT-4 和 DeepSeek-R1 等前沿 LLM 的执行推理最不连贯，多因自然语言捷径。尽管执行模拟相对连贯，LLM 在不同测试间的推理性能不一致，多为随机（48.87%）或弱（45.37%），可能解释其在需要路径敏感程序分析的任务上的弱点。作者还将 CES 与 bug 预测/定位/修复对比，发现 LLM 很少将执行推理纳入 bug 相关分析，其成功主要源于固有模式匹配、自然语言捷径或数据泄漏。
This paper proposes CES (C Execution Simulation), a task to evaluate the abilities of LLMs in simulating program execution and using that reasoning in programming tasks. Besides measuring the correctness of variable predictions during execution simulation, CES introduces the notion of coherence to determine whether the simulation complies with commonsense execution logic, even if the predicted values along the simulations are incorrect. This enables CES to rule out suspiciously correct output predictions due to reasoning shortcuts, hallucinations, or potential data leakage. CES also introduces a novel metric to measure reasoning consistency across tests with the same or different prime path coverage in a spectrum: strong, weak, and random. Evaluating 16 LLMs (including three reasoning LLMs) using CES indicates \(81.42\%\) coherent execution simulation on HumanEval, \(46.92\%\) and \(53.08\%\) of which result in correct and incorrect output predictions. Frontier LLMs such as GPT-4 and DeepSeek-R1 have the most incoherent execution reasoning, mostly due to natural-language shortcuts. Despite relatively coherent execution simulation, LLMs’ reasoning performance across different tests is inconsistent, mostly random (\(48.87\%\)) or weak (\(45.37\%\)), potentially explaining their weakness in programming tasks that require path-sensitive program analysis to succeed. We also compare CES with bug prediction/localization/repair, which intuitively requires control- and data-flow awareness. We observe that LLMs rarely incorporate execution reasoning into their analysis for bug-related tasks, and their success is primarily due to inherent pattern-matching capabilities, natural language shortcuts, or data leakage. Without reasoning, there is a threat to the generalizability of LLMs in dealing with unseen bugs or patterns in different contexts. CES can be used to vet the suspicious success of LLMs in these tasks systematically.