一句话：提出以 LLM 作为测试 Oracle，通过区分性输入与输出选择提升 LLM 生成代码的正确性。
LLM 代码生成常产生错误程序，而以自然语言描述作为非形式化 oracle 并不可靠。现有方法从多个候选中选程序，依据 LLM 生成测试上执行结果的一致性，但测试可能冗余或无法区分正误，投票等准则也缺乏可解释依据。本文提出以 LLM 作为测试 oracle 的程序选择：首先生成能捕捉候选程序语义差异的区分性输入，记录各程序输出；再让 LLM 结合自然语言问题描述，从多个输出中选出最可能正确的输出。实现 LLMCodeChoice 并在 HumanEval 和 MBPP 上评估，pass@1 相比 CodeT 提升 3.6–7%；所选输入-输出规约还揭示任务描述不完整与歧义及基准错误参考实现。
While large language models (LLMs) have shown significant advancements in code generation, their susceptibility to producing incorrect code poses a significant challenge to the adoption of LLM-generated programs. This issue largely stems from the reliance on natural language descriptions as informal oracles in code generation. Current strategies to mitigate this involve selecting the best program from multiple LLM-generated alternatives, judged by criteria like the consistency of their execution results on an LLM-generated test suite. However, this approach has crucial limitations: (1) LLMs often generate redundant tests or tests that cannot distinguish between correct and incorrect solutions, (2) the used consistency criteria, such as the majority vote, fail to foster developer trust due to the absence of transparent rationale behind the made choices. In this work, we propose a new perspective on increasing the quality of LLM-generated code via program selection using the LLM as a test oracle. Our method is based on our experimentally confirmed observation that LLMs serve more effectively as oracles when tasked with selecting the correct output from multiple choices. Leveraging this insight, we first generate distinguishing inputs that capture semantic discrepancies of programs sampled from an LLM, and record outputs produced by the programs on these inputs. An LLM then selects the most likely to be correct output from these, guided by the natural language problem description. We implemented this idea in a tool LLMCodeChoice and evaluated its accuracy in generating and selecting standalone programs. Our experiments demonstrated its effectiveness in improving pass@1 by 3.6-7% on HumanEval and MBPP benchmarks compared to the state-of-art CodeT. Most interestingly, the selected input-output specifications helped us to uncover incompleteness and ambiguities in task descriptions and also identify incorrect ground-truth implementations in the benchmarks.