一句话：提出 Code-DiTing，通过数据蒸馏将大模型推理能力迁移到小模型，实现无需参考或测试用例的代码生成自动评估。
论文研究神经代码生成的可信评估：传统方法依赖参考解或可执行测试用例，灵活性与可扩展性受限；LLM-as-Judge 方法则或需复杂提示且可解释性差，或依赖大模型导致开销高。作者在三个数据集上实证分析两类方法后，提出 Code-DiTing，用数据蒸馏将 DeepSeek-R1-671B 的推理能力迁移到 1.5B/7B 模型，并在推理时采用多数投票。Code-DiTing 1.5B 超过同量级模型并达到约 5 倍参数模型水平；7B 仅用约 1% 参数量即超过 GPT-4o 和 DeepSeek-V3 671B，且对偏好泄漏鲁棒。
Trustworthy evaluation methods for code snippets play a crucial role in neural code generation. Traditional methods, which either rely on reference solutions or require executable test cases, have inherent limitation in flexibility and scalability. The recent LLM-as-Judge methodology offers a promising alternative by directly evaluating functional consistency between the problem description and the generated code. To systematically understand the landscape of these LLM-as-Judge methods, we conduct a comprehensive empirical study across three diverse datasets. Our investigation reveals the pros and cons of two categories of LLM-as-Judge methods: the methods based on general foundation models can achieve good performance but require complex prompts and lack explainability, while the methods based on reasoning foundation models provide better explainability with simpler prompts but demand substantial computational resources due to their large parameter sizes. To address these limitations, we propose Code-DiTing, a novel code evaluation method that balances accuracy, efficiency and explainability. We develop a data distillation framework that effectively transfers reasoning capabilities from DeepSeek-R1-671B to our Code-DiTing 1.5B and 7B models, significantly enhancing evaluation explainability and reducing the computational cost. With the majority vote strategy in the inference process, Code-DiTing 1.5B outperforms all models with the same magnitude of parameters and achieves performance which would normally exhibit in a model with 5 times of parameter scale. Code-DiTing 7B surpasses GPT-4o and DeepSeek-V3 671B, even though it only uses 1% of the parameter volume of these large models. Further experiments show that Code-DiTing is robust to preference leakage and can serve as a promising alternative for code evaluation.
DOI 原文 ·
@inproceedings{paperbot9257,
title = {Code-DiTing: Automatic Evaluation of Code Generation without References or Test Cases},
author = {Guang Yang and Yu Zhou and Xiang Chen and Wei Zheng and Xing Hu and Xin Zhou and David Lo and Taolue Chen},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00021}
}