一句话：构建代码生成模型校准性评估框架，发现模型开箱即用校准不佳，并评估 Platt scaling 的改进效果与适用边界。
机器学习模型常会出错，用户需要可靠的置信度指示来判断是否该信任某个输出。校准研究此前多集中于非生成式分类场景。本文针对生成式代码模型，提出一个评估校准性的框架，涵盖多种任务、正确性判据、数据集与置信度估计方法。实验发现，所测试的生成式代码模型总体上开箱即用校准不佳，即置信度与正确性关联不强。作者进一步表明，使用 Platt scaling 等标准方法可以改善校准，但由于 Platt scaling 依赖预先获得正确性数据，本文评估了其在软件工程中的适用性与泛化性，讨论了它真正具有实用潜力的情形以及受限的情形，为后续研究提供了框架，并有助于开发者更理性地决定如何使用模型生成的代码。
Machine learning models are widely used, but can also often be wrong. Users would benefit from a reliable indication of whether a given output from a given model should be trusted, so a rational decision can be made whether to use the output or not. For example, outputs can be associated with a confidence measure; if this confidence measure is strongly associated with likelihood of correctness, then the model is said to be well-calibrated. A well-calibrated confidence measure can serve as a basis for rational, graduated decision-making on how much review and care is needed when using generated code. Calibration has so far been studied in mostly non-generative (e.g., classification) settings, especially in software engineering. However, generated code can quite often be wrong: Given generated code, developers must decide whether to use directly, use after varying intensity of careful review, or discard model-generated code. Thus, calibration is vital in generative settings. We make several contributions. We develop a framework for evaluating the calibration of code-generating models. We consider several tasks, correctness criteria, datasets, and approaches, and find that, by and large, generative code models we test are not well-calibrated out of the box. We then show how calibration can be improved using standard methods, such as Platt scaling. Since Platt scaling relies on the prior availability of correctness data, we evaluate the applicability and generalizability of Platt scaling in software engineering, discuss settings where it has good potential for practical use, and settings where it does not. Our contributions will lead to better-calibrated decision-making in the current use of code generated by language models, and offers a framework for future research to further improve calibration methods for generative models in software engineering.
DOI 原文 ·
@inproceedings{paperbot8816,
title = {Calibration and Correctness of Language Models for Code},
author = {Claudio Spiess and David Gros and Kunal Suresh Pai and Michael Pradel and Md Rafiqul Islam Rabin and Amin Alipour and Susmit Jha and Prem Devanbu and Toufique Ahmed},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00040}
}