一句话：基于 2400 万次 IDE 交互，发现通用校准难以稳定提升置信度可靠性，个性化校准在数据充足时有效，开发者偏好颜色编码指示器。
大型语言模型代码助手已嵌入 IDE，但开发者缺乏判断生成代码何时可信的信号。模型置信度可作为信号，前提是它能准确反映代码被接受的概率。本文从两方面研究 IDE 内置信度校准：面向开源模型的灵活校准方法，以及向开发者传达可靠性的界面设计。作者利用跨多语言的 2400 多万次真实 IDE 交互，将校准结果与开发者接受行为对比，发现通用 Platt 缩放校准器并不能稳定提升置信度作为可靠性信号的有用性，而个性化校准在拥有足够用户交互数据时能够带来帮助。此外，与专家设计师及 153 名职业开发者开展的多阶段设计研究表明，他们更偏好嵌入编辑器生成工作流中的非数值、颜色编码可靠性指示器。
Code assistants powered by large language models are now embedded in integrated development environments, yet developers lack reliable signals for when to trust generated code. Model confidence could serve as a signal, but only if it accurately reflects the likelihood of acceptance. Post-hoc calibration aims to achieve this alignment, though its efficacy in production settings remains understudied. We investigate in-IDE confidence calibration from two perspectives: (1) scalable methods for calibrating confidence signals and (2) interface design for communicating reliability to developers. We introduce a flexible calibration framework for open-source models and evaluate calibration against developer acceptance behavior using over 24 million real-world IDE interactions across multiple languages. We find that a general Platt-scaling calibrator does not, consistently improve the usefulness of confidence as a reliability signal, while personalized calibration can help when sufficient user interaction data is available. Complementing this, a multi-phase design study with expert designers and 153 professional developers indicates a preference for non-numerical, color-coded reliability indicators embedded in the in-editor generation workflow.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2510-22614,
title = {Does In-IDE Calibration of Large Language Models work at Scale?},
author = {Roham Koohestani and Agnia Sergeyuk and David Gros and Claudio Spiess and Sergey Titov and Premkumar Devanbu and Maliheh Izadi},
booktitle = {FSE},
year = {2026},
doi = {10.1145/3803437.3805234}
}