一句话：提出 TACO 框架与两个基准，从代码质量和用户意图对齐两方面细粒度评估 LLM 在编码辅助任务中的可信度。
大语言模型已深度融入软件开发流程，尤其在调试、实现与代码优化等编码辅助任务（CAT）中被广泛使用，但其输出中的幻觉会损害开发效率与软件可靠性，如何可信地评估这些回答成为关键问题。本文提出 TACO 框架，从代码质量与用户意图对齐两个维度联合评估 LLM 的回答，从而实现细粒度、可解释的可信度评价。为支撑评估，作者构建了两个基准：用于验证评估方法的人工标注数据集 TACO-Judge，以及面向真实 CAT 问题、大规模评测 LLM 表现的 TACO-Eval。实验结果表明 TACO 作为自动评估器具有较高准确性，并进一步考察了 LLM 在给自身输出打分时是否存在自偏好现象，同时比较了若干当前最优模型在 CAT 场景下的可信度，揭示了现有 LLM 的潜力与局限。
Large Language Models (LLMs) have rapidly become integral to software development workflows, particularly in coding assistance tasks (CAT) such as debugging, implementation, and code optimization. However, the trustworthiness of LLM-generated responses remains a critical concern, as hallucinations (incorrect or misleading outputs) can severely hinder developer productivity and software reliability. In this paper, we introduce TACO, a comprehensive framework for trust assessment of LLMs in CAT scenarios. TACO jointly evaluates both the code quality and the alignment with user intent of LLM responses, enabling fine-grained and interpretable trust evaluation. We construct two new benchmarks: TACO-Judge, a human-annotated dataset for validating evaluation methods, and TACO-Eval, a large-scale benchmark for assessing LLM performance on real-world CAT problems. Through extensive experiments, we (1) demonstrate the accuracy of TACO as an automated evaluator, (2) investigate whether LLMs exhibit self-preference when scoring their own outputs, and (3) compare the trustworthiness of several state-of-the-art LLMs in CAT scenarios. Our results highlight both the promise and limitations of current LLMs, and establish TACO as a reliable tool for their evaluation in software engineering contexts.
@inproceedings{paperbot11764,
title = {TACO: Trust Assessment of Large Language Models in Coding Assistance Tasks},
author = {Shihao Weng and Yang Feng and Jincheng Li and Yining Yin and Zhenlun Zhang and Lyuxi Liu and Jia Liu},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3787835}
}