一句话：系统研究用大语言模型自动评估提交信息质量，发现思维链加少样本提示可达到接近人工的评估水平。
提交信息用于记录和解释代码变更，但实践中质量普遍不高，空信息或不充分信息比例可观。尽管基于大语言模型（LLM）的提交信息自动生成已取得显著进展，其评估仍是难题：BLEU、ROUGE-L、METEOR 等基于参考的自动指标假设代码变更与提交信息一一对应，难以衡量真实质量，迫使研究者依赖昂贵的人工评估。本文系统研究了将 LLM 用作提交信息质量自动评估器的潜力，通过多种提示策略与当前最先进 LLM 的实验表明，结合思维链推理与少样本示例的 LLM 能达到接近人类评估者的水平。该 LLM 评估器显著优于传统指标，并在可复现性、鲁棒性与公平性上保持可接受水平（尽管存在一定固有波动），为大模型时代的提交信息评估提供了可扩展的替代方案。
Commit messages are essential in software development as they serve to document and explain code changes. Yet, their quality often falls short in practice, with studies showing significant proportions of empty or inadequate messages. While automated commit message generation has advanced significantly, particularly with Large Language Models (LLMs), the evaluation of generated messages remains challenging. Traditional reference-based automatic metrics like BLEU, ROUGE-L, and METEOR have notable limitations in assessing commit message quality, as they assume a one-to-one mapping between code changes and commit messages, leading researchers to rely on resource-intensive human evaluation. This study investigates the potential of LLMs as automated evaluators for commit message quality. Through systematic experimentation with various prompt strategies and state-of-the-art LLMs, we demonstrate that LLMs combining Chain-of-Thought reasoning with few-shot demonstrations achieve near human-level evaluation proficiency. Our LLM-based evaluator significantly outperforms traditional metrics while maintaining acceptable reproducibility, robustness, and fairness levels despite some inherent variability. This work conducts a comprehensive preliminary study on using LLMs for commit message evaluation, offering a scalable alternative to human assessment while maintaining high-quality evaluation.
@inproceedings{abs-2507-10906,
title = {Evaluating Generated Commit Messages with Large Language Models},
author = {Qunhong Zeng and Yuxia Zhang and Zexiong Ma and Bo Jiang and Ningyuan Sun and Klaas-Jan Stol and Xingyu Mou and Hui Liu},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3764569}
}