一句话：用11个LLM在20万余条代码评审反馈上分类情感（有害评论）与具体性，F1最高达0.83与0.67。
代码评审是协作开发的核心环节，但反馈质量差异很大，直接影响代码可维护性与开发者互动。本文研究如何用大语言模型从情感（重点关注有害评论）与具体性两个维度评估代码评审反馈质量。作者从30个开源软件仓库收集了超过20.4万条反馈线程，评测了11个LLM，情感分类F1最高达0.83，具体性分类最高0.67。结果显示，多数开源反馈为中性或具体性偏低，高度详尽或明显有害的评论仅占少数；来自45家组织的产业数据中高具体性反馈与极简评审更多，有害反馈依然罕见。在商业环境中的部署体现了实用价值：具体性分类能立即带来收益，例如揭示资深开发者给出的反馈比其收到的更具体这一指导缺口；而有害评论分类则需谨慎的交互设计以避免用户敏感。研究表明，自动化反馈质量评估在真实环境中可行且具有实用价值。
Code review is central to collaborative software development, yet feedback quality can vary widely, influencing code maintainability and developer interactions. This study investigates how large language models (LLMs) can assess code review feedback quality along two dimensions, sentiment (with a focus on harmful comments) and specificity, to support more constructive collaboration. Using over 204,000 feedback threads from 30 open-source software (OSS) repositories, we evaluate eleven LLMs, achieving F1-scores up to 0.83 for sentiment and 0.67 for specificity. Most OSS feedback is neutral or low in specificity, with highly detailed or overtly harmful comments comprising a small minority. Industry data from 45 organisations contains significantly more highly specific feedback and more minimal reviews, while harmful feedback remains rare. Deployment of our approach in commercial settings demonstrated practical value. Specificity classifications delivered immediate value, such as revealing mentorship gaps when senior developers provided more specific feedback than they received, while harmful comment classifications required careful UX framing to avoid user sensitivity. Our findings demonstrate the feasibility and practical utility of automated feedback-quality assessment in real-world environments.
@inproceedings{YouWLPB26,
title = {Assessing Harmful Comments and Specificity in Code Review Feedback at Scale using Large Language Models},
author = {Audrey You and Jingyi (Jenny) Wang and Youxiang Lei and Lauren Peate and Kelly Blincoe},
booktitle = {FSE},
year = {2026},
doi = {10.1145/3803437.3805264}
}