一句话：提出 AutoChecklist，通过自由文本反馈自动细化评估清单，提升 LLM 评判器的评估质量并降低人工成本。
在大规模评估 LLM 生成内容时，结构化清单能帮助人类和 LLM 给出细致、可解释且可靠的评价，但构建有效清单往往需要基于真实数据反复迭代，成本很高。本文提出 AutoChecklist——一个已在微软部署的系统，它在基于清单的 LLM 评判器中加入自由评论选项，并依据这些反馈动态更新清单，使清单能够适应异常样本并捕获初始设计可能遗漏的数据集特有标准。作者在三个生产相关领域中评估了该系统，结果表明：系统能持续发现有价值的新评估标准；评论选项能提升在原始清单下的答案质量；系统对种子清单的属性（如语气、答案类型）具有灵活性；且在不同数据集上表现鲁棒。这些结果说明，自动化清单细化能显著减少构建可靠评估流水线所需的人工投入。
Evaluating LLM-generated content at scale is a critical challenge in production systems. Structured checklists enable both humans and LLMs to provide detailed, interpretable, and reliable evaluations—but crafting effective checklists remains difficult and often requires iterative refinement based on real-world data. We present AutoChecklist, a system deployed at Microsoft that augments checklist-based LLM judges with a free-form comment option and dynamically updates the checklist based on this feedback. This ensures checklists adapt to outlier instances and capture dataset-specific criteria that initial designs may miss. We evaluate AutoChecklist across three production-relevant domains and demonstrate that (1) it consistently surfaces valuable new evaluation criteria, (2) the comment option improves answer quality on the original checklist, (3) it is flexible with respect to seed checklist properties like tone and answer type, and (4) it is robust across different datasets. The results show that automated checklist refinement significantly reduces the manual effort required to build reliable evaluation pipelines.