一句话：提出任务无关元模型与评估合成器，自动生成基础模型任务评估器并收集人类反馈。
针对基础模型应用中的幻觉与细微失败，现有评估方法多面向通用任务或特定基准，难以为缺乏指标和数据集的软件团队提供任务专属评估。本文提出合成基础模型任务评估器程序的方法，其核心包括：面向任意任务的元模型，用于刻画任务属性；高效利用人类反馈的交互协议；以及选择或生成合适评估项的评估合成器。作者在 TaskEval 中实现该方案，并在图表数据抽取和文档问答两个差异较大的任务上演示。初步评估显示所选评估的质量准确率分别为 93% 和 90%。
Hallucinations are a key concern when creating applications that rely on Foundation models (FMs). Understanding where and how these subtle failures occur in an application relies on evaluation methods known as \textit{evals}. Prior work focuses on defining new eval methods or benchmark datasets for specific tasks. However, neither helps a software team with a task-specific FM application when there is no metric or dataset. The demand for both automated approaches and deep integration of human insight makes this a challenging problem. We address this gap by proposing an approach to synthesise a FM task-specific evaluator program that provides automation and a custom UI for capturing feedback. The core novelty of our approach lies in: (1) a task-agnostic meta-model that captures properties of any FM task, (2) an interaction protocol for efficient use of human feedback, and (3) an eval synthesiser that selects or generates an appropriate set of evals. We implement our approach in \toolname and demonstrate the concept on two diverse FM tasks: chart data extraction and document question answering. A preliminary evaluation on the quality of our selected evals shows 93\% and 90\% accuracy respectively. Our research tackles a growing problem facing engineering teams, how to evaluate and review outputs from FM tasks.
@inproceedings{paperbot12003,
title = {TaskEval: Synthesised Evaluation for Foundation-Model Tasks},
author = {Dilani Widanapathiranage and Scott Barnett and Stefanus Kurniawan and Wannita Takerngsaksiri},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3786582.3786822}
}