paperbot · PL 论文追踪

全部论文 待读 收藏

Toward Authentic and Scalable Evaluation of Terminal Agents in Real-World Environments

走向真实世界环境中终端智能体的真实且可扩展评估
ASE Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering2026引用 0 inbox
Zhaoyang Chu
机器学习与程序语言
crossref: 2026-10-09

AI 速览

一句话：提出从真实终端活动轨迹反向构造评测任务，使终端智能体基准更真实且可扩展。

自主智能体越来越多通过终端操作软件系统，但现有评测仍依赖人工整理的基准，容易偏离真实开发者实践。该研究提出反向工程路线：从真实人类终端活动的轨迹中抽取并构造评测任务，从而让基准在构造上兼具真实性与可扩展性。前期工作 TerminalWorld 已将 80,870 条野外终端记录蒸馏为 1,530 个经校验任务，并揭示最佳前沿系统仅解决 62.5。本文作为研究愿景，旨在系统化这一从真实轨迹到可扩展评测的流程，为终端智能体提供更贴近实践、可持续扩展的评估基础。

问题
终端智能体的评测仍依赖人工整理基准，容易偏离真实开发者实践，且难以规模化。
方法
从真实人类终端活动轨迹中反向工程评测任务，将野外终端记录蒸馏并校验为可复现的基准任务。
评估
前期 TerminalWorld 将 80,870 条真实终端记录蒸馏为 1,530 个校验任务，并用前沿系统测试，显示最佳系统仅解决 62.5。
原文摘要（Abstract）

Autonomous agents increasingly act on software systems through the terminal, yet their evaluation still relies on manually curated benchmarks that drift away from real developer practice. My research pursues a different route: reverse-engineering evaluation tasks from traces of real human terminal activity, so that benchmarks become authentic and scalable by construction. As a first stage, my prior work TerminalWorld distilled 80,870 in-the-wild terminal recordings into 1,530 validated tasks, revealing that the best frontier system solves only 62.5

链接与引用

DOI 原文 · PDF（开放获取） ·

BibTeX 
@inproceedings{paperbot12486,
  title = {Toward Authentic and Scalable Evaluation of Terminal Agents in Real-World Environments},
  author = {Zhaoyang Chu},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3844572}
}