paperbot · PL 论文追踪

全部论文 待读 收藏

P1 Doc2Feat-Bench: Evaluating Documentation-Driven Feature Addition

Doc2Feat-Bench:评估文档驱动的功能添加
ASE Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering2026匹配度 5/10 · 论文面向 Coding Agent 在真实软件项目中的功能实现能力评测,属于 Agentic 程序与 LLM/Agent 工具方向,且符合用户对'构建重要且缺乏 benchmark 场景的评测集'的兴趣;但它聚焦软件工程任务本身,不涉及静态分析、浮点或贝叶斯程序分析等用户核心方向,也未将 PL 技术应用到 Agent 上。引用 22 inbox
Zhonghao Jiang, Le Deng, Jialun Cao, Michael Pradel, Zhongxin Liu
crossref: 2026-10-09

AI 速览

一句话:提出文档驱动功能添加基准 Doc2Feat-Bench,含 634 个真实任务与人工校验子集,最佳智能体成功率仅 37.72%。

已有智能体软件工程基准多依赖维护者协作用的 issue 报告,常含实现级细节。本文提出 Doc2Feat-Bench,面向文档驱动的功能添加:以成熟开源项目的文档变更(如 release notes)为需求,让智能体实现文档所描述的功能。基准经五阶段流水线构建,覆盖 10 个项目、634 个任务、约 11.4 万处代码变更,每个任务配对文档变更、对应实现与开发者测试;另人工校验出 114 个实例构成 Verified 子集,以支持低资源下的可靠评测。结果显示,即便消耗大量 token,最佳成功率仅 37.72%(OpenHands + Qwen3-Coder-480B),主要难点在于跨文件编辑、理解既有代码模块与正确解读文档语义。

问题
现有智能体软件开发基准多以面向维护者协作的 issue 报告为任务来源,常包含实现级细节,无法真实刻画仅凭公开文档更新实现新功能的场景。
方法
构建 Doc2Feat-Bench:以五阶段流水线从 release notes 出发,将文档变更与对应实现、开发者测试配对,形成覆盖 10 个成熟开源项目的 634 个任务,并人工校验出 114 个实例构成 Verified 子集。
评估
使用两种常用 scaffold 评测多种先进软件工程智能体,最佳任务成功率仅 37.72%,并分析其在跨文件编辑、既有模块理解与文档语义解读方面的失败模式。
原文摘要(Abstract)

Documentation changes in mature software projects often describe newly introduced or modified behavior. This makes them a natural basis for documentation-driven feature addition, where software engineering agents implement features from public-facing documentation updates. However, existing benchmarks for agentic software development primarily rely on issue reports written for maintainer coordination and often contain implementation-level details. Motivated by this observation, this work introduces Doc2Feat-bench, a benchmark for documentation-driven feature addition. Doc2Feat-bench consists of 634 tasks across 10 mature open-source software projects, involving about 114k code changes in total. Each task pairs a documentation change with the corresponding implementation and developer-written tests, enabling evaluation of whether a software engineering agent can implement the feature described by the documentation diff. The benchmark is constructed through a five-phase pipeline, starting from release notes, and offers broad coverage across libraries, developer tools, and frameworks while preserving real-world development settings. To facilitate lightweight and reliable evaluation under limited resources, we further curate a human-validated subset named Doc2Feat-bench Verified. It contains 114 high-quality instances whose task clarity and evaluation validity are manually verified. We use Doc2Feat-bench to assess a range of state-of-the-art software engineering agents with two widely used scaffolds. Experimental results show that despite significant token consumption, the best task success rate remains as low as 37.72% (OpenHands with Qwen3-Coder-480B). Our analysis reveals that software engineering agents face key challenges in performing cross-file edits, understanding existing code modules, and correctly interpreting documentation semantics.

链接与引用

DOI 原文 · arXiv ·

BibTeX
@inproceedings{paperbot12380,
  title = {Doc2Feat-Bench: Evaluating Documentation-Driven Feature Addition},
  author = {Zhonghao Jiang and Le Deng and Jialun Cao and Michael Pradel and Zhongxin Liu},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3837489}
}