一句话:提出文档驱动功能添加基准 Doc2Feat-Bench,含 634 个真实任务与人工校验子集,最佳智能体成功率仅 37.72%。
已有智能体软件工程基准多依赖维护者协作用的 issue 报告,常含实现级细节。本文提出 Doc2Feat-Bench,面向文档驱动的功能添加:以成熟开源项目的文档变更(如 release notes)为需求,让智能体实现文档所描述的功能。基准经五阶段流水线构建,覆盖 10 个项目、634 个任务、约 11.4 万处代码变更,每个任务配对文档变更、对应实现与开发者测试;另人工校验出 114 个实例构成 Verified 子集,以支持低资源下的可靠评测。结果显示,即便消耗大量 token,最佳成功率仅 37.72%(OpenHands + Qwen3-Coder-480B),主要难点在于跨文件编辑、理解既有代码模块与正确解读文档语义。
Documentation changes in mature software projects often describe newly introduced or modified behavior. This makes them a natural basis for documentation-driven feature addition, where software engineering agents implement features from public-facing documentation updates. However, existing benchmarks for agentic software development primarily rely on issue reports written for maintainer coordination and often contain implementation-level details. Motivated by this observation, this work introduces Doc2Feat-bench, a benchmark for documentation-driven feature addition. Doc2Feat-bench consists of 634 tasks across 10 mature open-source software projects, involving about 114k code changes in total. Each task pairs a documentation change with the corresponding implementation and developer-written tests, enabling evaluation of whether a software engineering agent can implement the feature described by the documentation diff. The benchmark is constructed through a five-phase pipeline, starting from release notes, and offers broad coverage across libraries, developer tools, and frameworks while preserving real-world development settings. To facilitate lightweight and reliable evaluation under limited resources, we further curate a human-validated subset named Doc2Feat-bench Verified. It contains 114 high-quality instances whose task clarity and evaluation validity are manually verified. We use Doc2Feat-bench to assess a range of state-of-the-art software engineering agents with two widely used scaffolds. Experimental results show that despite significant token consumption, the best task success rate remains as low as 37.72% (OpenHands with Qwen3-Coder-480B). Our analysis reveals that software engineering agents face key challenges in performing cross-file edits, understanding existing code modules, and correctly interpreting documentation semantics.