一句话：提出 SPICE 自动化流水线，为 SWE-bench 风格数据集标注问题清晰度、测试覆盖与工作量，将标注成本降低约 2 万倍。
高质量标注数据集对软件工程基础模型的训练与评估至关重要，但人工标注成本高昂且劳动密集。本文提出 SPICE，一个可扩展的自动化标注流水线，为 SWE-bench 风格数据集生成问题清晰度、测试覆盖与工作量估计三类标注。SPICE 结合上下文感知的代码导航、理由驱动的提示与多轮共识机制，使生成标签接近专家标注水平；其设计源于作者在标注 SWE-Gym 中 800 多个实例时的经验与痛点。评估表明，SPICE 与人工标注的 SWE-bench Verified 数据达成高度一致，并将 1000 个实例的标注成本从约 10 万美元（人工标注）降至仅 5.10 美元。作者还发布了 SPICE 工具与 SPICE Bench 数据集，后者包含从 SWE-Gym 的 291 个开源项目中整理的 6802 个 SPICE 标注实例，规模是 SWE-bench Verified 的 13 倍以上，旨在支持社区低成本、大规模地构建面向软件工程基础模型的数据集。
High-quality labeled datasets are crucial for training and evaluating foundation models in software engineering, but creating them is often prohibitively expensive and labor-intensive. We introduce SPICE, a scalable, automated pipeline for labeling SWE-bench-style datasets with annotations for issue clarity, test coverage, and effort estimation. SPICE combines context-aware code navigation, rationale-driven prompting, and multi-pass consensus to produce labels that closely approximate expert annotations. SPICE’s design was informed by our own experience and frustration in labeling more than 800 instances from SWE-Gym. SPICE achieves strong agreement with human-labeled SWE-bench Verified data while reducing the cost of labeling 1,000 instances from around $100,000 (manual annotation) to only $5.10. These results demonstrate SPICE’s potential to enable cost-effective, large-scale dataset creation for SE-focused FMs. To support the community, we release both SPICE tool and SPICE Bench, a new dataset of 6,802 SPICE-labeled instances curated from 291 open-source projects in SWE-Gym (over 13x larger than SWE-bench Verified).
DOI 原文 ·
@inproceedings{paperbot9339,
title = {SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation},
author = {Gustavo A. Oliva and Gopi Krishnan Rajbahadur and Aaditya Bhatia and Haoxiang Zhang and Yihao Chen and Zhilong Chen and Arthur Leung and Dayi Lin and Boyuan Chen and Ahmed E. Hassan},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00192}
}