一句话：构建可复用基准 AISmellBench，含 1,265 个标注文件与 3,714 个已验证 AI 代码异味实例，支撑检测器评测。
论文针对 ML 与基于 LLM 的系统中 AI 代码异味检测缺乏统一、可复用评测基准的问题，提出 AISmellBench。该基准汇集 888 个 ML 仓库与 333 个 LLM 仓库，共 1,221 个 Python 仓库；对 1,265 个 Python 文件人工标注了 31 类异味（24 类 ML 特有、7 类 LLM 特有）共 3,714 个已验证实例，并给出含 567 个实例的精简评测子集。数据包含仓库级元数据、文件级上下文、异味级标注、验证结论、置信度与溯源链接。构建采用可复现四阶段流水线：GitHub 仓库挖掘、预处理与资格过滤、工具辅助识别候选异味、人工验证与仲裁，标注者一致性 Fleiss' kappa = 0.78。该基准可用于评测新检测器、比较现有工具、研究异味分布与共现，并复现相关实证分析。
AISmellBench is a reusable benchmark for AI code smells in Machine Learning (ML) and Large Language Model (LLM)-based systems. The dataset is designed to support the empirical evaluation of AI code smell detection approaches by providing manually validated ground truth, multi-level metadata, and traceable provenance artifacts. The benchmark brings together two large-scale corpora covering 888 ML repositories and 333 LLM-based repositories, for a total of 1,221 AI-based Python repositories. It includes 1,265 annotated Python files with 3,714 validated AI code smell instances across 31 smell types (24 ML-specific, 7 LLM-specific). A compact evaluation subset of 567 validated instances is also provided to support efficient and comparable detector evaluation. AISmellBench was constructed through a reproducible four-stage pipeline: GitHub repository mining, preprocessing and eligibility filtering, tool-assisted identification of potential smell instances, and manual validation with adjudication artifacts (Fleiss' kappa = 0.78). The released package includes repository-level metadata, file-level context, smell-level annotations, validation verdicts, confidence scores, provenance links, construction artifacts, and evaluation files. This dataset is intended for researchers and practitioners studying AI code smells, static analysis, technical debt, software quality, ML engineering, LLM-based systems, and automated code improvement. It can be used to evaluate new AI code smell detectors, compare existing tools, study smell prevalence and co-occurrence, reproduce empirical analyses, and support future research on technical debt and automated correction in AI-based systems.
@inproceedings{paperbot12560,
title = {AISmellBench: A Reusable Benchmark of AI Code Smells in ML and LLM-Based Systems},
author = {Brahim Mahmoudi and Naouel Moha and Quentin Stiévenart and Florent Avellaneda},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834605}
}