一句话：首次系统评估 LLM 自动分类体系生成方法在 SE 工件上的质量、专家一致性、可靠性与效率，揭示质量-成本权衡。
论文关注软件工程（SE）研究中人工构建分类体系成本高且依赖领域专家，而现有 LLM 自动分类生成方法多来自其他领域，对技术复杂的 SE 工件是否适用尚不明确。作者提出多视角评估框架，从分类质量、与人类专家分类的一致性、独立标注可靠性和效率四个方面评估自动方法。实验系统收集 7 篇公开工件与人工分类的 SE 论文，使用 TnT-LLM 和 CLIMB 两种自动方法及 5 个先进 LLM。结果表明存在明显权衡：TnT-LLM 能生成接近人工质量但更复杂的分类体系，成本与运行时间显著更高；CLIMB 快 15–40 倍、便宜 8–49 倍，但在需要超越表面相似性的技术推理时质量较低。作者建议先用目标数据子集评估复杂度和成本，再选择自动方法或人工专家。
Taxonomies provide a shared conceptual framework for organizing heterogeneous observations in software engineering (SE) research. Manually constructing such taxonomies is labor-intensive and requires annotators with expertise in the SE domain. While advances in Large Language Models (LLMs) have led to the emergence of automated taxonomy generation methods outside the SE domain, their applicability to technically complex SE artifacts remains unclear. In this experience paper, we present the first comprehensive empirical evaluation of how state-of-the-art automated methods perform on SE artifacts through a multi-perspective evaluation framework, including taxonomy quality, alignment with taxonomies defined by human experts, reliability under independent annotation, and efficiency. To support this evaluation, we systematically collect seven SE papers with publicly available artifacts and human-defined taxonomies, and conduct experiments using two automated methods (TnT-LLM and CLIMB) with five state-of-the-art LLMs. Our evaluation reveals a clear trade-off: TnT-LLM constructs high-quality taxonomies comparable to human-defined ones but incurs substantially higher cost and runtime and tends to generate overly complex taxonomies, whereas CLIMB is 15–40× faster and 8–49× cheaper but tends to score lower on quality when technical inference beyond surface-level similarity is required. These findings suggest that TnT-LLM and CLIMB can be used in practical situations in the SE domain, while researchers should first assess the complexity of the generated taxonomies and their cost using a subset of the target data to decide whether to use automated methods or human experts. Our work represents a first step toward a systematic understanding of automated taxonomy generation in SE, offering actionable insights for future research and practice.
@inproceedings{paperbot12557,
title = {How Well Do LLMs Generate Taxonomies in the SE Domain? A Multi-perspective Evaluation Framework},
author = {Sota Nakashima and Yuta Ishimoto and Masanari Kondo and Tao Xiao and Yasutaka Kamei},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837551}
}