一句话：提出覆盖 12 个应用领域、15 种编程语言的 2400 个任务的多领域代码生成基准，揭示 LLM 通用编码能力与领域能力并不一致。
现有针对大语言模型代码生成能力的评测主要集中于通用领域任务，模型在真实应用领域中的表现尚不清楚，核心问题是：模型的通用编码能力能否可靠代表其专业领域能力？为此作者构建了 DomainCodeBench，一个多领域代码生成基准，覆盖 12 个软件应用领域与 15 种编程语言，包含 2400 个经人工校验的任务，配有 ground-truth、人工标注的 docstring 以及细粒度依赖信息，以更充分地覆盖领域特有挑战。基准的构建流程是先通过主题挖掘识别最流行的应用领域，再依据各领域常用框架与平台筛选编码任务。基于十个主流 LLM 的大量实验得到三点发现：(1) 性能解耦，通用领域顶尖模型在特定应用领域并不总是表现最优；(2) 领域特定弱点，模型失败多源于领域知识缺口与第三方库误用；(3) 上下文增强，在提示中加入领域特定知识可使性能提升约 38.17%，为性能优化提供了可操作的启示。基准、源码与实验结果均已公开。
With the rapid advancement of large language models (LLMs), extensive research has been conducted to investigate the code generation capabilities of LLMs. However, existing efforts primarily focus on general-domain tasks, leaving LLMs’ code generation performance in real-world application domains underexplored. This raises a critical question: can a model’s general-domain coding ability reliably represent its ability in specialized domains? In this paper, we introduce DomainCodeBench, a multi-domain code generation benchmark designed to systematically evaluate LLMs across 12 software application domains and 15 programming languages. DomainCodeBench contains 2,400 manually verified tasks with ground-truth, human-annotated docstrings, and fine-grained dependency information to ensure more coverage of domain-specific challenges. Specifically, we first identify the most popular application domains by topic mining. Then, we curate coding tasks based on commonly used frameworks and platforms in each domain. We obtain several findings through extensive experiments on DomainCodeBench with ten mainstream LLMs. (1) Performance decoupling: experiments reveal that top general-domain models do not consistently excel in specific application domains; (2) Domain-specific weaknesses: LLMs often fail due to domain knowledge gaps and third-party library misusage; (3) Contextual enhancement: we show that augmenting prompts with domain-specific knowledge improves performance by around 38.17%, providing actionable insights for performance optimization. Our replication package, including the benchmark, source code, and experimental results, is available at https://github.com/DeepSoftwareAnalytics/DomainCodeBench.
@inproceedings{paperbot11931,
title = {Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark},
author = {Dewu Zheng and Yanlin Wang and Ensheng Shi and Xilin Liu and Yuchi Ma and Hongyu Zhang and Zibin Zheng},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773271}
}