一句话：通过在 ASML 专有代码库上的案例研究，提出 build@k 指标，评估 LLM 在封闭工业环境中生成可用代码的能力。
大语言模型在开源领域的代码生成任务上表现优异，但在存在领域约束与代码相互依赖的专有工业环境中是否适用仍缺乏研究。本文与 ASML leveling 部门合作开展案例研究，构建了面向其专有代码库的评估框架并发布了新的基准，同时提出 build@k 指标，用于衡量 LLM 生成的代码能否成功编译并集成进真实工业仓库。作者考察了多种提示技术，比较了通用模型与代码专用模型，并分析了模型规模对代码生成能力的影响，评估同时采用基于匹配与基于执行的指标。结果显示，提示技术与模型规模对输出质量有显著影响，few-shot 与 chain-of-thought 提示取得最高的构建成功率；代码专用模型与通用模型之间的性能差异并不明显，且在不同模型家族间波动较大。
Large language models have shown impressive performance in various domains, including code generation across diverse open-source domains. However, their applicability in proprietary industrial settings, where domain-specific constraints and code interdependencies are prevalent, remains largely unexplored. We present a case study conducted in collaboration with the leveling department at ASML to investigate the performance of LLMs in generating functional, maintainable code within a closed, highly specialized software environment.We developed an evaluation framework tailored to ASML’s proprietary codebase and introduced a new benchmark. Additionally, we proposed a new evaluation metric, build@k, to assess whether LLM-generated code successfully compiles and integrates within real industrial repositories. We investigate various prompting techniques, compare the performance of generic and code-specific LLMs, and examine the impact of model size on code generation capabilities, using both match-based and execution-based metrics. The findings reveal that prompting techniques and model size have a significant impact on output quality, with few-shot and chain-of-thought prompting yielding the highest build success rates. The difference in performance between the code-specific LLMs and generic LLMs was less pronounced and varied substantially across different model families.
DOI 原文 ·
@inproceedings{paperbot9566,
title = {Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML},
author = {Yash Mundhra and Max Valk and Maliheh Izadi},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00287}
}