一句话：提出基于AST子树结构熵的无参考指标，用JS散度与结构交叉熵比刻画LLM代码生成的稳定性与一致性。
评估大语言模型代码生成的稳定性对判断其可靠性至关重要，但 pass@k、BLEU、CodeBLEU 等指标依赖参考实现或执行结果。本文将结构熵概念扩展到程序领域，对固定提示下生成的每个程序抽取深度受限的 AST 子树多重集，以其相对频率构成概率分布，并用两种互补指标度量稳定性：对称有界的 Jensen–Shannon 散度衡量结构重叠，结构交叉熵比揭示缺失的高概率模式；两者均有纯结构与 token 感知变体，分别反映控制流形态与标识符层变化。该指标无参考、语言无关、不依赖执行，复杂度为 O(n,d)，在标准代码生成任务上对多个主流 LLM 的基准测试表明其能揭示模型一致性与鲁棒性的细微差异。
Assessing the stability of code generation from large language models (LLMs) is essential for judging their reliability in real-world development. We extend prior "structural-entropy" concepts to the program domain by pairing entropy with abstract-syntax-tree (AST) analysis. For any fixed prompt, we collect the multiset of depth-bounded subtrees of AST in each generated program and treat their relative frequencies as a probability distribution. We then measure stability in two complementary ways: (i) Jensen–Shannon divergence, a symmetric, bounded indicator of structural overlap, and (ii) a Structural Cross-Entropy ratio that highlights missing high-probability patterns. Both metrics admit structural-only and token-aware variants, enabling separate views on control-flow shape and identifier-level variability. Unlike pass@k, BLEU, or CodeBLEU, our metrics are reference-free, language-agnostic, and execution-independent. We benchmark several leading LLMs on standard code generation tasks, demonstrating that AST-driven structural entropy reveals nuances in model consistency and robustness. The method runs in O(n,d) time with no external tests, providing a lightweight addition to the code-generation evaluation toolkit.
DOI 原文 ·
@inproceedings{paperbot9247,
title = {Measuring LLM Code Generation Stability via Structural Entropy},
author = {Yewei Song and Tiezhu Sun and Xunzhu Tang and Prateek Kumar Rajput and Tegawendé F. Bissyandé and Jacques Klein},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00343}
}