一句话：提出首个合约级 Solidity 代码生成基准 SolContractEval，并用基于历史交易重放的动态框架系统评测六个主流 LLM。
区块链的普及使智能合约广泛使用，也催生了对智能合约代码生成工具的需求。尽管大语言模型（LLM）在通用语言代码生成上表现优异，但其在 Solidity 上的效果仍缺乏研究：Solidity 在训练数据中占比小，且语法对版本敏感、灵活性有限，因此现有模型在合约开发中的可靠性存疑。现有评测多聚焦孤立函数与合成输入，无法衡量真实合约开发能力。为此，作者提出 SolContractEval——首个合约级 Solidity 代码生成基准，包含取自真实链上合约、覆盖九大领域的 124 个任务；每个任务由完整上下文依赖、结构化合约框架与简洁任务提示组成，并由资深开发者独立标注与交叉验证。为自动精确评估功能正确性，作者还构建了基于历史交易重放的动态评估框架。基于该基准对六个主流 LLM 的系统评测显示：Claude-3.7-Sonnet 总体表现最佳，但仍逊于这些模型在通用语言类级生成任务上的水平；模型在符合标准模式的任务上较好，在复杂逻辑与合约间依赖上表现不佳；同时对 Solidity 特有特性与上下文依赖的理解有限。
The rise of blockchain has brought smart contracts into mainstream use, creating a demand for smart contract generation tools. While large language models (LLMs) excel at generating code in general-purpose languages, their effectiveness on Solidity, the primary language for smart contracts, remains underexplored. Solidity constitutes only a small portion of typical LLM training data and differs from general-purpose languages in its version-sensitive syntax and limited flexibility. These factors raise concerns about the reliability of existing LLMs for Solidity code generation. Critically, existing evaluations, focused on isolated functions and synthetic inputs, fall short of assessing models’ capabilities in real-world contract development.To bridge this gap, we introduce SolContractEval, the first contract-level benchmark for Solidity code generation. It comprises 124 tasks drawn from real on-chain contracts across nine major domains. Each task input, consisting of complete context dependencies, a structured contract framework, and a concise task prompt, is independently annotated and cross-validated by experienced developers. To enable precise and automated evaluation of functional correctness, we also develop a dynamic evaluation framework based on historical transaction replay. Building on SolContractEval, we perform a systematic evaluation of six mainstream LLMs. We find that Claude-3.7-Sonnet achieves the highest overall performance, though evaluated models underper-form relative to their capabilities on class-level generation tasks in general-purpose programming languages. Second, current models perform better on tasks that follow standard patterns but struggle with complex logic and inter-contract dependencies. Finally, they exhibit limited understanding of Solidity-specific features and contextual dependencies.
DOI 原文 ·
@inproceedings{paperbot9384,
title = {SolContractEval: A Benchmark for Evaluating Contract-Level Solidity Code Generation},
author = {Zhifan Ye and Jiachi Chen and Zhenzhe Shao and Lingfeng Bao and Xiaohu Yang and Zhongxin Liu},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00235}
}