一句话：提出基于LLM的模块化框架evalSmarT，通过多模型与多提示策略评估智能合约生成注释的质量。
智能合约注释生成有助于提升区块链系统的代码理解与可维护性，但生成注释的质量评估仍具挑战：BLEU、ROUGE等传统指标难以捕捉领域语义，人工评估成本高且难以扩展。本文提出evalSmarT，一个模块化、可扩展的基于大语言模型（LLM）的评估框架。它通过组合约40个LLM与10种提示策略，支持超过400种评估器配置，并用于对注释生成工具进行基准测试及挑选信息量最大的输出。结果表明，提示设计会显著影响评估结果与人类判断的一致性；基于LLM的评估为现有方法提供了可扩展且语义更丰富的替代方案。
Smart contract comment generation has gained traction as a means to improve code comprehension and maintainability in blockchain systems. However, evaluating the quality of generated comments remains a challenge. Traditional metrics such as BLEU and ROUGE fail to capture domain-specific nuances, while human evaluation is costly and unscalable. In this paper, we present evalSmarT, a modular and extensible framework that leverages large language models (LLMs) as evaluators. The system supports over 400 evaluator configurations by combining approximately 40 LLMs with 10 prompting strategies. We demonstrate its application in benchmarking comment generation tools and selecting the most informative outputs. Our results show that prompt design significantly impacts alignment with human judgment, and that LLM-based evaluation offers a scalable and semantically rich alternative to existing methods.ResourcesVideo Demo: https://youtu.be/HXS_Yiszoz4Code and Data: https://anonymous.4open.science/r/SC_code_summarization-4653
@inproceedings{paperbot9520,
title = {evalSmarT: An LLM-Based Framework for Evaluating Smart Contract Generated Comments},
author = {Fatou Ndiaye Mbodji and Mame Marieme C. Sougoufara and Wendkûuni A. M. Christian Ouedraogo and Alioune Diallo and Kui Liu and Jacques Klein and Tegawendé F. Bissyandé},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00389}
}