一句话：复现 LAMNER 代码注释生成实验，并通过新数据集与 Chatbot Arena 评估检验其泛化性与鲁棒性。
在软件开发中，清晰注释对可读性与可维护性至关重要，而基于大语言模型的自动注释生成虽具潜力，其可靠性与泛化性仍需严格复现验证。本文复现 LAMNER（基于字符语言模型与命名实体识别的代码注释生成）的主要实验，沿用原始实验设置，并扩展评估：在另一个代码摘要数据集上测试，并采用 Chatbot Arena 式方法进行补充评价，以考察模型的泛化能力。研究发现，复现结果与原文报告不一致，且性能随数据集选择而变化。该工作揭示了 AI 驱动代码文档生成技术的鲁棒性差异，并讨论了复现此类技术时面临的挑战。
In the software development life cycle (SDLC), clear and comprehensive code comments play a critical role in improving code readability, maintainability, and collaboration among developers. As software complexity increases, automated code comment generation is one of the solutions to reduce the burden of documentation while ensuring code understandability. With the rise of generative AI, large language models (LLMs) have demonstrated significant potential in generating high-quality context-sensitive code comments. However, validating the reliability and generalizability of these models requires rigorous replication studies. In this paper, we replicate the results of LAMNER: Code Comment Generation Using Character Language Model and Named Entity Recognition [17]. We replicate the main experiment using the original research setup and extend the evaluation by testing on an alternative code summarization dataset and employing an additional evaluation method that is Chatbox Arena methodology to assess generalizability. Our study provides insights into the robustness of the LAMNER model, compares replicated results with the original reported findings, and highlights challenges in replicating AI-driven code documentation techniques. We observed results that differ from those reported in the original article, with performance varying depending on the choice of data set.
@inproceedings{paperbot8938,
title = {Insights into the Generalizability and Robustness of AI-Driven Code Comment Generation: A Replication Study},
author = {Eray Erer and Turgay Aytac and Aysun Bozanta and Ayse Basar},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728722}
}