Large Language Models (LLMs) are rapidly transforming software engineering, particularly in automated code generation. Yet empirical evaluation remains fragmented: studies vary widely in tasks, metrics, model configurations, and environments, limiting reproducibility, comparability, and the ability to build cumulative evidence. Meanwhile, adoption is accelerating; the 2025 Google DORA report indicates developers already spend substantial daily time integrating AI into core workflows. This creates a widening gap between how broadly LLMs are used across the software development lifecycle and how quickly the research community can accumulate reliable evidence about their capabilities, limitations, and risks. We argue that a healthier future for software engineering research requires community-standard empirical research protocols and a credible path to protocol automation. We propose a minimal protocol specification that standardizes essential elements of LLM code-generation experiments: (i) task definition and problem sources, (ii) quality attributes and evaluation metrics, (iii) reproducibility factors (e.g., model versioning, prompting strategy, sampling parameters), and (iv) required reporting artifacts (e.g., prompts, outputs, execution logs, environment details). We then frame protocols as a configurable protocol family with explicit commonality and variability points, enabling protocol instances to be generated under constraints (e.g., budget, domain, risk focus) and reused across studies. Building on this foundation, we outline an interactive protocol generator that derives complete controlled-experiment protocols from a Goal-Question-Metric (GQM) strategy and application context, and we sketch a pathway toward partial automation of execution and analysis. Finally, we propose addressing the recurring “benchmark-of-the-week” symptom by rotating protocol instances—with transparent configuration and reporting requirements—rather than ad hoc leaderboards, and we conclude with an action plan for authors, reviewers, organizers, and tool builders.

@inproceedings{paperbot12642,
  title = {Automating Empirical Protocols for LLM Code Generation: A Vision to Scale Empirical Investigation},
  author = {Nathalia Nascimento and Everton Guimaraes},
  booktitle = {ICSE},
  year = {2026},
  doi = {10.1145/3793657.3793897}
}