AI coding agents such as Aider, Claude Code, and AutoCodeRover are increasingly deployed in production software development workflows. Existing benchmarks measure correctness: whether an agent resolves a given issue. However, correctness alone does not capture operational reliability: whether the agent breaks the CI pipeline, requires human intervention, or behaves inconsistently across repeated runs. We present RELY-SE, a dashboard that measures five reliability metrics: Human Intervention Rate (HIR), Commit Failure Rate (CFR), Task Success Rate (TSR), Agent Goal Achievement Reliability (AGAR) and Decision Stability Metric (DSM) adapted from established safety-critical domains (SRE, autonomous vehicles, aviation, medical AI). The tool ingests execution traces, computes all five metrics, and renders comparative reliability profiles across agents and tasks through interactive views. We demonstrate RELY-SE on three contemporary agents evaluated on 15 SWE-bench Lite tasks (10 Django, 5 Matplotlib) with three trials each (135 total runs). Agent profiles are synthesised from published SWE-bench Lite leaderboard TSR values and representative CFR/HIR parameters rather than live agent execution, so results here illustrate the tool's output rather than a benchmark finding. On this illustrative data, Claude Code shows a TSR of 68.5% with a low CFR of 3.6%, while AutoCodeRover shows 14.7% TSR with 13.6% CFR — a disparity invisible to correctness-only evaluation but critical for CI stability. RELY-SE is publicly available at https://github.com/rishalab/Rely-SE. Demo video: https://youtu.be/RjYReUyznqI.

@inproceedings{paperbot12588,
  title = {RELY-SE: A Dashboard for Measuring Operational Reliability of AI Coding Agents},
  author = {Akilesh P and Shivadharshan S and Sridhar Chimalakonda and Vibhu Saujanya Sharma and Vikrant Kaulgud},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3834654}
}