COBug is the first leakage-controlled benchmark and evaluation pipeline designed specifically for file-level bug localization in COBOL systems. It is constructed from 12 query-bearing X-COBOL repositories and contains 552 deterministic queries, 400 distinct affected files, 476 repository-local candidate paths, and 27,047 report–candidate pairs. Each query is produced using an isolated source-code mutation with mechanically traceable ground truth, while the corresponding symptom-focused bug report is generated without accessing the affected source file. The benchmark evaluates lexical information retrieval, Logistic Regression, Random Forest, LambdaMART Learning-to-Rank, and frozen CodeBERT under strict Leave-One-Repository-Out evaluation with repository-grouped inner validation. The results show that the evaluated lexical, learned, and embedding-based approaches do not reliably outperform repository-matched random ranking. LambdaMART achieves the best aggregate performance, with a MAP of 0.1625, Top-1 accuracy of 0.0754, and Top-5 accuracy of 0.2109, but its improvement over lexical retrieval is inconsistent and largely influenced by one small repository. Frozen CodeBERT performs approximately at the random floor, and no method or floor comparison remains statistically significant after Holm correction. An additional LLM-based semantic-fault pilot admitted 15 of 29 adjudicated candidates, but no generator satisfied the predefined fault-type coverage requirement; therefore, it is retained only as bounded methodological evidence. A structured OpenCBS/ODC comparison further demonstrates category-level correspondence while identifying limitations in the benchmark’s static mutation-operator distribution.

@inproceedings{paperbot12569,
  title = {COBug: A Bug Localization Pipeline for COBOL: Does It Work? An Exploratory Study},
  author = {Kavyasri Gajula and Phanindra Kodela and Sridhar Chimalakonda},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3837518}
}