Group languages are regular languages recognized by finite groups, or equivalently by finite automata in which each letter induces a permutation on the set of states. We investigate the separation problem for this class of languages: given two arbitrary regular languages as input, we show how to decide if there exists a group language containing the first one while being disjoint from the second. We prove that covering, a problem generalizing separation, is decidable. A simple covering algorithm was already known: it can be obtained indirectly as a corollary of an algebraic theorem by Ash. Unfortunately, while deducing the algorithm from this algebraic result is straightforward, all proofs of Ash’s result itself require a strong background on algebraic concepts, and a wealth of technical machinery outside of automata theory. Our proof is independent previous ones. It relies exclusively on standard notions from automata theory: we directly deal with separation and work with input languages represented by nondeterministic finite automata.We also investigate two strict subclasses. First, the alphabet modulo testable languages are those defined by counting the occurrences of each letter modulo some fixed integer (equivalently, they are the languages recognized by a commutative group). Secondly, the modulo languages are those defined by counting the length of words modulo some fixed integer. We prove that covering is decidable for both classes, with algorithms that rely on the construction made for group languages.Our proofs lead to tight complexity bounds for separation for all three classes, as well as for covering for both alphabet modulo testable languages and for modulo testable languages.

@inproceedings{abs-2205-01632,
  title = {Group Separation Strikes Back},
  author = {Thomas Place and Marc Zeitoun},
  booktitle = {LICS},
  year = {2023},
  doi = {10.1109/lics56636.2023.10175683}
}