一句话：评估LLM驱动的运行时系统互操作策略，qwen2.5-coder:32b在多数场景有效，单位换算场景需CODEGEN。
针对系统之系统日益动态、异构且互操作构建成本高的问题，本文探索用LLM在运行时自主实现系统互操作。作者选取13个开源LLM，在具体互操作用例上构造四个版本的数据集，采用DIRECT与CODEGEN两种策略，每个模型和数据集版本运行三次，比较效果与一致性。结果显示，qwen2.5-coder:32b在四个版本中的三个上最有效：DIRECT平均pass@1≥0.99，CODEGEN平均pass@1≥0.89；在含单位换算的第四版中，DIRECT全部失败，而CODEGEN下该模型平均pass@1=0.75。结论认为部分LLM可自主实现系统互操作，建议跨领域评估并研究可靠性策略。
Background: Systems of systems are becoming increasingly dynamic and heterogeneous, and this adds pressure on the long-standing challenge of interoperability. Besides its technical aspect, interoperability has also an economic side, as development time efforts are required to build the interoperability artifacts. Objectives: With the recent advances in the field of large language models (LLMs), we aim at analyzing the effectiveness of LLM-based strategies to make systems interoperate autonomously, at runtime, without human intervention. Method: We selected 13 open source LLMs and curated four versions of a dataset in a concrete interoperability use case. We performed three runs of each model with each version of the dataset, using two different strategies. Then we compared the effectiveness of the models and the consistency of their results across multiple runs. Results: qwen2.5-coder:32b was the most effective model using both strategies DIRECT (average pass@1 ≥ 0.99) and CODEGEN (average pass@1 ≥ 0.89) in three out of four dataset versions. In the fourth dataset version, which included an unit conversion, all models using the strategy DIRECT failed, whereas using CODEGEN qwen2.5-coder:32b succeeded with an average pass@1 = 0.75. Conclusion: Some LLMs can make systems interoperate autonomously. Further evaluation in different domains is recommended, and further research on reliability strategies should be conducted.
@inproceedings{abs-2510-23893,
title = {Evaluating the effectiveness of LLM-based interoperability},
author = {Rodrigo Falcão and Stefan Schweitzer and Julien Siebert and Emily Calvet and Frank Elberzhager},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773251}
}