一句话：评估三种LLM在真实Python库迁移任务上的效果，发现可正确迁移多数代码改动，但测试通过率仍有限。
库迁移需要开发者理解两个库的API、建立映射并完成代码转换，人工操作耗时且易错。本文研究LLM在Python库迁移中的有效性，选取Llama 3.1、GPT-4o mini和GPT-4o，在PyMigBench的321个真实世界迁移、2989处迁移相关代码改动上进行评估。作者通过将LLM迁移结果与开发者迁移代码对比，并运行客户端仓库中的单元测试来衡量正确性。结果显示，三种模型分别正确迁移了89%、89%和94%的代码改动，但分别只有36%、52%和64%的迁移通过开发者迁移中通过的相同测试。为避免模型只是背诵已知迁移，作者还在10个从未发生迁移的新仓库上评估。总体而言，LLM可有效辅助库迁移，但仍存在开放挑战。
Library migration is the process of replacing one library with another library that provides similar functionality. Manual library migration is time consuming and error prone, as it requires developers to understand the APIs of both libraries, map them, and perform the necessary code transformations. Large Language Models (LLMs) are shown to be effective at generating and transforming code as well as finding similar code, which are necessary upstream tasks for library migration. Such capabilities suggest that LLMs may be suitable for library migration. Accordingly, this paper investigates the effectiveness of LLMs for migration between Python libraries. We evaluate three LLMs, LLama 3.1, GPT-4o mini, and GPT-4o on PyMigBench, where we migrate 321 real-world library migrations that include 2,989 migration-related code changes. To measure correctness, we (1) compare the LLM’s migrated code with the developers’ migrated code in the benchmark and (2) run the unit tests available in the client repositories. We find that LLama 3.1, GPT-4o mini, and GPT-4o correctly migrate 89%, 89%, and 94% of the migration-related code changes, respectively. We also find that 36%, 52% and 64% of the LLama 3.1, GPT-4o mini, and GPT-4o migrations pass the same tests that passed in the developer’s migration. To ensure the LLMs are not reciting the migrations, we also evaluate them on 10 new repositories where the migration never happened. Overall, our results suggest that LLMs can be effective in migrating code between libraries, but we also identify some open challenges.
DOI 原文 ·
@inproceedings{paperbot9570,
title = {An Empirical Study of Python Library Migration Using Large Language Models},
author = {Mohayeminul Islam and Ajay Kumar Jha and May Mahmoud and Ildar Akhmetov and Sarah Nadi},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00077}
}