一句话：首个针对 TensorFlow 与 PyTorch 的深度学习 API 误用大规模实证研究，并提出基于 LLM 的检测与修复方法 LLMAPIDet。
深度学习库的开发范式与传统软件差异显著，开发者在使用 DL API 时频繁出错，但已有 API 误用研究主要面向传统软件，对 DL API 误用的认识存在空白。本文首次系统研究 TensorFlow 与 PyTorch 中的 DL API 误用：从这两个库最热门的 200 个项目中收集 4,224 个 commit，人工确认 891 个 API 误用，并从误用类型、根因与症状三个维度刻画其特征；同时评估现有最先进的 API 误用检测器，发现其在该数据集上完全失效。为此作者提出 LLMAPIDet，通过在大规模误用样本上对 ChatGPT 提示链进行 prompt tuning 来实现 DL API 误用的检测与修复。在留出的 291 个误用测试集上，LLMAPIDet 检出 48 个而现有检测器一个都未检出；在 10 个最新 GitHub 项目上又发现 119 个此前未知的误用，并成功修复其中 46 个。
Deep Learning (DL) libraries have significantly impacted various domains in computer science over the last decade. However, developers often face challenges when using the DL APIs, as the development paradigm of DL applications differs greatly from traditional software development. Existing studies on API misuse mainly focus on traditional software, leaving a gap in understanding API misuse within DL APIs. To address this gap, we present the first comprehensive study of DL API misuse in TensorFlow and PyTorch. Specifically, we first collected a dataset of 4,224 commits from the top 200 most-starred projects using these two libraries and manually identified 891 API misuses. We then investigated the characteristics of these misuses from three perspectives, i.e., types, root causes, and symptoms. We have also conducted an evaluation to assess the effectiveness of the current state-of-the-art API misuse detector on our 891 confirmed API misuses. Our results confirmed that the state-of-the-art API misuse detector is ineffective in detecting DL API misuses. To address the limitations of existing API misuse detection for DL APIs, we propose LLMAPIDet, which leverages Large Language Models (LLMs) for DL API misuse detection and repair. We build LLMAPIDet by prompt-tuning a chain of ChatGPT prompts on 600 out of 891 confirmed API misuses and reserve the rest 291 API misuses as the testing dataset. Our evaluation shows that LLMAPIDet can detect 48 out of the 291 DL API misuses while none of them can be detected by the existing API misuse detector. We further evaluate LLMAPIDet on the latest versions of 10 GitHub projects. The evaluation shows that LLMAPIDet can identify 119 previously unknown API misuses and successfully fix 46 of them.
@inproceedings{paperbot11334,
title = {Demystifying and Detecting Misuses of Deep Learning APIs},
author = {Moshi Wei and Nima Shiri Harzevili and Yuekai Huang and Jinqiu Yang and Junjie Wang and Song Wang},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639177}
}