一句话：用领域知识感知提示引导LLM生成测试用例，自动发现深度学习模型优化器中的缺陷。
模型优化（如剪枝、量化）是深度学习模型部署前的关键阶段，但优化器中的模型优化缺陷（MOB）涉及复杂数据类型与层结构，传统静态分析和动态测试难以检测。本文提出用大语言模型加提示技术生成测试用例来检测MOB，并探索“聚焦与扩散”范式：从已知MOB的issue报告与修复等工件中提取代码结构和错误描述等领域知识，形成对问题域的精确刻画（聚焦），再以此为启发式方向引导LLM生成创新测试用例（扩散）。该思想实现为工具YanHui，并与现有基于少样本LLM的模糊测试技术对比。结果显示，YanHui生成的测试用例能更好地组合相关API与数据以暴露MOB，生成语法有效代码的比例提升11.4%，针对模型优化的on-target代码提升22.3%；共检测出17个MOB，其中5个为深层MOB，若无该提示技术则难以揭示。
Model optimization, such as pruning and quantization, has become the de facto pre-deployment phase when deploying deep learning (DL) models on resource-constrained platforms. However, the complexity of DL models often leads to non-trivial bugs in model optimizers, known as model optimization bugs (MOBs). These MOBs are characterized by involving complex data types and layer structures inherent to DL models, causing significant hurdles in detecting them through traditional static analysis and dynamic testing techniques. In this work, we leverage Large Language Models (LLMs) with prompting techniques to generate test cases for MOB detection. We explore how LLMs can draw an understanding of the MOB domain from scattered bug instances and generalize to detect new ones, a paradigm we term as concentration and diffusion. We extract MOB domain knowledge from the artifacts of known MOBs, such as their issue reports and fixes, and design knowledge-aware prompts to guide LLMs in generating effective test cases. The domain knowledge of code structure and error description provides precise in-depth depictions of the problem domain, i.e., the concentration, and heuristic directions to generate innovative test cases, i.e., the diffusion. Our approach is implemented as a tool named YanHui and benchmarked against existing few-shot LLM-based fuzzing techniques. Test cases generated by YanHui demonstrate enhanced capability to find relevant API and data combinations for exposing MOBs, leading to an 11.4% increase in generating syntactically valid code and a 22.3% increase in generating on-target code specific to model optimization. YanHui detects 17 MOBs, and among them, five are deep MOBs that are difficult to reveal without our prompting technique.
@inproceedings{paperbot8597,
title = {Large Language Models Can Connect the Dots: Exploring Model Optimization Bugs with Domain Knowledge-Aware Prompts},
author = {Hao Guan and Guangdong Bai and Yepang Liu},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680383}
}