一句话：提出 LAMPION 框架，用保语义的变形程序变换探测 ML 程序分析模型的鲁棒性，并揭示 CodeBERT 的脆弱行为。
变形测试已广泛用于深度学习模型的鲁棒性评估，但现有方法主要面向图像处理与目标识别任务，难以直接迁移到以程序为输入的 ML 程序分析模型。本文提出 LAMPION，一个针对这类模型的变形测试框架：它对测试数据集施加保语义的变形变换，生成与原测试集语义等价、但在标识符或语法结构上不同的新代码片段，从而在保持正确输出不变的前提下探测模型行为。作者以当前最先进的 Code-to-Text 模型 CodeBERT（为给定 Java 方法生成 Javadoc 摘要）为目标进行评估，结果表明即使是简单变换也会显著影响目标模型的行为，除经典性能指标外还能提供关于模型推理依据的额外信息。
Metamorphic testing is a well-established testing technique that has been successfully applied in various domains, including testing deep learning models to assess their robustness against data noise or malicious input. Currently, metamorphic testing approaches for machine learning (ML) models focused on image processing and object recognition tasks. Hence, these approaches cannot be applied to ML targeting program analysis tasks. In this paper, we extend metamorphic testing approaches for ML models targeting software programs. We present LAMPION, a novel testing framework that applies (semantics preserving) metamorphic transformations on the test datasets. LAMPION produces new code snippets equivalent to the original test set but different in their identifiers or syntactic structure. We evaluate LAMPION against CodeBERT, a state-of-the-art ML model for Code-To-Text tasks that creates Javadoc summaries for given Java methods. Our results show that simple transformations significantly impact the target model behavior, providing additional information on the models reasoning apart from the classic performance metric.
@inproceedings{paperbot6589,
title = {Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations},
author = {Leonhard Applis and Annibale Panichella and Arie van Deursen},
booktitle = {ASE},
year = {2021},
doi = {10.1109/ase51524.2021.9678706}
}