一句话：提出 SemSlicer，用大语言模型进行语义数据切片，无需预定义特征即可定位模型的系统性问题。
机器学习模型会犯错，但识别错误背后的系统性问题往往很困难。实践者通过错误分析、测试、审计和红队等方式形成模型问题假设，并用数据切片寻找相关样本；然而传统切片受限于已有特征和程序化切片函数。本文提出 SemSlicer 框架，支持语义数据切片：无需现有特征即可识别语义一致的切片。它利用大语言模型标注数据集，并根据用户定义的任意切片标准生成切片。评估表明，SemSlicer 能以低成本生成准确切片，在不同设计维度间灵活权衡，可靠识别表现不佳的数据切片，并帮助实践者发现反映系统性问题的有用切片。
Machine learning models make mistakes, yet sometimes it is difficult to identify the systematic problems behind the mistakes. Practitioners engage in various activities, including error analysis, testing, auditing, and red-teaming, to form hypotheses of what can go (or has gone) wrong with their models. To validate these hypotheses, practitioners employ data slicing to identify relevant examples. However, traditional data slicing is limited by available features and programmatic slicing functions. In this work, we propose SemSlicer, a framework that supports semantic data slicing, which identifies a semantically coherent slice, without the need for existing features. SemSlicer uses Large Language Models to annotate datasets and generate slices from any user-defined slicing criteria. We show that SemSlicer generates accurate slices with low cost, allows flexible trade-offs between different design dimensions, reliably identifies under-performing data slices, and helps practitioners identify useful data slices that reflect systematic problems.
@inproceedings{paperbot8426,
title = {What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing},
author = {Chenyang Yang and Yining Hong and Grace Lewis and Tongshuang Wu and Christian Kästner},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695033}
}