一句话：提出首个面向 ML Notebook 的自动断言生成工具 NBTestGen 与回归测试框架 NBTest。
Jupyter Notebook 是数据科学家和 ML 工程师进行原型设计与实验的事实标准，但其测试支持有限，持续开发中的静默回归常被忽略，导致不可靠且结果难以复现。为支持更系统的 ML Notebook 测试，本文提出首个自动断言生成方法 NBTestGen，为数据处理、模型构建与模型评估步骤生成回归断言；同时提出首个回归测试框架 NBTest，作为 Jupyter 插件支持 cell 作用域、非侵入式断言，并集成 pytest 与 CI。在 585 个 Kaggle notebook 上，NBTestGen 生成 21542 条断言（平均 36.82 条/notebook），杀死 72.21% 的 ML 特定变异体，通过率 100%；在 531 个旧版本中检测出 69.68% 的历史回归。SHAP 将 NBTest 集成到 CI，17 名 ML 开发者的用户研究也表明其直观且有用。
Jupyter Notebooks have become the de-facto choice for data scientists and machine learning (ML) engineers for prototyping and experimenting with ML pipelines. Notebooks provide a rich interactive interface with support for code, data, and visualization in one place. However, notebooks provide limited support for testing. As a result, during continuous development, many silent (non-crashing) regressions often go unnoticed and make notebooks unreliable and results hard to reproduce. To enable more systematic testing of ML notebooks, we introduce NBTestGen – the first automated assertion generation approach for ML notebooks. NBTestGen generates regression-based assertions that check properties related to data processing, model building, and model evaluation steps in a typical ML notebook. To support systematic integration of such assertions in notebooks, we introduce the first regression testing framework (called NBTest) that can be used as a Jupyter plugin and allows developers to write cell-scoped assertions in notebooks. Three key features of such assertions are that they are 1) cell-scoped: they are linked to specific notebook cells and execute only after those cells are executed, 2) non-intrusive: they do not block notebook execution (in a Jupyter session), so that development can continue when they fail, and 3) they integrate with pytest and CI pipelines, allowing developers to easily do regression testing of their notebooks. We evaluate NBTestGen on a corpus of 585 notebooks from the popular Kaggle platform. NBTestGen generates a total of 21542 assertions (36.82 on average per notebook). The generated assertions kill 72.21% of ML-specific mutations, while maintaining a high passrate of 100.00%. We also show that NBTestGen can detect 69.68% of historical regressions in 531 older versions of Kaggle notebooks. A popular ML library, SHAP, integrated NBTest into their CI. Further, we perform a user study with 17 ML developers that shows that such users find NBTest highly intuitive and useful.
@inproceedings{paperbot12435,
title = {Automated Assertion Generation and Regression Testing for Machine Learning Notebooks},
author = {Yingao (Elaine) Yao and Vedant Nimje and Varun Viswanath and Saikat Dutta},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834390}
}