一句话：通过220名人类评估者与自动验证器，实证揭示深度学习测试生成器所产输入的 validity 状况及标签保持问题。
深度学习系统测试需要大规模且具代表性的测试集，以评估其能否泛化到训练数据之外。已有多种测试输入生成器（TIG）被提出来生成人工输入，通过触发误行为暴露 DL 系统问题。然而，这类生成输入可能无效，即无法被识别为输入域的一部分，从而导致不可靠的质量评估。自动验证器可减轻人工检查负担，但输入有效性本身难以形式化，因此也难以自动化。本文开展大规模实证研究，覆盖2种自动验证器、220名人类评估者、5种 TIG 和3个分类任务，从自动与人工两个角度考察生成输入的有效性，并检查期望标签是否保持。结果表明，自动验证器判定84%的人工生成输入有效，但其期望标签并不总是保持；自动验证器与人类评估者达到约78%的一致率，但在特征丰富的数据集上仍存在局限。
Testing Deep Learning (DL) based systems inherently requires large and representative test sets to evaluate whether DL systems generalise beyond their training datasets. Diverse Test Input Generators (TIGs) have been proposed to produce artificial inputs that expose issues of the DL systems by triggering misbehaviours. Unfortunately, such generated inputs may be invalid, i.e., not recognisable as part of the input domain, thus providing an unreliable quality assessment. Automated validators can ease the burden of manually checking the validity of inputs for human testers, although input validity is a concept difficult to formalise and, thus, automate. In this paper, we investigate to what extent TIGs can generate valid inputs, according to both automated and human validators. We conduct a large empirical study, involving 2 different automated validators, 220 human assessors, 5 different TIGs and 3 classification tasks. Our results show that 84% artificially generated inputs are valid, according to automated validators, but their expected label is not always preserved. Automated validators reach a good consensus with humans (78% accuracy), but still have limitations when dealing with feature-rich datasets.