一句话：重新审视 AI 系统的测试基础，提出统计化可信性保证、测试过程要素与运行时监督，并连接测试与运行。
随着深度学习发展，AI 组件在图像、视频、文本和音频等任务上达到空前性能，并被集成进自主驾驶等需要认证可信性保证的复杂软件系统。本报告分析基于 AI 系统的独特特性及其故障类型，据此重新审视测试基础并重新定义测试目标，从统计视角讨论实际可提供的可信性保证。报告讨论修订后的测试过程关键要素，包括测试预言与测试输入生成，并引入运行时监督以处理现场意外错误。最后指出从测试到运行闭环所需的未来步骤，提出将测试输出重新连接到原始目标的实证框架。该工作为报告/路线图性质，未报告具体实验评估。
With the advent of deep learning, AI components have achieved unprecedented performance on complex, human competitive tasks, such as image, video, text and audio processing. Hence, they are increasingly integrated into sophisticated software systems, some of which (e.g., autonomous vehicles) are required to deliver certified dependability warranties. In this talk, I will consider the unique features of AI based systems and of the faults possibly affecting them, in order to revise the testing fundamentals and redefine the overall goal of testing, taking a statistical view on the dependability warranties that can be actually delivered. Then, I will consider the key elements of a revised testing process for AI based systems, including the test oracle and the test input generation problems. I will also introduce the notion of runtime supervision, to deal with unexpected error conditions that may occur in the field. Finally, I will identify the future steps that are essential to close the loop from testing to operation, proposing an empirical framework that reconnects the output of testing to its original goals.