一句话:提出自动故障注入平台,统一注入与数据采集,支持多云微服务并学习错误传播以辅助故障定位。
现代 IT 系统复杂度不断上升、故障代价高昂,利用 AI/ML 提升系统韧性已成为迫切需求,但构建 AIOps 模型所需的数据生成仍很困难。本文提出一个自动故障注入平台,将故障注入与数据采集统一在同一框架中,支持混合云与多云环境,使用时无需编程技能,并覆盖应用层和基础设施层的常见故障类型。平台还集成 AI 能力,特别是干预式因果学习。作者在云环境微服务应用上演示:当微服务之间的通信图未知且只有日志可用时,系统能够学习错误传播模型,并用于故障定位等后续任务。
In today’s IT environment with a growing number of costly outages, increasing complexity of the systems, and availability of massive operational data, there is a strengthening demand to effectively leverage Artificial Intelligence and Machine Learning (AI/ML) towards enhanced resiliency. In this paper, we present an automatic fault injection platform to enable and optimize the generation of data needed for building AI/ML models to support modern IT operations. The merits of our platform include the ease of use, the possibility to orchestrate complex fault scenarios and to optimize the data generation for the modeling task at hand. Specifically, we designed a fault injection service that (i) combines fault injection with data collection in a unified framework, (ii) supports hybrid and multi-cloud environments, and (iii) does not require programming skills for its use. Our current implementation covers the most common fault types both at the application and infrastructure levels. The platform also includes some AI capabilities. In particular, we demonstrate the interventional causal learning capability currently available in our platform. We show how our system is able to learn a model of error propagation in a micro-service application in a cloud environment (when the communication graph among micro-services is unknown and only logs are available) for use in subsequent applications such as fault localization.
@inproceedings{paperbot7346,
title = {A fault injection platform for learning AIOps models},
author = {Frank Bagehorn and Jesus Rios and Saurabh Jha and Robert Filepp and Larisa Shwartz and Naoki Abe and Xi Yang},
booktitle = {ASE},
year = {2022},
doi = {10.1145/3551349.3559503}
}