一句话：提出 MARTENS 框架，将 Isaac Sim 与进化搜索结合，在仿真中测试并修复机械臂深度学习视觉模型。
针对自主机械臂中视觉与控制交互复杂、真实数据难采集且标注昂贵，导致深度学习物体检测模型难以测试和改进的问题，本文提出 MARTENS 框架，将照片级 NVIDIA Isaac Sim 仿真器与进化搜索结合，自动识别关键测试场景，以提升视觉模型并暴露系统设计缺陷。在两个工业案例中，MARTENS 相比随机测试多发现 25%–50% 的失败且多样性更高；经其训练和修复的模型在未用真实数据重训时 mAP 达 0.91 和 0.82，少量微调（<10 epoch）后提升至 0.95 和 0.89。仅用真实数据训练的基线分别需超过 25 epoch 才达 0.8 和 0.75。
Testing autonomous robotic manipulators is challenging due to the complex software interactions between vision and control components. A crucial element of modern robotic manipulators is the deep learning based object detection model. The creation and assessment of this model requires real world data, which can be hard to label and collect, especially when the hardware setup is not available. The current techniques primarily focus on using synthetic data to train deep neural networks (DDNs) and identifying failures through offline or online simulation-based testing. However, the process of exploiting the identified failures to uncover design flaws early on, and leveraging the optimized DNN within the simulation to accelerate the engineering of the DNN for real-world tasks remains unclear. To address these challenges, we propose the MARTENS (Manipulator Robot Testing and Enhancement in Simulation) framework, which integrates a photorealistic NVIDIA Isaac Sim simulator with evolutionary search to identify critical scenarios aiming at improving the deep learning vision model and uncovering system design flaws. Evaluation of two industrial case studies demonstrated that MARTENS effectively reveals robotic manipulator system failures, detecting 25% to 50% more failures with greater diversity compared to random test generation. The model trained and repaired using the MARTENS approach achieved mean average precision (mAP) scores of 0.91 and 0.82 on real-world images with no prior retraining. Further fine-tuning on real-world images for a few epochs (less than 10) increased the mAP to 0.95 and 0.89 for the first and second use cases, respectively. In contrast, a model trained solely on real-world data achieved mAPs of 0.8 and 0.75 for use case 1 and use case 2 after more than 25 epochs.
@inproceedings{paperbot8310,
title = {In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators},
author = {Dmytro Humeniuk and Houssem Ben Braiek and Thomas Reid and Foutse Khomh},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695281}
}