一句话：提出完全非侵入的多模态机器人框架 RFNIT，用外部感知与机械执行对手机 GUI 做拟人化物理交互测试。
移动应用的自动化 GUI 测试常需复现真实用户交互，但插桩或 ADB 等系统级手段具有侵入性，限制了在真机上的适用范围；已有的 DLD、R-DLD 等方法虽能检测活动重启导致的数据丢失，却仍依赖系统级访问。本博士研究提出 RFNIT——一个完全非侵入的多模态机器人框架，用于智能手机上的物理 GUI 交互测试。RFNIT 通过外部传感感知设备状态并由机器人执行物理手势，实现端到端自动化：视觉感知模块结合外部摄像头图像与 OCR 文本识别 GUI 元素，LLM 驱动的推理代理据此决定后续测试动作，机械执行器完成点击、滑动与打字等手势。初步结果表明，系统可从外部摄像头输入准确识别按钮、文本框与虚拟键盘，视觉检测准确率超过 85%。后续工作将聚焦于提升长时间测试会话的鲁棒性、可重复性与 GUI 状态覆盖。
Automated testing of mobile applications remains challenging when realistic user interactions must be reproduced and GUI behavior verified without relying on invasive mechanisms such as instrumentation or ADB-level control. Existing approaches, including DLD and R-DLD, detect data loss triggered by activity restarts but still depend on system-level access, which limits their applicability on real devices. This doctoral research proposes RFNIT, a multi-modal, fully non-invasive robotic framework for physical GUI interaction testing on smartphones. RFNIT supports end-to-end automation by perceiving the device state through external sensing and executing physical interactions, enabling human-like actions while preserving reproducibility and execution accuracy. The framework integrates a vision-based perception module, an LLM-driven reasoning agent, and a robotic actuator capable of performing touch, swipe, and typing gestures. Multimodal data, including external camera images and OCR-extracted text, are combined to identify GUI elements and guide subsequent testing decisions. Preliminary results indicate that RFNIT accurately recognizes buttons, text fields, and virtual keyboards from external camera input, achieving over 85% visual detection accuracy. Ongoing and future work will focus on improving robustness, repeatability, and coverage of GUI states during long-running testing sessions.