ExtendAIST: Exploring the Space of AI-in-the-Loop System Testing

• Main Authors: Tingting Wu, Yunwei Dong, Yu Zhang, Aziz Singa
• Format: Article
• Language: English
• Published: MDPI AG 2020-01-01
• Series: Applied Sciences
• Subjects: ai-in-the-loop system; machine learning; ais testing; ais testing strategy
• Online Access: https://www.mdpi.com/2076-3417/10/2/518

The AI-in-the-loop system (AIS) has been widely used in various autonomous decision and control systems, such as computing vision, autonomous vehicle, and collision avoidance systems. AIS generates and updates control strategies through learning algorithms, which make the control behaviors non-deterministic and bring about the test oracle problem in AIS testing procedure. The traditional system mainly concerns about properties of safety, reliability, and real-time, while AIS concerns more about the correctness, robustness, and stiffness of system. To perform an AIS testing with the existing testing techniques according to the testing requirements, this paper presents an extendable framework of AI-in-the-loop system testing by exploring the key steps involved in the testing procedure, named ExtendAIST, which contributes to define the execution steps of ExtendAIST and design space of testing techniques. Furthermore, the ExtendAIST framework provides three concerns for AIS testing, which include: (a) the extension points; (b) sub-extension points; and (c) existing techniques commonly present in each point. Therefore, testers can obtain the testing strategy using existing techniques directly for corresponding testing requirements or extend more techniques based on these extension points.