Deep Learning in Next-Frame Prediction: A Benchmark Review

• Main Authors: Yufan Zhou, Haiwei Dong, Abdulmotaleb El Saddik
• Format: Article
• Language: English
• Published: IEEE 2020-01-01
• Series: IEEE Access
• Subjects: Frame prediction architecture; loss function design; state-of-the-art evaluation
• Online Access: https://ieeexplore.ieee.org/document/9063513/

As an unsupervised representation problem in deep learning, next-frame prediction is a new, promising direction of research in computer vision, predicting possible future images by presenting historical image information. It provides extensive application value in robot decision making and autonomous driving. In this paper, we introduce recent state-of-the-art next-frame prediction networks and categorize them into two architectures: sequence-to-one architecture and sequence-to-sequence architecture. After comparing these approaches by analyzing the network architecture and loss function design, the pros and cons are analyzed. Based on the off-the-shelf data-sets and the corresponding evaluation metrics, the performance of the aforementioned approaches is quantitatively compared. The future promising research directions are pointed out at last.