Segmentation-Based Seam Cutting for High-Resolution 360-Degree Video Stitching

• Main Authors: Taeha Kim, Seongyeop Yang, Byeongkeun Kang, Heekyung Lee, Jeongil Seo, Yeejin Lee
• Format: Article
• Language: English
• Published: IEEE 2021-01-01
• Series: IEEE Access
• Subjects: Video stitching; image stitching; seam estimation; 360-degree video; instance segmentation; deep neural network
• Online Access: https://ieeexplore.ieee.org/document/9465807/

We present a novel segmentation-based seam cutting algorithm to generate visually plausible high-resolution 360-degree video efficiently. While the demand for an efficient video stitching algorithm for generating immersive videos has increased, it has received limited attention in the literature. Furthermore, stitched videos often suffer from distorted objects, temporal inconsistency and time constraints. Thus, in this paper, we propose an efficient seam finding algorithm that preserves objects from distortion, minimizes temporal inconsistency, and reduces processing time. One of the fundamental steps in image and video stitching is the estimation of seam boundary. To do this, the proposed algorithm leverages a convolutional neural networks-based instance segmentation algorithm that provides more accurate object regions. It computes energy surfaces considering the regions and then estimates seam boundary by discovering a minimal energy path with minimal computations. We validate the proposed algorithm using real-world high-resolution 360-degree sequences. The experimental results verify that the proposed algorithm can produce seam boundaries that avoid objects with better temporary consistency. The proposed algorithm reduces the number of pixels passed through objects by approximately 30% on average compared to the existing algorithms. The qualitative comparisons furthermore demonstrate that the proposed algorithm consistently produces more perceptually pleasing results.