Algorithm and Architecture Design of Free-viewpointView Synthesis Engine

• Main Authors: Ku-Chu Wei, 魏谷竹
• Other Authors: 簡韶逸
• Format: Others
• Language: en_US
• Published: 2013
• Online Access: http://ndltd.ncl.edu.tw/handle/40252109313159461321

碩士 === 國立臺灣大學 === 電子工程學研究所 === 101 === Free-viewpoint TV (FTV) is considered to be the next-generation TV. Compared to conventional 3D TV, free-viewpoint TV can provide infinite viewpoints, a fully 6DoF transformation. However, only limited views can be captured by real-world camera, and free-viewpoint view synthesis is the key technology for such application: synthesize the video with arbitrary viewpoint. The target of this thesis is to propose a free-viewpoint view synthesis engine for free-viewpoint TV. The design space is from algorithm level to architecture level. In the literature, free-viewpoint view synthesis is composed of depth-image-based rendering followed by hole-filling. To realize view synthesis in practical systems, the efficiency of view synthesis must be considered to achieve a good balance between the image quality and the computational complexity. We propose an efficient DIBR implementation that, when compared to state-of-the-art backward warping, requires only half of the computation complexity with comparable quality. Specifically, the proposed scheme uses ray casting and pull-push processing to render in one pass, which can be regarded as applying 3D filters in the depth-image-based rendering. Due to the imperfect input depth maps, noticeable temporal fluctuation can be perceived when playing back the synthesized video. To cope with the problem, a robust temporal filtering technique is also integrated into the view synthesis scheme to eliminate such a temporal fluctuation. With the proposed one-pass depth-image-based rendering flow, the following hardware architecture design can be benefited, since the one-pass flow is suitable for pipeline design. Moreover, a simple depth-aware convolution-based inpainting, where the weight is the combination of the distance and the depth value, is employed to achieve smoother data flow for hardware design with lower cost. The main design challenge of free-viewpoint view synthesis engine is the huge bandwidth requirement and the highly random access to DRAM in depth-image-based rendering. In our design, block-based pipeline is utilized to reduce the huge bandwidth with small on-chip SRAM. To reduce the random access problem, an extra module, pre-warping, is added to pre-fetch the blocks that DIBR requires only. With the combination of pre-warping and carefully considered block-based design, the free-viewpoint view synthesis engine achieves a nice trade-off among bandwidth, on-chip memory, and computational resources.