Abstract

High-resolution (HR) videos have become popular due to the widespread adoption of high-definition displays. Super-resolution (SR) techniques aim to recover HR frames from low-resolution (LR) frames. Recently, deep neural network (DNN)-based SR methods have achieved superior quality compared to traditional methods. FPGA-based SR accelerators have been proposed to optimize performance and power efficiency. However, most accelerators tailored for video SR only accept uncompressed video frames and operate per-frame DNN inference, ignoring the temporal-spatial information in compressed video bitstreams. In contrast, we observe that non-key frames can be directly constructed using codec information and HR key-frames, saving a significant amount of DNN computing. In this paper, we propose a novel compressed video SR flow and a specific FPGA accelerator called Co-ViSu that integrates decoder, SR, and encoder engines. Co-ViSu exploits codec information reuse scheme to skip non-key frame decoding, avoid complex DNN computation and speed up encoding. Our experimental results show that Co-ViSu achieves 3.6x to 9.4x performance, 4.2x energy efficiency gain with only 0.17dB quality loss compared to the traditional flow, and 2.1x throughput than state-of-the-art.