Multi-Dimensional Dynamic Model Compression for Efficient Image Super-Resolution

Zejiang Hou, Sun Yuan Kung

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Modern single image super-resolution (SR) system based on convolutional neural networks achieves substantial progress. However, most SR deep networks are computationally expensive and require excessively large activation memory footprints, impeding their effective deployment to resource-limited devices. Based on the observation that the activation patterns in SR networks exhibit high input-dependency, we propose Multi-Dimensional Dynamic Model Compression method that can reduce both spatial and channel wise redundancy in an SR deep network for different input images. To reduce the spatial-wise redundancy, we propose to perform convolution on scaled-down feature-maps where the down-scaling factor is made adaptive to different input images. To reduce the channel-wise redundancy, we introduce a low-cost channel saliency predictor for each convolution to dynamically skip the computation of unimportant channels based on the Gumbel-Softmax. To better capture the feature-maps information and facilitate input-adaptive decision, we employ classic image processing metrics, e.g., Spatial Information, to guide the saliency predictors. The proposed method can be readily applied to a variety of SR deep networks and trained end-to-end with standard super-resolution loss, in combination with a sparsity criterion. Experiments on several benchmarks demonstrate that our method can effectively reduce the FLOPs of both lightweight and non-compact SR models with negligible PSNR loss. Moreover, our compressed models achieve competitive PSNR-FLOPs Pareto frontier compared with SOTA NAS-based SR methods.

Original languageEnglish (US)
Title of host publicationProceedings - 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3492-3502
Number of pages11
ISBN (Electronic)9781665409155
DOIs
StatePublished - 2022
Event22nd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022 - Waikoloa, United States
Duration: Jan 4 2022Jan 8 2022

Publication series

NameProceedings - 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022

Conference

Conference22nd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022
Country/TerritoryUnited States
CityWaikoloa
Period1/4/221/8/22

All Science Journal Classification (ASJC) codes

  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Keywords

  • Deep Learning Deep Learning
  • Efficient Training and Inference Methods for Networks

Fingerprint

Dive into the research topics of 'Multi-Dimensional Dynamic Model Compression for Efficient Image Super-Resolution'. Together they form a unique fingerprint.

Cite this