Convolutional Sparse Coding for RGB+NIR Imaging

Xuemei Hu, Felix Heide, Qionghai Dai, Gordon Wetzstein

Research output: Contribution to journalArticlepeer-review

30 Scopus citations


Emerging sensor designs increasingly rely on novel color filter arrays (CFAs) to sample the incident spectrum in unconventional ways. In particular, capturing a near-infrared (NIR) channel along with conventional RGB color is an exciting new imaging modality. RGB+NIR sensing has broad applications in computational photography, such as low-light denoising, it has applications in computer vision, such as facial recognition and tracking, and it paves the way toward low-cost single-sensor RGB and depth imaging using structured illumination. However, cost-effective commercial CFAs suffer from severe spectral cross talk. This cross talk represents a major challenge in high-quality RGB+NIR imaging, rendering existing spatially multiplexed sensor designs impractical. In this work, we introduce a new approach to RGB+NIR image reconstruction using learned convolutional sparse priors. We demonstrate high-quality color and NIR imaging for challenging scenes, even including high-frequency structured NIR illumination. The effectiveness of the proposed method is validated on a large data set of experimental captures, and simulated benchmark results which demonstrate that this work achieves unprecedented reconstruction quality.

Original languageEnglish (US)
Article number8170267
Pages (from-to)1611-1625
Number of pages15
JournalIEEE Transactions on Image Processing
Issue number4
StatePublished - Apr 2018
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design


  • Computational photography
  • convolutional sparse coding
  • structured illumination


Dive into the research topics of 'Convolutional Sparse Coding for RGB+NIR Imaging'. Together they form a unique fingerprint.

Cite this