Hyperparameter optimization in black-box image processing using differentiable proxies

Ethan Tseng, Felix Yu, Yuting Yang, Karl St. Arnaud, Derek Nowrouzezahrai, Jean François Lalonde, Felix Heide, Fahim Mannan

Research output: Contribution to journalArticlepeer-review

45 Scopus citations


Nearly every commodity imaging system we directly interact with, or indirectly rely on, leverages power efficient, application-adjustable black-box hardware image signal processing (ISPs) units, running either in dedicated hardware blocks, or as proprietary software modules on programmable hardware. The configuration parameters of these black-box ISPs often have complex interactions with the output image, and must be adjusted prior to deployment according to application-specific quality and performance metrics. Today, this search is commonly performed manually by "golden eye" experts or algorithm developers leveraging domain expertise. We present a fully automatic system to optimize the parameters of black-box hardware and software image processing pipelines according to any arbitrary (i.e., application-specific) metric. We leverage a differentiable mapping between the configuration space and evaluation metrics, parameterized by a convolutional neural network that we train in an end-to-end fashion with imaging hardware in-the-loop. Unlike prior art, our differentiable proxies allow for high-dimension parameter search with stochastic first-order optimizers, without explicitly modeling any lower-level image processing transformations. As such, we can efficiently optimize black-box image processing pipelines for a variety of imaging applications, reducing application-specific configuration times from months to hours. Our optimization method is fully automatic, even with black-box hardware in the loop.We validate our method on experimental data for real-time display applications, object detection, and extreme low-light imaging. The proposed approach outperforms manual search qualitatively and quantitatively for all domain-specific applications tested. When applied to traditional denoisers, we demonstrate that-just by changing hyperparameters-traditional algorithms can outperform recent deep learning methods by a substantial margin on recent benchmarks.

Original languageEnglish (US)
Article number27
JournalACM Transactions on Graphics
Issue number4
StatePublished - Jul 2019

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design


  • Image processing


Dive into the research topics of 'Hyperparameter optimization in black-box image processing using differentiable proxies'. Together they form a unique fingerprint.

Cite this