TY - JOUR
T1 - Hyperparameter optimization in black-box image processing using differentiable proxies
AU - Tseng, Ethan
AU - Yu, Felix
AU - Yang, Yuting
AU - St. Arnaud, Karl
AU - Nowrouzezahrai, Derek
AU - Lalonde, Jean François
AU - Heide, Felix
AU - Mannan, Fahim
N1 - Publisher Copyright:
© 2019 Association for Computing Machinery.
PY - 2019/7
Y1 - 2019/7
N2 - Nearly every commodity imaging system we directly interact with, or indirectly rely on, leverages power efficient, application-adjustable black-box hardware image signal processing (ISPs) units, running either in dedicated hardware blocks, or as proprietary software modules on programmable hardware. The configuration parameters of these black-box ISPs often have complex interactions with the output image, and must be adjusted prior to deployment according to application-specific quality and performance metrics. Today, this search is commonly performed manually by "golden eye" experts or algorithm developers leveraging domain expertise. We present a fully automatic system to optimize the parameters of black-box hardware and software image processing pipelines according to any arbitrary (i.e., application-specific) metric. We leverage a differentiable mapping between the configuration space and evaluation metrics, parameterized by a convolutional neural network that we train in an end-to-end fashion with imaging hardware in-the-loop. Unlike prior art, our differentiable proxies allow for high-dimension parameter search with stochastic first-order optimizers, without explicitly modeling any lower-level image processing transformations. As such, we can efficiently optimize black-box image processing pipelines for a variety of imaging applications, reducing application-specific configuration times from months to hours. Our optimization method is fully automatic, even with black-box hardware in the loop.We validate our method on experimental data for real-time display applications, object detection, and extreme low-light imaging. The proposed approach outperforms manual search qualitatively and quantitatively for all domain-specific applications tested. When applied to traditional denoisers, we demonstrate that-just by changing hyperparameters-traditional algorithms can outperform recent deep learning methods by a substantial margin on recent benchmarks.
AB - Nearly every commodity imaging system we directly interact with, or indirectly rely on, leverages power efficient, application-adjustable black-box hardware image signal processing (ISPs) units, running either in dedicated hardware blocks, or as proprietary software modules on programmable hardware. The configuration parameters of these black-box ISPs often have complex interactions with the output image, and must be adjusted prior to deployment according to application-specific quality and performance metrics. Today, this search is commonly performed manually by "golden eye" experts or algorithm developers leveraging domain expertise. We present a fully automatic system to optimize the parameters of black-box hardware and software image processing pipelines according to any arbitrary (i.e., application-specific) metric. We leverage a differentiable mapping between the configuration space and evaluation metrics, parameterized by a convolutional neural network that we train in an end-to-end fashion with imaging hardware in-the-loop. Unlike prior art, our differentiable proxies allow for high-dimension parameter search with stochastic first-order optimizers, without explicitly modeling any lower-level image processing transformations. As such, we can efficiently optimize black-box image processing pipelines for a variety of imaging applications, reducing application-specific configuration times from months to hours. Our optimization method is fully automatic, even with black-box hardware in the loop.We validate our method on experimental data for real-time display applications, object detection, and extreme low-light imaging. The proposed approach outperforms manual search qualitatively and quantitatively for all domain-specific applications tested. When applied to traditional denoisers, we demonstrate that-just by changing hyperparameters-traditional algorithms can outperform recent deep learning methods by a substantial margin on recent benchmarks.
KW - Image processing
UR - http://www.scopus.com/inward/record.url?scp=85073886848&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85073886848&partnerID=8YFLogxK
U2 - 10.1145/3306346.3322996
DO - 10.1145/3306346.3322996
M3 - Article
AN - SCOPUS:85073886848
SN - 0730-0301
VL - 38
JO - ACM Transactions on Graphics
JF - ACM Transactions on Graphics
IS - 4
M1 - 27
ER -