Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution

Lanlan Liu, Jia Deng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Scopus citations

Abstract

We introduce Dynamic Deep Neural Networks (D 2 NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D 2 NN neurons are executed, and the particular subset is determined by the D 2 NN itself. By pruning unnecessary computation depending on input, D 2 NNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a D 2 NN augments a feed-forward deep neural network (directed acyclic graph of differentiable modules) with controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A D 2 NN is trained end to end. Both regular and controller modules in a D 2 NN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various D 2 NN architectures on image classification tasks, we demonstrate that D 2 NNs are general and flexible, and can effectively optimize accuracy-efficiency trade-offs.

Original languageEnglish (US)
Title of host publication32nd AAAI Conference on Artificial Intelligence, AAAI 2018
PublisherAAAI press
Pages3675-3682
Number of pages8
ISBN (Electronic)9781577358008
StatePublished - 2018
Externally publishedYes
Event32nd AAAI Conference on Artificial Intelligence, AAAI 2018 - New Orleans, United States
Duration: Feb 2 2018Feb 7 2018

Publication series

Name32nd AAAI Conference on Artificial Intelligence, AAAI 2018

Conference

Conference32nd AAAI Conference on Artificial Intelligence, AAAI 2018
CountryUnited States
CityNew Orleans
Period2/2/182/7/18

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution'. Together they form a unique fingerprint.

Cite this