Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution

We introduce Dynamic Deep Neural Networks (D ² NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D ² NN neurons are executed, and the particular subset is determined by the D ² NN itself. By pruning unnecessary computation depending on input, D ² NNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a D ² NN augments a feed-forward deep neural network (directed acyclic graph of differentiable modules) with controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A D ² NN is trained end to end. Both regular and controller modules in a D ² NN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various D ² NN architectures on image classification tasks, we demonstrate that D ² NNs are general and flexible, and can effectively optimize accuracy-efficiency trade-offs.