We introduce Dynamic Deep Neural Networks (D 2 NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D 2 NN neurons are executed, and the particular subset is determined by the D 2 NN itself. By pruning unnecessary computation depending on input, D 2 NNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a D 2 NN augments a feed-forward deep neural network (directed acyclic graph of differentiable modules) with controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A D 2 NN is trained end to end. Both regular and controller modules in a D 2 NN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various D 2 NN architectures on image classification tasks, we demonstrate that D 2 NNs are general and flexible, and can effectively optimize accuracy-efficiency trade-offs.