Scalable training of 3D convolutional networks on multi- and many-cores

Aleksandar Zlateski, Kisuk Lee, Hyunjune Sebastian Seung

Research output: Contribution to journalArticlepeer-review

7 Scopus citations


Convolutional networks (ConvNets) have become a popular approach to computer vision. Here we consider the parallelization of ConvNet training, which is computationally costly. Our novel parallel algorithm is based on decomposition into a set of tasks, most of which are convolutions or FFTs. Theoretical analysis suggests that linear speedup with the number of processors is attainable. To attain such performance on real shared-memory machines, our algorithm computes convolutions converging on the same node of the network with temporal locality to reduce cache misses, and sums the convergent convolution outputs via an almost wait-free concurrent method to reduce time spent in critical sections. Benchmarking with multi-core CPUs shows speedup roughly equal to the number of physical cores. We also demonstrate 90x speedup on a many-core CPU (Xeon Phi Knights Corner). Our algorithm can be either faster or slower than certain GPU implementations depending on specifics of the network architecture, kernel sizes, and density and size of the output patch.

Original languageEnglish (US)
Pages (from-to)195-204
Number of pages10
JournalJournal of Parallel and Distributed Computing
StatePublished - Aug 1 2017

All Science Journal Classification (ASJC) codes

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture
  • Computer Networks and Communications
  • Artificial Intelligence


  • Convolutional neural networks
  • Deep learning
  • Dynamic scheduling
  • FFT convolution
  • Multi-core algorithms
  • Wait-free summation


Dive into the research topics of 'Scalable training of 3D convolutional networks on multi- and many-cores'. Together they form a unique fingerprint.

Cite this