Convolutional neural networks with low-rank regularization

Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, E. Weinan

Research output: Contribution to conferencePaper

Abstract

Large CNNs have delivered impressive performance in various computer vision applications. But the storage and computation requirements make it problematic for deploying these models on mobile devices. Recently, tensor decompositions have been used for speeding up CNNs. In this paper, we further develop the tensor decomposition technique. We propose a new algorithm for computing the low-rank tensor decomposition for removing the redundancy in the convolution kernels. The algorithm finds the exact global optimizer of the decomposition and is more effective than iterative methods. Based on the decomposition, we further propose a new method for training low-rank constrained CNNs from scratch. Interestingly, while achieving a significant speedup, sometimes the low-rank constrained CNNs delivers significantly better performance than their non-constrained counterparts. On the CIFAR-10 dataset, the proposed low-rank NIN model achieves 91.31% accuracy (without data augmentation), which also improves upon state-of-the-art result. We evaluated the proposed method on CIFAR-10 and ILSVRC12 datasets for a variety of modern CNNs, including AlexNet, NIN, VGG and GoogleNet with success. For example, the forward time of VGG-16 is reduced by half while the performance is still comparable. Empirical success suggests that low-rank tensor decompositions can be a very useful tool for speeding up large CNNs.

Original languageEnglish (US)
StatePublished - Jan 1 2016
Event4th International Conference on Learning Representations, ICLR 2016 - San Juan, Puerto Rico
Duration: May 2 2016May 4 2016

Conference

Conference4th International Conference on Learning Representations, ICLR 2016
CountryPuerto Rico
CitySan Juan
Period5/2/165/4/16

All Science Journal Classification (ASJC) codes

  • Education
  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint Dive into the research topics of 'Convolutional neural networks with low-rank regularization'. Together they form a unique fingerprint.

  • Cite this

    Tai, C., Xiao, T., Zhang, Y., Wang, X., & Weinan, E. (2016). Convolutional neural networks with low-rank regularization. Paper presented at 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico.