Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks

Xiaoliang Dai, Hongxu Yin, Niraj K. Jha

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

Deep neural networks (DNNs) have become a widely deployed model for numerous machine learning applications. However, their fixed architecture, substantial training cost, and significant model redundancy make it difficult to efficiently update them to accommodate previously unseen data. To solve these problems, we propose an incremental learning framework based on a grow-and-prune neural network synthesis paradigm. When new data arrive, the neural network first grows new connections based on the gradients to increase the network capacity to accommodate new data. Then, the framework iteratively prunes away connections based on the magnitude of weights to enhance network compactness, and hence recover efficiency. Finally, the model rests at a lightweight DNN that is both ready for inference and suitable for future grow-and-prune updates. The proposed framework improves accuracy, shrinks network size, and significantly reduces the additional training cost for incoming data compared to conventional approaches, such as training from scratch and network fine-tuning. For the LeNet-300-100 (LeNet-5) neural network architectures derived for the MNIST dataset, the framework reduces training cost by up to 64% (67%), 63% (63%), and 69% (73%) compared to training from scratch, network fine-tuning, and grow-and-prune from scratch, respectively. For the ResNet-18 architecture derived for the ImageNet dataset (DeepSpeech2 for the AN4 dataset), the corresponding training cost reductions against training from scratch, network fine-tunning, and grow-and-prune from scratch are 64% (67%), 60% (62%), and 72% (71%), respectively. Our derived models contain fewer network parameters but achieve higher accuracy relative to conventional baselines.

Original languageEnglish (US)
JournalIEEE Transactions on Emerging Topics in Computing
DOIs
StateAccepted/In press - 2020

All Science Journal Classification (ASJC) codes

  • Computer Science (miscellaneous)
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications

Keywords

  • Computational modeling
  • Computer architecture
  • Data models
  • Deep learning
  • Logic gates
  • Neural networks
  • Task analysis
  • Training
  • grow-and-prune paradigm
  • incremental learning
  • machine learning
  • neural network

Fingerprint

Dive into the research topics of 'Incremental Learning Using a Grow-and-Prune Paradigm with Efficient Neural Networks'. Together they form a unique fingerprint.

Cite this