Evolving transferable neural pruning functions

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Structural design of neural networks is crucial for the success of deep learning. While most prior works in evolutionary learning aim at directly searching the structure of a network, few attempts have been made on another promising track, channel pruning, which recently has made major headway in designing efficient deep learning models. In fact, prior pruning methods adopt human-made pruning functions to score a channel's importance for channel pruning, which requires domain knowledge and could be sub-optimal. To this end, we pioneer the use of genetic programming (GP) to discover strong pruning metrics automatically. Specifically, we craft a novel design space to express high-quality and transferable pruning functions, which ensures an end-to-end evolution process where no manual modification is needed on the evolved functions for their transferability after evolution. Unlike prior methods, our approach can provide both compact pruned networks for efficient inference and novel closed-form pruning metrics which are mathematically explainable and thus generalizable to different pruning tasks. While the evolution is conducted on small datasets, our functions shows promising results when applied to more challenging datasets, different from those used in the evolution process. For example, on ILSVRC-2012, an evolved function achieves state-of-the-art pruning results.

Original languageEnglish (US)
Title of host publicationGECCO 2022 - Proceedings of the 2022 Genetic and Evolutionary Computation Conference
PublisherAssociation for Computing Machinery, Inc
Pages385-394
Number of pages10
ISBN (Electronic)9781450392372
DOIs
StatePublished - Jul 8 2022
Event2022 Genetic and Evolutionary Computation Conference, GECCO 2022 - Virtual, Online, United States
Duration: Jul 9 2022Jul 13 2022

Publication series

NameGECCO 2022 - Proceedings of the 2022 Genetic and Evolutionary Computation Conference

Conference

Conference2022 Genetic and Evolutionary Computation Conference, GECCO 2022
Country/TerritoryUnited States
CityVirtual, Online
Period7/9/227/13/22

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Software
  • Theoretical Computer Science

Keywords

  • Channel Pruning
  • Convolutional Neural Network
  • Efficient Deep Learning
  • Genetic Programming

Fingerprint

Dive into the research topics of 'Evolving transferable neural pruning functions'. Together they form a unique fingerprint.

Cite this