TY - GEN
T1 - Low Precision Representations for High Dimensional Models
AU - Saha, Rajarshi
AU - Pilanci, Mert
AU - Goldsmith, Andrea J.
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - The large memory footprint of high dimensional models require quantization to a lower precision for deployment on resource constrained edge devices. With this motivation, we consider the problems of learning a (i) linear regressor, and a (ii) linear classifier from a given training dataset, and quantizing the learned model parameters subject to a pre-specified bit-budget. The error metric is the prediction risk of the quantized model, and our proposed randomized embedding-based quantization methods attain near-optimal error while being computationally efficient. We provide fundamental bounds on the bit-budget constrained minimax risk that, together with our proposed algorithms, characterize the minimum threshold budget required to achieve a risk comparable to the unquantized setting. We also show the efficacy of our strategy by quantizing a two-layer ReLU neural network for non-linear regression. Numerical simulations show the improved performance of our proposed scheme as well as its closeness to the lower bound.
AB - The large memory footprint of high dimensional models require quantization to a lower precision for deployment on resource constrained edge devices. With this motivation, we consider the problems of learning a (i) linear regressor, and a (ii) linear classifier from a given training dataset, and quantizing the learned model parameters subject to a pre-specified bit-budget. The error metric is the prediction risk of the quantized model, and our proposed randomized embedding-based quantization methods attain near-optimal error while being computationally efficient. We provide fundamental bounds on the bit-budget constrained minimax risk that, together with our proposed algorithms, characterize the minimum threshold budget required to achieve a risk comparable to the unquantized setting. We also show the efficacy of our strategy by quantizing a two-layer ReLU neural network for non-linear regression. Numerical simulations show the improved performance of our proposed scheme as well as its closeness to the lower bound.
KW - Classification
KW - Minimax lower bounds
KW - Quantization
KW - Randomized Hadamard
KW - Regression
UR - http://www.scopus.com/inward/record.url?scp=85175517861&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85175517861&partnerID=8YFLogxK
U2 - 10.1109/ICASSP49357.2023.10095529
DO - 10.1109/ICASSP49357.2023.10095529
M3 - Conference contribution
AN - SCOPUS:85175517861
T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
BT - ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023
Y2 - 4 June 2023 through 10 June 2023
ER -