TY - JOUR
T1 - A Novel Framework for the Analysis and Design of Heterogeneous Federated Learning
AU - Wang, Jianyu
AU - Liu, Qinghua
AU - Liang, Hao
AU - Gauri, Joshi
AU - Poor, H. Vincent
N1 - Funding Information:
Manuscript received December 29, 2020; revised April 22, 2021 and July 8, 2021; accepted August 3, 2021. Date of publication August 24, 2021; date of current version September 24, 2021. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Ketan Rajawat. This work was supported in part by NSF under Grants CCF-1850029 and CCF-2045694, in part by the 2018 IBM Faculty Research Award, and in part by the Qualcomm Innovation fellowship. (Corresponding author: Jianyu Wang.) Jianyu Wang and Gauri Joshi are with the Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA 15213 USA (e-mail: jianyuw1@andrew.cmu.edu; gaurij@andrew.cmu.edu).
Publisher Copyright:
© 1991-2012 IEEE.
PY - 2021
Y1 - 2021
N2 - In federated learning, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated optimization algorithms with heterogeneous local training progress at clients. The analyses are conducted for both smooth non-convex and strongly convex settings, and can also be extended to partial client participation case. Additionally, it subsumes previously proposed methods such as FedAvg and FedProx, and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.
AB - In federated learning, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated optimization algorithms with heterogeneous local training progress at clients. The analyses are conducted for both smooth non-convex and strongly convex settings, and can also be extended to partial client participation case. Additionally, it subsumes previously proposed methods such as FedAvg and FedProx, and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.
KW - Federated learning
KW - distributed optimization
UR - http://www.scopus.com/inward/record.url?scp=85113902031&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85113902031&partnerID=8YFLogxK
U2 - 10.1109/TSP.2021.3106104
DO - 10.1109/TSP.2021.3106104
M3 - Article
AN - SCOPUS:85113902031
SN - 1053-587X
VL - 69
SP - 5234
EP - 5249
JO - IEEE Transactions on Signal Processing
JF - IEEE Transactions on Signal Processing
ER -