Convergence Time Minimization for Federated Reinforcement Learning over Wireless Networks

Sihua Wang, Mingzhe Chen, Changchuan Yin, H. Vincent Poor

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, the convergence time of federated reinforcement learning (FRL) that is deployed over a realistic wireless network is studied. In the considered model, several devices and the base station (BS) jointly participate in the iterative training of an FRL algorithm. Due to limited wireless resources, the BS must select a subset of devices to exchange FRL training parameters at each iteration, which will significantly affect the training loss and convergence time of the considered FRL algorithm. This joint learning, wireless resource allocation, and device selection problem is formulated as an optimization problem aiming to minimize the FRL convergence time while meeting the FRL temporal difference (TD) error requirement. To solve this problem, a deep Q network based algorithm is designed. The proposed method enables the BS to dynamically select an appropriate subset of devices to join the FRL training. Given the selected devices, a resource block allocation scheme can be derived to further minimize the FRL convergence time. Simulation results with real data show that the proposed approach can reduce the FRL convergence time by up to 44.7% compared to a baseline that randomly determines the subset of participating devices and their occupied resource blocks.

Original languageEnglish (US)
Title of host publication2022 56th Annual Conference on Information Sciences and Systems, CISS 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages246-251
Number of pages6
ISBN (Electronic)9781665417969
DOIs
StatePublished - 2022
Event56th Annual Conference on Information Sciences and Systems, CISS 2022 - Princeton, United States
Duration: Mar 9 2022Mar 11 2022

Publication series

Name2022 56th Annual Conference on Information Sciences and Systems, CISS 2022

Conference

Conference56th Annual Conference on Information Sciences and Systems, CISS 2022
Country/TerritoryUnited States
CityPrinceton
Period3/9/223/11/22

All Science Journal Classification (ASJC) codes

  • Information Systems and Management
  • Artificial Intelligence
  • Computer Science Applications
  • Information Systems

Fingerprint

Dive into the research topics of 'Convergence Time Minimization for Federated Reinforcement Learning over Wireless Networks'. Together they form a unique fingerprint.

Cite this