Reinforcement Learning for Minimizing Age of Information under Realistic Physical Dynamics

Sihua Wang, Mingzhe Chen, Walid Saad, Changchuan Yin, Shuguang Cui, H. Vincent Poor

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, the problem of minimizing the weighted sum of age of information (AoI) and total energy consumption of Internet of Things (IoT) devices is studied. In particular, each IoT device monitors a physical process that follows nonlinear dynamics. As the dynamic of the physical process varies over time, each device must sample the real-time status of the physical system and send the status information to a base station (BS) so as to monitor the physical process. The dynamics of the realistic physical process will influence the sampling frequency and status update scheme of each device. In particular, as the physical process varies rapidly, the sampling frequency of each device must be increased to capture these physical dynamics. Meanwhile, changes in the sampling frequency will also impact the energy usage of the device. Thus, it is necessary to determine a subset of devices to sample the physical process at each time slot so as to accurately monitor the dynamics of the physical process using minimum energy. This problem is formulated as an optimization problem whose goal is to minimize the weighted sum of AoI and total device energy consumption. To solve this problem, a machine learning framework based on the repeated update Q-learning (RUQL) algorithm is proposed. The proposed method enables the BS to overcome the biased action selection problem (e.g., an agent always takes a subset of actions while ignoring other actions), and hence, dynamically and quickly finding a device sampling and status update policy so as to minimize the sum of AoI and energy consumption of all devices. Simulations with real data of PM 2.5 pollution in Beijing from the Center for Statistical Science at Peking University show that the proposed algorithm can reduce the sum of AoI by up to 26.9% compared to the conventional Q-learning method.

Original languageEnglish (US)
Title of host publication2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728182988
DOIs
StatePublished - Dec 2020
Event2020 IEEE Global Communications Conference, GLOBECOM 2020 - Virtual, Taipei, Taiwan, Province of China
Duration: Dec 7 2020Dec 11 2020

Publication series

Name2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings

Conference

Conference2020 IEEE Global Communications Conference, GLOBECOM 2020
Country/TerritoryTaiwan, Province of China
CityVirtual, Taipei
Period12/7/2012/11/20

All Science Journal Classification (ASJC) codes

  • Media Technology
  • Modeling and Simulation
  • Instrumentation
  • Artificial Intelligence
  • Computer Networks and Communications
  • Hardware and Architecture
  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'Reinforcement Learning for Minimizing Age of Information under Realistic Physical Dynamics'. Together they form a unique fingerprint.

Cite this