TY - JOUR
T1 - UAV-to-Device Underlay Communications
T2 - Age of Information Minimization by Multi-Agent Deep Reinforcement Learning
AU - Wu, Fanyi
AU - Zhang, Hongliang
AU - Wu, Jianjun
AU - Han, Zhu
AU - Poor, H. Vincent
AU - Song, Lingyang
N1 - Funding Information:
Manuscript received August 7, 2020; revised January 12, 2021; accepted March 2, 2021. Date of publication March 10, 2021; date of current version July 15, 2021. This work was supported in part by National Natural Science Foundation of China under Grant 61625101 and 61941101, in part by the National Key Research and Development Project of China under Grant 2020AAA0130401, and in part by the U.S. National Science Foundation under Grants EARS-1839818, CNS1717454, CNS-1731424, CNS-1702850, CCF-0939370, and CCF-1908308. This article was presented in part at the 2020 IEEE GLOBECOM, Taipei, Taiwan. The associate editor coordinating the review of this article and approving it for publication was R. F. Schaefer. (Corresponding author: Jianjun Wu.) Fanyi Wu, Jianjun Wu, and Lingyang Song are with the Department of Electronics Engineering, Peking University, Beijing 100871, China (e-mail: fanyi.wu@pku.edu.cn; just@pku.edu.cn; lingyang.song@pku.edu.cn).
Publisher Copyright:
© 1972-2012 IEEE.
PY - 2021/7
Y1 - 2021/7
N2 - In recent years, unmanned aerial vehicles (UAVs) have unlocked numerous sensing applications, which are expected to add billions of dollars to the world economy in the next decade. To further improve the Quality-of-Service in these applications, the 3rd Generation Partnership Project has considered the use of terrestrial cellular networks to support UAV sensing services, also known as the cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device (U2D) communications. To evaluate the freshness of the sensory data, the concept of age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs' AoIs are determined by their trajectories during sensing and transmission, we investigate the AoI minimization problem for UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus we utilize multi-agent deep reinforcement learning to approximate the state-action space. Then, we propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a lower AoI than a greedy algorithm, policy gradient algorithm, and overlay U2D scheme.
AB - In recent years, unmanned aerial vehicles (UAVs) have unlocked numerous sensing applications, which are expected to add billions of dollars to the world economy in the next decade. To further improve the Quality-of-Service in these applications, the 3rd Generation Partnership Project has considered the use of terrestrial cellular networks to support UAV sensing services, also known as the cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device (U2D) communications. To evaluate the freshness of the sensory data, the concept of age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs' AoIs are determined by their trajectories during sensing and transmission, we investigate the AoI minimization problem for UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus we utilize multi-agent deep reinforcement learning to approximate the state-action space. Then, we propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our proposed algorithm can achieve a lower AoI than a greedy algorithm, policy gradient algorithm, and overlay U2D scheme.
KW - UAV-to-Device communication
KW - age of information
KW - cellular Internet of UAVs
KW - multi-agent deep reinforcement learning
UR - http://www.scopus.com/inward/record.url?scp=85102620921&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85102620921&partnerID=8YFLogxK
U2 - 10.1109/TCOMM.2021.3065135
DO - 10.1109/TCOMM.2021.3065135
M3 - Article
AN - SCOPUS:85102620921
SN - 0090-6778
VL - 69
SP - 4461
EP - 4475
JO - IEEE Transactions on Communications
JF - IEEE Transactions on Communications
IS - 7
M1 - 9374461
ER -