AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning

Fanyi Wu; Hongliang Zhang; Jianjun Wu; Lingyang Song; Zhu Han; H. Vincent Poor

doi:10.1109/GLOBECOM42002.2020.9322539

AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning

Fanyi Wu, Hongliang Zhang, Jianjun Wu, Lingyang Song, Zhu Han, H. Vincent Poor

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Scopus citations

Abstract

In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device communications. To evaluate the freshness of the sensory data, the age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs' AoIs are determined by their trajectories during sensing and transmission, we aim to minimize the AoIs of UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus, we propose a multi-UAV trajectory design algorithm by leveraging multi-agent deep reinforcement learning to solve it. Simulation results show that our proposed algorithm outperforms both a greedy algorithm and a policy gradient algorithm.

Original language	English (US)
Title of host publication	2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728182988
DOIs	https://doi.org/10.1109/GLOBECOM42002.2020.9322539
State	Published - Dec 2020
Event	2020 IEEE Global Communications Conference, GLOBECOM 2020 - Virtual, Taipei, Taiwan, Province of China Duration: Dec 7 2020 → Dec 11 2020

Publication series

Name	2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings

Conference

Conference	2020 IEEE Global Communications Conference, GLOBECOM 2020
Country/Territory	Taiwan, Province of China
City	Virtual, Taipei
Period	12/7/20 → 12/11/20

All Science Journal Classification (ASJC) codes

Media Technology
Modeling and Simulation
Instrumentation
Artificial Intelligence
Computer Networks and Communications
Hardware and Architecture
Software
Safety, Risk, Reliability and Quality

Keywords

UAV-to-Device communication
cellular Internet of UAVs
multi-agent deep reinforcement learning

Access to Document

10.1109/GLOBECOM42002.2020.9322539

Cite this

Wu, F., Zhang, H., Wu, J., Song, L., Han, Z., & Poor, H. V. (2020). AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning. In 2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings [9322539] (2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/GLOBECOM42002.2020.9322539

@inproceedings{cad0e2c612b141e189cacefdd5938007,

title = "AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning",

abstract = "In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device communications. To evaluate the freshness of the sensory data, the age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs' AoIs are determined by their trajectories during sensing and transmission, we aim to minimize the AoIs of UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus, we propose a multi-UAV trajectory design algorithm by leveraging multi-agent deep reinforcement learning to solve it. Simulation results show that our proposed algorithm outperforms both a greedy algorithm and a policy gradient algorithm.",

keywords = "UAV-to-Device communication, cellular Internet of UAVs, multi-agent deep reinforcement learning",

author = "Fanyi Wu and Hongliang Zhang and Jianjun Wu and Lingyang Song and Zhu Han and Poor, {H. Vincent}",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 2020 IEEE Global Communications Conference, GLOBECOM 2020 ; Conference date: 07-12-2020 Through 11-12-2020",

year = "2020",

month = dec,

doi = "10.1109/GLOBECOM42002.2020.9322539",

language = "English (US)",

series = "2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings",

address = "United States",

}

Wu, F, Zhang, H, Wu, J, Song, L, Han, Z & Poor, HV 2020, AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning. in 2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings., 9322539, 2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2020 IEEE Global Communications Conference, GLOBECOM 2020, Virtual, Taipei, Taiwan, Province of China, 12/7/20. https://doi.org/10.1109/GLOBECOM42002.2020.9322539

AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning. / Wu, Fanyi; Zhang, Hongliang; Wu, Jianjun et al.

2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. 9322539 (2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning

AU - Wu, Fanyi

AU - Zhang, Hongliang

AU - Wu, Jianjun

AU - Song, Lingyang

AU - Han, Zhu

AU - Poor, H. Vincent

PY - 2020/12

Y1 - 2020/12

N2 - In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device communications. To evaluate the freshness of the sensory data, the age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs' AoIs are determined by their trajectories during sensing and transmission, we aim to minimize the AoIs of UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus, we propose a multi-UAV trajectory design algorithm by leveraging multi-agent deep reinforcement learning to solve it. Simulation results show that our proposed algorithm outperforms both a greedy algorithm and a policy gradient algorithm.

AB - In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to the base station via cellular links, or to the mobile devices by underlay UAV-to-Device communications. To evaluate the freshness of the sensory data, the age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs' AoIs are determined by their trajectories during sensing and transmission, we aim to minimize the AoIs of UAVs by designing their trajectories. This problem is a Markov decision problem with an infinite state-action space, and thus, we propose a multi-UAV trajectory design algorithm by leveraging multi-agent deep reinforcement learning to solve it. Simulation results show that our proposed algorithm outperforms both a greedy algorithm and a policy gradient algorithm.

KW - UAV-to-Device communication

KW - cellular Internet of UAVs

KW - multi-agent deep reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=85100394064&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85100394064&partnerID=8YFLogxK

U2 - 10.1109/GLOBECOM42002.2020.9322539

DO - 10.1109/GLOBECOM42002.2020.9322539

M3 - Conference contribution

AN - SCOPUS:85100394064

T3 - 2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings

BT - 2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2020 IEEE Global Communications Conference, GLOBECOM 2020

Y2 - 7 December 2020 through 11 December 2020

ER -

Wu F, Zhang H, Wu J, Song L, Han Z, Poor HV. AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning. In 2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2020. 9322539. (2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings). doi: 10.1109/GLOBECOM42002.2020.9322539

AoI Minimization for UAV-to-Device Underlay Communication by Multi-agent Deep Reinforcement Learning

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this