Networked markov decision processes with delays

Sachin Adlakha, Sanjay Lall, Andrea Goldsmith

Research output: Contribution to journalArticlepeer-review

20 Scopus citations


We consider a networked control system, where each subsystem evolves as a Markov decision process with some extra inputs from other systems. Each subsystem is coupled to its neighbors via communication links over which the signals are delayed, but are otherwise transmitted noise-free. A centralized controller receives delayed state information from each subsystem. The control action applied to each subsystem takes effect after a certain delay rather than immediately. We give an explicit bound on the finite history of measurement and control that is required for the optimal control of such networked Markov decision processes. We also show that these bounds depend only on the underlying graph structure as well as the associated delays. Thus, the partially observed Markov decision process associated with a networked Markov decision process can be converted into an information state Markov decision process, whose state does not grow with time.

Original languageEnglish (US)
Article number6018989
Pages (from-to)1013-1018
Number of pages6
JournalIEEE Transactions on Automatic Control
Issue number4
StatePublished - Apr 2012
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Computer Science Applications
  • Electrical and Electronic Engineering


  • Delayed systems
  • Markov decision processes
  • networked systems


Dive into the research topics of 'Networked markov decision processes with delays'. Together they form a unique fingerprint.

Cite this