ML Seminar: Distributed Stochastic Approximation: Reinforcement Learning and Optimization with Communication Constraints

Friday, January 31, 2020
11:00am - 12:00pm
EER 1.516

We will discuss two problems that have different application spaces but share a common mathematical core. These problems combine stochastic approximation, an iterative method for finding the fixed point of a function from noisy observations, and consensus, a general averaging technique for multiple agents to cooperatively solve a distributed problem.

In the first part of the talk, we will discuss the policy evaluation problem in multi-agent reinforcement learning. In this problem, a set of agents operate in a common environment under a fixed control policy, working together to discover the value (accumulative reward) associated with each environmental state. We give a finite-time analysis on the performance of the well-known "TD-lambda" algorithm that depends on the connectivity of the agents and the intrinsic properties of the Markov decision process driving the agents decisions.

In the second part, we discuss distributed optimization problems over a network when the communication between the nodes is constrained, and so information that is exchanged between the nodes must be quantized. We propose a modified consensus-based gradient method that communicates using random (dithered) quantization. We derive an upper bounds on the rate of convergence of this method as a function of the bandwidth available between the nodes and the underlying network topology, and derive rates for both convex and strongly convex objective functions.


Georgia Institute of Technology

Dr. Justin Romberg is a Professor in the School of Electrical and Computer Engineering at the Georgia Institute of Technology. Dr. Romberg received the B.S.E.E. (1997), M.S. (1999) and Ph.D. (2004) degrees from Rice University in Houston, Texas. From Fall 2003 until Fall 2006, he was a Postdoctoral Scholar in Applied and Computational Mathematics at the California Institute of Technology. He spent the Summer of 2000 as a researcher at Xerox PARC, the Fall of 2003 as a visitor at the Laboratoire Jacques-Louis Lions in Paris, and the Fall of 2004 as a Fellow at UCLA’s Institute for Pure and Applied Mathematics. In the Fall of 2006, he joined the Georgia Tech ECE faculty. In 2008 he received an ONR Young Investigator Award, in 2009 he received a PECASE award and a Packard Fellowship, and in 2010 he was named a Rice University Outstanding Young Engineering Alumnus. In 2006-2007 he was a consultant for the TV show “Numb3rs”.  He was an Associate Editor for the IEEE Transactions on Information Theory from 2008-2011, for the SIAM Journal on Imaging Science from 2013-2018, and the SIAM Journal on the Mathematics of Data Science from 2018-present.  In 2018, he was named an IEEE Fellow.