A Mean Field Games Approach to Consensus Problems Mojtaba Nourian - - PowerPoint PPT Presentation

A Mean Field Games Approach to Consensus Problems

Mojtaba Nourian

McGill University, Montr´ eal, Canada Information and Control in Networks LCCC, Lunds University 23 October 2012 Joint work with Professors Peter Caines, Roland Malham´ e and Minyi Huang

1 / 22

Outline

Background: Mean Field Game (MFG) Theory Standard Consensus Algorithms (SCAs) MFG Consensus Formulation and Solution – Homogenous Case MFG Consensus Formulation and Solution – Heterogeneous Case

2 / 22

Background – Mean Field Game (MFG) Theory

The Modeling Setup of Mean Field Game Theory (Huang, Caines, Malham´ e (’03,’06,’07), Lasry-Lions (’06,’07)): For a class of dynamic games with a large number of minor agents Each minor agent interacts with the average or so-called mass effect of

• ther agents via couplings in their individual cost functions and individual

dynamics A minor agent is an agent which, asymptotically as the population size goes to infinity, has a negligible influence on the overall system while the

• verall population’s effect on it is significant

3 / 22

Background – Mean Field Game (MFG) Theory

Key Idea of Mean Field Game (MFG) Theory (HCM (’03,’06,’07)): Establish the existence of an equilibrium relationship between the individual strategies and the mass effect in the infinite population limit Such that the individual strategy of each agent is a best response to the mass effect, and the set of the strategies collectively replicate that mass effect Apply the resulting infinite population strategies to a finite population system and obtain suitable approximate equilibrium

4 / 22

Background – MFG-LQG Problem Formulation

Basic Linear-Qudratic-Gaussian (LQG) Dynamic Game Problem Individual Agent’s Dynamics: dzi(t) =

• aizi(t) + bui(t)
• dt + idwi(t),

1  i  N N: population size, zi: state of agent i, ui: control input, wi: disturbance Individual Agent’s Cost Function: Ji(ui, ⌫) , E Z 1 eρt⇣ zi(t) ⌫(t) 2 + ru2

i (t)

⌘ dt ⇢ > 0: discount factor, r > 0: control penalty, and ⌫(·) , 1 N

N

X

k=1

zk(·) + ⌘

• Main feature:

Agents are coupled via their costs Stochastic tracked process ⌫: (i) depends on other agents’ control laws (ii) not feasible for zi to track all zk trajectories for large N

5 / 22

Background – Preliminary LQG Tracking Problem

Preliminary LQG Tracking Problem For One Agent Only: x⇤(·) known and deterministic dzi(t) =

• aizi(t) + bui(t)
• dt + idwi(t)

Ji(ui, x⇤) = E Z 1 eρt⇣ zi(t) x⇤(t) 2 + ru2

i (t)

⌘ dt Computation of the Optimal Tracking Control: ui(·) = b r

• Πizi(·) + si(·)
• Riccati Equation:

⇢Πi = 2aiΠi b2 r Π2

i + 1,

Πi > 0 Mass Offset Control: dsi dt = ⇢si + aisi b2 r Πisi x⇤ Boundedness condition on x⇤(·) implies existence of unique solution si

6 / 22

Background – The Fundamental MFG-LQG System

Continuum of Systems under Optimal LQG Tracking Control: a 2 A; common b for simplicity dsa dt = ⇢sa + asa b2 r Πasa x⇤ (Tracking mass equation) d¯ za dt = (a b2 r Πa)¯ za b2 r sa (The mean state equation) ¯ z(t) = Z

A

¯ za(t)dF(a) (The mean field function) x⇤(t) = (¯ z(t) + ⌘) t 0 (The mass function) Riccati Equation : ⇢Πa = 2aΠa b2 r Π2

a + 1,

Πa > 0 F(·): The limit empirical distribution of {ai : i > 1} ⇢ A Individual control action ua = b

r (Πaza + sa) is optimal w.r.t tracked x⇤

Does there exist a solution (¯ za, sa, x⇤; a 2 A)? Yes: Fixed Point Theorem

7 / 22

Background – Properties of MFG-LQG Solution

Theorem (HCM’03,’07) Subject to technical conditions, the MFG system has a unique solution for which the resulting set of MFG controls U N

mf = {u0 i = b

r (Πizi + si); 1  i  N}, 1  N < 1 yields an ✏-Nash equilibrium for all ✏, i.e. 8✏ > 0 9N(✏) s.t. 8N N(✏) Ji(u0

i , u0 i) ✏  inf ui Ji(ui, u0 i)  Ji(u0 i , u0 i)

where ui is adapted to the set of full information admissible controls. Agent y is a maximizer Agent x is a minimizer

−2 −1 1 2 −2 −1 1 2 −4 −3 −2 −1 1 2 3 4 x y

8 / 22

Background – Properties of MFG-LQG Solution

Counterintuitive Nature of MFG controls: Intrinsically decentralized agent’s feedback = feedback of agent’s local stochastic state + feedback of deterministic precomputable mass (No communication among agents!) Applying MFG Controls to the Finite Population System: ✏-Nash equilibrium (with respect to all possible controls among the full information pattern) exists between the individuals of a large N population system with ✏ ! 0 as N goes to infinity

9 / 22

Background – Standard Consensus Algorithms

Definition A consensus process is a process for achieving an agreement among the members of a group of agents on some common state property such as velocity

• r information.

Standard Consensus Algorithms (SCAs): A network of N agents with dynamics dzi(t) = ui(t)dt, t 0, 1  i  N, where an agreement is achieved via local communications with their neighbours based on the network topology G = (V, E) (V : the set of vertices, E ⇢ V ⇥ V : an ordered set of edges)

10 / 22

Background – Standard Consensus Algorithms

Time-Invariant SCAs: dzi(t) = X

j2Ni

aij

• zj(t) zi(t)
• dt,

t 0, 1  i  N where Ni = {j 2 V : (i, j) 2 E}. Definition Consensus is said to be achieved asymptotically for a group of N agents if limt!1 |zi(t) zj(t)| = 0 for any i and j, 1  i 6= j  N. Theorem (see e.g. (Ren et.al. ’05)) If the undirected graph G is connected (i.e., there is a path between every pair

• f nodes), then

the system achieves consensus asymptotically as time goes to infinity the consensus value is the average of initial states

1 N

PN

j=1 zj(0). 11 / 22

Why MFG Consensus Formulation?

The connectivity of the network structure needed for the SCAs (even for the less demanding “frequently connected” hypotheses) may not hold Communication is costly and may be distorted SCAs are fragile in the presence of noise in the agents’ dynamics MFG approach with no communication but prior statistical information In this approach we seek to synthesize the collective behaviour of the group from fundamental principles

12 / 22

MFG Consensus Formulation – Homogenous Case

Dynamics: dzi(t) = ui(t)dt + dwi(t), t 0, 1  i  N Cost Functions: JN

i (ui, ui) := E

Z 1 eρt⇣ zi(t) 1 N 1

N

X

j=1,j6=i

zj(t) 2 + ru2

i (t)

⌘ dt N; population size, zi: state of agent i, ui: control input wi: disturbance (standard Wiener process), ⇢ > 0: discount factor r > 0: control penalty Each agent in the group seeks a strategy to be as close as possible to the average of the population Let F(·) be the limit empirical distribution of {zi(0) : i > 1} ⇢ C

13 / 22

MFG Consensus Solution – Homogenous Case

Mean Field Game System of the Consensus Formulation:

• Computation of Best Response Control for a Generic Agent with Initial ↵ 2 C

and Mass Trajectory 1(·): uo

α(t) = 1

r

• pzα(t) + s(t)
• (Best Response Control)

p2 + r⇢p r = 0 = ) p = (r⇢ + p (r⇢)2 + 4r)/2 (Riccati Equation) ds(t) dt =

• ⇢ + p

r

• s(t) + 1(t)

(Tracking equation)

• Mass behavior equation in the consensus formulation under uo

α(·):

dzα(t) = 1 r

• pzα(t) + s(t)
• dt + dwi(t)

(The generic agenet process) d¯ zα(t) dt = 1 r

• p¯

zα(t) + s(t)

• , ¯

zα(0) = ↵ (The mean state equation) 1(t) = Z

C

¯ zα(t)dF(↵), t 0 (The mass function)

14 / 22

MFG Consensus Solution – Homogenous Case

Theorem (NCMH’10) The unique solution of MFG system: (s(t), 1(t)) = (p1(0), 1(0)), t 0. Applying the MFG control uo

i (t) = p r

• zi(t) 1(0)
• yields:

zo

i (t) = 1(0) + e p

r t

zi(0) 1(0)

• +

Z t e p

r (tτ)dwi(⌧), t 0.

Definition Mean-consensus is said to be achieved asymptotically for a group of N agents if limt!1 |¯ zi(t) ¯ zj(t)| = 0 for any i and j, 1  i 6= j  N. Theorem (NCMH’10) (i) A mean-consensus is reached asymptotically as time goes to infinity with individual asymptotic variance σ2r

2p .

(ii) The set of MFG control strategies {uo

i : 1  i  N} generates an ✏N-Nash

equilibrium such that limN!1 ✏N = 0.

15 / 22

MFG Consensus Solution – Homogenous Case

Simulation Result (500 agents) (A) Trajectories of agents’ states, (B) Histogram of the system at time t = 20

16 / 22

MFG Consensus Formulation – Heterogeneous Case

Dynamics: dzi(t) = ui(t)dt + dwi(t), 1  i  N, li 2 Θ li: type of agent i, Θ := {✓1, · · · , ✓K} to model K subpopulations, Nk: the number of agents of type ✓k Cost Functions: JN

i (ui, ui) := E

Z 1 eρt⇣ zi(t) PN

j=1 !(N) lilj zj(t)

PN

j=1 !(N) lilj

2 + ru2

i (t)

⌘ dt, with the weight coefficients: !(N)

lilj =

⇢ 1/Nk for li, lj = ✓k, !θiθj/Nk0 for li = ✓k, lj = ✓k0. where !θiθj 0 for any ✓i, ✓j 2 Θ and PK

j=1 !θiθj 6= 0 for each ✓i 2 Θ. 17 / 22

MFG Consensus Solution – Heterogenous Case

Assumption: There exists a probability vector ⇡ such that lim

N!1(N1

N , · · · , Nk N ) = ⇡ := (⇡1, · · · , ⇡K) where min1kK ⇡k > 0. The Fundamental MFG System dsθ(t) dt =

• ⇢ + p

r

• sθ(t) + 1

θ (t),

✓ 2 Θ (Tracking mass equation) d¯ zθ(t) dt = p r ¯ zθ(t) 1 r sθ(t), ¯ zθ(0) (The mean state equation) 1

θ (·) =

P

θ02Θ ⇡θ0!θθ0 ¯

zθ0(·) P

θ02Θ ⇡θ0!θθ0

(The mass function) Riccati Equation : p2 + r⇢p r = 0 = ) p = (r⇢ + p (r⇢)2 + 4r)/2 Individual best response action uo

θ(t) = 1 r

• pzθ(t) + s(t)
• is optimal

w.r.t tracked 1

θ 18 / 22

MFG Consensus Solution – Heterogenous Case

Let (W)ij := ⇡j!θiθj PK

k=1 ⇡k!θiθk

, 1  i, j  K Matrix W is a row-stochastic matrix since all its row sums are 1. Definition A stochastic matrix is irreducible if its corresponding digraph is strongly connected. Theorem If W is irreducible then the unique stationary solution of the MFG system is (s1, ¯ z1) = ✓ pT ¯ z(0) T 1K 1K, T ¯ z(0) T 1K 1K ◆ where T is the unique left-hand Perron vector for W. Hence, agents reach mean-consensus in γT ¯

z(0) γT 1K 1K. 19 / 22

MFG Consensus Solution – Heterogenous Case

Simulation Result: 500 agents in a system of 5 subpopulations such that W corresponds to an adjacency matrix of a strongly connected graph (A) Trajectories of agents’ states, (B) Histogram of the system at time t = 20

20 / 22

MFG Consensus Solution – Heterogenous Case

Simulation Result: 500 agents in a system of 5 subpopulations such that W corresponds to an adjacency matrix of a graph with two connected components (A) Trajectories of agents’ states, (B) Histogram of the system at time t = 20

21 / 22

Conclusion

Extensions and Generalizations: Analysis extends to the cooperative social optimization with the social cost JN

soc(u) = PN i=1 JN i (ui, ui)

Analysis extends to MFG flocking formulation Future Research: Consensus algorithms by the use of: A priori statistical information (MFG) Local communications (SCAs)

22 / 22