SLIDE 1
Measure and cost dependent properties of information strucutres - - PowerPoint PPT Presentation
Measure and cost dependent properties of information strucutres - - PowerPoint PPT Presentation
Measure and cost dependent properties of information strucutres Aditya Mahajan Serdar Yksel Yale University Queen's University ACC 2010 2/17 Why are information structures useful? 2/17 Why are information structures useful? Info
SLIDE 2
SLIDE 3
2/17
Why are information structures useful?
Info structures capture the design difficulties of decentralized control
SLIDE 4
2/17
Why are information structures useful?
Info structures capture the design difficulties of decentralized control Classical info structures are centralized systems, hence easy to design Non-classical info structures are decentralized systems, hence hard to design
SLIDE 5
2/17
Why are information structures useful?
Info structures capture the design difficulties of decentralized control Classical info structures are centralized systems, hence easy to design Non-classical info structures are decentralized systems, hence hard to design Is this really true? Can we have two systems with identical information structures that behave differently?
SLIDE 6
3/17
A controller with no memory
Plant Controller Channel
-
State Equation: =
, ,
Observation Equation:
= ℎ, 𝑂
Controller with no memory: =
SLIDE 7
3/17
A controller with no memory
Plant Controller Channel
-
Non-classical info structure
State Equation: =
, ,
Observation Equation:
= ℎ, 𝑂
Controller with no memory: =
SLIDE 8
3/17
A controller with no memory
Plant Controller Channel
-
Non-classical info structure
State Equation: =
, ,
Observation Equation:
= ℎ, 𝑂
Controller with no memory: =
The info structure does not depend on channel ℎ
SLIDE 9
3/17
A controller with no memory
Plant Controller Channel
-
Non-classical info structure
State Equation: =
, ,
Observation Equation:
= ℎ, 𝑂
Controller with no memory: =
The info structure does not depend on channel ℎ When the channel is noiseless, the system is an MDP --- a centralized system
SLIDE 10
3/17
A controller with no memory
Plant Controller Channel
-
Non-classical info structure
State Equation: =
, ,
Observation Equation:
= ℎ, 𝑂
Controller with no memory: =
The info structure does not depend on channel ℎ When the channel is noiseless, the system is an MDP --- a centralized system
Two systems with identical info structures Perfect observations ⇒ centralized Imperfect observations ⇒ decentralized
SLIDE 11
4/17
What is missing?
Information structures do not completely characterize the design difficulties of decentralized systems
SLIDE 12
4/17
What is missing?
Information structures do not completely characterize the design difficulties of decentralized systems Information structures capture who knows what and when, but do not capture usefulness of available data
SLIDE 13
4/17
What is missing?
Information structures do not completely characterize the design difficulties of decentralized systems Information structures capture who knows what and when, but do not capture usefulness of available data We present a generalization of information structures, which we call -generalization, that captures the usefulness of information. This generalization depends on the coupling of the cost function and the independence properties of the probability measure
SLIDE 14
5/17
Contributions of the paper
Defined a -generalization of an info structure The solution technique for any info structure is also applicable to its
- generalization
SLIDE 15
5/17
Contributions of the paper
Defined a -generalization of an info structure The solution technique for any info structure is also applicable to its
- generalization
Implications: Follow a two step approach Define info structure in the usual manner (keeps analysis simple) Define the -generalization of an info structure We get the solution technique for -generalized info structure for free!
SLIDE 16
5/17
Contributions of the paper
Defined a -generalization of an info structure The solution technique for any info structure is also applicable to its
- generalization
Implications: Follow a two step approach Define info structure in the usual manner (keeps analysis simple) Define the -generalization of an info structure We get the solution technique for -generalized info structure for free! Present coupled dynamic programs to find pbpo solution of quasiclassical info structures Works for non-linear systems Need to only solve parametric optimization problem
SLIDE 17
6/17
Outline of the paper
Model Information Structures
- generalization of info structures
Coupled dynamic programs for quasiclassical info structure Example
SLIDE 18
7/17
The intrinsic model
Originally proposed by Witsenhausen, 1971 and 1975
SLIDE 19
7/17
The intrinsic model
Originally proposed by Witsenhausen, 1971 and 1975 Intrinsic event: 𝜕 taking values in a probability space
SLIDE 20
7/17
The intrinsic model
Originally proposed by Witsenhausen, 1971 and 1975 Intrinsic event: 𝜕 taking values in a probability space 𝑂 agents
SLIDE 21
7/17
The intrinsic model
Originally proposed by Witsenhausen, 1971 and 1975 Intrinsic event: 𝜕 taking values in a probability space 𝑂 agents Observations of agent :
taking value in a measurable space
- =
𝜕,
where ⊂ [ − 1]
SLIDE 22
7/17
The intrinsic model
Originally proposed by Witsenhausen, 1971 and 1975 Intrinsic event: 𝜕 taking values in a probability space 𝑂 agents Observations of agent :
taking value in a measurable space
- =
𝜕,
where ⊂ [ − 1] Action of agent : taking values in a measurable space =
SLIDE 23
7/17
The intrinsic model
Originally proposed by Witsenhausen, 1971 and 1975 Intrinsic event: 𝜕 taking values in a probability space 𝑂 agents Observations of agent :
taking value in a measurable space
- =
𝜕,
where ⊂ [ − 1] Action of agent : taking values in a measurable space =
Cost: Additive terms. Agents coupled by 𝑙-th cost term: ⊂ [𝑂]
∑
𝜍𝜕,
SLIDE 24
7/17
The intrinsic model
Originally proposed by Witsenhausen, 1971 and 1975 Intrinsic event: 𝜕 taking values in a probability space 𝑂 agents Observations of agent :
taking value in a measurable space
- =
𝜕,
where ⊂ [ − 1] Action of agent : taking values in a measurable space =
Cost: Additive terms. Agents coupled by 𝑙-th cost term: ⊂ [𝑂]
∑
𝜍𝜕, Objective: Choose , . . . , to minimize expected cost
SLIDE 25
8/17
Salient Features
Agents are coupled in two ways:
Coupling through dynamics
*
: set of agents that can influence the observations of agent
∈ *
⇒ there exist = , , . . . , ℓ = such that
∈ , 𝑗 = 1, . . . , ℓ
Coupling through cost
*
: agents coupled to agent through cost
*
=
⋃
𝟚{ ∈ }
SLIDE 26
9/17
Information Structures
Information Structure
Collection of information known to each agent
SLIDE 27
9/17
Information Structures
Information Structure
Collection of information known to each agent
Classification of info structures
Classical info structure Each agent knows the data available to all agents that act before it Quasiclassical info structure Each agent knows the data available to all agents that can influence its
- bservation
SLIDE 28
9/17
Information Structures
Information Structure
Collection of information known to each agent
Classification of info structures
Classical info structure Each agent knows the data available to all agents that act before it Quasiclassical info structure Each agent knows the data available to all agents that can influence its
- bservation
Strictly classical info structures Each agent . . . data and control actions . . . Strictly quasiclassical info structure Each agent . . . data and control actions . . .
SLIDE 29
10/17
Expansion of info structures
Classical expansion of info structure
A new system obtained by
- ↦
, [], []
SLIDE 30
10/17
Expansion of info structures
Classical expansion of info structure
A new system obtained by
- ↦
, [], []
Quasiclassical expansion of info structure
A new system obtained by
- ↦
, *
, *
SLIDE 31
11/17
The main idea (1)
Dynamic programming works only for strictly classical info structure.
SLIDE 32
11/17
The main idea (1)
Dynamic programming works only for strictly classical info structure. Nevertheless, we can design for classical info structure (not strict) as follows:
SLIDE 33
11/17
The main idea (1)
Dynamic programming works only for strictly classical info structure. Nevertheless, we can design for classical info structure (not strict) as follows: Denote the classical system by .
SLIDE 34
11/17
The main idea (1)
Dynamic programming works only for strictly classical info structure. Nevertheless, we can design for classical info structure (not strict) as follows: Denote the classical system by . Let 𝑇 be the classical expansion of . 𝑇 is strictly classical. Find optimal policy for 𝑇 (using dynamic programing)
SLIDE 35
11/17
The main idea (1)
Dynamic programming works only for strictly classical info structure. Nevertheless, we can design for classical info structure (not strict) as follows: Denote the classical system by . Let 𝑇 be the classical expansion of . 𝑇 is strictly classical. Find optimal policy for 𝑇 (using dynamic programing) The difficulty is that may not be implementable in
SLIDE 36
11/17
The main idea (1)
Dynamic programming works only for strictly classical info structure. Nevertheless, we can design for classical info structure (not strict) as follows: Denote the classical system by . Let 𝑇 be the classical expansion of . 𝑇 is strictly classical. Find optimal policy for 𝑇 (using dynamic programing) The difficulty is that may not be implementable in By successive substitution, we can find a corresponding policy * such that and * have the same performance in 𝑇 * is implementable in
SLIDE 37
11/17
The main idea (1)
Dynamic programming works only for strictly classical info structure. Nevertheless, we can design for classical info structure (not strict) as follows: Denote the classical system by . Let 𝑇 be the classical expansion of . 𝑇 is strictly classical. Find optimal policy for 𝑇 (using dynamic programing) The difficulty is that may not be implementable in By successive substitution, we can find a corresponding policy * such that and * have the same performance in 𝑇 * is implementable in Question: Instead of a classical system, can we start with a more relaxed system such that this procedure still works?
SLIDE 38
11/17
The main idea (1)
Dynamic programming works only for strictly classical info structure. Nevertheless, we can design for classical info structure (not strict) as follows: Denote the classical system by . Let 𝑇 be the classical expansion of . 𝑇 is strictly classical. Find optimal policy for 𝑇 (using dynamic programing) The difficulty is that may not be implementable in By successive substitution, we can find a corresponding policy * such that and * have the same performance in 𝑇 * is implementable in Question: Instead of a classical system, can we start with a more relaxed system such that this procedure still works?
- classical info structure:
Let ∶=
∑
𝜍𝜕, 𝟚{{ ∈ } ∪ {∃ ∈ : ∈ * }}.
Then, an info structure is -classical if 𝔽{ |
, } = 𝔽{ | [], []}
SLIDE 39
12/17
The main idea (2)
We ask a similar question for quasiclassical info structures. What is the most relaxed info structure that we can start with such that if we take its quasiclassical expansion find the optimal policy for the quasiclassical expansion then, can find a corresponding optimal policy that is implementable in the original system
SLIDE 40
12/17
The main idea (2)
We ask a similar question for quasiclassical info structures. What is the most relaxed info structure that we can start with such that if we take its quasiclassical expansion find the optimal policy for the quasiclassical expansion then, can find a corresponding optimal policy that is implementable in the original system Difficulty: No appropriate solution technique for quasiclassical systems Solutions for LQG quasiclassical systems rely convexity of static LQG
- teams. These results do not extend to non-LQG systems.
Sequential decomposition for optimal design gives a functional
- ptimization problem. This makes it extremely hard to find a
corresponding policy (revisit later)
SLIDE 41
12/17
The main idea (2)
We ask a similar question for quasiclassical info structures. What is the most relaxed info structure that we can start with such that if we take its quasiclassical expansion find the optimal policy for the quasiclassical expansion then, can find a corresponding optimal policy that is implementable in the original system Difficulty: No appropriate solution technique for quasiclassical systems Solutions for LQG quasiclassical systems rely convexity of static LQG
- teams. These results do not extend to non-LQG systems.
Sequential decomposition for optimal design gives a functional
- ptimization problem. This makes it extremely hard to find a
corresponding policy Find pbpo solutions using coupled dynamic programs (revisit later)
SLIDE 42
12/17
The main idea (2)
We ask a similar question for quasiclassical info structures. What is the most relaxed info structure that we can start with such that if we take its quasiclassical expansion find the optimal policy for the quasiclassical expansion then, can find a corresponding optimal policy that is implementable in the original system Difficulty: No appropriate solution technique for quasiclassical systems Solutions for LQG quasiclassical systems rely convexity of static LQG
- teams. These results do not extend to non-LQG systems.
Sequential decomposition for optimal design gives a functional
- ptimization problem. This makes it extremely hard to find a
corresponding policy Find pbpo solutions using coupled dynamic programs (revisit later)
- quasiclassical info structure:
Let ∶=
∑
𝜍𝜕, 𝟚{{ ∈ } ∪ {∃ ∈ : ∈ * }}.
Then, an info structure is -quasiclassical if 𝔽{ |
, } = 𝔽{ | , , *
, * }
SLIDE 43
13/17
Proof outline
The proof for both cases is constructive Take expanded info structure Find an optimal (or pbpo) policy Construct a corresponding policy that is implementable in original system The details of each step conceptually simple, but notationally cumbersome due to generality of the model
SLIDE 44
14/17
Coupled Dynamic programs for quasiclassical info structure
Any quasiclassical system can be broken into a collection of coupled systems where each subsystem has a classical info structure
SLIDE 45
14/17
Coupled Dynamic programs for quasiclassical info structure
Any quasiclassical system can be broken into a collection of coupled systems where each subsystem has a classical info structure
SLIDE 46
14/17
Coupled Dynamic programs for quasiclassical info structure
Any quasiclassical system can be broken into a collection of coupled systems where each subsystem has a classical info structure Subsystem A
SLIDE 47
14/17
Coupled Dynamic programs for quasiclassical info structure
Any quasiclassical system can be broken into a collection of coupled systems where each subsystem has a classical info structure Subsystem B Subsystem A
SLIDE 48
14/17
Coupled Dynamic programs for quasiclassical info structure
Any quasiclassical system can be broken into a collection of coupled systems where each subsystem has a classical info structure Subsystem B Subsystem A Subsystem C
SLIDE 49
14/17
Coupled Dynamic programs for quasiclassical info structure
Any quasiclassical system can be broken into a collection of coupled systems where each subsystem has a classical info structure Subsystem B Subsystem A Subsystem C Subsystems A, B, and C are classical
SLIDE 50
14/17
Coupled Dynamic programs for quasiclassical info structure
Any quasiclassical system can be broken into a collection of coupled systems where each subsystem has a classical info structure Subsystem B Subsystem A Subsystem C Subsystems A, B, and C are classical Write a DP for each subsystem and solve them iteratively Idea originally proposed in Teneketzis and Ho, 1987
SLIDE 51
15/17
An Example
Sys 1 Sys 2 Ctr 1 Ctr 2
= , 𝑣 ,
= , , 𝑣 ,
= ℎ ,
= ℎ ,
𝑣
= [], 𝑣 []
𝑣
= [], [], 𝑣 [], 𝑣 []
Choose 𝐻 ∶=
, . . . , and 𝐻 ∶= , . . . , to minimize
𝔽 {
∑
𝜍
, , 𝑣 , 𝑣 }
SLIDE 52
15/17
An Example
Sys 1 Sys 2 Ctr 1 Ctr 2
= , 𝑣 ,
= , , 𝑣 ,
= ℎ ,
= ℎ ,
𝑣
= [], 𝑣 []
𝑣
= [], [], 𝑣 [], 𝑣 []
Choose 𝐻 ∶=
, . . . , and 𝐻 ∶= , . . . , to minimize
𝔽 {
∑
𝜍
, , 𝑣 , 𝑣 }
Quasiclassical info structure Non-linear dynamics Noisy observations
SLIDE 53
16/17
An Example
1.1 1.2 1.3 1.4 2.1 2.2 2.3 2.4
SLIDE 54
16/17
An Example
1.1 1.2 1.3 1.4 2.1 2.2 2.3 2.4
Subsystem 1
Fix policy 𝐻 and solve for 𝐻
[], 𝑣 [] = 𝔽{𝜍 , , 𝑣 , 𝑣 | [], 𝑣 []}
[], 𝑣 [] = 𝔽{𝜍 , , 𝑣 , 𝑣
+
[], 𝑣 []| [], 𝑣 []}
SLIDE 55
16/17
An Example
1.1 1.2 1.3 1.4 2.1 2.2 2.3 2.4
Subsystem 1
Fix policy 𝐻 and solve for 𝐻
[], 𝑣 [] = 𝔽{𝜍 , , 𝑣 , 𝑣 | [], 𝑣 []}
[], 𝑣 [] = 𝔽{𝜍 , , 𝑣 , 𝑣
+
[], 𝑣 []| [], 𝑣 []}
Subsystem 2
Fix policy 𝐻 and solve for 𝐻
[], [], 𝑣 []𝑣 [] = 𝔽{𝜍 , , 𝑣 , 𝑣 | [], [], 𝑣 [], 𝑣 []}
[], [], 𝑣 [], 𝑣 [] = 𝔽{𝜍 , , 𝑣 , 𝑣
+
[], [], 𝑣 [], 𝑣 []| [], [], 𝑣 [], 𝑣 []}
SLIDE 56
17/17
Conclusion
Defined a -generalization of info structure The solution technique for any info structure is also applicable to its -generalization Present coupled dynamic programs to find person by person optimal solution of quasiclassical info structures
SLIDE 57