[PPT] - Ancillary Service to the Grid Using Intelligent Deferrable Loads PowerPoint Presentation

SLIDE 1

Ancillary Service to the Grid Using Intelligent Deferrable Loads

PGMO Days 2015

Ana Buˇ si´ c Inria, DI ENS

In collaboration with S. Meyn and P. Barooah

Thanks to PGMO, NSF, and Google

SLIDE 2

Outline

1 Challenges of Renewable Energy Integration 2 Virtual Energy Storage 3 Control of Deferrable Loads: Goals and Architecture 4 Mean Field Model 5 Local Control Design 6 Conclusions and Future Directions

0 / 19

SLIDE 3

March 8th 2014: Impact of wind and solar on net-load at CAISO Ramp limitations cause price-spikes

Price spike due to high net-load ramping need when solar production ramped out Negative prices due to high mid-day solar production

1200 15 2 4 19 17 21 23 27 25 800 1000 600 400 200

200

GW GW Toal Load Wind and Solar Load and Net-load Toal Wind Toal Solar Net-load: Toal Load, less Wind and Solar $/MWh 24 hrs 24 hrs Peak ramp Peak Peak ramp Peak

Challenges

SLIDE 4

Challenges of Renewable Energy Integration

Some of the Challenges

1 Ducks

MISO, CAISO, and others: seek markets for ramping products

1 / 19

SLIDE 5

Challenges of Renewable Energy Integration

Some of the Challenges

1 Ducks 2 Ramps

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

GW (t) = Wind generation in BPA, Jan 2015

Ramps 1 / 19

SLIDE 6

Challenges of Renewable Energy Integration

Some of the Challenges

1 Ducks 2 Ramps 3 Regulation 1 / 19

SLIDE 7

Challenges of Renewable Energy Integration

Some of the Challenges

1 Ducks 2 Ramps 3 Regulation

One potential solution: Large-scale storage with fast charging/discharging rates

1 / 19

SLIDE 8

Challenges of Renewable Energy Integration

Some of the Challenges

1 Ducks 2 Ramps 3 Regulation

One potential solution: Large-scale storage with fast charging/discharging rates Let’s consider some alternatives

1 / 19

SLIDE 9

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

Gr(t) G1 G2 G

Traditional generation DD: Chillers & Pool Pumps DD: HVAC Fans

3

Gr = G1 + G2 + G3

Virtual Energy Storage

SLIDE 10

Virtual Energy Storage

Control Architecture

Frequency Decomposition

Today: PJM decomposes regulation signal based on bandwidth, R = RegA + RegD Proposal: Each class of DR (and other) resources will have its own bandwidth of service, based on QoS constraints and costs.

2 / 19

SLIDE 11

Virtual Energy Storage

Frequency Decomposition

Taming the Duck

March 8th 2014: Impact of wind and solar on net-load at CAISO Ramp limitations cause price-spikes

Price spike due to high net-load ramping need when solar production ramped out Negative prices due to high mid-day solar production

1200 15 2 4 19 17 21 23 27 25 800 1000 600 400 200

200

GW GW Toal Load Wind and Solar Load and Net-load Toal Wind Toal Solar Net-load: Toal Load, less Wind and Solar $/MWh 24 hrs 24 hrs Peak ramp Peak Peak ramp Peak

ISOs need help: ... ramp capability shortages could result in a single, five-minute dispatch interval or multiple consecutive dispatch intervals during which the price of energy can increase significantly due to scarcity pricing, even if the event does not present a significant reliability risk

http://tinyurl.com/FERC-ER14-2156-000 3 / 19

SLIDE 12

Virtual Energy Storage

Frequency Decomposition

Taming the Duck

One Day at CAISO 2020 ISO/RTOs are seeking ramping products to address engineering challenges, and to avoid scarcity prices Do we need ramping products?

Net Load Curve

In c r ea s ed ramp

GW

5

5 10 15 20 25

12am 12am 3am 6am 9am 12pm 3pm 6pm 9pm

4 / 19

SLIDE 13

Virtual Energy Storage

Frequency Decomposition

Taming the Duck

One Day at CAISO 2020

Net Load Curve

T a m i n g t h e D u c k GW

5

5 10 15 20 25

12am 12am 3am 6am 9am 12pm 3pm 6pm 9pm

This doesn’t look at all scary! We need resources, but anyone here knows how to track this tame duck

4 / 19

SLIDE 14

Virtual Energy Storage

Frequency Decomposition

Taming the Duck

One Day at CAISO 2020

Net Load Curve Low pass Mid pass High pass

The duck is a sum of a smooth energy signal, and two zero-energy services GW

5

5 10 15 20 25

12am 12am 3am 6am 9am 12pm 3pm 6pm 9pm

4 / 19

SLIDE 15

Virtual Energy Storage

Frequency Decomposition

Regulation

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

GW (t) = Wind generation in BPA, Jan 2015

Ramps

5 / 19

SLIDE 16

Virtual Energy Storage

Frequency Decomposition

Regulation

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

G Goal:

W (t) = Wind generation in BPA, Jan 2015 Ramps

GW (t) + Gr(t) ≡ 4GW

5 / 19

SLIDE 17

Virtual Energy Storage

Frequency Decomposition

Regulation

Ra

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

G Goal:

W (t) = Wind generation in BPA, Jan 2015 Ramps Ramps Ramps Ramps Ramps

GW (t) + Gr(t) ≡ 4GW

5 / 19

SLIDE 18

Virtual Energy Storage

Frequency Decomposition

Regulation

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

Goal: GW (t) + Gr(t) ≡ 4GW

btained from

generation? Gr(t) Gr(t)

Ramp

5 / 19

SLIDE 19

Virtual Energy Storage

Frequency Decomposition

Regulation

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

Gr(t) Gr = G1 + G2 + G3 G1

5 / 19

SLIDE 20

Virtual Energy Storage

Frequency Decomposition

Regulation

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

Gr(t) Gr = G1 + G2 + G3 G1 G2

5 / 19

SLIDE 21

Virtual Energy Storage

Frequency Decomposition

Regulation

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

Gr(t) Gr = G1 + G2 + G3 G1 G2 G3

5 / 19

SLIDE 22

Virtual Energy Storage

Frequency Decomposition

Regulation

Jan 01 Jan 02 Jan 03 Jan 04 Jan 05 Jan 06

GW

1 2 3 4

Gr(t) Gr = G1 + G2 + G3 G1 G2 G3 Where do we find these resources?

5 / 19

SLIDE 23

Virtual Energy Storage

Demand Dispatch

Responsive Regulation and desired QoS Gr Gr = G1 + G2 + G3 G1 G2 G3 ?

6 / 19

SLIDE 24

Virtual Energy Storage

Demand Dispatch

Responsive Regulation and desired QoS Gr Gr = G1 + G2 + G3 G1 G2 G Traditional generation

3 6 / 19

SLIDE 25

Virtual Energy Storage

Demand Dispatch

Responsive Regulation and desired QoS Gr Gr = G1 + G2 + G3 G1 G2 G Traditional generation Water pumping (e.g. pool pumps) Fans in commercial HVAC

Responsive Regulation and desired QoS – A partial list of the needs of the grid operator, and the consumer

High quality Ancillary Service? Reliable? Cost effective? Customer QoS constraints satisfied? Virtual energy storage: achieve these goals simultaneously through distributed control

7 / 19

SLIDE 31

Control of Deferrable Loads

SLIDE 32

Control of Deferrable Loads: Goals and Architecture

Control Goals and Architecture

Prefilter and decision rules designed to respect needs of load and grid

Two components to local control Local feedback loop Local Control Load i

ζt Y i

t

U i

t

Prefilter Decision

ζt U i

t

Xi

t

Xi

t

Requirements Minimal communication: Each load monitors its state and a regulation signal from the grid Aggregate must be controllable: Randomized policies required for finite-state loads

8 / 19

SLIDE 33

Control of Deferrable Loads: Goals and Architecture

Control Goals and Architecture

Prefilter and decision rules designed to respect needs of load and grid

Two components to local control Local feedback loop Local Control Load i

ζt Y i

t

U i

t

Prefilter Decision

ζt U i

t

Xi

t

Xi

t

Requirements Minimal communication: Each load monitors its state and a regulation signal from the grid Aggregate must be controllable: Randomized policies required for finite-state loads Questions

How to analyze aggregate of similar loads?
Local control design?

8 / 19

SLIDE 34

yt

Control @ Utility Gain One Million Pools Disturbance to be rejected Proportion of pools on

desired

ζt

µ t+1 = µtPζt yt = µt, U

Aggregate of similar deferrable loads

SLIDE 35

Mean Field Model

Control Architecture

Aggregate of similar deferrable loads

...

Load 1

BA

Reference (MW)

Load 2 Load N

ζ

r

+

Gc

Power Consumption (MW)

Examples: Chillers in HVAC systems, water heaters, residential TCLs, ... ... residential pool pumps

9 / 19

SLIDE 36

Mean Field Model

Load Model

Controlled Markovian Dynamics

...

Load 1

BA

Reference (MW)

Load 2 Load N

ζ r

+

Gc

Power Consumption (MW)

Assumptions:

10 / 19

SLIDE 37

Mean Field Model

Load Model

Controlled Markovian Dynamics

...

Load 1

BA

Reference (MW)

Load 2 Load N

ζ r

+

Gc

Power Consumption (MW)

Assumptions: Discrete time: ith load Xi(t) evolves on finite state space X

10 / 19

SLIDE 38

Mean Field Model

Load Model

Controlled Markovian Dynamics

...

Load 1

BA

Reference (MW)

Load 2 Load N

ζ r

+

Gc

Power Consumption (MW)

Assumptions: Discrete time: ith load Xi(t) evolves on finite state space X Each load is subject to common controlled Markovian dynamics. Signal ζ = {ζt} is broadcast to all loads

10 / 19

SLIDE 39

Mean Field Model

Load Model

Controlled Markovian Dynamics

...

Load 1

BA

Reference (MW)

Load 2 Load N

ζ r

+

Gc

Power Consumption (MW)

Assumptions: Discrete time: ith load Xi(t) evolves on finite state space X Each load is subject to common controlled Markovian dynamics. Signal ζ = {ζt} is broadcast to all loads Controlled transition matrix {Pζ : ζ ∈ R}: P{Xi

t+1 = x′ | Xi t = x, ζt = ζ} = Pζ(x, x′)

10 / 19

SLIDE 40

Mean Field Model

Load Model

Controlled Markovian Dynamics

...

Load 1

BA

Reference (MW)

Load 2 Load N

ζ r

+

Gc

Power Consumption (MW)

Assumptions: Discrete time: ith load Xi(t) evolves on finite state space X Each load is subject to common controlled Markovian dynamics. Signal ζ = {ζt} is broadcast to all loads Controlled transition matrix {Pζ : ζ ∈ R}: P{Xi

t+1 = x′ | Xi t = x, ζt = ζ} = Pζ(x, x′)

U : X → R models the needs of the grid

10 / 19

SLIDE 41

Mean Field Model

Aggregate Model

N loads running independently, each under the command ζ.

11 / 19

SLIDE 42

Mean Field Model

Aggregate Model

N loads running independently, each under the command ζ. Empirical Distributions: µN

t (x) = 1

N

i=1

I{Xi(t) = x}, x ∈ X yN

t = 1

N

i=1

U(Xi

t) =

x

µN

t (x)U(x)

11 / 19

SLIDE 43

Mean Field Model

Aggregate Model

N loads running independently, each under the command ζ. Empirical Distributions: µN

t (x) = 1

N

i=1

I{Xi(t) = x}, x ∈ X yN

t = 1

N

i=1

U(Xi

t) =

x

µN

t (x)U(x)

Limiting model: µt+1 = µtPζt, yt :=

x

µt(x)U(x) via Law of Large Numbers for martingales

11 / 19

SLIDE 44

Mean Field Model

Aggregate Model

N loads running independently, each under the command ζ. Empirical Distributions: µN

t (x) = 1

N

i=1

I{Xi(t) = x}, x ∈ X yN

t = 1

N

i=1

U(Xi

t) =

x

µN

t (x)U(x)

Mean-field model: µt+1 = µtPζt, yt = µt(U) ζt = ft(µ0, . . . , µt) by design

11 / 19

SLIDE 45

Mean Field Model

Aggregate Model

N loads running independently, each under the command ζ. Empirical Distributions: µN

t (x) = 1

N

i=1

I{Xi(t) = x}, x ∈ X yN

t = 1

N

i=1

U(Xi

t) =

x

µN

t (x)U(x)

Mean-field model: µt+1 = µtPζt, yt = µt(U) ζt = ft(µ0, . . . , µt) by design

Question: How to design Pζ?

11 / 19

SLIDE 46

Local Design

SLIDE 47

Local Control Design

Local Design

Goal: Construct a family of transition matrices {Pζ : ζ ∈ R}

Design: Consider first the finite-horizon control problem:

pζ(x1, . . . , xT ) =

T −1

i=0

Pζ(xi, xi+1) , x0 ∈ X

12 / 19

SLIDE 48

Local Control Design

Local Design

Goal: Construct a family of transition matrices {Pζ : ζ ∈ R}

Design: Consider first the finite-horizon control problem:

pζ(x1, . . . , xT ) =

T −1

i=0

Pζ(xi, xi+1) , x0 ∈ X

Choose distribution pζ to maximize ζEpζ T

t=1

U(Xt)

− D(pp0)

D denotes relative entropy. p0 denotes nominal Markovian model.

12 / 19

SLIDE 49

Local Control Design

Local Design

Goal: Construct a family of transition matrices {Pζ : ζ ∈ R}

Design: Consider first the finite-horizon control problem:

pζ(x1, . . . , xT ) =

T −1

i=0

Pζ(xi, xi+1) , x0 ∈ X

Choose distribution pζ to maximize ζEpζ T

t=1

U(Xt)

− D(pp0)

D denotes relative entropy. p0 denotes nominal Markovian model.

Explicit solution for finite T: p∗

ζ(xT 0 ) ∝ exp

ζ

T

t=0

U(xt)

p0(xT

0 )

12 / 19

SLIDE 50

Local Control Design

Local Design

Goal: Construct a family of transition matrices {Pζ : ζ ∈ R}

pζ(x1, . . . , xT ) =

T −1

i=0

Pζ(xi, xi+1) , x0 ∈ X

Explicit solution for finite T: p∗

ζ(xT 0 ) ∝ exp

ζ

T

t=0

U(xt)

p0(xT

0 )

Markovian, but not time-homogeneous.

12 / 19

SLIDE 51

Local Control Design

Local Design

Goal: Construct a family of transition matrices {Pζ : ζ ∈ R}

pζ(x1, . . . , xT ) =

T −1

i=0

Pζ(xi, xi+1) , x0 ∈ X

Explicit solution for finite T: p∗

ζ(xT 0 ) ∝ exp

ζ

T

t=0

U(xt)

p0(xT

0 )

As T → ∞, we obtain transition matrix Pζ

12 / 19

SLIDE 52

Local Control Design

Local Design

Goal: Construct a family of transition matrices {Pζ : ζ ∈ R}

pζ(x1, . . . , xT ) =

T −1

i=0

Pζ(xi, xi+1) , x0 ∈ X

Explicit solution for finite T: p∗

ζ(xT 0 ) ∝ exp

ζ

T

t=0

U(xt)

p0(xT

0 )

As T → ∞, we obtain transition matrix Pζ Explicit construction via eigenvector problem: Pζ(x, y) = 1 λ v(y) v(x) ˆ Pζ(x, y) , x, y ∈ X, where ˆ Pζv = λv, ˆ Pζ(x, y) = exp(ζU(x))P0(x, y)

Extension/reinterpretation of [Todorov 2007] + [Kontoyiannis & M 200X]

12 / 19

SLIDE 53

µ t+1 = µtP ζt yt = µt, U Φ

t+1 = AΦt + Bζt

γt = CΦt

Linearized Dynamics

SLIDE 54

Φt ∈ R|X|,

a column vector with Φt(x) ≈ µt(x) − π(x), x ∈ X

13 / 19

SLIDE 57

Linearized Dynamics

Mean Field Model

Linearized Dynamics

Mean-field model: µt+1 = µtPζt, yt = µt(U) Linear state space model: Φt+1 = AΦt + Bζt γt = CΦt Interpretations: |ζt| is small, and π denotes invariant measure for P0.

Φt ∈ R|X|,

a column vector with Φt(x) ≈ µt(x) − π(x), x ∈ X

γt ≈ yt − y0; deviation from nominal steady-state

13 / 19

SLIDE 58

Linearized Dynamics

Mean Field Model

Linearized Dynamics

Mean-field model: µt+1 = µtPζt, yt = µt(U) Linear state space model: Φt+1 = AΦt + Bζt γt = CΦt Interpretations: |ζt| is small, and π denotes invariant measure for P0.

Φt ∈ R|X|,

a column vector with Φt(x) ≈ µt(x) − π(x), x ∈ X

γt ≈ yt − y0; deviation from nominal steady-state
A = P T

0 , Ci = U(xi), and input dynamics linearized:

13 / 19

SLIDE 59

Linearized Dynamics

Mean Field Model

Linearized Dynamics

Mean-field model: µt+1 = µtPζt, yt = µt(U) Linear state space model: Φt+1 = AΦt + Bζt γt = CΦt Interpretations: |ζt| is small, and π denotes invariant measure for P0.

Φt ∈ R|X|,

a column vector with Φt(x) ≈ µt(x) − π(x), x ∈ X

γt ≈ yt − y0; deviation from nominal steady-state
A = P T

0 , Ci = U(xi), and input dynamics linearized:

BT = d dζ πPζ

ζ=0

13 / 19

SLIDE 60

Linearized Dynamics

Example: One Million Pools in Florida

How Pools Can Help Regulate The Grid

1,5KW 400V

Needs of a single pool ⊲ Filtration system circulates and cleans: Average pool pump uses 1.3kW and runs 6-12 hours per day, 7 days per week ⊲ Pool owners are oblivious, until they see frogs and algae ⊲ Pool owners do not trust anyone: Privacy is a big concern Single pool dynamics:

1 2 T−1 T

. . .

T On Off 1 2 T−1

. . . 14 / 19

SLIDE 61

Linearized Dynamics

Pools in Florida Supply G2 – BPA regulation signal∗

Stochastic simulation using N = 105 pools

Reference Output deviation (MW)

−300 −200 −100 100 200 300 20 40 60 80 100 120 140 160 t/hour 20 40 60 80 100 120 140 160

PI control: ζt = 19et + 1.4eI

t ,

et = rt − yt and eI

t = t k=0 ek

Each pool pump turns on/off with probability depending on 1) its internal state, and 2) the BPA reg signal

∗transmission.bpa.gov/Business/Operations/Wind/reserves.aspx

15 / 19

SLIDE 62

10-2 10-1 100 101 Frequency (rad/s) 10-5 10-4 10-3 Frequency (rad/s) Magnitude (dB)

15
10
5

5 10 15 20 Phase (deg)

90
45

45 G ri d T r a n s fe r F u nc t i

n

Uncertainty Here Fans in Commercial Buildings Residential Water Heaters Refrigerators Water Pumping Pool Pumps Chiller Tanks

Bandwidth centered around its natural cycle

Reference (from Bonneville Power Authority)

10,000 pools

Output deviation

−300 −200 −100 100 200 300

Tracking BPA Regulation Signal (MW)

20 40 60 80 100 120 140 160 t/hour 20 40 60 80 100 120 140 160

Conclusions and Future Directions

SLIDE 63

Conclusions and Future Directions

Challenges: intermittence and volatility of renewable generation In the absence of grid-level efficient storage, increased need for responsive fossil-fuel generators, negating the environmental benefits of renewables Approach: creating Virtual Energy Storage through direct control of flexible loads - helping the grid while respecting user QoS (MDP on the local level and mean-field analysis of the aggregate)

16 / 19

SLIDE 64

Conclusions and Future Directions

Challenges: intermittence and volatility of renewable generation In the absence of grid-level efficient storage, increased need for responsive fossil-fuel generators, negating the environmental benefits of renewables Approach: creating Virtual Energy Storage through direct control of flexible loads - helping the grid while respecting user QoS (MDP on the local level and mean-field analysis of the aggregate) Current and future research directions Extending local control design to include disturbance from the nature Investigating needs for communication and forecast (minimizing communication and computation costs while providing reliable service to the grid) Integrating VES with traditional generation and batteries (resource allocation optimization problems involving different time scales)

16 / 19

SLIDE 65

Conclusions and Future Directions

Conclusions

Thank You!

17 / 19

SLIDE 66

Conclusions and Future Directions

References: Demand Response

S. Meyn, P. Barooah, A. Buˇ

si´ c, and J. Ehren. Ancillary service to the grid from deferrable loads: the case for intelligent pool pumps in Florida (Invited). In Proceedings of the 52nd IEEE Conf. on Decision and Control, 2013. Journal version to appear, Trans. Auto. Control.

A. Buˇ

si´ c and S. Meyn. Passive dynamics in mean field control. ArXiv e-prints: arXiv:1402.4618. 53rd IEEE Conf. on Decision and Control (Invited), 2014.

S. Meyn, Y. Chen, and A. Buˇ

si´

c. Individual risk in mean-field control models for

decentralized control, with application to automated demand response. 53rd IEEE Conf.

n Decision and Control (Invited), 2014.
J. L. Mathieu. Modeling, Analysis, and Control of Demand Response Resources. PhD

thesis, Berkeley, 2012.

D. Callaway and I. Hiskens, Achieving controllability of electric loads. Proceedings of the

IEEE, 99(1):184–199, 2011.

S. Koch, J. Mathieu, and D. Callaway, Modeling and control of aggregated heterogeneous

thermostatically controlled loads for ancillary services, in Proc. PSCC, 2011, 1–7.

H. Hao, A. Kowli, Y. Lin, P. Barooah, and S. Meyn Ancillary Service for the Grid Via

Control of Commercial Building HVAC Systems. ACC 2013

18 / 19

SLIDE 67

Conclusions and Future Directions

References: Markov Models

I. Kontoyiannis and S. P. Meyn. Spectral theory and limit theorems for geometrically

ergodic Markov processes. Ann. Appl. Probab., 13:304–362, 2003.

I. Kontoyiannis and S. P. Meyn. Large deviations asymptotics and the spectral theory of

multiplicatively regular Markov processes. Electron. J. Probab., 10(3):61–123 (electronic), 2005.

E. Todorov. Linearly-solvable Markov decision problems. In B. Sch¨
lkopf, J. Platt, and
T. Hoffman, editors, Advances in Neural Information Processing Systems, (19) 1369–1376.

MIT Press, Cambridge, MA, 2007.

M. Huang, P. E. Caines, and R. P. Malhame. Large-population cost-coupled LQG problems

with nonuniform agents: Individual-mass behavior and decentralized ε-Nash equilibria. IEEE Trans. Automat. Control, 52(9):1560–1571, 2007.

H. Yin, P. Mehta, S. Meyn, and U. Shanbhag. Synchronization of coupled oscillators is a
game. IEEE Transactions on Automatic Control, 57(4):920–935, 2012.
P. Guan, M. Raginsky, and R. Willett. Online Markov decision processes with

Kullback-Leibler control cost. In American Control Conference (ACC), 2012, 1388–1393, 2012. V.S.Borkar and R.Sundaresan Asympotics of the invariant measure in mean field models with jumps. Stochastic Systems, 2(2):322-380, 2012.

19 / 19