Imprecise Markov chains From basic theory to applications II prof. - - PowerPoint PPT Presentation

Imprecise Markov chains

From basic theory to applications II

• prof. Jasper De Bock

Imprecise continuous-time Markov chains

Imprecise continuous-time Markov chains

Continuous-time Markov chains

Continuous-time Markov chains Markov assumption

Continuous-time Markov chains…

≈ I(x, y) + ∆ Qt(x, y)

…that are nice enough

Continuous-time Markov chains

≈ I(x, y) + ∆ Qt(x, y)

Continuous-time Markov chains…

∆ Qt(x, y)

Let’s assume that this does not depend

• n time!

Continuous-time Markov chains… …that are homogeneous

Q(x, y)

Continuous-time Markov chains

Q(x, y)

Continuous-time Markov chains

Q(x, y)

that’s just a probability mass function

P

yQ(x, y) = 0

(8y 6= x) Q(x, y) 0 (∀x) Q(x, x) ≤ 0

initial distribution transition rate matrix

π0(x)

Amorous Bickering Confusion Depression

What is ?

P(Xt = y|X0 = x)

Tt(x, y) := P(Xt = y|X0 = x) d dtTt = QTt , with T0 = I

transition matrix backward Kolmogorov differential equation

⇒ Tt = eQt =

lim

n→+∞(I + t

nQ)n

What is ?

P(Xt = y|X0 = x)

Tt(x, y) := P(Xt = y|X0 = x) d dtTt = QTt , with T0 = I

transition matrix backward Kolmogorov differential equation

eQt(x, y)

⇒ Tt = eQt =

lim

n→+∞(I + t

nQ)n

What is ?

P(Xt = y|X0 = x)

What is ? What is ?

P(Xt = y|X0 = x)

What is ?

E(f(Xt)|X0 = x) eQt(x, y) eQtf(x) P(Xt = y) π0eQt(y) π0eQtf

What is ?

E(f(Xt))

… …

What is ?

P(Xt = y|X0 = x) eQt(x, y) lim

t→+∞ P(Xt = y|X0 = x) =

lim

t→+∞ eQt(x, y)

The following limit always exists! And often does not depend on !

x π∞(y) = lim

t→+∞ P(Xt = y) =

lim

t→+∞ π0eQt(y)

That’s all fine and well, but what can you use it for?

Reliability engineering (failure probabilities, …) Queuing theory (waiting in line …)

• optimising supermarket waiting times
• dimensioning of call centers
• airport security lines
• router queues on the internet

Cell division in biology (how long does it take?) …

Message passing in optical links

m1

channels type I messages require 1 channel type II messages require channels

n2

We want to minimise the blocking probability of messages by finding an optimal policy

Message passing in optical links

superchannels

m2 = m1 n2

type I messages require 1 channel type II messages require channels

n2 m1

channels We want to minimise the blocking probability of messages by finding an optimal policy

So how about imprecision?

Imprecise continuous-time Markov chains

Imprecise continuous-time Markov chains

Q(x, y)

?

What if we don’t know these (exactly)

Imprecise continuous-time Markov chains

Q(x, y)

?

What if we don’t know these (exactly)

∈

∈

Q

P

What is ? What is ?

P(Xt = y|X0 = x)

What is ?

E(f(Xt)|X0 = x) eQt(x, y) eQtf(x) P(Xt = y) π0eQt(y) π0eQtf

What is ?

E(f(Xt))

Optimising with respect to and yields lower and upper bounds

π0 ∈ P Q ∈ Q

? ? ? ? ? ?

Imprecise continuous-time Markov chains

∆ Qt(x, y)

Let’s assume that this does not depend

• n time!

Imprecise continuous-time Markov chains

∆ Qt(x, y)

Let’s assume that this does not depend

• n time!

Imprecise continuous-time Markov chains

≈ I(x, y) + ∆ Qt(x, y)

∈

Q

In that case, all we know is that

What is ? What is ?

P(Xt = y|X0 = x)

What is ?

E(f(Xt)|X0 = x) eQt(x, y) eQtf(x) P(Xt = y) π0eQt(y) π0eQtf

What is ?

E(f(Xt))

Optimising with respect to and yields lower and upper bounds

π0 ∈ P Qt ∈ Q

Optimising with respect to and yields lower and upper bounds

π0 ∈ P Qt ∈ Q

! !

this turns

• ut to be

surprisingly simple

(in many cases)

Lower transition operator backward Kolmogorov differential equation

⇒

What is ?

E(f(Xt)|X0 = x) T tf(x) = E(f(Xt)|X0 = x) = min

Q∈Q E(f(Xt)|X0 = x)

d dtT t = QT t, with T 0 = I T t = eQt = lim

n→+∞(I + t

nQ)n Qf(x) = min

Q∈Q Qf(x)

Lower transition rate operator

eQtf(x)

eQtf(x)

Lower transition operator backward Kolmogorov differential equation

⇒

What is ?

E(f(Xt)|X0 = x) T tf(x) = E(f(Xt)|X0 = x) = min

Q∈Q E(f(Xt)|X0 = x)

d dtT t = QT t, with T 0 = I T t = eQt = lim

n→+∞(I + t

nQ)n Qf(x) = min

Q∈Q Qf(x)

Lower transition rate operator

≥

… …

What is ?

E(f(Xt)|X0 = x) eQtf(x)

≤ −(eQt(−f))(x)

What is ?

P(Xt = y|X0 = x)

≤ −(eQt(−Iy))(x)

≥ eQtIy(x)

≥ ≥ ≥

What is ?

E(f(Xt))

≥

The following limit always exists! And often does not depend on !

x

What is ?

E(f(Xt)|X0 = x) eQtf(x) lim

t→+∞ E(f(Xt)|X0 = x) =

lim

t→+∞ eQtf(x)

E∞f = lim

t→+∞ E(f(Xt))

with E(f(Xt)) = min

π0∈P min Q∈Q E(f(Xt) = min π0∈P π0eQtf

≥

Imprecise continuous-time Markov chains

≈ I(x, y) + ∆ Qt(x, y)

Markov assumption

Imprecise continuous-time Markov chains Markov assumption

≈ I(x, y) + ∆ Qt,x1,...,xn(x, y)

Imprecise continuous-time Markov chains

≈ I(x, y) + ∆ Qt,x1,...,xn(x, y)

In that case, all we know is that

∈

Q

What is ? What is ?

P(Xt = y|X0 = x)

What is ?

E(f(Xt)|X0 = x) eQt(x, y) eQtf(x) P(Xt = y) π0eQt(y) π0eQtf

What is ?

E(f(Xt))

Optimising with respect to and yields lower and upper bounds

π0 ∈ P Qt,x1,...,xn ∈ Q

! !

(in many cases) Optimising with respect to and yields lower and upper bounds

π0 ∈ P

this turns 


• ut to (still) be

surprisingly simple

Qt,x1,...,xn ∈ Q

… …

What is ?

E(f(Xt)|X0 = x) eQtf(x)

≤ −(eQt(−f))(x)

What is ?

P(Xt = y|X0 = x)

≤ −(eQt(−Iy))(x)

≥ eQtIy(x)

≥ ≥ ≥

What is ?

E(f(Xt))

≥

That’s enough! Too confusing! And time is running out…

Partially specified and are allowed Time homogeneity can be dropped The Markov assumption can be dropped Advantages of imprecise (continuous-time) Markov chains over their precise counterpart

π0 Q

Efficient computations remain possible …

NOT 
 GOOD GOOD Amorous Bickering Confusion Depression

Thomas Krak, Jasper De Bock, Arno Siebes. Imprecise continuous-time Markov chains. International Journal of Approximate Reasoning, 88: 452-528. 2017. Jasper De Bock. The limit behaviour of imprecise continuous- time Markov chains. Journal of nonlinear Science, 27(1): 159-196. 2017.

References

Damjan Skulj. Efficient computation of the bounds of continuous time imprecise Markov chains. Applied mathematics and computation, 250(C): 165-180, 2015. Matthias C.M. Troffaes, Jacob Gledhill, Damjan Skulj, Simon

• Blake. Using imprecise continuous time Markov chains for

assessing the reliability of power networks with common cause failure and non-immediate repair. Proceedings of ISIPTA ’15: 287-294, 2015. [1] [2] [3] [4]

References (2)

Cristina Rottondi, Alexander Erreygers, Giacomo Verticale, Jasper De Bock. Modelling spectrum assignment in a two- service flexi-grid optical link with imprecise continuous-time Markov chains. Proceedings of DRCN 2017: 39-46. 2017. Alexander Erreygers, Jasper De Bock. Imprecise continuous- time Markov chains: efficient computational methods with guaranteed error bounds. PMLR: proceedings of machine learning research, 62 (proceedings of ISIPTA ’17): 145-156. 2017. Thomas Krak, Jasper De Bock, Arno Siebes. Efficient computation of updated lower expectations for imprecise continuous-time hidden Markov chains. PMLR: proceedings of machine learning research, 62 (proceedings of ISIPTA ’17): 193-204. 2017. [5] [6] [7]