Estimation of a rare event probability on a complex system modeled - - PowerPoint PPT Presentation

Kriging and Rare Events

Estimation of a rare event probability on a complex system modeled with a kriging algorithm.

R.PASTEL1,2 J.MORIO2 H.PIET-LAHANIER2

1ENSTA-ParisTech

FRANCE

2ONERA

FRANCE

Rare Events Simulation, Rennes, 2008

Kriging and Rare Events Introduction

Introduction

Probabilities and statistics are more and more frequently used in mathematical modeling to: structure what is unknown in a system, quantify the errors, measure a system reliability. What is, in average, the output of a complex system with random inputs?

Kriging and Rare Events Introduction

Unfortunately, the problem has often no analytical solution , and/or is not analytical itself, = ⇒ we have to go through a numerical resolution. Two numerical resolution drawbacks

1 Numerical techniques require a long calculation time, 2 Random number generators are not perfectly random!

= ⇒ no rare event is generated.

Kriging and Rare Events Introduction

Hereby, we will conjugate two approaches to solve those issues:

1 to free ourselves from the system’s analytical expression and

shorten the calculation time, we will use kriging, a statistical interpolation technique.

2 to quantify the randomness taking rare events into account,

we will compare three methods:

The Crude Monte-Carlo (CMC), The histogram, The Importance Sampling (IS).

Kriging and Rare Events Summary

Summary

1 Kriging

The problem intuition Elements of theory An example Use and limitations

Kriging and Rare Events Summary

Summary

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

Kriging and Rare Events Summary

Summary

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Kriging

Current Section

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Kriging The problem intuition

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Kriging The problem intuition

Kriging

The problem intuition

Kriging is an interpolation technique originated in Geophysics. Measurement drillings have to be in limited number to reduce costs and shorten the exploration time of a given subsoil.

Kriging and Rare Events Kriging The problem intuition

Figure: Subsoil to explore to find the gold.

Kriging and Rare Events Kriging The problem intuition

Figure: Some gold concentration measurements are done.

Kriging and Rare Events Kriging The problem intuition

Figure: What is the gold concentration elsewhere?

Kriging and Rare Events Kriging Elements of theory

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Kriging Elements of theory

Kriging

Formal idea

One wants to know the values of the criterion c on a domain of interest D. However, it is impossible to have infinitely many exact values as: c has no analytical expression, AND/OR c is costly to evaluate numerically. Problem framework It is therefore about building up a good approximation of the criterion

• n the whole domain of interest, based on few exact values measured

in well-chosen locations.

Kriging and Rare Events Kriging Elements of theory

A classical regression approach

Let us review a classical regression approach.

Kriging and Rare Events Kriging Elements of theory

A classical regression approach

Let us review a classical regression approach. Basic idea: Criterion = Model + Error For a given x ∈ D: c(x) = F(β, x) + ǫ(β, x)

Kriging and Rare Events Kriging Elements of theory

A classical regression approach

Let us review a classical regression approach. Basic idea: Criterion = Model + Error For a given x ∈ D: c(x) = F(β, x) + ǫ(β, x) Usually: c(x) = f (x)β + ǫ(β, x)

Kriging and Rare Events Kriging Elements of theory

A classical regression approach

Let us review a classical regression approach. Basic idea: Criterion = Model + Error For a given x ∈ D: c(x) = F(β, x) + ǫ(β, x) Usually: c(x) = f (x)β + ǫ(β, x) ⇓ ⇓ ⇓ Measured data matrices: C = Fβ + E

Kriging and Rare Events Kriging Elements of theory

A classical regression approach

Let us review a classical regression approach. Basic idea: Criterion = Model + Error For a given x ∈ D: c(x) = F(β, x) + ǫ(β, x) Usually: c(x) = f (x)β + ǫ(β, x) ⇓ ⇓ ⇓ Measured data matrices: C = Fβ + E = ⇒ β is defined

Kriging and Rare Events Kriging Elements of theory

A classical regression approach

Let us review a classical regression approach. Basic idea: Criterion = Model + Error For a given x ∈ D: c(x) = F(β, x) + ǫ(β, x) Usually: c(x) = f (x)β + ǫ(β, x) ⇓ ⇓ ⇓ Measured data matrices: C = Fβ + E = ⇒ β is defined as the solution of minβ E2

2.

Kriging and Rare Events Kriging Elements of theory

Two steps further

ˆ c,the kriging estimator The two extra hypothesis behind kriging are

Kriging and Rare Events Kriging Elements of theory

Two steps further

ˆ c,the kriging estimator The two extra hypothesis behind kriging are

1 ǫ(β, x)

Kriging and Rare Events Kriging Elements of theory

Two steps further

ˆ c,the kriging estimator The two extra hypothesis behind kriging are

1 ǫ(β, x) is the trajectory of a random process.

Kriging and Rare Events Kriging Elements of theory

Two steps further

ˆ c,the kriging estimator The two extra hypothesis behind kriging are

1 ǫ(β, x) is the trajectory of a random process. 2 ˆ

c(x)

Kriging and Rare Events Kriging Elements of theory

Two steps further

ˆ c,the kriging estimator The two extra hypothesis behind kriging are

1 ǫ(β, x) is the trajectory of a random process. 2 ˆ

c(x) = κT(x)C with κ : Rn → Rm.

Kriging and Rare Events Kriging Elements of theory

Two steps further

ˆ c,the kriging estimator The two extra hypothesis behind kriging are

1 ǫ(β, x) is the trajectory of a random process. 2 ˆ

c(x) = κT(x)C with κ : Rn → Rm. = ⇒ ∀x ∈ D, (β, κ) is defined

Kriging and Rare Events Kriging Elements of theory

Two steps further

ˆ c,the kriging estimator The two extra hypothesis behind kriging are

1 ǫ(β, x) is the trajectory of a random process. 2 ˆ

c(x) = κT(x)C with κ : Rn → Rm. = ⇒ ∀x ∈ D, (β, κ) is defined as the solution of

• minβ,κ E[ˆ

c(x) − c(x)2

2]

Under the no bias constraint F Tc = f (x)

Kriging and Rare Events Kriging An example

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Kriging An example

Kriging and Rare Events Kriging An example

Kriging and Rare Events Kriging An example

Kriging and Rare Events Kriging An example

Kriging and Rare Events Kriging Use and limitations

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Kriging Use and limitations

Kriging

Use and limitations

Kriging in a nutshell Kriging enables to interpolate any given criterion, analytical or not,

• n a restrained domain thanks to sufficiently many well-spread

exact values. Error is nil at the measure sites and increases when moving away from them. In particular: Kriging interpolates but does not forecast → the proxy is bad in subdomains with no measure. The proxy quality is good in a limited area around the measurement site → a good proxy requires a sufficient measurement site density.

Kriging and Rare Events Probabilistic methods

Current Section

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Probabilistic methods The experience framework

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Probabilistic methods The experience framework

The experience framework

Kriging and Rare Events Probabilistic methods The experience framework

The experience framework

We consider a dummy mapping Z : R2 → R.

Kriging and Rare Events Probabilistic methods The experience framework

The experience framework

We consider a dummy mapping Z : R2 → R. We assume its parameters X = (X1, X2) are distributed according to a centered, very low variance Gaussian.

Kriging and Rare Events Probabilistic methods The experience framework

The experience framework

We consider a dummy mapping Z : R2 → R. We assume its parameters X = (X1, X2) are distributed according to a centered, very low variance Gaussian. For some given thresholds d, we will estimate the non-exceedance in absolute value probabilities P(|Z(X)| ≤ d)

Kriging and Rare Events Probabilistic methods The experience framework

The experience framework

We consider a dummy mapping Z : R2 → R. We assume its parameters X = (X1, X2) are distributed according to a centered, very low variance Gaussian. For some given thresholds d, we will estimate the non-exceedance in absolute value probabilities P(|Z(X)| ≤ d) twice:

Kriging and Rare Events Probabilistic methods The experience framework

The experience framework

We consider a dummy mapping Z : R2 → R. We assume its parameters X = (X1, X2) are distributed according to a centered, very low variance Gaussian. For some given thresholds d, we will estimate the non-exceedance in absolute value probabilities P(|Z(X)| ≤ d) twice:

1 with the original mapping Z.

Kriging and Rare Events Probabilistic methods The experience framework

The experience framework

We consider a dummy mapping Z : R2 → R. We assume its parameters X = (X1, X2) are distributed according to a centered, very low variance Gaussian. For some given thresholds d, we will estimate the non-exceedance in absolute value probabilities P(|Z(X)| ≤ d) twice:

1 with the original mapping Z. 2 with the kriged mapping ˆ

Z.

Kriging and Rare Events Probabilistic methods The experience framework

Kriging and Rare Events Probabilistic methods The experience framework

Kriging and Rare Events Probabilistic methods The experience framework

The domain of interest D

Two constraints on D

Kriging and Rare Events Probabilistic methods The experience framework

The domain of interest D

Two constraints on D

1 D has to be small enough to have a decent measure site

density

Kriging and Rare Events Probabilistic methods The experience framework

The domain of interest D

Two constraints on D

1 D has to be small enough to have a decent measure site

density and to be accurate.

Kriging and Rare Events Probabilistic methods The experience framework

The domain of interest D

Two constraints on D

1 D has to be small enough to have a decent measure site

density and to be accurate.

2 D has to be wide enough to encompass the major part of fX’s

support

Kriging and Rare Events Probabilistic methods The experience framework

The domain of interest D

Two constraints on D

1 D has to be small enough to have a decent measure site

density and to be accurate.

2 D has to be wide enough to encompass the major part of fX’s

support and to allow rare events.

Kriging and Rare Events Probabilistic methods The Crude Monte-Carlo (CMC)

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Probabilistic methods The Crude Monte-Carlo (CMC)

The Crude Monte-Carlo

Basic idea

CMC basic idea The empirical average legitimated by the law of great numbers and the central limit theorem: = ⇒ A random variable expectancy can be estimated by the arithmetical mean of a large realization sample and the larger the sample, the better the proxy. Then, one should:

1 Generate a sample as large as possible. 2 Make the arithmetical mean of the realizations.

Kriging and Rare Events Probabilistic methods The Crude Monte-Carlo (CMC)

Two major CMC drawbacks

1 One does not always know how to simulate the random

variable of interest.

2 A rare event, i.e. whose probability is, say, lower than 10−6, is

almost never generated through CMC.

Kriging and Rare Events Probabilistic methods Importance Sampling

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Probabilistic methods Importance Sampling

Importance Sampling

Basic idea If the random variable X is not convenient, use another one Y ! Y is designed to meet two needs:

1 Enhance the simulation conditions. 2 Encourage the realization of rare events.

Kriging and Rare Events Probabilistic methods Importance Sampling

The formal trick

1 Choose a new random variable. 2 Link it with the original one through a weighting. 3 Use CMC to estimate the new expectancy.

E[Z(X)] =

• Rn Z(x)fX(x)dx

=

• dom(fX )

Z(x)fX(x) fY (x)fY (x)dx =

• Rn Z(y)fX(y)

fY (y)fY (y)dy = E[Z(Y )fX(Y ) fY (Y )]

Kriging and Rare Events Probabilistic methods Importance Sampling

Kriging and Rare Events Probabilistic methods Importance Sampling

Kriging and Rare Events Probabilistic methods Importance Sampling

Kriging and Rare Events Probabilistic methods The Histogram

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Probabilistic methods The Histogram

The histogram

The fundamental technique In practice, most of the time, one observes a random feature and has to content himself with a global modeling:

1 One splits the observations into subsets called bins. 2 One forecasts based on each bin’s observations frequency.

Formally, we make the problem discreet and build up the image probabilty by "transferring the probability weights" .

Kriging and Rare Events Probabilistic methods The Histogram

Figure: Probability transfer in dimension one to build up the image probability.

Kriging and Rare Events Probabilistic methods Experimental results

Progress

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Probabilistic methods Experimental results

Results comparison

For CMC and IS, the given figures are an average of 100 estimations based on 100000 points each and their relative error. To build up the histogram, D was divided with a

1 400 step in both directions.

d 1 3 5 P(|Z(X)| ≤ d)MCC 0.4391 1 − 7 ∗ 10−7 1 ρMCC 0.26% 418.84% P(|Z(X)| ≤ d)IS 0.4343 1 − 3.6653 ∗ 10−7 1 − 3.4480 ∗ 10−9 ρIS 1.69% 2.23% 2.89% P(|Z(X)| ≤ d)hist 0.4391 1 − 3.6755 ∗ 10−7 1 − 3.4576 ∗ 10−9

Table: P(|Z(X)| ≤ d) estimates for three thresholds d in the three stochastic ways.

Kriging and Rare Events Probabilistic methods Experimental results

Using ˆ Z instead of Z

The very same estimates are made but substituting ˆ Z to Z. d 1 3 5 P(|ˆ Z(X)| ≤ d)MCC 0.4421 1 − 7 ∗ 10−7 1 ρMCC 0.26% 418.84% P(|ˆ Z(X)| ≤ d)IS 0.4385 1 − 3.6292 ∗ 10−7 1 − 3.4633 ∗ 10−9 ρIS 1.56% 2.56% 2.81% P(|ˆ Z(X)| ≤ d)hist 0.4423 1 − 3.6160 ∗ 10−7 1 − 3.4320 ∗ 10−9

Table: P(|Z(X)| ≤ d) estimates for three thresholds d in the three stochastic ways using ˆ Z instead of Z.

Kriging and Rare Events Probabilistic methods Experimental results

Three approaches and a substitution

1 CMC is very instinctive but does not make the rare events

appear.

2 IS focuses on rare events but requires more a priori knowledge. 3 The histogram makes the probability calculations easier but

needs turning the problem discreet before hand. The Main Result Substituting ˆ Z to Z when estimating the expectancy can lead to a very good approximation!

Kriging and Rare Events Application to missile firing

Current Section

1 Kriging

The problem intuition Elements of theory An example Use and limitations

2 Probabilistic methods

The experience framework The Crude Monte-Carlo (CMC) Importance Sampling The Histogram Experimental results

3 Application to missile firing

Kriging and Rare Events Application to missile firing

Application to missile impact scattering.

Figure: Method application system.

Kriging and Rare Events Application to missile firing

PIMSBAL

Kriging and Rare Events Application to missile firing

Figure: Kriged missile mapping estimate based on PIMSBAL.

Kriging and Rare Events Application to missile firing

For CMC and IS, the given figures are an average of 100 estimations based on 100000 points each. To build up the histogram, D was divided with a

1 1500 step in both directions.

d 5.7033 ∗ 105 5.8913 ∗ 105 6.2668 ∗ 105 P(|Z(X)| ≤ d)CMC 0.9947 1 − 0.0003 1 P(|Z(X)| ≤ d)IS 0.9947 0.99968 1 − 1.1511 ∗ 10−7 P(|Z(X)| ≤ d)hist 0.9947 0.99963 1 − 1.1053 ∗ 10−7

Table: P(|Z(X)| ≤ d) estimates for three thresholds d in the three stochastic ways in the PIMSBAL missile fire simulator.

Kriging and Rare Events Conclusion

Conclusion We developed an efficient method to estimate an expectancy written as E[Z(X)] when The pdf of X is known. Z’s analytical expression is either unknown or Z is hard to evaluate numerically.

• ne wants to take the rare events into account.

Actually, we have shown that The Importance Sampling or an histogram proved accurate estimations for any level of scarcity. E[ˆ Z(X)] was a good approximate to E[Z(X)].

Kriging and Rare Events Conclusion

Future Work: My PhD! To start my PhD at the ONERA under the Direction of Mr.LEGLAND from IRISA, I will: Study the impact of the interpolation quality. Look for a better IS new random variable through Cross-Entropy. Try new probabilistic methods such as Splitting Techniques and Interacting Particles.

Kriging and Rare Events Conclusion

Thank you for your attention!