Bayes Statistics: the continuous case Hans-Peter Helfrich - - PowerPoint PPT Presentation

Bayes’ Statistics: the continuous case

Hans-Peter Helfrich

University of Bonn

Theodor-Brinkmann-Graduate School

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 1 / 19

Overview

1

Empirical distributions

2

Marginal and conditional density

3

Gaussian prior and likelihood density functions

4

Updating the posterior

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 2 / 19

Multi-dimensional data

Data given by [Johnson, 1996]

NR FAT BMI ABDOMEN HIP THIGH 1 12.6 23.7 85.2 94.5 59 2 6.9 23.4 83 98.7 58.7 3 24.6 24.7 87.9 99.2 59.6 4 10.9 24.9 86.4 101.2 60.1 5 27.8 25.6 100 101.9 63.2 6 20.6 26.5 94.4 107.8 66 7 19 26.2 90.7 100.3 58.4 8 12.8 23.6 88.5 97.1 60 ... ...

Random variables

We consider each row as a sample of random variables X1, X2, . . . , X5. In the cited paper N = 252 samples are given.

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 3 / 19

Discrete example

Counts

Fat/ circ. [60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 18 18 (10,20] 12 81 6 (20,30] 65 30 (30,40] 2 17 1 1 (40,50] 1

Probabilities

Fat/circ. [60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 0.071 0.071 0.000 0.000 0.000 (10,20] 0.048 0.321 0.024 0.000 0.000 (20,30] 0.000 0.258 0.119 0.000 0.000 (30,40] 0.000 0.008 0.067 0.004 0.004 (40,50] 0.000 0.000 0.000 0.004 0.000

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 4 / 19

Discrete example

Marginal probabilities

Fat/circ. [60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 0.071 0.071 0.000 0.000 0.000 0.143 (10,20] 0.048 0.321 0.024 0.000 0.000 0.393 (20,30] 0.000 0.258 0.119 0.000 0.000 0.377 (30,40] 0.000 0.008 0.067 0.004 0.004 0.083 (40,50] 0.000 0.000 0.000 0.004 0.000 0.004 0.119 0.659 0.210 0.008 0.004 1

Conditional probabilities

P(A∣B) = P(A, B) P(B) , P(B∣A) = P(A, B) P(A) P(B) = ∑ P(B∣Aj)P(Aj)

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 5 / 19

Discrete example

Conditional probabilities for fixed fat percentage 휃

[60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 0.500 0.500 0.000 0.000 0.000 0.143 (10,20] 0.121 0.818 0.061 0.000 0.000 0.393 (20,30] 0.000 0.684 0.316 0.000 0.000 0.377 (30,40] 0.000 0.095 0.810 0.048 0.048 0.083 (40,50] 0.000 0.000 0.000 1.000 0.000 0.004 0.119 0.659 0.210 0.008 0.004 1

P(20 < 휃 ≤ 30∣80 < x ≤ 100) = P(80 < x ≤ 100∣20 < 휃 ≤ 30)P(20 < 휃 ≤ 30) P(80 < x ≤ 100) = 0.684 ⋅ 0.377 0.659 = 0.392

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 6 / 19

Discrete example

Conditional probabilities for fixed abdomen circumference x

[60,80] (80,100] (100,120] (120,140] (140,160] [0,10] 0.600 0.108 0.000 0.000 0.000 (10,20] 0.400 0.488 0.113 0.000 0.000 (20,30] 0.000 0.392 0.566 0.000 0.000 (30,40] 0.000 0.012 0.321 0.500 1.000 (40,50] 0.000 0.000 0.000 0.500 0.000 0.119 0.659 0.210 0.008 0.004 1

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 7 / 19

Two-dimensional distribution

Joint distribution

For two random variables, X and Θ, we can assign a joint density distribution f (x, 휃).

Explanation

f (x, 휃)ΔxΔ휃 is the probability that a value lies in the range (x, x + Δx), (휃, 휃 + Δ휃)

Example

X Abdomen circumference 휃 Fat percentage

A b d

• m

e n c i r c . 60 80 100 120 140 F a t p e r c e n t a g e 10 20 30 40 D e n s i t y [ % ] 0.0 0.1 0.2 0.3 0.4 0.5

Figure: Empirical distribution

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 8 / 19

Empirical distribution f (x, 휃)

A b d

• m

e n c i r c . 60 80 100 120 140 F a t p e r c e n t a g e 10 20 30 40 Density [%] 0.0 0.1 0.2 0.3 0.4 0.5 0.0 0.1 0.2 0.3 0.4 0.5 60 80 100 120 140 10 20 30 40 Abdomen circ. Fat percentage

Figure: Joint distribution of body fat and abdomen c.

Data given here

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 9 / 19

Twodimensional normal density distribution

A b d

• m

e n c i r c . 60 80 100 120 140 F a t p e r c e n t a g e 10 20 30 40 Density [%] 0.0 0.1 0.2 0.3 0.4 0.5 0.0 0.1 0.2 0.3 0.4 0.5 60 80 100 120 140 10 20 30 40 Abdomen circ. Fat percentage

Figure: Joint distribution of body fat and abdomen c.

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 10 / 19

Marginal density distribution

The marginal distribution gives the distribution of one variable regardless

• f the other variable. We obtain it by integrating the densities with

respect to the other variable.

Marginal density distribution for 휃

p(휃) = ∫ f (x, 휃)dx

Marginal densities distribution for x

g(x) = ∫ f (x, 휃)d휃

Example

p(휃) Distribution of fat percentage 휃 regardless of abdomen circumference x g(x) Distribution of abdomen circumference x

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 11 / 19

Conditional densities

Density distribution p(휃∣x) of 휃 for given x p(휃∣x) = f (x, 휃) g(x) Density distribution f (x∣휃) of 휃 for given x f (x∣휃) = f (x, 휃) p(휃) By elimination of f (x, 휃) we come to

Bayes’ theorem

p(휃∣x) = f (x∣휃)p(휃) ∫ f (x∣휃)p(휃)d휃

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 12 / 19

Bayes’ theorem

Determination of an unknown parameter 휃

Given a prior knowledge by analyzing a former experiment We want to determine an unknown parameter We get further information by a new experiment The outcome should give us a credible interval for the parameter

Assumptions

The prior distribution p(휃) of the unknown parameter 휃 An observation x for the unknown parameter The likelihood L(x∣휃) for getting x for a given 휃

Bayes’ theorem

p(휃∣x) = L(x∣휃)p(휃) ∫ L(x∣휃)p(휃)d휃 ∝ L(x∣휃)p(휃)

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 13 / 19

Likelihood and density function

Prior density and likelihood

We consider the case that the prior and the likelihood are normally distributed cf. [Gelman et al., 2004, Carlin and Louis, 1996]

Prior with mean 휇0 and standard deviation 휎0

p(휃) = 1 √ 2휋휎0 exp ( −(휃 − 휇0)2 2휎2 ) = N(휃∣휇0, 휎2

0)

Likelihood with mean 휃 and standard deviation 휎

L(x∣휃) = 1 √ 2휋휎 exp ( −(x − 휃)2 2휎2 ) = N(x∣휃, 휎2) Later, we will treat more general cases by Monte Carlo simulations.

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 14 / 19

Precision

If X is a random variable with standard deviation 휎, we call 휏 =

1 휎2

precision of X. Precision is the reciprocal value of the variance. High precision implies low variance.

Theorem

Assume p(휃) has normally distributed density with mean 휇0 and precision 휏0 L(x∣휃) has normal distributed density with mean 휇 and precision 휏 Then the posterior density p(휃∣x) is also normal with mean value ¯ 휇 and precision ¯ 휏 ¯ 휇 = 휏0 휏0 + 휏 휇0 + 휏 휏0 + 휏 x ¯ 휏 = 휏0 + 휏 The mean value ¯ 휇 is the weighted mean of the mean value 휇0 and the

• utcome x, the weights are proportional to the precisions.

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 15 / 19

Proof of the theorem

We write L(x∣휃) ∝ exp ( −(x − 휃)2 2휎2 ) , p(휃) ∝ exp ( −(휃 − 휇0)2 2휎2 ) . That means the terms on the right and left hand side are proportional, the proportionality constants are independent of x and of 휃, but may depend

• n 휎 and 휎0. By multiplying both terms, we get

p(휃∣x) ∝ L(x∣휃)p(휃) ∝ exp ( −휏(x − 휃)2 2 ) exp ( −휏0(휃 − 휇0)2 2 ) ∝ exp ( −1 2(휏 + 휏0)(휃2 − 2휏x + 휏0휇0 휏 + 휏0 휃) ) . By quadratic completion, we get a normal density function, and can get the mean value and standard deviation by comparing the coefficients.

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 16 / 19

Updating the posterior

Several studies

For the first study with outcome x1, we get by Bayes’ theorem p1(휃∣x1) ∝ L1(x1∣휃)p(휃) Now for the second study, we take p1(휃∣x1) as a priori density and we

• btain

p2(휃∣x2) ∝ L2(x2∣휃)p1(휃∣x1) By repeating that procedure for normally distributed priors and likelihoods, we eventually get that pn(휃∣xn) is normally distributed with mean value 휇 = w0휇0 + w1x1 + ⋅ ⋅ ⋅ + wnxn, and precision 휏 = 휏0 + 휏1 + ⋅ ⋅ ⋅ + 휏n, and weights wi = 휏i/휏, i = 0, 1, 2, ⋅ ⋅ ⋅ , n, where

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 17 / 19

Special case

Equal variances

Now, we assume 휏0 = 0, i.e., 휎0 = ∞. We say that p has an non-informative prior. We assume further that all variances are equal. i.e., 휎1 = 휎2 = ⋅ ⋅ ⋅ = 휎n = 휎 We obtain 휏 = n 휎2 The weights are given by wi = 1 n, i = 1, 2, . . . , n. The Bayes estimate for all trials is given by 휇 = 1 n(x1 + x2 + ⋅ ⋅ ⋅ + xn) with

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 18 / 19

References

Carlin, B. P. and Louis, T. A. (1996). Bayes and empirical Bayes methods for data analysis, volume 69 of Monographs on Statistics and Applied Probability. Chapman & Hall, London. Gelman, A. B., Carlin, J. S., Stern, H., and Rubin, D. (2004). Bayesian Data Analysis. Chapman & Hall. Johnson, R. W. (1996). fitting percentage of body fat to simple body measurements. Journal of Statistics Education, 4.

H.-P. Helfrich (University of Bonn) Bayes’ Statistics Brinkmann School 19 / 19