Lectures 2 and 3: Goodness of Fit Applied Statistics 2014 1 / 36 - - PowerPoint PPT Presentation

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Lectures 2 and 3: Goodness of Fit

Applied Statistics 2014

1 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Goodness of Fit (GoF) testing

Let the observations X1, . . . , Xn be i.i.d from some unknown distribution F.

1 Simple GOF

H0 : F = F0 H1 : F = F0. F0 completely specified.

2 Composite GOF

H0 : F ∈ F H1 : F ∈ F. F some parametric class of distribution functions.

2 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Empirical distribution function (EDF) based tests

It is used for continuous distributions. It is typically more powerful than chi-square tests.

Chi-square tests

It is applicable to both continuous and discrete random variables. It is also useful for random vectors, so multi-dimensional.

Probability plotting – graphical tool

3 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Simple GoF: EDF-based statistics

Under H0, we have (by Glivenko-Cantelli theorem), sup

t | ˆ

Fn(t) − F0(t)|

a.s.

→ 0, n → ∞. Any discrepancy measure between ˆ Fn and F0 serves as a test statistic. Well known test statistics for continuous F0 Dn = sup

x∈R

| ˆ Fn(x) − F0(x)| Kolmogorov-Smirnov Cn = n ∞

−∞

( ˆ Fn(x) − F0(x))2dF0(x) Cram´ er-Von-Mises An = n ∞

−∞

( ˆ Fn(x) − F0(x))2 F0(x)(1 − F0(x))dF0(x) = ∞

−∞

ˆ Fn(x) − E0 ˆ Fn(x) sd0 ˆ Fn(x) 2 dF0(x) Anderson-Darling

4 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Simple GoF: Computational formulae

Put U(i) = F0(X(i)). Then, we have, Dn = max

1≤i≤n max{| ˆ

Fn(X(i)) − F0(X(i))|, | ˆ Fn(X(i)−) − F0(X(i))|} = max

1≤i≤n max{|i/n − U(i)|, |U(i) − (i − 1)/n|};

Cn = 1 12n +

n

• i=1
• U(i) − 2i − 1

2n 2 ; An = −n − 1 n

n

• i=1

(2i − 1)[log U(i) + log(1 − U(n−i+1))].

5 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Simple GoF: Distribution of test statistic under H0 (1)

Probability integral transform

If X has a continuous distribution F0, then F0(X) has a uniform distribu- tion on [0, 1], i.e. UN [0, 1]. Under H0, {U(i), 1 ≤ i ≤ n} are order statistics from UN[0, 1].

Corollary

Under H0, the distributions of Dn, Cn and An are all independent of F0. Thus, the tests are distribution free. To compute the critical value, Small n: tables. Large n: asymptotics

6 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Simple GoF: Distribution of test statistic under H0 (2)

Under H0, sup

t∈R

| ˆ Fn(t)−F0(t)|

d

= sup

t∈[0,1]

|Un(t)−t| where Un(t) = 1 n

n

• 1

1{Ui≤t}, with Ui i.i.d from UN[0, 1]. Theorem (Donsker) {√n(Un(t)−t)}t∈[0,1] converges in distribution (in the space D[0, 1]) to a standard Brownian Bridge B0, which is a Gaussian process with E(B0(t)) = 0 and E(B0(t1)B0(t2)) = t1(1 − t2), for 0 ≤ t1 ≤ t2 ≤ 1. This implies (by continuous-mapping theorem), under H0, √nDn

d

→ sup

t∈[0,1]

|B0(t)|. and Cn

d

→ 1 B2

0(t)dt

and Dn

d

→ 1 B2

0(t)

t(1 − t)dt.

7 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Simple GoF: Consistency under alternatives

Let Gn(t) = PF0(√nDn ≤ t) . Reject H0 : F = F0 (with significance α) if √nDn > G−1

n (1 − α),

with 0 < α < 1.

Lemma (consistency under any alternative)

If the data {Xi}n

i=1 comes from a distribution F = F0 then

PF (√nDn > G−1

n (1 − α)) → 1 ,

as n → ∞.

8 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Some comments

The KS test, Dn, is probably most well known. However it is often (much) less powerful than the quadratic statistics Cn and An. An behaves similarly to Cn, but is more powerful when F0 departs from the true (underlying) distribution in the tails. EDF statistics are usually more powerful than the Pearson chi-square statistics, which we shall discuss later on.

9 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Composite tests: location-scale families

A location-scale family of distributions: Fθ(x) = H

• x−θ1

θ2

• where H

is some known distribution function and θ = (θ1, θ2), with location parameter θ1 ∈ R and the scale parameter θ2 > 0. H0 : F = Fθ, for some θ ∈ Θ ⊆ R × (0, ∞). General idea: compare ˆ Fn and Fˆ

θn, where ˆ

θn is an efficient estimator

• f θ.

Theorem (Antle and Bain 1969): Let (ˆ θ1, ˆ θ2) be the MLE of (θ1, θ2). Then the quantities ˆ θ2/θ2, (ˆ θ1 − θ1)/θ2 and (ˆ θ1 − θ1)/ˆ θ2 are each distributed independently of θ1 and θ2. Replacing F0 with Fˆ

θ in the definition of Dn, Cn and An, we obtain

the (EDF) test statistics for the composite test. The distribution of the obtained test statistics, under H0, depends on the d.f. H. The critical values are often computed via simulations.

10 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Composite tests for normality

H0 : F0 is a normal distribution function. Example: AD-test statistic for normality An = n ( ˆ Fn(t) − Φn(t))2 Φn(t)(1 − Φn(t))dΦn(t) Φn(t) = Φ t − ¯ Xn Sn

• .

Computational formula An = −n − 1 n

n

• i=1

(2i − 1)[log U(i) + log(1 − U(n−i+1))]. with U(i) = Φ X(i) − ¯ Xn Sn

• .

Similarly: Cr´ amer - Von Mises Cn and KS-test Dn (known as Lilliefors test).

11 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Other specialized tests for assessing normality

Jarque-Bera test (1980).

• emp. skewness

b1 =

1 n

n

1(Xi − ¯

Xn)3 S3

• emp. kurtosis

b2 =

1 n

n

1(Xi − ¯

Xn)4 S4 Now √nb1

w

− → N(0, 6) and √n(b2 − 3)

w

− → N(0, 24). The Jarque-Bera statistic is given by JB = n b2

1

6 + (b2 − 3)2 24

• .

12 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Other specialized tests for assessing normality

Shapiro-Wilk Test statistic W = n

i=1 aiX(i)

2 n

i=1(Xi − ¯

Xn)2 (∈ (0, 1]), for certain a1, . . . , an (fixed). Under H0, the numerator is an estimator for a multiple of σ2, the denominator is an estimator for (n − 1)σ2. Under H0, W ≈ 1. Under H1, the numerator is usually smaller.

13 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Tests for normality in R

In R: ks.test(x,"pnorm") # can also be used # for other parametric families shapiro.test(x) library(nortest) ad.test(x) cvm.test(x) library(tseries) jarque.bera.test(x) These are tests for a composite null hypothesis. Simulation studies: Anderson-Darling test preferable.

14 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

A simulation study to assess the performance of tests for normality.

We compute the fraction of times that the null hypothesis of normality is rejected for a number of distributions (in total we simulated 1000 times). Results for n= 20 norm cauchy exp t5 t10 t15 Shapiro 0.054 0.852 0.852 0.182 0.095 0.080 KS 0.038 0.206 1.000 0.067 0.050 0.046 AD 0.043 0.863 0.799 0.166 0.092 0.074 CvM 0.050 0.864 0.751 0.157 0.081 0.070 JB 0.025 0.807 0.516 0.162 0.067 0.060

15 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

A simulation study...

Results for n= 50 norm cauchy exp t5 t10 t15 Shapiro 0.065 0.994 1.000 0.360 0.152 0.100 KS 0.062 0.472 1.000 0.066 0.045 0.054 AD 0.055 0.994 0.998 0.289 0.123 0.073 CvM 0.055 0.738 0.989 0.249 0.113 0.070 JB 0.043 0.993 0.953 0.396 0.172 0.106

16 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

A simulation study...

Results for n= 200 norm cauchy exp t5 t10 t15 Shapiro 0.054 1.000 1.000 0.825 0.362 0.223 KS 0.044 0.999 1.000 0.084 0.058 0.047 AD 0.052 NA NA NA 0.258 0.136 CvM 0.044 0.003 0.981 0.689 0.213 0.107 JB 0.049 1.000 1.000 0.869 0.436 0.291

17 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

A simulation study...

Results for n= 5000 norm cauchy exp t5 t10 t15 Shapiro 0.056 1 1 1.000 1.000 0.997 KS 0.047 1 1 1.000 0.693 0.205 AD 0.058 NA NA NA NA 0.989 CvM 0.061 1 1 1.000 1.000 0.962 JB 0.057 1 1 1.000 1.000 1.000

18 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Simple GoF: Chi-square type tests

H0 : F = F0 H1 : F = F0. Let S = supp(F0). Fix a positive integer k. Let S =

k

• i=1

Ak,i be a partition of S. Define Ni := #{j : Xj ∈ Ak,i}. Under H0: ei := E0Ni = nP0(X ∈ Ak,i) and we expect ei and Ni to be close.

19 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Simple GoF: Chi-square type tests

Chi-square statistic Q =

k

• i=1

(Ni − ei)2 ei . For n large, Q is distributed as χ2

k−1.

Example Genetic theory predicts that certain fruit flies will fall into four categories in proportions 9:3:3:1. Data showed counts of 59, 20, 11 and 10.

20 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Simple GoF: Chi-square type tests

In R: chisq.test(c(59,20,11,10),p=c(9/16,3/16,3/16,1/16)) Chi-squared test for given probabilities data: c(59, 20, 11, 10) X-squared = 5.6711, df = 3, p-value = 0.1288

21 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Composite GoF: Chi-Square type tests

H0 : F = Fθ, where θ is an unknown parameter vector. The test statistics is still Q whereas now ei’s are unknown under H0. In order to get ei, we need to estimate θ first.

If θ is estimated with MLE, ˆ θ1, based on N1, . . . , Nk, then asymptotically, Q1 = Qˆ

θ1 ∼ χ2 k−1−c,

where c is the length of θ, the number of the unknown parameters. If θ is estimated with MLE, ˆ θ2, based on X1, . . . , Xn, then asymptotically, Q2 = Qˆ

θ2

is not necessary χ2 distributed but it is bounded between χ2

k−1 and

χ2

k−1−c.

Reference: Chernoff and Lehmann (1954)

22 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Choice of k

A common practical recommendation is k = 2n

2 5 .

n Integer nearest 2n2/5 25 7 50 10 80 12 100 13

23 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Probability plotting: location-scale families

Not a formal test, but visual check if assuming certain distribution is reasonable. Requires some training! Idea behind probability plotting: Suppose X1, . . . , Xn are i.i.d. from F. H0 : F = Fθ(x) = H

• x−θ1

θ2

• f a location-scale family .

Under H0, Fθ(X(i)) ≈ ˆ Fn(X(i)) = i n. X(i) ≈ F −1

θ

i n

• = θ1 + θ2H−1

i n

• .

24 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Probability plotting

Under H0, the points

• H−1

i n+1

• , X(i)
• should be approximately on a

straight line. Some authors suggest that it is slightly better to use

• H−1

i−3/8 n+1/4

• n ≤ 10

H−1 i−0.5

n

• n ≥ 11

In R x<-rnorm(10) plot(qnorm(ppoints(x)),sort(x)) # or qqplot(qnorm(ppoints(x)),x) # in case of normality we can also use qqnorm(x) (Use e.g. qexp, for exponential location scale family.)

25 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Probability plotting in R

Figure: Normal QQ-plot for 20 data from the standard normal distribution.

Min. 1st Qu. Median Mean 3rd Qu. Max.

• 0.94280 -0.50670 -0.11250

0.01467 0.47890 1.59000

26 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Probability plotting vs. testing

For most practical purposes (residual analysis), the difference between a N(0, 1) and t15 random variable is unimportant. Simulate data from both distributions and use the AD-test for normality. P-values: n normal t\_15 10 0.14 0.63 100 0.47 0.52 10000 0.90 5.29e-06 Eventually, (as n gets large), we will always detect the difference.

27 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Probability plotting vs. testing

Figure: Normal QQ-plots. Upper: Normal data. Lower: t15 data. From left to right: n = 10, n = 100 and n = 10000.

28 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Why probability plots can mislead

Theorem (correlation of probability plots for normality)

Put zi = Φ−1

i n+1

• . Let rn =

n

i=1 ziXi

√n

i=1 z2 i

n

i=1(Xi− ¯

X)2 , i.e. rn is the

correlation coefficient of the pairs

• X(i), zi
• . Let F be the true CDF with

variance σ2, then lim

n→∞ rn = 1

σ 1 F −1(x)Φ−1(x)dx =: ρF a.s.

F ρF normal 1 uniform 0.98 double exp 0.98 t3 0.90 t5 0.98 χ2

3

0.96 exponential 0.90 logistic 0.97

29 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Large datasets: comparing GOF-tests with prob. plotting

Figure: Normal QQ-plots, n = 100

Note: correlation of points in lower-right panel equals 0.9847.

30 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Large datasets: comparing GOF-tests with prob. plotting

Figure: Normal QQ-plots, n = 1000

Note: correlation of points in lower-right panel equals 0.9770.

31 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Group Presentation (Feb 16) – group 2

Develop your own R code to implement the Anderson-Darling test for normality. How do you compute the AD test statistic for a given data set? The table below gives asymptotic critical values (from Stephens 1974). α 0.010 0.025 0.050 0.100 0.15 0.25 q 1.035 0.873 0.752 0.631 0.561 0.470 You can use linear interpolation to compute critical values for other significant levels. For instance q(0.2) = (q(0.15) + q(0.25))/2 = 0.516. Assess the power of the AD test for normality against the alternative

• f an exponential distribution for n = 1000 and α = 0.05.

The following is a real data set consisting of 100 measurements of the speed of light in air. This classic experiment was carried out by Michelson in 1879. Use your code to test the normality of the data.

32 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

299.85 300 299.96 299.83 299.88 299.88 299.89 299.74 299.98 299.94 299.79 299.88 299.91 299.81 299.9 299.93 299.96 299.81 299.88 299.85 299.81 300.07 299.65 299.94 299.88 299.86 299.87 299.82 299.93 299.76 299.88 299.88 299.72 299.84 299.8 299.85 299.81 299.8 299.83 299.72 299.84 299.77 299.95 300 299.85 299.8 299.62 299.85 299.76 299.98 300 299.88 299.79 299.86 299.84 299.74 299.98 299.96 299.9 299.76 299.97 299.84 299.75 299.88 299.96 299.84 299.8 299.95 299.84 299.76 299.91 299.92 299.89 299.86 299.88 299.72 299.84 299.85 299.85 299.78 299.89 299.84 299.78 299.81 299.76 299.81 299.79 299.81 299.82 299.85 299.87 299.87 299.81 299.74 299.81 299.94 299.95 299.8 299.81 299.87

33 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Group Presentation (Feb 16) – group 3

Present the paper “Interval Estimation for a Binomial Proportion” (Brown-Cai-DasGupta 2001); focus on the paper itself and no need to cover the section of ’Comment’. The authors prefer the Agresti-Coull method to the Wald method (also called standard interval in the paper). Do you agree on this

• pinion? Please write an R code to compare the performance of the

two methods.

34 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Group Presentation (Feb 23) – group 4

Write an R code to implement the chi-sqaure test for the following

• hypothesis. H0 : X1, . . . , Xn are i.i.d from N(0, 1). Choose different

alternative distributions to demenstrate that Cram´ er-Von-Mises test is more powful than chi-square test. Write an R code to implement the chi-sqaure test for a bivariate distribution. H0 : X1, . . . , Xn are i.i.d from a standard bivariate normal distribution. You can devide R2 into k = 10 spherical shells with equal probability under the null. You might need to install R package mvtnorm.

35 / 36

GoF testing EDF tests Chi-square tests Probability plotting Assignment

Group Presentation (Feb 23) – group 5

Concerning the strength of In reliability tests, researchers often rely

• n parametric assumptions to characterize observed lifetimes. To

study the lifetime of certain airplane polished window, 31 measurements are collected. To see if a normal or Weibull distribution is appropriate for this data, please apply different approaches for assessing goodness of fit. You might consider graphical tool also. Give your conclusion. 18.830, 20.800, 21.657, 23.030, 23.230, 24.050, 24.321, 25.500, 25.520, 25.800, 26.690, 26.770, 26.780, 27.050, 27.670, 29.900, 31.110, 33.200, 33.730, 33.760, 33.890, 34.760, 35.750, 35.910, 36.980, 37.080, 37.090, 39.580, 44.045, 45.290, 45.381

36 / 36