Hand Calculations Calculation of 0 , 1 , and SS E from the - - PowerPoint PPT Presentation

ST 370 Probability and Statistics for Engineers

Hand Calculations

Calculation of ˆ β0, ˆ β1, and SSE from the formulas is tedious; these shortcuts can help: Sxx =

n

• i=1

(xi − ¯ x)2 =

n

• i=1

x2

i − 1

n

• n
• i=1

xi 2 Sxy =

n

• i=1

(xi − ¯ x)(yi − ¯ y) =

n

• i=1

xiyi − 1 n

• n
• i=1

xi

n

• i=1

yi

• SST = Syy =

n

• i=1

(yi − ¯ y)2 =

n

• i=1

y 2

i − 1

n

• n
• i=1

yi 2 SSE = SST − ˆ β1Sxy

1 / 18 Simple Linear Regression

ST 370 Probability and Statistics for Engineers

Standard errors Calculations of standard errors begin with an estimate of σ, the standard deviation of the noise term ǫ. The least squares residuals are ei = yi − (ˆ β0 + ˆ β1xi) so the residual sum of squares is SSE =

n

• i=1

e2

i .

Because two parameters were estimated in finding the residuals, the residual degrees of freedom are n − 2, and the estimate of σ2 is ˆ σ2 = MSE = SSE n − 2.

2 / 18 Simple Linear Regression

ST 370 Probability and Statistics for Engineers

The estimated standard errors of the least squares estimates are se(ˆ β1) = ˆ σ

• 1

Sxx and se(ˆ β0) = ˆ σ

• 1

n + ¯ x2 Sxx .

3 / 18 Simple Linear Regression

ST 370 Probability and Statistics for Engineers

These estimated standard errors, especially se(ˆ β1), are used to set up confidence intervals like ˆ β1 ± tα/2,ν × estimated standard error and test statistics like tobs = ˆ β1 estimated standard error.

4 / 18 Simple Linear Regression

ST 370 Probability and Statistics for Engineers

Predicting a New Observation

Regression equations are often used to predict what the response would be in a new experiment, in which the predictor value is say xnew. If we carried out many experiments with x = xnew, we would expect the average response to be β0 + β1xnew, which we estimate by ˆ Ynew = ˆ β0 + ˆ β1xnew.

5 / 18 Simple Linear Regression Predicting a New Observation

ST 370 Probability and Statistics for Engineers

When this mean response is the quantity of interest, you use its estimated standard error se( ˆ Ynew) = ˆ σ

• 1

n + (xnew − ¯ x)2 Sxx to set up confidence intervals. Note that if xnew = 0, ˆ Ynew is just ˆ β0, and this expression reduces to the estimated standard error given earlier. Note also that se( ˆ Ynew) is always at least ˆ σ/√n, and increases as xnew gets farther from ¯ x.

6 / 18 Simple Linear Regression Predicting a New Observation

ST 370 Probability and Statistics for Engineers

Often, however, what is wanted is not a confidence interval for the mean response, but a prediction interval for a single new observation Ynew at x = xnew. Now Ynew = β0 + β1xnew + ǫnew so our uncertainty about Ynew comes from two sources: We have to use estimates ˆ β0 and ˆ β1 instead of the true values; We have no information about the value of ǫnew.

7 / 18 Simple Linear Regression Predicting a New Observation

ST 370 Probability and Statistics for Engineers

The best prediction of Ynew is still ˆ Ynew, but the estimated prediction standard error is pse(Ynew) = ˆ σ

• 1 + 1

n + (xnew − ¯ x)2 Sxx We use the estimated prediction standard error to set up a 100(1 − α) prediction interval for Ynew: ˆ Ynew ± tα/2,ν × pse(Ynew).

8 / 18 Simple Linear Regression Predicting a New Observation

ST 370 Probability and Statistics for Engineers

In R The predict() method can produce either A confidence interval, for the mean response; A prediction interval, for a single new observation.

• xygenLm <- lm(Purity ~ HC, oxygen)

# confidence interval for mean response: predict(oxygenLm, newdata = data.frame(HC = 1.0), interval = "confidence") # prediction interval for single observation: predict(oxygenLm, newdata = data.frame(HC = 1.0), interval = "prediction")

9 / 18 Simple Linear Regression Predicting a New Observation

ST 370 Probability and Statistics for Engineers

Indicator variables

Typically, the predictor variable x is a controllable or measured variable; however, sometimes it is an artificial quantity, constructed to convey some information. An indicator variable is a variable that takes only the values 0 and 1; the observed data can then be divided into two groups: those where x = 0 and those where x = 1.

10 / 18 Simple Linear Regression Indicator variables

ST 370 Probability and Statistics for Engineers

When x = 0, the regression model Y = β0 + β1x + ǫ simplifies to Y = β0 + ǫ so the mean response for this group is β0. When x = 1, the regression model becomes Y = β0 + β1 + ǫ so the mean response for this group is β0 + β1.

11 / 18 Simple Linear Regression Indicator variables

ST 370 Probability and Statistics for Engineers

So the interpretation of the coefficients β0 and β1 is: β0 is the mean response for group 0; β1 is the difference between the mean response for group 1 and the mean response for group 0. When we discussed the factorial design with one factor having two levels, we used the model Yi,j = µ + τi + ǫi,j, i = 1 or 2, j = 1, 2, . . . , n with the constraint τ1 = 0; so: µ is the mean response for the first (baseline) level of the factor; τ2 is the difference between the mean response for the second level and the mean response for the first level.

12 / 18 Simple Linear Regression Indicator variables

ST 370 Probability and Statistics for Engineers

The two models represent the same features, in a different notation. That is: the factorial model with a single factor having two levels is essentially the same as the regression model with an indicator variable as the predictor. That insight is not especially helpful in this context, but when we move on to regression models with more than one predictor it is very useful. Using indicator variables allows us to put regression models and factorial models into a unified framework, called the general linear model.

13 / 18 Simple Linear Regression Indicator variables

ST 370 Probability and Statistics for Engineers

Coefficient of Determination

How well does a regression model fit a given set of data? A careful answer must look at the context: does the model give usefully precise predictions? What is “usefully precise” varies from

• ne context to another.

A quick but not careful answer is to look at the coefficient of determination, R2: R2 = 1 − SSE SST , the fraction of variability in Y explained by the predictor x.

14 / 18 Simple Linear Regression Coefficient of Determination

ST 370 Probability and Statistics for Engineers

In R For the Oxygen example, part of the regression output is:

Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 74.283 1.593 46.62 < 2e-16 *** HC 14.947 1.317 11.35 1.23e-09 ***

• Signif. codes:

0 *** 0.001 ** 0.01 * 0.05 . 0.1 1 Residual standard error: 1.087 on 18 degrees of freedom Multiple R-squared: 0.8774,Adjusted R-squared: 0.8706 F-statistic: 128.9 on 1 and 18 DF, p-value: 1.227e-09

The R2 is reported as the Multiple R-squared, 0.8774. We might report that “87.74% of the variability in oxygen purity is explained by variations in hydrocarbon levels”.

15 / 18 Simple Linear Regression Coefficient of Determination

ST 370 Probability and Statistics for Engineers

Regression with Transformed Variables

In some applications of regression, either the response Y or the predictor x may need to be transformed, and sometimes both. Example: wind turbine Response: DC Output Predictor: Wind speed In R

turbine <- read.csv("Data/Table-11-05.csv") with(turbine, plot(WindSpeed, Output)) with(turbine, plot(-1 / WindSpeed, Output))

16 / 18 Simple Linear Regression Regression with Transformed Variables

ST 370 Probability and Statistics for Engineers

No straight line will give a good fit to the plot of Output versus Wind

• Speed. The plot of Output versus (-1 / Wind Speed) looks more

appropriate for a straight line fit. Try both:

summary(lm(Output ~ WindSpeed, turbine)) summary(lm(Output ~ I(-1 / WindSpeed), turbine))

Note: many symbols like “-” and “/” that usually represent arithmetic operations have special meanings in a formula, so an expression like -1 / WindSpeed must be “wrapped” in the identity function I(.).

17 / 18 Simple Linear Regression Regression with Transformed Variables

ST 370 Probability and Statistics for Engineers

Note that the second model has R2 = 0.98, while the first has R2 = 0.8745. That is, using -1 / WindSpeed as the predictor accounts for much more of the variability in Output than using WindSpeed. Interestingly, using -1 / WindSpeed also results in the model equation ˆ y = 2.98 − 6.93 x , which predicts that the Output will never be higher than 2.98, no matter how high the Wind Speed.

18 / 18 Simple Linear Regression Regression with Transformed Variables