[PPT] - Evaluating the Treatment Effect in the ADM Study and Lords Paradox PowerPoint Presentation

SLIDE 1

Lord’s paradox comparison of the approaches ADM study

Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

Perman Gochyyev

GSE, UC Berkeley BEAR Center

October 17, 2017

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 2

Lord’s paradox comparison of the approaches ADM study

utline

1 Lord’s paradox (15 mins) 2 comparison of the approaches (20 mins) 3 ADM study (10 mins) Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 3

Lord’s paradox comparison of the approaches ADM study

utline

1 Lord’s paradox (15 mins) 2 comparison of the approaches (20 mins) 3 ADM study (10 mins) Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 4

Lord’s paradox comparison of the approaches ADM study

utline

1 Lord’s paradox (15 mins) 2 comparison of the approaches (20 mins) 3 ADM study (10 mins) Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 5

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Lord, 1967

A large university is interested in investigating the effects on the students of the diet provided in the university dining halls and any sex difference in these

effects. Various types of data are gathered. In

particular, the weight of each student at the time of his arrival in September and his weight the following June are recorded. (p.304)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 7

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 8

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 9

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 10

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 11

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 12

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 13

Lord’s paradox comparison of the approaches ADM study Lord, 1967

Lord, 1967

statistician 1: change score approach: Ypost − Ypre = β1 + β2W + ǫ statistician 2: regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ

who is right?

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 14

Lord’s paradox comparison of the approaches ADM study Lord, 1967

Lord, 1967

statistician 1: change score approach: Ypost − Ypre = β1 + β2W + ǫ statistician 2: regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ

who is right?

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 15

Lord’s paradox comparison of the approaches ADM study Lord, 1967

Lord, 1967

statistician 1: change score approach: Ypost − Ypre = β1 + β2W + ǫ statistician 2: regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ

who is right?

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 16

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

utcome: frequency of negative social encounters

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 17

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

utcome: frequency of negative social encounters

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 18

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

utcome: frequency of negative social encounters

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 19

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

utcome: frequency of negative social encounters

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 20

Lord’s paradox comparison of the approaches ADM study Lord, 1967

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

utcome: frequency of negative social encounters

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 21

Lord’s paradox comparison of the approaches ADM study Lord, 1967

the two approaches

Why does the approach taken by statistician 2 — the method that currently dominates social science methodology — give an unintuitive and misleading result?

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 22

Lord’s paradox comparison of the approaches ADM study Lord, 1967

the two approaches

change score approach: Ypost − Ypre = β1 + β2W + ǫ regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ still regressor variable approach Ypost − Ypre = β1 + β2W + (β3 − 1)Ypre + ǫ

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 23

Lord’s paradox comparison of the approaches ADM study Lord, 1967

the two approaches

change score approach: Ypost − Ypre = β1 + β2W + ǫ regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ still regressor variable approach Ypost − Ypre = β1 + β2W + (β3 − 1)Ypre + ǫ

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 24

Lord’s paradox comparison of the approaches ADM study Lord, 1967

the two approaches

change score approach: Ypost − Ypre = β1 + β2W + ǫ regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ still regressor variable approach Ypost − Ypre = β1 + β2W + (β3 − 1)Ypre + ǫ

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 25

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 26

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 27

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 28

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 29

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 30

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 31

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 32

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 33

Lord’s paradox comparison of the approaches ADM study Lord, 1967

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics

statistician 1: FD approach (same as FE estimator when two time-points) (also: DID) statistician 2: lagged dependent variable approach

experimental design

statistician 1: ANOVA (RANOVA: repeated measures ANOVA) statistician 2: ANCOVA (analysis of covariance)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 34

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

the two approaches

change score approach: intuitive, easy to interpret

John Snow‘s finding that the cholera was a water-borne infectious disease

regressor variable approach:

as long as temporal order allows, seems reasonable to learn from the data

extreme example: pretest and posttest of weight group 1: mice group 2: elephants

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 36

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

the two approaches

change score approach: intuitive, easy to interpret

John Snow‘s finding that the cholera was a water-borne infectious disease

regressor variable approach:

as long as temporal order allows, seems reasonable to learn from the data

extreme example: pretest and posttest of weight group 1: mice group 2: elephants

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 37

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

the two approaches

change score approach: intuitive, easy to interpret

John Snow‘s finding that the cholera was a water-borne infectious disease

regressor variable approach:

as long as temporal order allows, seems reasonable to learn from the data

extreme example: pretest and posttest of weight group 1: mice group 2: elephants

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 38

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

the two approaches

change score approach: intuitive, easy to interpret

John Snow‘s finding that the cholera was a water-borne infectious disease

regressor variable approach:

as long as temporal order allows, seems reasonable to learn from the data

extreme example: pretest and posttest of weight group 1: mice group 2: elephants

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 39

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

Lord’s paradox

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 40

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

treatment assigned at random

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 41

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

n Lord, 1967

Lord (1967): “... there are as many different explanations as there are explainers” “... there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled preexisting differences between groups” Senn (2008): “In a disturbing paper in the Psychological Bulletin in 1967, Lord considered a case..,” Rubin, Stuart & Zanutto (2003): “A Classic Example of Poorly Formulated Causal Assessment—Lord’s paradox” Wainer & Brown (2007): “..by far, the most difficult paradox to disentangle and requires clear thinking”

change score method: underdog in educational and social methodology (but not in econometrics)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 42

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

n Lord, 1967

Lord (1967): “... there are as many different explanations as there are explainers” “... there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled preexisting differences between groups” Senn (2008): “In a disturbing paper in the Psychological Bulletin in 1967, Lord considered a case..,” Rubin, Stuart & Zanutto (2003): “A Classic Example of Poorly Formulated Causal Assessment—Lord’s paradox” Wainer & Brown (2007): “..by far, the most difficult paradox to disentangle and requires clear thinking”

change score method: underdog in educational and social methodology (but not in econometrics)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 43

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

n Lord, 1967

Lord (1967): “... there are as many different explanations as there are explainers” “... there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled preexisting differences between groups” Senn (2008): “In a disturbing paper in the Psychological Bulletin in 1967, Lord considered a case..,” Rubin, Stuart & Zanutto (2003): “A Classic Example of Poorly Formulated Causal Assessment—Lord’s paradox” Wainer & Brown (2007): “..by far, the most difficult paradox to disentangle and requires clear thinking”

change score method: underdog in educational and social methodology (but not in econometrics)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 44

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

n Lord, 1967

Lord (1967): “... there are as many different explanations as there are explainers” “... there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled preexisting differences between groups” Senn (2008): “In a disturbing paper in the Psychological Bulletin in 1967, Lord considered a case..,” Rubin, Stuart & Zanutto (2003): “A Classic Example of Poorly Formulated Causal Assessment—Lord’s paradox” Wainer & Brown (2007): “..by far, the most difficult paradox to disentangle and requires clear thinking”

change score method: underdog in educational and social methodology (but not in econometrics)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 45

Lord’s paradox comparison of the approaches ADM study understanding the two approaches

n Lord, 1967

Lord (1967): “... there are as many different explanations as there are explainers” “... there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled preexisting differences between groups” Senn (2008): “In a disturbing paper in the Psychological Bulletin in 1967, Lord considered a case..,” Rubin, Stuart & Zanutto (2003): “A Classic Example of Poorly Formulated Causal Assessment—Lord’s paradox” Wainer & Brown (2007): “..by far, the most difficult paradox to disentangle and requires clear thinking”

change score method: underdog in educational and social methodology (but not in econometrics)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 46

Lord’s paradox comparison of the approaches ADM study

three camps

debates over this paradox spread into mainly three directions:

psychometricians: debates over reliability, measurement error, regression to the mean (Cronbach & Furby, 1970; Linn & Slinde, 1977 vs. Rogosa & Willett, 1983; Zimmerman & Williams, 1982) whose causal framework is better armed to explain the paradox (Holland & Rubin, 1982; Wainer & Brown, 2007; Pearl, 2014) ”it depends” camp (Kenny, 1975, Allison, 1990)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 47

Lord’s paradox comparison of the approaches ADM study

three camps

debates over this paradox spread into mainly three directions:

psychometricians: debates over reliability, measurement error, regression to the mean (Cronbach & Furby, 1970; Linn & Slinde, 1977 vs. Rogosa & Willett, 1983; Zimmerman & Williams, 1982) whose causal framework is better armed to explain the paradox (Holland & Rubin, 1982; Wainer & Brown, 2007; Pearl, 2014) ”it depends” camp (Kenny, 1975, Allison, 1990)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 48

Lord’s paradox comparison of the approaches ADM study

three camps

debates over this paradox spread into mainly three directions:

psychometricians: debates over reliability, measurement error, regression to the mean (Cronbach & Furby, 1970; Linn & Slinde, 1977 vs. Rogosa & Willett, 1983; Zimmerman & Williams, 1982) whose causal framework is better armed to explain the paradox (Holland & Rubin, 1982; Wainer & Brown, 2007; Pearl, 2014) ”it depends” camp (Kenny, 1975, Allison, 1990)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 49

Lord’s paradox comparison of the approaches ADM study

three camps

debates over this paradox spread into mainly three directions:

psychometricians: debates over reliability, measurement error, regression to the mean (Cronbach & Furby, 1970; Linn & Slinde, 1977 vs. Rogosa & Willett, 1983; Zimmerman & Williams, 1982) whose causal framework is better armed to explain the paradox (Holland & Rubin, 1982; Wainer & Brown, 2007; Pearl, 2014) ”it depends” camp (Kenny, 1975, Allison, 1990)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 50

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Cronbach & Furby (1970): “It appears that investigators who ask

questions regarding gain scores would ordinarily be better advised to frame their questions in other ways.”

Linn & Slinde (1977): “Problems in measuring change abound

and the virtues in doing so are hard to find.”

main “issues”:

gain scores will be higher for individuals with a lower pretest (“unfairness”) unreliable: Gulliksen’s (1950) formula

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 52

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Cronbach & Furby (1970): “It appears that investigators who ask

questions regarding gain scores would ordinarily be better advised to frame their questions in other ways.”

Linn & Slinde (1977): “Problems in measuring change abound

and the virtues in doing so are hard to find.”

main “issues”:

gain scores will be higher for individuals with a lower pretest (“unfairness”) unreliable: Gulliksen’s (1950) formula

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 53

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Cronbach & Furby (1970): “It appears that investigators who ask

questions regarding gain scores would ordinarily be better advised to frame their questions in other ways.”

Linn & Slinde (1977): “Problems in measuring change abound

and the virtues in doing so are hard to find.”

main “issues”:

gain scores will be higher for individuals with a lower pretest (“unfairness”) unreliable: Gulliksen’s (1950) formula

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 54

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Cronbach & Furby (1970): “It appears that investigators who ask

questions regarding gain scores would ordinarily be better advised to frame their questions in other ways.”

Linn & Slinde (1977): “Problems in measuring change abound

and the virtues in doing so are hard to find.”

main “issues”:

gain scores will be higher for individuals with a lower pretest (“unfairness”) unreliable: Gulliksen’s (1950) formula

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 55

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Cronbach & Furby (1970): “It appears that investigators who ask

questions regarding gain scores would ordinarily be better advised to frame their questions in other ways.”

Linn & Slinde (1977): “Problems in measuring change abound

and the virtues in doing so are hard to find.”

main “issues”:

gain scores will be higher for individuals with a lower pretest (“unfairness”) unreliable: Gulliksen’s (1950) formula

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 56

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Willett (1997): “change and previous status will always be related

since current status is the product of prior change”

Rogosa, Brandt, & Zimowski (1982): “The correlation between

true change and true initial status (zero or otherwise) is an interesting fact of life.” (..but no more than that)

Allison (1990): “regression toward the mean is not an issue when

the objective is to compare two groups” [assuming equivalence at baseline]

counterexample in education: those who are high in the initial status might be better suited to understand the new instruction and gain more than those with a lower initial status

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 57

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Willett (1997): “change and previous status will always be related

since current status is the product of prior change”

Rogosa, Brandt, & Zimowski (1982): “The correlation between

true change and true initial status (zero or otherwise) is an interesting fact of life.” (..but no more than that)

Allison (1990): “regression toward the mean is not an issue when

the objective is to compare two groups” [assuming equivalence at baseline]

counterexample in education: those who are high in the initial status might be better suited to understand the new instruction and gain more than those with a lower initial status

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 58

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Willett (1997): “change and previous status will always be related

since current status is the product of prior change”

Rogosa, Brandt, & Zimowski (1982): “The correlation between

true change and true initial status (zero or otherwise) is an interesting fact of life.” (..but no more than that)

Allison (1990): “regression toward the mean is not an issue when

the objective is to compare two groups” [assuming equivalence at baseline]

counterexample in education: those who are high in the initial status might be better suited to understand the new instruction and gain more than those with a lower initial status

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 59

Lord’s paradox comparison of the approaches ADM study psychometricians on change scores

psychometricians on change scores

Willett (1997): “change and previous status will always be related

since current status is the product of prior change”

Rogosa, Brandt, & Zimowski (1982): “The correlation between

true change and true initial status (zero or otherwise) is an interesting fact of life.” (..but no more than that)

Allison (1990): “regression toward the mean is not an issue when

the objective is to compare two groups” [assuming equivalence at baseline]

counterexample in education: those who are high in the initial status might be better suited to understand the new instruction and gain more than those with a lower initial status

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 60

Lord’s paradox comparison of the approaches ADM study attempts using causal framework

Neyman-Rubin framework

Rubin, et al., (2003); Holland & Rubin (1983):

Lord’s example was a “poorly formulated causal assessment” since the potential outcome under the control diet is missing “...researcher investigating gain wouldn’t know if changes in scores would have occurred with no treatment anyway”

the hypothetical researcher in Lord (1967) is interested in gender differences and not in the effect of the diet

Lord (1967): “differential effect” of the diet

under the Neyman-Rubin (a.k.a. potential outcomes) causal framework the effect of gender cannot be a causal research question

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 62

Lord’s paradox comparison of the approaches ADM study attempts using causal framework

Neyman-Rubin framework

Rubin, et al., (2003); Holland & Rubin (1983):

Lord’s example was a “poorly formulated causal assessment” since the potential outcome under the control diet is missing “...researcher investigating gain wouldn’t know if changes in scores would have occurred with no treatment anyway”

the hypothetical researcher in Lord (1967) is interested in gender differences and not in the effect of the diet

Lord (1967): “differential effect” of the diet

under the Neyman-Rubin (a.k.a. potential outcomes) causal framework the effect of gender cannot be a causal research question

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 63

Lord’s paradox comparison of the approaches ADM study attempts using causal framework

Neyman-Rubin framework

Rubin, et al., (2003); Holland & Rubin (1983):

Lord’s example was a “poorly formulated causal assessment” since the potential outcome under the control diet is missing “...researcher investigating gain wouldn’t know if changes in scores would have occurred with no treatment anyway”

the hypothetical researcher in Lord (1967) is interested in gender differences and not in the effect of the diet

Lord (1967): “differential effect” of the diet

under the Neyman-Rubin (a.k.a. potential outcomes) causal framework the effect of gender cannot be a causal research question

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 64

Lord’s paradox comparison of the approaches ADM study attempts using causal framework

Neyman-Rubin framework

Rubin, et al., (2003); Holland & Rubin (1983):

Lord’s example was a “poorly formulated causal assessment” since the potential outcome under the control diet is missing “...researcher investigating gain wouldn’t know if changes in scores would have occurred with no treatment anyway”

the hypothetical researcher in Lord (1967) is interested in gender differences and not in the effect of the diet

Lord (1967): “differential effect” of the diet

under the Neyman-Rubin (a.k.a. potential outcomes) causal framework the effect of gender cannot be a causal research question

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 65

Lord’s paradox comparison of the approaches ADM study attempts using causal framework

Neyman-Rubin framework

Rubin, et al., (2003); Holland & Rubin (1983):

Lord’s example was a “poorly formulated causal assessment” since the potential outcome under the control diet is missing “...researcher investigating gain wouldn’t know if changes in scores would have occurred with no treatment anyway”

the hypothetical researcher in Lord (1967) is interested in gender differences and not in the effect of the diet

Lord (1967): “differential effect” of the diet

under the Neyman-Rubin (a.k.a. potential outcomes) causal framework the effect of gender cannot be a causal research question

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 66

Lord’s paradox comparison of the approaches ADM study attempts using causal framework

Neyman-Rubin framework

Rubin, et al., (2003); Holland & Rubin (1983):

Lord’s example was a “poorly formulated causal assessment” since the potential outcome under the control diet is missing “...researcher investigating gain wouldn’t know if changes in scores would have occurred with no treatment anyway”

the hypothetical researcher in Lord (1967) is interested in gender differences and not in the effect of the diet

Lord (1967): “differential effect” of the diet

under the Neyman-Rubin (a.k.a. potential outcomes) causal framework the effect of gender cannot be a causal research question

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 67

Lord’s paradox comparison of the approaches ADM study a closer look

regressor variable model

Y2j = β1 + β2Wj + β3Y1j + ǫj OLS yields unbiased estimates assuming:

ǫj uncorrelated with Wj and Y1j correct specification i.i.d. no measurement error

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 69

Lord’s paradox comparison of the approaches ADM study a closer look

regressor variable model

Y2j = β1 + β2Wj + β3Y1j + ǫj OLS yields unbiased estimates assuming:

ǫj uncorrelated with Wj and Y1j correct specification i.i.d. no measurement error

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 70

Lord’s paradox comparison of the approaches ADM study a closer look

regressor variable model

Y2j = β1 + β2Wj + β3Y1j + ǫj OLS yields unbiased estimates assuming:

ǫj uncorrelated with Wj and Y1j correct specification i.i.d. no measurement error

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 71

Lord’s paradox comparison of the approaches ADM study a closer look

regressor variable model

Y2j = β1 + β2Wj + β3Y1j + ǫj OLS yields unbiased estimates assuming:

ǫj uncorrelated with Wj and Y1j correct specification i.i.d. no measurement error

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 72

Lord’s paradox comparison of the approaches ADM study a closer look

regressor variable model

Y2j = β1 + β2Wj + β3Y1j + ǫj OLS yields unbiased estimates assuming:

ǫj uncorrelated with Wj and Y1j correct specification i.i.d. no measurement error

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 73

Lord’s paradox comparison of the approaches ADM study a closer look

regressor variable model

Y2j = β1 + β2Wj + β3Y1j + ǫj OLS yields unbiased estimates assuming:

ǫj uncorrelated with Wj and Y1j correct specification i.i.d. no measurement error

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 74

Lord’s paradox comparison of the approaches ADM study a closer look

regressor variable model

Y2j = β1 + β2Wj + β3Y1j + ǫj OLS yields unbiased estimates assuming:

ǫj uncorrelated with Wj and Y1j correct specification i.i.d. no measurement error

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 75

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 76

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 77

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 78

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 79

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 80

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 81

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 82

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 83

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 84

Lord’s paradox comparison of the approaches ADM study a closer look

change score model (following Allison, 1990)

assume Gj is binary: 1 if person j ends up in the treatment group; 0 otherwise pre: Y1j = β0 + δGj + ǫ1j

δ: group differences that are stable

post: Y2j = β0 + β1 + δGj + β2Wj + ǫ2j

Wj is treatment indicator Gj = Wj (collinear) β1 represents the change that is occurring in both groups (e.g., gained knowledge during a school-year)

Y2j − Y1j = (β0 − β0) + β1 + (δGj − δGj) + β2Wj + (ǫ2j − ǫ1j) ∆Yj = β1 + β2Wj + ǫ∆

j

assuming ǫ∆

j

is not correlated with Wj, OLS is consistent and hence the estimates are unbiased

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 85

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 86

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 87

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 88

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 89

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 90

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 91

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 92

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 93

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

change score method: Y2j − Y1j = β1 + β2Wj + ǫ∆

j

rewrite: Y2j = β1 + β2Wj + (1)Y1j + ǫ∆

j

some say/imply that this is a special case of the: Y2j = β1 + β2Wj + β3Y1j + ǫj see for instance:

Hedeker & Gibbons, 2006 (p. 8) Van Breukelen, 2013, (p. 903) Gelman & Hill, 2006 (p. 177) “an unnecessary assumption, namely, that [β3 = 1]”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 94

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

inconsistent estimates since ǫ∆

j

is negatively correlated with Y1j by construction

Y2j − Y1j = β1 + β2Wj + ǫ∆

j

not a special case

the two approaches represent two completely different models!

verlooking this crucial distinction has been the most common

error in comparisons of the two approaches any discussion of Lord’s paradox that does not acknowledge this distinction is likely to be misleading

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 95

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

inconsistent estimates since ǫ∆

j

is negatively correlated with Y1j by construction

Y2j − Y1j = β1 + β2Wj + ǫ∆

j

not a special case

the two approaches represent two completely different models!

verlooking this crucial distinction has been the most common

error in comparisons of the two approaches any discussion of Lord’s paradox that does not acknowledge this distinction is likely to be misleading

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 96

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

inconsistent estimates since ǫ∆

j

is negatively correlated with Y1j by construction

Y2j − Y1j = β1 + β2Wj + ǫ∆

j

not a special case

the two approaches represent two completely different models!

verlooking this crucial distinction has been the most common

error in comparisons of the two approaches any discussion of Lord’s paradox that does not acknowledge this distinction is likely to be misleading

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 97

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

inconsistent estimates since ǫ∆

j

is negatively correlated with Y1j by construction

Y2j − Y1j = β1 + β2Wj + ǫ∆

j

not a special case

the two approaches represent two completely different models!

verlooking this crucial distinction has been the most common

error in comparisons of the two approaches any discussion of Lord’s paradox that does not acknowledge this distinction is likely to be misleading

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 98

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

inconsistent estimates since ǫ∆

j

is negatively correlated with Y1j by construction

Y2j − Y1j = β1 + β2Wj + ǫ∆

j

not a special case

the two approaches represent two completely different models!

verlooking this crucial distinction has been the most common

error in comparisons of the two approaches any discussion of Lord’s paradox that does not acknowledge this distinction is likely to be misleading

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 99

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

inconsistent estimates since ǫ∆

j

is negatively correlated with Y1j by construction

Y2j − Y1j = β1 + β2Wj + ǫ∆

j

not a special case

the two approaches represent two completely different models!

verlooking this crucial distinction has been the most common

error in comparisons of the two approaches any discussion of Lord’s paradox that does not acknowledge this distinction is likely to be misleading

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 100

Lord’s paradox comparison of the approaches ADM study a closer look

special case?

Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

inconsistent estimates since ǫ∆

j

is negatively correlated with Y1j by construction

Y2j − Y1j = β1 + β2Wj + ǫ∆

j

not a special case

the two approaches represent two completely different models!

verlooking this crucial distinction has been the most common

error in comparisons of the two approaches any discussion of Lord’s paradox that does not acknowledge this distinction is likely to be misleading

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 101

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 102

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 103

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 104

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 105

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 106

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 107

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 108

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 109

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 110

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

assume RV model is the correct model and β2 = 0

Y2j = β1 + β2Wj + β3Y1j + ǫj Y2j − β3Y1j = β1 + β2Wj + ǫj if β2 = 0 Y Wj=0

2j

− β3Y Wj=0

1j

= Y Wj=1

2j

− β3Y Wj=1

1j

β3Y Wj=1

1j

− β3Y Wj=0

1j

= Y Wj=1

2j

− Y Wj=0

2j

β3(Y Wj=1

1j

− Y Wj=0

1j

) = Y Wj=1

2j

− Y Wj=0

2j

this implies that the mean difference on the posttest will be less than the mean difference in the pretest the mean of each of the two groups will regress toward the grand mean this implication is not plausible when the exchangeability of the two groups is not a reasonable assumption

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 111

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

if RV is the correct model:

Y2j − Y1j = β1 + β2Wj + β3Y1j + ǫj

and CS model is used incorrectly:

Y2j − Y1j = β1 + β2Wj + ǫj

then we also have an omitted variable bias

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 112

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

if RV is the correct model:

Y2j − Y1j = β1 + β2Wj + β3Y1j + ǫj

and CS model is used incorrectly:

Y2j − Y1j = β1 + β2Wj + ǫj

then we also have an omitted variable bias

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 113

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

if RV is the correct model:

Y2j − Y1j = β1 + β2Wj + β3Y1j + ǫj

and CS model is used incorrectly:

Y2j − Y1j = β1 + β2Wj + ǫj

then we also have an omitted variable bias

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 114

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

if RV is the correct model:

Y2j − Y1j = β1 + β2Wj + β3Y1j + ǫj

and CS model is used incorrectly:

Y2j − Y1j = β1 + β2Wj + ǫj

then we also have an omitted variable bias

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 115

Lord’s paradox comparison of the approaches ADM study a closer look

if RV model is the correct model

if RV is the correct model:

Y2j − Y1j = β1 + β2Wj + β3Y1j + ǫj

and CS model is used incorrectly:

Y2j − Y1j = β1 + β2Wj + ǫj

then we also have an omitted variable bias

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 116

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

assume CS is the correct model and β2 = 0 Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

let ρ1W : the correlation between the pretest and the treatment indicator let ρ2W : the correlation between the posttest and the treatment indicator let ρ21: the correlation between the pretest and the posttest The partial regression coefficient of Y1 controlling for W in the RV approach is

β21·W = β21−ρ2W ρ1W

1−ρ2

1W

= β21−ρ2

1W

1−ρ2

1W

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 117

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

assume CS is the correct model and β2 = 0 Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

let ρ1W : the correlation between the pretest and the treatment indicator let ρ2W : the correlation between the posttest and the treatment indicator let ρ21: the correlation between the pretest and the posttest The partial regression coefficient of Y1 controlling for W in the RV approach is

β21·W = β21−ρ2W ρ1W

1−ρ2

1W

= β21−ρ2

1W

1−ρ2

1W

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 118

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

assume CS is the correct model and β2 = 0 Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

let ρ1W : the correlation between the pretest and the treatment indicator let ρ2W : the correlation between the posttest and the treatment indicator let ρ21: the correlation between the pretest and the posttest The partial regression coefficient of Y1 controlling for W in the RV approach is

β21·W = β21−ρ2W ρ1W

1−ρ2

1W

= β21−ρ2

1W

1−ρ2

1W

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 119

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

assume CS is the correct model and β2 = 0 Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

let ρ1W : the correlation between the pretest and the treatment indicator let ρ2W : the correlation between the posttest and the treatment indicator let ρ21: the correlation between the pretest and the posttest The partial regression coefficient of Y1 controlling for W in the RV approach is

β21·W = β21−ρ2W ρ1W

1−ρ2

1W

= β21−ρ2

1W

1−ρ2

1W

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 120

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

assume CS is the correct model and β2 = 0 Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

let ρ1W : the correlation between the pretest and the treatment indicator let ρ2W : the correlation between the posttest and the treatment indicator let ρ21: the correlation between the pretest and the posttest The partial regression coefficient of Y1 controlling for W in the RV approach is

β21·W = β21−ρ2W ρ1W

1−ρ2

1W

= β21−ρ2

1W

1−ρ2

1W

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 121

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

assume CS is the correct model and β2 = 0 Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

let ρ1W : the correlation between the pretest and the treatment indicator let ρ2W : the correlation between the posttest and the treatment indicator let ρ21: the correlation between the pretest and the posttest The partial regression coefficient of Y1 controlling for W in the RV approach is

β21·W = β21−ρ2W ρ1W

1−ρ2

1W

= β21−ρ2

1W

1−ρ2

1W

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 122

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

assume CS is the correct model and β2 = 0 Y2j = β1 + β2Wj + β3Y1j + ǫ∆

j

let ρ1W : the correlation between the pretest and the treatment indicator let ρ2W : the correlation between the posttest and the treatment indicator let ρ21: the correlation between the pretest and the posttest The partial regression coefficient of Y1 controlling for W in the RV approach is

β21·W = β21−ρ2W ρ1W

1−ρ2

1W

= β21−ρ2

1W

1−ρ2

1W

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 123

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

partial regression coefficient for W, controlling for Y1

β2W ·1 = β2W −ρ21ρ1W

1−ρ2

1W

= β1W −ρ21ρ1W

1−ρ2

1W

= =

σ2√ 1−ρ2

21

σW√ 1−ρ2

W 1

ρ1W −ρ21ρ1W

√

1−ρ2

21

√

1−ρ2

1W

= (1−ρ21)ρ1W

1−ρ2

1W

recall Allison (1990) approach: Y1 = β0 + δG + ǫ1 further decompose ǫ1 = U + V1 , where U is the stable component of Y then, the correlation of the pretest and the treatment indicator can be expressed as

ρ1W = δ + cov(W ,U)

var(W )

where cov(W ,U)

var(W )

is the correlation between the stable component (U) and the treatment indicator (W)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 124

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

partial regression coefficient for W, controlling for Y1

β2W ·1 = β2W −ρ21ρ1W

1−ρ2

1W

= β1W −ρ21ρ1W

1−ρ2

1W

= =

σ2√ 1−ρ2

21

σW√ 1−ρ2

W 1

ρ1W −ρ21ρ1W

√

1−ρ2

21

√

1−ρ2

1W

= (1−ρ21)ρ1W

1−ρ2

1W

recall Allison (1990) approach: Y1 = β0 + δG + ǫ1 further decompose ǫ1 = U + V1 , where U is the stable component of Y then, the correlation of the pretest and the treatment indicator can be expressed as

ρ1W = δ + cov(W ,U)

var(W )

where cov(W ,U)

var(W )

is the correlation between the stable component (U) and the treatment indicator (W)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 125

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

partial regression coefficient for W, controlling for Y1

β2W ·1 = β2W −ρ21ρ1W

1−ρ2

1W

= β1W −ρ21ρ1W

1−ρ2

1W

= =

σ2√ 1−ρ2

21

σW√ 1−ρ2

W 1

ρ1W −ρ21ρ1W

√

1−ρ2

21

√

1−ρ2

1W

= (1−ρ21)ρ1W

1−ρ2

1W

recall Allison (1990) approach: Y1 = β0 + δG + ǫ1 further decompose ǫ1 = U + V1 , where U is the stable component of Y then, the correlation of the pretest and the treatment indicator can be expressed as

ρ1W = δ + cov(W ,U)

var(W )

where cov(W ,U)

var(W )

is the correlation between the stable component (U) and the treatment indicator (W)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 126

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

partial regression coefficient for W, controlling for Y1

β2W ·1 = β2W −ρ21ρ1W

1−ρ2

1W

= β1W −ρ21ρ1W

1−ρ2

1W

= =

σ2√ 1−ρ2

21

σW√ 1−ρ2

W 1

ρ1W −ρ21ρ1W

√

1−ρ2

21

√

1−ρ2

1W

= (1−ρ21)ρ1W

1−ρ2

1W

recall Allison (1990) approach: Y1 = β0 + δG + ǫ1 further decompose ǫ1 = U + V1 , where U is the stable component of Y then, the correlation of the pretest and the treatment indicator can be expressed as

ρ1W = δ + cov(W ,U)

var(W )

where cov(W ,U)

var(W )

is the correlation between the stable component (U) and the treatment indicator (W)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 127

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

partial regression coefficient for W, controlling for Y1

β2W ·1 = β2W −ρ21ρ1W

1−ρ2

1W

= β1W −ρ21ρ1W

1−ρ2

1W

= =

σ2√ 1−ρ2

21

σW√ 1−ρ2

W 1

ρ1W −ρ21ρ1W

√

1−ρ2

21

√

1−ρ2

1W

= (1−ρ21)ρ1W

1−ρ2

1W

recall Allison (1990) approach: Y1 = β0 + δG + ǫ1 further decompose ǫ1 = U + V1 , where U is the stable component of Y then, the correlation of the pretest and the treatment indicator can be expressed as

ρ1W = δ + cov(W ,U)

var(W )

where cov(W ,U)

var(W )

is the correlation between the stable component (U) and the treatment indicator (W)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 128

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

partial regression coefficient for W, controlling for Y1

β2W ·1 = β2W −ρ21ρ1W

1−ρ2

1W

= β1W −ρ21ρ1W

1−ρ2

1W

= =

σ2√ 1−ρ2

21

σW√ 1−ρ2

W 1

ρ1W −ρ21ρ1W

√

1−ρ2

21

√

1−ρ2

1W

= (1−ρ21)ρ1W

1−ρ2

1W

recall Allison (1990) approach: Y1 = β0 + δG + ǫ1 further decompose ǫ1 = U + V1 , where U is the stable component of Y then, the correlation of the pretest and the treatment indicator can be expressed as

ρ1W = δ + cov(W ,U)

var(W )

where cov(W ,U)

var(W )

is the correlation between the stable component (U) and the treatment indicator (W)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 129

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

β2W ·1 =

(1−ρ21)[δ+ cov(W ,U)

var(W ) ]

1−ρ2

1W

is non-zero and will be zero only when:

no difference between two groups (δ = 0) AND treatment assignment independent of stable component of Y (i.e., cov(W , U) = 0)

the paradox is inevitable when the RV approach is used incorrectly

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 130

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

β2W ·1 =

(1−ρ21)[δ+ cov(W ,U)

var(W ) ]

1−ρ2

1W

is non-zero and will be zero only when:

no difference between two groups (δ = 0) AND treatment assignment independent of stable component of Y (i.e., cov(W , U) = 0)

the paradox is inevitable when the RV approach is used incorrectly

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 131

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

β2W ·1 =

(1−ρ21)[δ+ cov(W ,U)

var(W ) ]

1−ρ2

1W

is non-zero and will be zero only when:

no difference between two groups (δ = 0) AND treatment assignment independent of stable component of Y (i.e., cov(W , U) = 0)

the paradox is inevitable when the RV approach is used incorrectly

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 132

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

β2W ·1 =

(1−ρ21)[δ+ cov(W ,U)

var(W ) ]

1−ρ2

1W

is non-zero and will be zero only when:

no difference between two groups (δ = 0) AND treatment assignment independent of stable component of Y (i.e., cov(W , U) = 0)

the paradox is inevitable when the RV approach is used incorrectly

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 133

Lord’s paradox comparison of the approaches ADM study a closer look

if CS model is the correct model

β2W ·1 =

(1−ρ21)[δ+ cov(W ,U)

var(W ) ]

1−ρ2

1W

is non-zero and will be zero only when:

no difference between two groups (δ = 0) AND treatment assignment independent of stable component of Y (i.e., cov(W , U) = 0)

the paradox is inevitable when the RV approach is used incorrectly

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 134

Lord’s paradox comparison of the approaches ADM study a closer look

which is appropriate and when?

it depends when randomized: both are fine, CS has less power when the treatment is assigned based on the pretest: – it becomes necessary to control for the pretest

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 135

Lord’s paradox comparison of the approaches ADM study a closer look

which is appropriate and when?

it depends when randomized: both are fine, CS has less power when the treatment is assigned based on the pretest: – it becomes necessary to control for the pretest

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 136

Lord’s paradox comparison of the approaches ADM study a closer look

which is appropriate and when?

it depends when randomized: both are fine, CS has less power when the treatment is assigned based on the pretest: – it becomes necessary to control for the pretest

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 137

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

Lindley & Novick (1981) and Novick (1983): inference must be based on careful specification of the relevant subpopulation (or assumption of exchangeability) are groups exchangeable with respect to the outcome of interest?

it might be that Group A and Group B are exchangeable if the

utcome of interest is the mean number of hours spent in the

gym and not exchangeable if the outcome of interest is the mean number of calories burned subject matter expertise is necessary to answer this question

exchangeable: if and only if they have identical distributions

f response patterns

guaranteed by randomization

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 138

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

Lindley & Novick (1981) and Novick (1983): inference must be based on careful specification of the relevant subpopulation (or assumption of exchangeability) are groups exchangeable with respect to the outcome of interest?

it might be that Group A and Group B are exchangeable if the

utcome of interest is the mean number of hours spent in the

gym and not exchangeable if the outcome of interest is the mean number of calories burned subject matter expertise is necessary to answer this question

exchangeable: if and only if they have identical distributions

f response patterns

guaranteed by randomization

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 139

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

Lindley & Novick (1981) and Novick (1983): inference must be based on careful specification of the relevant subpopulation (or assumption of exchangeability) are groups exchangeable with respect to the outcome of interest?

it might be that Group A and Group B are exchangeable if the

utcome of interest is the mean number of hours spent in the

gym and not exchangeable if the outcome of interest is the mean number of calories burned subject matter expertise is necessary to answer this question

exchangeable: if and only if they have identical distributions

f response patterns

guaranteed by randomization

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 140

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

Lindley & Novick (1981) and Novick (1983): inference must be based on careful specification of the relevant subpopulation (or assumption of exchangeability) are groups exchangeable with respect to the outcome of interest?

it might be that Group A and Group B are exchangeable if the

utcome of interest is the mean number of hours spent in the

gym and not exchangeable if the outcome of interest is the mean number of calories burned subject matter expertise is necessary to answer this question

exchangeable: if and only if they have identical distributions

f response patterns

guaranteed by randomization

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 141

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

Lindley & Novick (1981) and Novick (1983): inference must be based on careful specification of the relevant subpopulation (or assumption of exchangeability) are groups exchangeable with respect to the outcome of interest?

it might be that Group A and Group B are exchangeable if the

utcome of interest is the mean number of hours spent in the

gym and not exchangeable if the outcome of interest is the mean number of calories burned subject matter expertise is necessary to answer this question

exchangeable: if and only if they have identical distributions

f response patterns

guaranteed by randomization

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 142

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

Lindley & Novick (1981) and Novick (1983): inference must be based on careful specification of the relevant subpopulation (or assumption of exchangeability) are groups exchangeable with respect to the outcome of interest?

it might be that Group A and Group B are exchangeable if the

utcome of interest is the mean number of hours spent in the

gym and not exchangeable if the outcome of interest is the mean number of calories burned subject matter expertise is necessary to answer this question

exchangeable: if and only if they have identical distributions

f response patterns

guaranteed by randomization

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 143

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

Lindley & Novick (1981) and Novick (1983): inference must be based on careful specification of the relevant subpopulation (or assumption of exchangeability) are groups exchangeable with respect to the outcome of interest?

it might be that Group A and Group B are exchangeable if the

utcome of interest is the mean number of hours spent in the

gym and not exchangeable if the outcome of interest is the mean number of calories burned subject matter expertise is necessary to answer this question

exchangeable: if and only if they have identical distributions

f response patterns

guaranteed by randomization

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 144

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

the exchangeability assumption is related to the assumption of the regression to the grand mean if a researcher suspects that the regression to the group-specific means is more plausible assumption, then s/he should prefer the CS approach what if there is a measurement error in pre and post? – RV approach will produce biased estimates what if pre and post have different scales? – CS approach requires a common scale

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 145

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

the exchangeability assumption is related to the assumption of the regression to the grand mean if a researcher suspects that the regression to the group-specific means is more plausible assumption, then s/he should prefer the CS approach what if there is a measurement error in pre and post? – RV approach will produce biased estimates what if pre and post have different scales? – CS approach requires a common scale

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 146

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

the exchangeability assumption is related to the assumption of the regression to the grand mean if a researcher suspects that the regression to the group-specific means is more plausible assumption, then s/he should prefer the CS approach what if there is a measurement error in pre and post? – RV approach will produce biased estimates what if pre and post have different scales? – CS approach requires a common scale

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 147

Lord’s paradox comparison of the approaches ADM study a closer look

exchangeability assumption

the exchangeability assumption is related to the assumption of the regression to the grand mean if a researcher suspects that the regression to the group-specific means is more plausible assumption, then s/he should prefer the CS approach what if there is a measurement error in pre and post? – RV approach will produce biased estimates what if pre and post have different scales? – CS approach requires a common scale

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 148

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

ADM study

Data Modeling curriculum: designed to improve statistical reasoning skills cluster-randomized trial pretest and posttest: ADM Statistical Reasoning Measure developed by Rich Lehrer at Vanderbilt in conjunction with the BEAR Center the measure has five dimensions (domains): Data Display (DAD), Models of Variability (MOV), Chance (CHA), Concepts of Statistics (COS), and Informal Inference (INI)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 150

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

ADM study

Data Modeling curriculum: designed to improve statistical reasoning skills cluster-randomized trial pretest and posttest: ADM Statistical Reasoning Measure developed by Rich Lehrer at Vanderbilt in conjunction with the BEAR Center the measure has five dimensions (domains): Data Display (DAD), Models of Variability (MOV), Chance (CHA), Concepts of Statistics (COS), and Informal Inference (INI)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 151

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

ADM study

Data Modeling curriculum: designed to improve statistical reasoning skills cluster-randomized trial pretest and posttest: ADM Statistical Reasoning Measure developed by Rich Lehrer at Vanderbilt in conjunction with the BEAR Center the measure has five dimensions (domains): Data Display (DAD), Models of Variability (MOV), Chance (CHA), Concepts of Statistics (COS), and Informal Inference (INI)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 152

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

ADM study

Data Modeling curriculum: designed to improve statistical reasoning skills cluster-randomized trial pretest and posttest: ADM Statistical Reasoning Measure developed by Rich Lehrer at Vanderbilt in conjunction with the BEAR Center the measure has five dimensions (domains): Data Display (DAD), Models of Variability (MOV), Chance (CHA), Concepts of Statistics (COS), and Informal Inference (INI)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 153

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

common scale for pre and post

no common items between pre and post we need common scale (especially for the CS approach) linked through a third data set—a large set of ADM items administered in 2011

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 154

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

common scale for pre and post

no common items between pre and post we need common scale (especially for the CS approach) linked through a third data set—a large set of ADM items administered in 2011

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 155

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

common scale for pre and post

no common items between pre and post we need common scale (especially for the CS approach) linked through a third data set—a large set of ADM items administered in 2011

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 156

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

ADM study

768 students nested in 40 teachers from 21 schools and 4 districts

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 157

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

complications

three groups

group A: treatment group group C: control group group B: schools assigned to treatment that started the new curriculum before the pretest

selection bias: 7 schools assigned to the treatment and 5 schools assigned to the control group opted out from the study after the randomization results were known

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 158

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

complications

three groups

group A: treatment group group C: control group group B: schools assigned to treatment that started the new curriculum before the pretest

selection bias: 7 schools assigned to the treatment and 5 schools assigned to the control group opted out from the study after the randomization results were known

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 159

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

complications

three groups

group A: treatment group group C: control group group B: schools assigned to treatment that started the new curriculum before the pretest

selection bias: 7 schools assigned to the treatment and 5 schools assigned to the control group opted out from the study after the randomization results were known

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 160

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

complications

three groups

group A: treatment group group C: control group group B: schools assigned to treatment that started the new curriculum before the pretest

selection bias: 7 schools assigned to the treatment and 5 schools assigned to the control group opted out from the study after the randomization results were known

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 161

Lord’s paradox comparison of the approaches ADM study Data Modeling curriculum

complications

three groups

group A: treatment group group C: control group group B: schools assigned to treatment that started the new curriculum before the pretest

selection bias: 7 schools assigned to the treatment and 5 schools assigned to the control group opted out from the study after the randomization results were known

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 162

Lord’s paradox comparison of the approaches ADM study analysis of the data

late pretest

group B: 9 of the 19 teachers in the treatment group (due to the delayed IRB approval) units 1 and 2 of the ADM curriculum: DAD module of the curriculum the DAD dimension is highly correlated with the other dimensions

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 164

Lord’s paradox comparison of the approaches ADM study analysis of the data

late pretest

group B: 9 of the 19 teachers in the treatment group (due to the delayed IRB approval) units 1 and 2 of the ADM curriculum: DAD module of the curriculum the DAD dimension is highly correlated with the other dimensions

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 165

Lord’s paradox comparison of the approaches ADM study analysis of the data

late pretest

group B: 9 of the 19 teachers in the treatment group (due to the delayed IRB approval) units 1 and 2 of the ADM curriculum: DAD module of the curriculum the DAD dimension is highly correlated with the other dimensions

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 166

Lord’s paradox comparison of the approaches ADM study analysis of the data

balance in covariates

note: assignment to group A vs. B can be assumed at random (delay in IRB approval)

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 167

Lord’s paradox comparison of the approaches ADM study analysis of the data

balance at pretest

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 168

Lord’s paradox comparison of the approaches ADM study analysis of the data

exploiting the multidimensional nature of the measure: groups B and A at pretest

at pretest: are differences between groups A and B most pronounced in DAD?

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 169

Lord’s paradox comparison of the approaches ADM study analysis of the data

groups B and A at posttest

at posttest: both groups receive the same treatment by the time they take the posttest

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 170

Lord’s paradox comparison of the approaches ADM study analysis of the data

gains: groups B and A

group A is significantly higher in gains in DAD domain than group B due to the positive correlation: higher gains in CHA, COS, INI and MOV in group A compared to group B gains for the group B on dimensions that have highest correlation with the DAD domain are smaller than other dimensions when DAD items are excluded in the estimation of the composite score, group A and group B differences in gain from pre to post is not significant when DAD items are included in the estimation of the composite score, group A’s gain from pre to post is significantly higher

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 171

Lord’s paradox comparison of the approaches ADM study analysis of the data

gains: groups B and A

group A is significantly higher in gains in DAD domain than group B due to the positive correlation: higher gains in CHA, COS, INI and MOV in group A compared to group B gains for the group B on dimensions that have highest correlation with the DAD domain are smaller than other dimensions when DAD items are excluded in the estimation of the composite score, group A and group B differences in gain from pre to post is not significant when DAD items are included in the estimation of the composite score, group A’s gain from pre to post is significantly higher

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 172

Lord’s paradox comparison of the approaches ADM study analysis of the data

gains: groups B and A

group A is significantly higher in gains in DAD domain than group B due to the positive correlation: higher gains in CHA, COS, INI and MOV in group A compared to group B gains for the group B on dimensions that have highest correlation with the DAD domain are smaller than other dimensions when DAD items are excluded in the estimation of the composite score, group A and group B differences in gain from pre to post is not significant when DAD items are included in the estimation of the composite score, group A’s gain from pre to post is significantly higher

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 173

Lord’s paradox comparison of the approaches ADM study analysis of the data

gains: groups B and A

group A is significantly higher in gains in DAD domain than group B due to the positive correlation: higher gains in CHA, COS, INI and MOV in group A compared to group B gains for the group B on dimensions that have highest correlation with the DAD domain are smaller than other dimensions when DAD items are excluded in the estimation of the composite score, group A and group B differences in gain from pre to post is not significant when DAD items are included in the estimation of the composite score, group A’s gain from pre to post is significantly higher

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 174

Lord’s paradox comparison of the approaches ADM study analysis of the data

gains: groups B and A

group A is significantly higher in gains in DAD domain than group B due to the positive correlation: higher gains in CHA, COS, INI and MOV in group A compared to group B gains for the group B on dimensions that have highest correlation with the DAD domain are smaller than other dimensions when DAD items are excluded in the estimation of the composite score, group A and group B differences in gain from pre to post is not significant when DAD items are included in the estimation of the composite score, group A’s gain from pre to post is significantly higher

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 175

Lord’s paradox comparison of the approaches ADM study analysis of the data

comparing all three groups: composite score

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 176

Lord’s paradox comparison of the approaches ADM study analysis of the data

three groups: logit scores

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 177

Lord’s paradox comparison of the approaches ADM study analysis of the data

CS approach

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 178

Lord’s paradox comparison of the approaches ADM study analysis of the data

RV approach

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 179

Lord’s paradox comparison of the approaches ADM study analysis of the data

in [mostly harmless] econometrics

Angrist & Pischke (2009), section 5.4 Wooldridge (2010, p. 974) “... a commonly used alternative in

the statistics is to add the first-period outcome as a control” ∆Yj = β1 + β2Wj + β3Y1j + ǫj Y1j overcontrols and leads to inconsistency “..if [treatment] indicates a job training program and less productive workers are more likely to participate, then regression that controls for Y1 underestimates the job training

effect. If more productive workers participate, it overestimates

the effect of job training.”, adding “[...] covariates can induce bias when there was none.”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 180

Lord’s paradox comparison of the approaches ADM study analysis of the data

in [mostly harmless] econometrics

Angrist & Pischke (2009), section 5.4 Wooldridge (2010, p. 974) “... a commonly used alternative in

the statistics is to add the first-period outcome as a control” ∆Yj = β1 + β2Wj + β3Y1j + ǫj Y1j overcontrols and leads to inconsistency “..if [treatment] indicates a job training program and less productive workers are more likely to participate, then regression that controls for Y1 underestimates the job training

effect. If more productive workers participate, it overestimates

the effect of job training.”, adding “[...] covariates can induce bias when there was none.”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 181

Lord’s paradox comparison of the approaches ADM study analysis of the data

in [mostly harmless] econometrics

Angrist & Pischke (2009), section 5.4 Wooldridge (2010, p. 974) “... a commonly used alternative in

the statistics is to add the first-period outcome as a control” ∆Yj = β1 + β2Wj + β3Y1j + ǫj Y1j overcontrols and leads to inconsistency “..if [treatment] indicates a job training program and less productive workers are more likely to participate, then regression that controls for Y1 underestimates the job training

effect. If more productive workers participate, it overestimates

the effect of job training.”, adding “[...] covariates can induce bias when there was none.”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 182

Lord’s paradox comparison of the approaches ADM study analysis of the data

in [mostly harmless] econometrics

Angrist & Pischke (2009), section 5.4 Wooldridge (2010, p. 974) “... a commonly used alternative in

the statistics is to add the first-period outcome as a control” ∆Yj = β1 + β2Wj + β3Y1j + ǫj Y1j overcontrols and leads to inconsistency “..if [treatment] indicates a job training program and less productive workers are more likely to participate, then regression that controls for Y1 underestimates the job training

effect. If more productive workers participate, it overestimates

the effect of job training.”, adding “[...] covariates can induce bias when there was none.”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 183

Lord’s paradox comparison of the approaches ADM study analysis of the data

in [mostly harmless] econometrics

Angrist & Pischke (2009), section 5.4 Wooldridge (2010, p. 974) “... a commonly used alternative in

the statistics is to add the first-period outcome as a control” ∆Yj = β1 + β2Wj + β3Y1j + ǫj Y1j overcontrols and leads to inconsistency “..if [treatment] indicates a job training program and less productive workers are more likely to participate, then regression that controls for Y1 underestimates the job training

effect. If more productive workers participate, it overestimates

the effect of job training.”, adding “[...] covariates can induce bias when there was none.”

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 184

Lord’s paradox comparison of the approaches ADM study analysis of the data

comparing all three groups: DAD

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 185

Lord’s paradox comparison of the approaches ADM study analysis of the data

comparing all three groups: CHA

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 186

Lord’s paradox comparison of the approaches ADM study analysis of the data

comparing all three groups: COS

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 187

Lord’s paradox comparison of the approaches ADM study analysis of the data

comparing all three groups: MOV

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 188

Lord’s paradox comparison of the approaches ADM study analysis of the data

comparing all three groups: INI

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 189

Lord’s paradox comparison of the approaches ADM study analysis of the data

final notes

Wainer & Brown (2007): “[Lord’s paradox is ] by far, the most difficult paradox to disentangle and requires clear thinking” Freedman: none of the statistical methods, no matter how fancy and sophisticated they are, will be able to compensate for the sloppy study design

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 190

Lord’s paradox comparison of the approaches ADM study analysis of the data

final notes

Wainer & Brown (2007): “[Lord’s paradox is ] by far, the most difficult paradox to disentangle and requires clear thinking” Freedman: none of the statistical methods, no matter how fancy and sophisticated they are, will be able to compensate for the sloppy study design

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

SLIDE 191

Lord’s paradox comparison of the approaches ADM study analysis of the data

thank you

questions? perman@berkeley.edu

Perman Gochyyev GSE, UC Berkeley BEAR Center Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

Evaluating the Treatment Effect in the ADM Study and Lord’s Paradox

Perman Gochyyev

GSE, UC Berkeley BEAR Center

October 17, 2017

Table of Contents

1

Lord’s paradox Lord, 1967 understanding the two approaches

2

comparison of the approaches psychometricians on change scores attempts using causal framework a closer look

3

ADM study Data Modeling curriculum analysis of the data

example from Lord, 1967

A large university is interested in investigating the effects on the students of the diet provided in the university dining halls and any sex difference in these

particular, the weight of each student at the time of his arrival in September and his weight the following June are recorded. (p.304)

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

example from Lord, 1967

statistician 1

examined gains in student body weight between boys and girls found no significant changes between the beginning and end of the year

statistician 2

covaried out each student’s weight in September from his/her weight in June discovered that the average weight in June was greater for boys than for girls

Lord, 1967

statistician 1: change score approach: Ypost − Ypre = β1 + β2W + ǫ statistician 2: regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ

who is right?

Lord, 1967

statistician 1: change score approach: Ypost − Ypre = β1 + β2W + ǫ statistician 2: regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ

who is right?

Lord, 1967

statistician 1: change score approach: Ypost − Ypre = β1 + β2W + ǫ statistician 2: regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ

who is right?

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

example from Allison, 1990

treatment: plastic surgery for craniofacial abnormalities (n=18) control group: children with same age range (n=30)

statistician 1: β2 = 0 statistician 2: β2 > 0 at p=0.03

the two approaches

Why does the approach taken by statistician 2 — the method that currently dominates social science methodology — give an unintuitive and misleading result?

the two approaches

change score approach: Ypost − Ypre = β1 + β2W + ǫ regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ still regressor variable approach Ypost − Ypre = β1 + β2W + (β3 − 1)Ypre + ǫ

the two approaches

change score approach: Ypost − Ypre = β1 + β2W + ǫ regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ still regressor variable approach Ypost − Ypre = β1 + β2W + (β3 − 1)Ypre + ǫ

the two approaches

change score approach: Ypost − Ypre = β1 + β2W + ǫ regressor variable approach Ypost = β1 + β2W + β3Ypre + ǫ still regressor variable approach Ypost − Ypre = β1 + β2W + (β3 − 1)Ypre + ǫ

in other disciplines and traditions

causal SEM framework

statistician 1: total effect statistician 2: direct effect (adjusting for pretest)

econometrics