Ordering Individuals with Sum Scores: the Introduction of the - - PowerPoint PPT Presentation

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Ordering Individuals with Sum Scores: the Introduction of the Nonparametric Rasch Model

Robert Zwitser Gunter Maris

Cito

July 26th 2013

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Scoring

How to score a test? latent variable models How to report test results? parameter estimates versus observed scores

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Monotone Latent Variable Models

Monotone latent variable models: assuming (at least) Unidimensionality Local Independence Monotonicity In particular: Rasch Model (RM) Monotone Homogeneity Model (MHM)

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Monotone Homogeneity Model

Stochastic ordering of θ by X+ (SOL): (Θ|X+ = a) >

st (Θ|X+ = b),

if a > b, which equals E(h(Θ)|X+ = a) > E(h(Θ)|X+ = b) for all a > b, and all bounded increasing functions h.

0.0 0.2 0.4 0.6 0.8 1.0

Θ F(Θ|X+)

• ● ●
• ●
• ●
• X+ = a

X+ = b

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Individual Measurement

Consider a test with 3 items: Guttman item two items with P(Xi = 1) = 0.5 (e.g., two coins) (Θ|X+ = 2) >

st (Θ|X+ = 1)

(Θ|X = [0, 1, 1]) <

st (Θ|X = [1, 0, 0])

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

(Ordinal) Sufficiency

Sufficiency of statistic H implies that (Θ|X = x2) >

st (Θ|X = x1),

if H(x2) > H(x1), (1) and (Θ|X = x2) =

st (Θ|X = x1),

if H(x2) = H(x1). (2) Ordinal sufficiency (OS) implies only (1).

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Some models and OS of X+

OS of X+ for θ∗? MHM: no MHM + invariant item ordering + monotone traceline ratio: no Normal Ogive Model: no 2PL model: depends on the discrimination parameters

∗proofs provided in the paper Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Some models and OS of X+

OS of X+ for θ∗? MHM: no MHM + invariant item ordering + monotone traceline ratio: no Normal Ogive Model: no 2PL model: depends on the discrimination parameters New model: nonparametric Rasch Model (npRM) UD, LI, M and OS.

∗proofs provided in the paper Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Testable Implications of OS (1)

Lemma OS of the sum score for a set of items implies OS of the sum score for any subset of items. Lemma If OS of the sum score holds in all subsets of (p − 1) items, then it also holds for all p items, provided p is even.

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Testable Implications of OS (2)

Let S be a subset of X.

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Testable Implications of OS (2)

Let S be a subset of X. The null hypotheses: H(X) is ordinal sufficient for θ. Then ∀s1, s2 : (Θ|s2) >

st (Θ|s1),

if H(s2) > H(s1).

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Testable Implications of OS (2)

Let S be a subset of X. The null hypotheses: H(X) is ordinal sufficient for θ. Then ∀s1, s2 : (Θ|s2) >

st (Θ|s1),

if H(s2) > H(s1). These multiple sub-hypotheses can be tested by determining whether ∀s1, s2 : (X

¯ S +|s2)> st(X ¯ S +|s1),

if H(s2) > H(s1).

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Example (1)

Dutch Entrance Test (in Dutch: Entreetoets) multiple subtests administered annually to 125,000 grade 5 pupils subtest with 120 math items

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Example (2)

One-Parameter Logistic Model (OPLM):

Pi(θ) = exp[ai(θ − δi)] 1 + exp[ai(θ − δi)]

item discrimination 1 2 2 4 3 3 · · · · · · 6 2 · · · · · · 8 4 · · · · · · 10 5 · · · · · ·

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Example (3): Results

item discrimination 1 2 2 4 3 3 Kolmogorov-Smirnov Test: D− = 0.0002, p = .9998

20 40 60 80 100 120 0.0 0.2 0.4 0.6 0.8 1.0 F(X+

S|s)

• ●
• X+

S|s

• s = {0,0,1}

s = {1,1,0}

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Example (3): Results

item discrimination 1 2 6 2 10 5 Kolmogorov-Smirnov Test: D− = 0.0829, p < .001

20 40 60 80 100 120 0.0 0.2 0.4 0.6 0.8 1.0 F(X+

S|s)

• X+

S|s

• s = {0,0,1}

s = {1,1,0}

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Example (3): Results

item discrimination 1 2 6 2 8 4 Kolmogorov-Smirnov Test: D− = 0.0016, p = .9792

20 40 60 80 100 120 0.0 0.2 0.4 0.6 0.8 1.0 F(X+

S|s)

• X+

S|s

• s = {0,0,1}

s = {1,1,0}

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Future Directions

If X+ is not OS, find another statistic How to equate two tests that both have an OS statistic

Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions

Thank you!

robert.zwitser@cito.nl

Robert Zwitser, Gunter Maris Presentation IMPS 2013