What Do We Get From CR and TE Items? Findings from the study The - - PowerPoint PPT Presentation

▶
what do we get from cr and te items
SMART_READER_LITE
LIVE PREVIEW

What Do We Get From CR and TE Items? Findings from the study The - - PowerPoint PPT Presentation

Jun 07, 2023 754 likes •1.19k views

What Do We Get From CR and TE Items? Findings from the study The Utility of Online Mathematics Constructed-Response Items: Maintaining Important Mathematics in State Assessments and Providing Appropriate Access to Students Phoebe C. Winter, Scott

slide-1
SLIDE 1

What Do We Get From CR and TE Items?

Findings from the study The Utility of Online Mathematics Constructed-Response Items: Maintaining Important Mathematics in State Assessments and Providing Appropriate Access to Students

Phoebe C. Winter, Scott W. Wood, Susan M. Lottridge, Thomas B. Hughes, & Teresa E. Walker

Presentation at the National Conference on Student Assessment, June 20, 2013

slide-2
SLIDE 2

This work was supported by a Grant for Enhanced Assessment Instruments awarded by the U.S. Department of Education to the North Carolina Department of Public Instruction as lead state, award S368A100003. The

  • pinions in this document are those of the authors and do not necessarily reflect

those of the North Carolina Department of Public Instruction, the U.S. Department of Education, or the participating project states. For more information, including sample items and the full report, visit http://www.pacificmetrics.com/omap/

slide-3
SLIDE 3

Overview

  • Background
  • Overall results

– Reliability – Information functions

  • Item set results

– TE vs. MC

  • Information functions

– CR vs. MC

  • Information functions
  • Student responses
slide-4
SLIDE 4

Purpose

  • To investigate the feasibility and utility of using mathematics

assessments consisting solely of constructed-response (CR) and technology-enhanced (TE) items to measure mathematics knowledge and skills.

slide-5
SLIDE 5

Design

  • 4 states
  • Grade 7 and Algebra I

– Subset of CCSS in mathematics

  • Two core test forms per content area

– Cores designed to measure the same KS – Form 1 consisted of CR/TE items – Form 2 consisted of KS-matched MC items – Same number of points for each core form – Additional items added to include all item formats and balance response time across forms

  • Tests delivered online
  • All items machine scored
  • Forms randomly assigned within content area
slide-6
SLIDE 6

Grade 7 Forms

Form Core Items N Items on T est CR/TE MC T

  • tal

Form 1 CR/TE 13 5 18 Form 2 MC 5 16 21

slide-7
SLIDE 7

Algebra I Forms

Form Core Items N Items on T est CR/TE MC Total Form 1 CR/TE 14 5 19 Form 2 MC 4 17 21

slide-8
SLIDE 8

CR/TE Item Types

Graphing items allow the student to draw lines or points (or both) on a grid. Drag-and-Drop items allow the student to match images to their correct counterpart

  • r sort and order components of a response.

Constrained Response items have an input tool that pops up when the student clicks in the answer box. The student can either click on the tool buttons or use the matching keys on the keyboard to enter the response. Table Entry items have students use the constrained response tool to enter responses in a table. Free Text Entry items allow the student to use the entire keyboard to respond; restrictions on what the student can enter are removed. Mixed items use both constrained response and free text entry response in different parts of the item.

slide-9
SLIDE 9

Research Sample: Schools and Students by State

Grade 7 Algebra I Schools Students Schools Students State A 14 (31%) 1,395 (24%) 12 (20%) 746 (22%) State B 13 (29%) 2,026 (34%) 18 (31%) 904 (27%) State C 10 (22%) 1,134 (19%) 15 (25%) 604 (18%) State D 8 (18%) 1,361 (23%) 14 (24%) 1,151 (34%) T

  • tal

45 5,916 59 3,405

slide-10
SLIDE 10

Test Information Functions, Grade 7

Form1_Info

1 2 3 4 5 6 7 8 9 10

Theta

  • 3
  • 2
  • 1

1 2 3

Blue X = Form 1 Red + = Form 2 Test Information Functions -- Grade 7

Red = MC Form, Blue = CR/TE Form

slide-11
SLIDE 11

Comparison of Item Information by Set, Grade 7

Theta Value KS Assessed CR/TE Type –2 –1 1 2 7SP8b TE      7RP2ac TE    7SP6* ET      7RP3* ET    7EE4a ET     7NS3 ET    7G5* ET      7EE4a* ET    7NS3* ET     7G5 ET      7RP3* ET      7NS3 ET     7G5 ET    

slide-12
SLIDE 12

Test Information Functions, Algebra I

Form1_Info

1 2 3 4 5 6 7

Theta

  • 3
  • 2
  • 1

1 2 3

Blue X = Form 1 Red + = Form 2 Test Information Functions -- Algebra

Red = MC Form, Blue = CR/TE Form

slide-13
SLIDE 13

Test Information Functions, Algebra I with Anomalous Item Set Removed

Form1_Info

1 2 3 4 5 6 7

Theta

  • 3
  • 2
  • 1

1 2 3

Blue X = Form 1 Red + = Form 2 Test Information Functions -- Algebra (Item 17,19 from Form 2, Item 16 from Form 1 Removed)

Red = MC Form, Blue = CR/TE Form

slide-14
SLIDE 14

Comparison of Item Information by Set, Algebra I

Theta Value KS Assessed CR/TE Type –2 –1 1 2 SID6 GR   FIF4 GR    AREI1 MX    AREI6 TE    FIF4 DD      FLE5 FT     AAPR1 ET      SID6 FT   ACED ET     AREI1 DD    AAPR1 ET    AREI6 ET      FLE5* FT    

slide-15
SLIDE 15

Technology Enhanced Items

slide-16
SLIDE 16

Algebra I, Set B, TE

KS Statement: Identify graphs given a verbal description of a relationship between quantities.

Statistics Aspect Value 0 pt 31% 1 pt 4% 2 pt 65% b-parameter

  • 0.64

point biserial 0.50

slide-17
SLIDE 17

Algebra I, Set B, MC1

KS Statement: Identify graphs given a verbal description of a relationship between quantities. Statistics Aspect Value A 2% B 3% C 92% D 2% b-parameter –2.17 point biserial 0.20

slide-18
SLIDE 18

Algebra I, Set B, MC 2

KS Statement: Identify graphs given a verbal description of a relationship between quantities. Statistics Aspect Value A 70% B 9% C 6% D 15% b-parameter

  • 0.61

point biserial 0.25

slide-19
SLIDE 19

Information

1 2

Theta

  • 3
  • 2
  • 1

1 2 3

Blue = Form 1 Red = Form 2 Item Information Functions -- Algebra -- Item 4 vs. Item 3 & 5

Item Information Functions for Algebra I Set B Red = MC Form, Blue = CR/TE Form

slide-20
SLIDE 20

Algebra I, Set J, TE

KS Statement: Understand that solving an equation is a logical process that maintains equality while moving toward a solution.

Statistics Aspect Value 0 pt 31% 1 pt 69% b-parameter

  • 0.70

point biserial 0.49

slide-21
SLIDE 21

Algebra I, Set J, MC

KS Statement: Understand that solving an equation is a logical process that maintains equality while moving toward a solution. Statistics Aspect Value A 49% B 8% C 24% D 19% b-parameter 0.43 point biserial 0.37

slide-22
SLIDE 22

Information

1 2

Theta

  • 3
  • 2
  • 1

1 2 3

Blue = Form 1 Red = Form 2 Item Information Functions -- Algebra -- Item 14 vs. Item 18

Item Information Functions for Algebra I Set J Red = MC Form, Blue = CR/TE Form

slide-23
SLIDE 23

Constructed Response Items

slide-24
SLIDE 24

Grade 7, Set B, CR

KS 1: Find a matching value given a proportional relationship and a given value. KS 2: Represent a proportional relation with an equation of type y = kx.

Statistics Aspect Value 0 pt 35% 1 pt 48% 2 pt 18% b-parameter 0.39 point biserial 0.57

Exemplary Response DVDs Cost 10 $160.00 15 $240 20 $320 n 16n

slide-25
SLIDE 25

Grade 7, Set B, MC #1

KS Statement 1: Find a matching value given a proportional relationship and a given value. Statistics Aspect Value A 11% B 4% C 10% D 75% b-parameter –0.89 point biserial 0.37

slide-26
SLIDE 26

Grade 7, Set B, MC #2

KS Statement 2: Represent a proportional relation with an equation of type y = kx.

Statistics Aspect Value A 83% B 7% C 6% D 5% b-parameter –1.07 point biserial 0.39

slide-27
SLIDE 27

Information

1 2

Theta

  • 3
  • 2
  • 1

1 2 3

Blue = Form 1 Red = Form 2 Item Information Functions -- Grade 7 -- Item 3 vs. Item 2 & 3

Item Information Functions for Grade 7 Set B Red = MC Form, Blue = CR/TE Form

slide-28
SLIDE 28

Grade 7, Set B, CR

KS 1: Find a matching value given a proportional relationship and a given value. KS 2: Represent a proportional relation with an equation of type y = kx.

Statistics Aspect Value 0 pt 35% 1 pt 48% 2 pt 18% b-parameter 0.39 point biserial 0.57

Exemplary Response DVDs Cost 10 $160.00 15 $240 20 $320 n 16n

slide-29
SLIDE 29

Additional Information from CR Item

  • Students use proportional reasoning to find the values, but

do not find the unit rate needed for the expression.

  • Students determine the unit rate for the correct expression,

but do not correctly calculate values.

  • Students do not write an accurate algebraic expression.
slide-30
SLIDE 30

Grade 7, Set G, CR

Exemplary Response: x = 40

KS Statement: Use facts about supplementary, complementary, vertical, and adjacent angles to find an unknown angle in a single step. Statistics Aspect Value 0 pt 58% 1 pt 42% b-parameter 0.31 point biserial 0.50

slide-31
SLIDE 31

Grade 7, Set G, MC

KS Statement: Use facts about supplementary, complementary, vertical, and adjacent angles to find an unknown angle in a single step. Statistics Aspect Value A 10% B 13% C 65% D 13% b-parameter 0.41 point biserial 0.28

slide-32
SLIDE 32

Information

1 2

Theta

  • 3
  • 2
  • 1

1 2 3

Blue = Form 1 Red = Form 2 Item Information Functions -- Grade 7 -- Item 10 vs. Item 13

Item Information Functions for Grade 7 Set G Red = MC Form, Blue = CR/TE Form

slide-33
SLIDE 33

Algebra I, Set G, CR

Exemplary Response: 3x – 2y

Statistics Aspect Value 0 pt 67% 1 pt 33% b-parameter 0.62 point biserial 0.51 KS Statement: Add and subtract polynomials, combining like terms as needed.

slide-34
SLIDE 34

Algebra I, Set G, MC

KS Statement: Add and subtract polynomials, combining like terms as needed. Statistics Aspect Value A 16% B 48% C 14% D 22% b-parameter 0.67 point biserial 0.28

slide-35
SLIDE 35

Information

1 2

Theta

  • 3
  • 2
  • 1

1 2 3

Blue = Form 1 Red = Form 2 Item Information Functions -- Algebra -- Item 10 vs. Item 12

Item Information Functions for Algebra I Set G Red = MC Form, Blue = CR/TE Form

slide-36
SLIDE 36

Algebra I, Set G

slide-37
SLIDE 37

Common Incorrect Responses to CR Item

  • Example #1: 2x2–3y2
  • Example #2: 3x+2xy–4y
slide-38
SLIDE 38

Algebra I, Set H, CR

KS Statement: Define or describe the relationship between two variables based on data presented in a scatter plot.

Exemplary Response: a line with a negative slope

Statistics Aspect Value 0 pt 48% 1 pt 52% b-parameter –0.10 point biserial 0.31

slide-39
SLIDE 39

Algebra I, Set H, MC

KS Statement: Define or describe the relationship between two variables based on data presented in a scatter plot. Statistics Aspect Value A 65% B 14% C 17% D 4% b-parameter

  • 0.30

point biserial 0.26

slide-40
SLIDE 40

Information

1 2

Theta

  • 3
  • 2
  • 1

1 2 3

Blue = Form 1 Red = Form 2 Item Information Functions -- Algebra -- Item 12 vs. Item 1

Item Information Functions for Algebra I Set H Red = MC Form, Blue = CR/TE Form

slide-41
SLIDE 41

Incorrect Response Examples

  • Example #1: The X values seem to all be positive. All the

Y values are negative.

  • Example #2: They are both almost the same and both cross

at 4.

  • Example #3: There is an equal amount of points on each

axis.

slide-42
SLIDE 42
  • So what do we get?

– From this particular study – limited number of mathematics items, 2 grade levels

  • Overall, more information, better reliability
  • For CR items, the ability to better understand student

thinking

  • Possibilities

– Useful especially for classroom assessment – Potential for enhancing machine scoring algorithms – Better selected response and TE items based on CR responses

Summary

slide-43
SLIDE 43

For more information, including sample items and the full report, visit http://www.pacificmetrics.com/omap/

Prepared by Pacific Metrics for the OMAP NCEAG Partner States