The Effect of Socio-economic Status on the Think-Aloud Quality in - - PowerPoint PPT Presentation

The Effect of Socio-economic Status on the Think-Aloud Quality in Children

Mila Sugovic PhD; Anh-Thu Ton MA; Cecilia Teal MS, MSW; Michael Plotkin

Background

2

 Cognitive interviewing (Tourangeau, et al., 2000) is one of the most effective methods of eliciting in- depth, multi-faceted feedback from the target populations.  Think aloud techniques have been widely studied however, little research has involved survey pretesting, in particular with children.  Young children typically find the think-aloud method to be difficult (van Someren et al, 1994).  Student’s socio-economic status (SES) may affect their ability to accurately recall information (Ginet et al., 2014) but it is unclear whether the quality of verbal reporting produced by children also differs across SES.

Objective and Measures

3

 Objective:

 To evaluate whether students’ socio-economic status (SES) affects the quality of the think-aloud produced by grade 4 students.

 Measures:

 Response Length: number of words used by students during the think-aloud.  Response Relevance: overall relevance of reported think-aloud to questiontopic on a 3- point Likert Scale (1 – Not at all, 2 – Somewhat, 3 – Relevant).  Reported Issues: average number of issues identified by the student.

Methods

4

20 Students

High SES (>$50K)

N=7

Introduction 1. Instructions 2. Demonstration 3. Practice Administer Questions & Think-Aloud Debriefing Administer Questions & Think-Aloud Debriefing Low SES (≤$50K)

N=13

Introduction 1. Instructions 2. Demonstration 3. Practice

Task

5

6 computer-based Science NAEP Assessment Questions (2009)  Question Content: 2 Physical Science, 2 Earth and Space Science, 2 Life Science  Question Difficulty: 2 Easy, 2 Medium, 2 Hard  Question Type: 3 Multiple Choice, 3 Short Constructed Response (SCR)

Sample Demographics

6

 Total of 20 fourth grade students completed the cognitive interview.  The sample consisted of a mix of SES, race/ethnicity and gender.

Table 1: Race/Ethnicity by SES Table 2: Gender by SES

High SES Low SES Total Black or African American 1 3 4 White or Caucasian 4 2 6 Hispanic or Latino(a) 8 8 Asian 2 2 Total 7 13 20 High SES Low SES Total Female 3 7 10 Male 4 6 10 Total 7 13 20

Results – Response Length by SES

7

96 85 119 137 136 110 81 98 104 108 100 94 20 40 60 80 100 120 140 Q1 Q2 Q3 Q4 Q5 Q6

Number of Reported Words Question Number

High SES Low SES

Results – Relevance of Verbal Reports

8

2.8 2.6 3.0 2.8 2.6 2.4 2.6 2.5 2.8 2.4 2.5 2.6 .0 1.0 2.0 3.0 Q1 Q2 Q3 Q4 Q5 Q6

Relevance Rating Question Number

High SES Low SES

Results – Issues Identified

9

.10 .33 .00 .62 .24 .39 .14 .10 .00 .41 .43 .15 .00 .10 .20 .30 .40 .50 .60 .70 Q1 Q2 Q3 Q4 Q5 Q6

Number of Identified Issues Question Number

High SES Low SES

Conclusion

10

 Results appear to indicate a modest trend showing that High SES students produced comparably more words than Low SES students.  There is no difference in verbal report relevance between High and Low SES students. Students across both SES groups generally provided relevant responses during the think-alouds.  Results suggest that there is no difference in the number of issues identified across the two SES groups of grade 4 students. Results may be affected by the overall low number of issues identified between the two groups across all questions.  Overall, findings suggest that the quality of information gathered from think-aloud interviewing is not considerably affected by SES among grade 4 students.

Limitations and Future Research

11

 Limitations include:

 Small sample size (n = 20)  Imbalance in the number of students across the two SES groups(7 High SES, 13 Low SES)  Already revised and tested Science NAEP Questions administered in 2009.

 Newly developed NAEP assessments are more interactive and engaging.

 To what extent does item engagement promote and interfere with student’s ability to remain on topic?  Does item engagement affect the overall quality of student’s think-aloud?

Anh-Thu Ton, MA Research Analyst

tona@eurekafacts.com

Small Business - US SBA SDB Certified Certified to the International Standard for Market, Opinion and Social Research (ISO 20252) DBE & MBE certified nationally & in CA , FL, IL, MA, MD, NJ, NY, PA, TX, VA