for Assessing Online Inquiry in ELA Jesse R. Sparks Educational - - PowerPoint PPT Presentation

Measuring What Matters with Innovative Assessments of 21st Century Skills for College and Career Readiness:

NAEP SAIL Virtual World for Assessing Online Inquiry in ELA

Jesse R. Sparks Educational Testing Service

CCSSO National Conference on Student Assessment June 25, 2019

NAEP SAIL Initiative

2

The Survey Assessment Innovations Laboratory (SAIL) is a hothouse dedicated to explorations in the cognitive sciences, assessment, and technology that will enable a continuously updated NAEP which leads the field of educational assessment

What we assess How we assess How we score and report what we assess

New models of 21st century skills and competencies New technologies to elicit and capture complex human interactions Computational advances for automated scoring, complex modeling, score reporting

Motivation

• There is a growing interest in teaching and learning of complex

cognitive constructs:

• Conducting inquiry from multiple sources in the English Language Arts
• There is a need to develop quality large-scale assessments

that provide valid information about students’ knowledge, skills, and abilities as related to authentic, complex inquiry tasks.

3

NAEP SAIL Project Aims

• Research and develop virtual environments with authentic contexts
• To gather evidence of students’ inquiry skills using naturalistic simulations reflecting real-world

applications of the targeted skills

• Explore students’ interactions in simulated inquiry scenarios that require
• Locating, evaluating, reading, and writing syntheses from multiple sources, and reconciling

conflicting, unreliable, and inaccurate information

• Leverage technology from games and simulations
• to enhance student motivation and engagement in assessment tasks
• (important for NAEP, which is “low stakes” for the students)
• to provide evidence of students’ cognitive processes as they unfold over time.

4

SAIL Virtual World for ELA: Environment & Task Design

Construct: Online Multiple-Source Inquiry

6

SAIL ELA Virtual World Environment

7

SAIL ELA Virtual World Environment: Tools and Resources

Planning Inquiry with Virtual Partner (Guide) Compose Written Response to Inquiry Task using Collected Resources Simulated Web Search & Library Search Tools to retrieve documents Supports for Source Evaluation & Note-taking for Saved Resources Simulated Questioning of Virtual Characters (experts or laypeople)

8

9

SAIL ELA Virtual World – Scenario-Based Task Logic

Scenario-based tasks in the ELA Virtual World include three phases:

• Setup: a fixed sequence of

introductory material and tutorials for each tool

• Free Roam: a period of “free

exploration” of available resources (ideal & optional)

• Conclusion: a culminating

phase where students work to integrate information and compose a final, written response to the inquiry question, using information from the available sources.

Define Problem and Information Needs

Use Partner to define research questions Use Partner to define known, needed information Use Map Tool to make plans for information gathering

Locate Sources

Use Inspector to examine sources Use Internet Search to retrieve websites Use Librarian to retrieve print sources Use Conversations with Experts or Laypeople to retrieve information

Evaluate Sources; Process, Analyze & Synthesize

Use Evidence Manager to support close reading and analysis of sources Use Evidence Manager to support critical evaluation of sources

Communicate Results

Use Communicator to plan a response (e.g., identify intended recipients) Use Communicator to compose a response to the task Use Evidence Manager and Communicator to support a response with evidence drawn from sources (i.e., citations, excerpts)

Relations between Online Inquiry Construct & SAIL ELA Virtual World digital tools (white boxes):

SAIL ELA Student/Evidence Model Alignment

Each aspect of the SAIL ELA Virtual World is designed to measure a specific aspect of the online inquiry

• construct. All student actions are mapped to a specific dimension, and are evaluated with specific scoring
• rules. In ECD parlance, this represents our Evidence Model.

In Communicator Tool, student includes citations

• f relevant and reliable

sources in their final response to the task.

Scored by number of key references cited (0-3)

In Evidence Manager Tool, student evaluates the usefulness of sources for one’s inquiry task.

Scored by number of usefulness ratings of 4 or 5

• f 5 for sources containing

key information for completing the task

Example Evidence Model Statements

10

• Goal: Evaluate historical accuracy of claims contained in artifact—Should it go in museum?
• Three phases: Setup, Free Roam, and Conclusion.
• Free Roam allows for nonlinear, free exploration and investigation of inquiry process.

11

SAIL ELA Scenario-Based Task & Structure

Phase Construct(s) Score Components Points (99 total) Setup: Planning Inquiry

Define problem/info needs, Locate, Evaluate

Identify best first step with Partner; Identify key information that must be investigated; Identify 2 useful locations; Questioning an authority figure; Evaluations in Evidence Manager tutorial

13

1 best first step 3 claims to examine 2 useful locations 1 question to authority figure 6 Evidence Manager evaluations

Free Roam: Gather and Evaluate Resources

Locate, Evaluate, Process/Analyze/ Synthesize

Navigation; Search behaviors in Internet Café & Library; Questioning behaviors in Faculty Offices; Resources viewed/saved to Evidence Manager; Evaluations in Evidence Manager

55

9 Library (2 search, 1 save, 6 EM Evaluations) 20 Faculty Offices (6 questions, 2 save, 12 EM Evaluations) 26 Internet Café (2 search, 3 view, 3 save, 18 EM evaluations)

Conclusion: Integration Task

Evaluate, Process/Analyze/ Synthesize

True/false judgments, Source attributions, Misinformation corrections

26

10 true/false judgments 10 source attributions 6 corrections

Conclusion: Argument Writing Task

Process/Analyze/ Synthesize, Communicate

Final decision about inquiry task; Argument writing; Use of excerpts; Use of citations

5 [11]

1 final decision 3 citations 1 excerpt [6 argument quality]

SAIL Virtual World for ELA: Empirical Work

SAIL ELA Empirical Work 2015-2017

• Phase 1: Developing the Virtual World Platform (N=14)
• Play testing – Groups A&B – Storyboard (n=5)
• Play testing – Group C – Playable Platform (n=5)
• Play testing – Group D – Playable Platform (n=4)
• Phase 2: Developing the Scenario-Based Task (N=31)
• Play testing – Group E – Playable Task (n=6)
• Play testing – Group F – Playable Task (n=5)
• Play testing – Group G – Playable Task (n=5)
• Cognitive Lab – Group H – Complete Task w/Think-Alouds (n=15)
• Small-Scale Tryouts: Group SE – Complete Task in Classroom

Conditions (n=134)

13

Setup Phase Free Roam Writing Letter Evaluation

Coglab Data – Time per Scene/Tool

14

Key Findings from Empirical Work

• Students demonstrated a variety of skilled performances during the virtual

world inquiry task—multiple paths to solution were enabled and pursued by students.

• Task scores correlated with external measures of ELA proficiency,

including class grades and state test scores (e.g., PARCC, ACT Aspire)

• Students struggled with information locating/search tasks and critical

evaluation activities, including difficulty correcting inaccurate information.

• However, students’ ability to correct inaccurate information predicted their

performance on the final essay task.

• Students found the virtual world task to be engaging, despite some

tedious or time-consuming aspects.

15

16

Students Collaborating on ELA Inquiry Task in “Face to Face” Context Digital Platform Enables Structured Collaboration in “Remote” Context

Extending the SAIL ELA Virtual World to Assess Collaborative Inquiry

In 2016, we began development of a collaborative version of the SAIL ELA Virtual World task, in order to elicit evidence of students’ collaborative skills, and to elicit better evidence of students’ online inquiry skills VIA collaborative dialogue.

• Conducted “face to face” playtesting

with pairs of students to refine collaborative prompts

• Conducted “remote” playtesting using a

digital platform for collaboration, which enables delivery of collaborative prompts, real-time audio/video and text chat, and a shared task interface.

These Virtual Environments Enable Us To…

• Deliver innovative assessment tasks using digital platforms
• Engage students in simulated versions of authentic real-world tasks
• Collect moment-by-moment data on students’ actions
• Provide multiple paths to correct solutions, and opportunities to

make students’ thinking visible

• Better understand how students’ solution processes contribute to

final outcomes

17

References

Coiro, J., Sparks, J.R., & Kulikowich, J.M. (2018). Assessing online collaborative inquiry and social deliberation skills as learners navigate multiple sources and

• perspectives. In J.L.G. Braasch, I. Braten, & M.T. McCrudden (Eds.), Handbook of

Multiple Source Use (pp. 485-501). New York: Routledge. Coiro, J., Sparks, J.R., Kiili, C., Castek, J., Lee, C-H., & Holland, B.R. (accepted). Students engaging in multiple-source inquiry tasks: Capturing dimensions of collaborative online inquiry and social deliberation. Literacy Research: Theory, Method, and Practice. Sparks, J.R., Appel, C., Gao, J., & Zhu, M. (2018, April). NAEP SAIL Virtual World for Assessment of ELA Inquiry. Paper in symposium session presented at the annual meeting of the American Educational Research Association, New York, NY. Sparks, J.R., & Deane, P . (2015). Cognitively based assessment of research and inquiry skills: Defining a key practice in the English language arts (ETS Research Report No. RR-15-35). Princeton, NJ: Educational Testing Service.

18

To follow up with questions or comments, please contact: jsparks@ets.org

Acknowledgements

ELA Virtual World: Jesse R. Sparks (PI), Brian Young, Madeline Keehner, Jie Gao, Mengxiao Zhu, Colleen Appel, Gary Feng, Hilary Persky, Heather Nadelman, Irv Katz, Julie Coiro (University of Rhode Island) Developer: Intelligent Automation, Inc.