Going to Scale: Immediate and Long-Term Effects of a Randomized - - PowerPoint PPT Presentation

▶
going to scale immediate and long term effects of a
SMART_READER_LITE
LIVE PREVIEW

Going to Scale: Immediate and Long-Term Effects of a Randomized - - PowerPoint PPT Presentation

Oct 12, 2022 46 likes •272 views

Going to Scale: Immediate and Long-Term Effects of a Randomized Control Trial of a State Pre-K Dale C. Farran, Mark W. Lipsey, Kerry G. Hofer Dale C. Farran, Mark W. Lipsey, Kerry G. Hofer Nianbo Nianbo Dong, Carol Bilbr Dong, Carol Bilbrey

slide-1
SLIDE 1

Going to Scale: Immediate and Long-Term Effects of a Randomized Control Trial of a State Pre-K

Dale C. Farran, Mark W. Lipsey, Kerry G. Hofer Dale C. Farran, Mark W. Lipsey, Kerry G. Hofer Nianbo Nianbo Dong, Carol Bilbr Dong, Carol Bilbrey ey

Peabody Research Institute Vanderbilt University AERA, Philadelphia, PA, April 2014

slide-2
SLIDE 2

Research Team

  • Senior Research Associate

– Kerry Hofer

  • Research Associate

– Nianbo Dong

  • Research Analysts

– Chris Ham – Rick Feldser – Ilknur Sekmen

  • TN Education Consultant

– Bob Taylor

  • Child assessors across the state
  • Principal Investigators

– Mark Lipsey – Dale Farran

  • TN Dept of Education Partner

– Connie Casha

  • Project Manager

– Carol Bilbrey

  • Project Coordinator

– Janie Hughart

  • Regional Coordinator

– Diane Hughes

Funded by the Funded by the Institute for Ed Institute for Education Sciences Grant #R305E090009 ucation Sciences Grant #R305E090009

April 2014 Farran AERA 2

slide-3
SLIDE 3

The Tennessee Voluntary Pre-K Program (TN-VPK)

  • 4-year old children

– priority for children who qualify for the FRPL programs – after that, children at-risk for other reasons.

  • Full day program

– 5.5 hours per day; minimum of 180 days per year.

  • Program meets 9 of 10 NIEER recommendations for quality, e.g.,

– State licensed teachers with early childhood education endorsements. – Aide in each classroom; class size maximum of 20. – Approved age-appropriate curriculum aligned with the Tennessee ECE Developmental Standards.

  • 934 pre-k classrooms in 133 of the 136 Tennessee school systems

across all 95 Tennessee counties; serves more than 18,000 children.

April 2014 Farran AERA 3

slide-4
SLIDE 4

RCT (Randomized Control Trial)

  • Full Sample

– Randomization to wait lists in oversubscribed schools – 2 cohorts; 3025 students across TN

  • Cohort 1: Pre-k year = 2009-10; N=1764
  • Cohort 2: Pre-k year = 2010-11; N=1261

– Tracking through the state data system to 3rd grade

  • Intensive Substudy of Consented Children

– Assessed each year by the research team – 1076 students in the analysis sample – 773 in TN-VPK; 303 not in TN-VPK (control) – Analyses focused on Treatment on Treated (TOT)

April 2014 Farran AERA 4

slide-5
SLIDE 5

Achievement (Cognitive) Outcomes

[Available only for the Intensive Substudy Sample]

Woodcock Johnson III Scales

  • Literacy: Letter-Word Identification, Spelling
  • Language: Picture Vocabulary, Oral Comprehension,

Passage Comprehension (K and 1st grade only)

  • Math: Applied Problems, Quantitative Concepts, Calculation

(K & 1st grade only)

  • Overall WJ Composite

– Mean W score across all WJ scales

  • Composite of 6 scales used at end of pre-k and beyond
  • Composite of 8 scales used at end of K and beyond

April 2014 Farran AERA 5

slide-6
SLIDE 6

Characteristics of the Children in the ISS Analysis Sample

Charact Characteristic ristic Mean Mean Age start of pre-k year 4.3 Male 45% FRPL 100% Race/ethnicity White 57% Black 23% Hispanic 18% Asian and other 2% Non-native English speaker 20%

April 2014 Farran AERA 6

slide-7
SLIDE 7

Preschool Experience of ISS Control Children

49% 6% 5% 15% 11% 6% 4% 5%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Home with parent Home with Other Homebased Care Private Child Care Head Start Mix (some formal) Mix (no formal) Unknown

Percent of Nonparticipants

60% at home 15% private childcare; 11% Head Start 10% mixed 7

slide-8
SLIDE 8

Outcom e Measure Effect Size % Im provem ent for Pre-K Overall Com posite .33* 45% Literacy Letter-Word ID .46* 89% Spelling .25* 30% Language Picture Vocabulary .20* 83% Oral Comprehension .10* 26% Math Applied Problems .14* 21% Quantitative Concepts .31* 49%

* p<.05

VPK Effects at End of Pre-k on the WJII Subscales

April 2014 Farran AERA 8

slide-9
SLIDE 9

Outcom e Measure PK ES K ES 1st ES Overall Com posite (6) .33* .04

  • .09

Overall Com posite (8)

  • .00
  • .09

Literacy Letter-Word ID .46* .04

  • .06

Spelling .25* .01

  • .09

Language Picture Vocabulary .20* .08 .02 Oral Comprehension .10* .09

  • .07

Passage Comprehension

  • .11†
  • .07

Math Applied Problems .14* .02

  • .05

Quantitative Concepts .31*

  • .07
  • .21*

Calculation

  • .07
  • .06

* p<.05 † p<.10

Effects at End of PK, K, & 1st on WJII Subscales

April 2014 Farran AERA 9

slide-10
SLIDE 10

Non-Cognitive Outcomes

[Available for ISS sample & portions of full sample]

Teacher ratings at beginning of K and end of 1st grade

  • Cooper-Farran Behavioral Rating Scales (ISS sample)

– Work-Related Skills – Social Behavior

  • Academic Child Behavior Record (ISS sample)

– Preparation for Grade – Peer Relations – Behavior Problems – Feelings About School

Measures from the State Education Information System

  • Retention in grade
  • Attendance
  • Disciplinary Action
  • ELL status
  • IEPs

April 2014 Farran AERA 10 Too early to collect valid data

slide-11
SLIDE 11

Outcom e Measure Beginning

  • f K ES

End of 1st Grade ES CF Social Skills .30 .06 CF Work-Related Skills .37*

  • .03

ACBR Preparation for Grade .44*

  • .10

ACBR Peer Relations .19* .08 ACBR Behavior Problems

  • .39*
  • .05

ACBR Feelings About School .05

  • .06

* p<.05

Effects on Teacher Ratings at Beginning

  • f K and End of First Grade (ISS sample)

April 2014 Farran AERA 11

slide-12
SLIDE 12

Grade Retention in Kindergarten

No VPK 6.2% No VPK 6.8% VPK 4.1% VPK 5.0%

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% ISS Sample Full Sample Percent Retained in K

(p<.05) (p<.05)

April 2014 Farran AERA 12

slide-13
SLIDE 13

Grade and Sam ple

TN-VPK Participants Mean Days TN-VPK Nonparticipants Mean Days p for difference

K attendance, ISS sample 160.0 160.4 .534 K attendance, Full sample 158.0 156.6 .0 8 4 1st grade attendance, Cohort 1 164.0 160.6 .0 17

Attendance in Kindergarten & 1st Grade

April 2014 Farran AERA 13

slide-14
SLIDE 14

Summary of TN-VPK Effects Found So Far

Achievement (Cognitive) Outcomes Achievement (Cognitive) Outcomes

  • Positive effects at the end of pre-k on all measures
  • Effects were not sustained through kindergarten and first grade

Non-Cognit Non-Cognitive Outcomes e Outcomes

  • Teacher ratings

– Positive effects at beginning of K on ratings of preparedness for grade, work related skills, and classroom behavior – Effects were not sustained through end of first grade

  • School records

– Fewer TN-VPK participants retained in kindergarten but group difference in rate not sustained through 1st grade – More days attended by TN-VPK participants in K and 1st grade

April 2014 Farran AERA 14

slide-15
SLIDE 15

PUTTING TN-VPK RESULTS IN PUTTING TN-VPK RESULTS IN CONTEXT* CONTEXT*

April 2014 Farran AERA 15

*With thanks to Greg Duncan for Meta Analysis Slides

slide-16
SLIDE 16
  • 0.50

0.00 0.50 1.00 1.50 2.00

1950 1960 1970 1980 1990 2000 2010

Average erage ef effect ct size in sd units size in sd units

Average cognitiv erage cognitive im e impact at end of treatment pact at end of treatment

Perry Preschool Abecedarian

slide-17
SLIDE 17

Evidentiary Basis for Scaling Up

  • Both Perry and Abecedarian

– Implemented in the 1960s and 70s – Counterfactual much more limited than today – Small, focused, intense programs – Heavily guided by on site, invested researchers – Much more extensive

  • Abecedarian intervention from 6 weeks to age 5
  • Full working day not school day
  • Nonetheless these are studies cited to justify expanding

pre-k intervention for poor children though larger, more current studies do not replicate early effects

April 2014 Farran AERA 17

slide-18
SLIDE 18
  • 0.50

0.00 0.50 1.00 1.50 2.00

1950 1960 1970 1980 1990 2000 2010

Average erage ef effect ct size in sd units size in sd units

Average cognitiv erage cognitive im e impact at end of treatment pact at end of treatment

Head Start Non Head Start

Perry Preschool Abecedarian National Head Start

slide-19
SLIDE 19
  • 0.50

0.00 0.50 1.00 1.50 2.00

1950 1960 1970 1980 1990 2000 2010

Average erage ef effect ct size in sd units size in sd units

Average cognitiv erage cognitive im e impact at end of treatment pact at end of treatment

Head Start Non Head Start

Perry Preschool Abecedarian National Head Start

TN-VPK

slide-20
SLIDE 20

Issues with Scaling Up Pre-K

  • We lack coherent vision for what Pre-K should accomplish
  • Variation among domains most strongly affected

– Concrete school-like readiness skills most affected – Language and math skills least affected – Improving language and math requires rigorous research efforts

  • Programs that have minimum cost, minimum required program

elements, scale up easily, but perhaps without important or lasting effects

  • Strong vision critical when scaling up at a statewide level

– School districts are highly independent

  • Even for an effective program, it will be a challenge at scale to

maintain consistent positive effects across all classrooms

April 2014 Farran AERA 20

slide-21
SLIDE 21

Conclusion

As prekindergarten programs are scaled up, rigorous data should be collected to determine variations among individual classrooms with systems for improvement. As researchers and advocates, we need to be much more intentional and systematic in our efforts to help pre-k achieve its potential.

April 2014 Farran AERA 21

slide-22
SLIDE 22

Thanks!

More information available at:

http://peabody.vanderbilt.edu/research/pri/projects/by_content_area/tennessee_state_pre-k_evaluation/

  • r Google “Peabody Research Institute”

Contact email: dale.farran@vanderbilt.edu

April 2014 Farran AERA 22