and Diagnostic Testing to the High Schools Lessons Learned from - - PowerPoint PPT Presentation

Transitioning College Placement and Diagnostic Testing to the High Schools – Lessons Learned from P.E.R.T.

Matthew Schultz, PhD McCann Associates

2

College Placement Testing in Florida

• Statutorily required placement testing
• Acceptable instruments defined in rule

– SAT, ACT, Accuplacer, and PERT

• Students who do not meet the scores placed

in developmental education

• Required developmental education

coursework must be completed before students may enroll in entry-level college credit courses

From Accuplacer to PERT

American Diploma Project

4

Faculty Involvement

• Over 100 faculty from five school districts, two non-public

colleges, nine universities and 24 colleges

• Identified Florida’s Postsecondary Readiness Competencies
• Prioritized the readiness competencies in terms of

determining a student’s readiness for college-level coursework

• Helped develop the PERT blueprints based on the prioritized

list of readiness competencies

• Reviewed test items to ensure they are aligned to the

competencies and to ensure accuracy and clarity

• Took the test “as a student” to ensure the test is functioning

as expected

5

Postsecondary Readiness Competencies

• PERT is aligned to Florida’s Postsecondary

Readiness Competencies (PRCs)

– PRCs in reading, writing and mathematics – Prioritized list of competencies for success in initial college credit courses (living document) – True assessment of what students need to know before enrolling in introductory courses – Follow up diagnostics that identify which competencies are not met for the purposes of remediation

Blueprint Development

• Series of meetings with stake holders from

department and colleges

• PRC’s for each content area reviewed and

prioritized by SME’s

• Critical PRC’s identified and weighted
• Blueprints developed to reflect most critical

PRC’s for success

Test Design Process

• PRC’s and Blueprints used to drive content

development

• College Faculty developed content exemplars
• McCann developed item banks of content

aligned specifically to PRC’s

• College Faculty reviewed all content and

approved alignment

• DOE, Florida Colleges and McCann

collaborated on pilot to test content

Test Development Process

• Pilot included all colleges and most campuses
• 900+ items administered to samples of

student test takers

• Online platform used to deliver content
• Psychometric review of item performance
• Item calibrations (3 parameter IRT) completed

and CAT developed

• PERT launched operationally Fall 2010

– Includes ongoing field testing of content

9

Diagnostic Competencies

• Diagnostic exams are available for students

not meeting the cut-scores

– Lower and upper level in reading, writing and mathematics – Based on statewide common course descriptions – Prioritized list of competencies for success in given course – No passing/failing score, just results by competency

10

PERT Components

• PERT System Components

– Placement Tests (3 sub-tests)

• Identifies appropriate course placement (developmental

education through college credit)

– Reading – Writing – Mathematics – Diagnostic Tests

• Lower and upper level for each subject
• Based on separate competencies for exiting remedial

courses

11

PERT Test Design

• 30 questions per placement subtest

– 25 operational items will be the basis of the student’s placement score – 5 field test items to continually enhance the

• perational test bank
• Each test item is aligned with a PRC per the

test blueprint

• All items have been approved and aligned by

Florida faculty

PERT Test Formats

• PERT Placement is delivered using a CAT

engine

– Draws on item banks of 200+ items per content area – Item banks constantly being expanded

• PERT Diagnostics are linear

– Content delivered collaboratively between Florida College Faculty and McCann

16

PERT Interim Cut Score Setting

Data

• Pilot data
• Data reflecting current student enrollment – first course, first year
• f enrollment
• Data reflecting CPT/Accuplacer scores

Target

• Establish PERT cut scores/ranges that result in course placement

comparable to current enrollment numbers

– Two ranges for developmental education – lower and upper – One score for entry level college credit – Additional score for MAC 1105 College Algebra

Cut Scores and Expected Placement

Math Placement Level 3-Year Average Percent of Students Cut Score Range Expected Percent at Placement Advanced Math 0.5% NA

• College Algebra or Equiv

6.1% 123-150 7.1% Intermediate Algebra 13.9% 113-122 14.1% Upper Level Dev Ed 40.6% 96-112 41.9% Lower Level Dev Ed 38.9% 50-95 36.9% Reading Placement Level 3-Year Average Percent

• f Students

Cut Score Range Expected Percent at Placement College Ready (ENC 1101) 40.5% 104-150 42.0% Upper Level Dev Ed 46.3% 84-103 45.1% Lower Level Dev Ed 13.2% 50-83 12.9% Writing Placement Level 3-Year Average Percent

• f Students

Cut Score Range Expected Percent at Placement College Ready (ENC 1101) 54.9% 99-150 55.3% Upper Level Dev Ed 20.4% 90-98 21.5% Lower Level Dev Ed 24.7% 50-89 23.2%

PERT in the Colleges

Population Math Reading Writing College N 385,978 329,066 325,758 Mean 97.9 103.5 101.7

PERT in the High Schools

Population Math Reading Writing High School N 207,687 178,258 163,392 Mean 103.2 97.7 99.9

Sample Distribution

1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 10,000 50 53 56 59 62 65 68 71 74 77 80 83 86 89 92 95 98 101 104 107 110 113 116 119 122 125 128 131 134 137 140 143 146 149 Test Taker Counts PERT Writing Scores

PERT Writing Score Distribution

Contrasting Colleges and High Schools

Population Math Reading Writing High School Mean 103.2 97.7 99.9 College Mean 97.9 103.5 101.7

Future Directions with PERT

• Standard Setting
• Legislation

Contact Information

Scott Kramer

Vice President of Strategic Accounts McCann Associates skramer@mccanntesting.com

Matthew Schultz

Director Psychometric Services McCann Associates mschultz@mccanntesting.com