SLIDE 1
Conducting Social Policy Research in the Private Sector
November 1, 2019 Michael Frye, Ph.D.
A Day in the Life of a Member
- f the Abt Associates Team
Conducting Social Policy Research in the Private Sector A Day in - - PowerPoint PPT Presentation
Conducting Social Policy Research in the Private Sector A Day in - - PowerPoint PPT Presentation
Conducting Social Policy Research in the Private Sector A Day in the Life of a Member of the Abt Associates Team November 1, 2019 Michael Frye, Ph.D. Agenda Introductions Overview of Abt Associates Description of my day-to-day work at
SLIDE 2
SLIDE 3
How did I get here? § BS in statistics from Virginia Tech in 2006 § Discovered social policy research in the private sector after graduation § Worked for RMC Research Corporation from 2006-2012 § Started working at Abt Associates in 2012 § Completed MS (2008) and Ph.D. (2015) programs while working
SLIDE 4
Who We Are
4
SLIDE 5
Core Capabilities
5
§ Communications & Behavior Change § Data Capture & Surveys § Digital Transformations § Research, Monitoring, and Evaluation § Technical Assistance & Implementation
SLIDE 6
Key Markets & Global Expertise
6
§ Education § Environment & Energy § Food Security & Agriculture § Governance & Justice § Health
§ Behavioral Health § Chronic/Non-Communicable Diseases § Health Systems § Infectious Diseases § Private Sector Health § Reproductive, Maternal, Newborn, and Child Health
§ Housing, Communities, & Asset Building § Workforce & Economic Mobility
SLIDE 7
Our clients
7
§ U.S. federal, state, and local governments
§ Health and Human Services (HHS) § Department of Labor (DOL) § Department of Education (Ed) § Food and Nutrition Service, USDA § Environmental Protection Agency (EPA) § Centers for Disease Control (CDC) § United States Agency for International Development (USAID)
§ International governments
§ Department for International Development (DFID) § Department of Foreign Affairs and Trade (DFAT)
§ Universities and foundations
§ Bill and Melinda Gates Foundation § CDC Foundation § Medtronic
SLIDE 8
SEP projects
8
§ Abt’s Social and Economic Policy (SEP) division conducts social policy research in the following market areas:
§ Education § Housing, Communities, & Asset Building § Workforce & Economic Mobility
§ SEP research projects tend to be contracts with federal agencies like Institute of Education Sciences, the Department of Education’s research office. § We compete with other similar organizations for these projects. § Sometimes, we will partner with other organizations, universities, or individuals to form the strongest team for a complex research project. § Project scope and duration can vary.
SLIDE 9
SEP’s education practice
9
§ Abt currently has 50 different education projects with 19 different clients. § Topics include:
§ Child welfare and youth development § Teacher preparation § Literacy § College transitions and success § English language learners § Adult education § Early childhood § STEM education and training § Expanded learning time § Private school vouchers § Charter schools
§ Methods include: randomized controlled trials (RCTs), evidence reviews, evaluation technical assistance, quasi-experimental designs (QEDs), descriptive studies.
SLIDE 10
Education projects
10
§ What Works Clearinghouse Postsecondary Education and Postsecondary Preparation Evidence Reporting (WWC-PEPPER) contract. § DC Opportunity Scholarships Program: a QED study that examines a voucher program that allows students in DC to attend private schools. § College Access: RCTs that test enhancements to two programs: (1) the use of text messaging within Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) and (2) a combination of student materials, advisor development, and messaging (Find the Fit) within the Upward Bound program.
SLIDE 11
What do I do at Abt?
§ Project Director on the WWC-PEPPER contract, and also advisor on another WWC project to develop Practice Guides § Lead an evidence review for a project to examine how districts nationwide spend Title IV-A money § Manage the Federal Student Aid Experimental Sites Initiatives: a series of experiments that test variations in the disbursement of student aid and the delivery of financial aid services to students § Support an implementation study that examines variation in instructional quality and education experience of students receiving DC-OSP vouchers § Supervise multiple staff an run an initiative to mentor junior staff
SLIDE 12
What Is the WWC?
§ The What Works Clearinghouse (WWC) reviews the existing research on different programs, products, practices, and policies in education. § Its goal is to provide educators with the information they need to make evidence-based decisions using the results from high-quality research. § It disseminates information through products such as Intervention Reports and Practice Guides.
12
SLIDE 13
WWC Contractors
WWC PGs
IRG (with MPR) 2M (with Abt)
WWC Grants
Development Services Group
WWC PK-12
Mathematica Policy Research (with Abt)
Department of Education (ED) WWC Postsecondary
Abt Associates
Institute of Education Sciences (IES) National Center for Education Evaluation & Regional Assistance National Center for Education Statistics National Center for Education Research National Center for Special Education Research WWC Coordination
American Institutes for Research 13
SLIDE 14
What Is WWC-PEPPER?
§ Covers WWC reviews of research on programs, products, practices, and policies designed to support success at the postsecondary level. § Began in 2012. Abt’s WWC-PEPPER contract began in October 2016 and ends October 2021. § Continues the WWC’s reputation as a “central and trusted” resource for evidence by delivering actionable information to education practitioners, namely in the form
- f WWC Intervention Reports and Practice Guides.
14
SLIDE 15
How does the WWC work?
§ The WWC has developed a set of Evidence Standards that can be applied to group design studies to determine if the findings are credible. § Guided by these standards, the WWC conducts systematic reviews in the following steps:
1. Relevant studies are identified via a literature search. 2. Studies are screened to confirm eligibility. 3. Eligible studies are reviewed by WWC certified staff 4. Findings from studies that meet standards are synthesized and published in WWC products, which are posted on the website.
15
SLIDE 16
The WWC website
16
SLIDE 17
WWC products
§ Intervention Reports that summarize research findings on the effectiveness of a particular curriculum, program, practice, or policy. § Practice Guides that provide education practitioners with a set of recommendations to improve practice in their classrooms and schools around a broad education topic (e.g. using technology, teaching students to write)
§ These recommendations capture the intersection of subject matter expertise and supporting evidence from studies that meet the WWC standards.
§ Information on individual studies that were reviewed either for the above mentioned products or other purposes (e.g. grant competitions). § Data on all of the studies reviewed by the WWC. § Handbooks and protocols that define the standards and procedures that guide
- ur reviews.
§ Webinars that inform the public of the WWC practices and products that are available.
17
SLIDE 18
WWC Intervention Reports
18
SLIDE 19
WWC Practice Guides
19
SLIDE 20
WWC Review Process
§ All reviews are guided by a protocol that specifies:
§ Topic statement § Interventions of interest § Eligible populations § Outcomes included § Technical issues § Literature search strategy
§ Studies are double-coded by WWC certified reviewers and then reconciled. § If a study receives a rating of Does Not Meet WWC Standards from the first reviewer, it will be single coded and reconciled.
20
SLIDE 21
WWC Evidence Standards
§ WWC standards cover four research designs that introduce a strong counterfactual (i.e., establishing what would have happened in the absence of the intervention):
§ Randomized controlled trials (RCTs): Randomly assigning subjects (students, classrooms, schools, districts) to a treatment group or a control group. § Quasi-experimental designs (QEDs): Developing a comparison group in a non-random fashion (usually through matching). § Regression discontinuity design (RDDs): Using a “forcing variable” to assign treatment; impacts are assessed at the cut point. § Single-case designs (SCDs): Studying a single participant (or group of participants) using outcomes that are measured repeatedly across different conditions. The subject effectively serves as its own control.
21
SLIDE 22
Key Concepts
§ Attrition: RCTs must have (a) low overall attrition, (b) low differential attrition, and (c) low attrition at both the cluster and subcluster levels in cluster designs. § Baseline equivalence: QEDs must establish baseline equivalence on the analytic sample for each contrast; baseline measure must be correlated with outcome measure. § Confounding factors: An aspect of the intervention was perfectly aligned (“confounded”) with either the intervention or the comparison
- group. In these cases, it is not possible to tell whether the intervention
- r the confounding factor is responsible for the difference in outcomes.
– N=1 problem – A characteristic of the intervention is systematically associated with the outcome (e.g., teachers in treatment group all have more experience than controls) – Bundled interventions – Time confounds (treatment and control groups formed in different years)
§ Valid outcomes: Outcomes must meet standards for face validity, reliability, consistency, lack of overalignment.
22
SLIDE 23
Determinants of a WWC Rating
23
SLIDE 24
Why care about the WWC? § Even if you are not doing WWC reviews, the WWC standards will likely affect your work if you do education research. § IES grants are typically held to the WWC standards. § ESSA regulations are increasingly requiring states and districts to pick education solutions that are evidence- based.
SLIDE 25
What’s next? § Abt is looking for new colleagues § If you are interested in working at a firm that does social policy research, I encourage you to apply!
SLIDE 26
What are we looking for? § Desire to impact economic well-being of
- thers via social programs that we study
§ Strong research skills § Proclivity to working collaboratively while also being individually motivated § Interest in innovative approaches to solving problems
SLIDE 27
The Recruitment Process
27
*** If you apply, use requisition number 61640 ***
SLIDE 28
Questions?
28
Ask now! Or feel free to follow up…
§ Technical questions about our work can be sent to me: Michael_Frye@abtassoc.com § Recruiting questions can be sent to Samantha Hicks: Samantha_Hicks@abtassoc.com
SLIDE 29
Connect With Us
29