Evaluating Complex Interventions Gareth Parry Senior Scientist, - - PowerPoint PPT Presentation
Evaluating Complex Interventions Gareth Parry Senior Scientist, IHI Astou Coly June 27, 2017 Senior Improvement Advisor, URC 2 http://www.academyhealth.org/evaluationguide This report was made possible by generous support from the Robert
Gareth Parry Senior Scientist, IHI Astou Coly Senior Improvement Advisor, URC June 27, 2017
2
This report was made possible by generous support from the Robert Wood Johnson Foundation.
– Evaluators and implementers have agreed on the overall theory
– Key stakeholders have agreed on the evaluation questions
– Leviton LC et al Evaluability assessment to improve public health
policies, programs, and practices. Annual Review of Public
3
5
Yes
6
7
12-month outcomes of community engagement versus technical assistance to implement depression collaborative care: a partnered, cluster, randomized, comparative effectiveness trial
A cluster RCT conducted in Los Angeles, to compare the effectiveness
depression collaborative care trainings Outcome: Mental health-related quality
12-months. CEP was associated with a decrease in poor MHRQL compared to RS at 6 months (OR=0.71; 95% CI: 0.55-0.91) and 12 months (OR=0.77; 95% CI: 0.61-0.97).
B Chung et al 12-month outcomes of community engagement versus technical assistance to implement depression collaborative care: a partnered, cluster, randomized, comparative effectiveness trial. Annals of internal medicine. 2014; 161(10 Suppl): S23-34
The authors concluded that while CEP did not indicate an effect at 12 months, policymakers and communities should still consider this strategy given the lack of alternative approaches that have demonstrated higher effectiveness.
8
“Gold Standard” Randomization allows for difference in outcomes between the intervention and control groups to be attributed to the intervention Randomization may not be practical or ethical Randomization at Cluster level Limited external validity Control group may be exposed to the intervention Retention of participants may differ between intervention and control groups Fixed-protocols limit adaptation of the interventions
9
Yes Yes
10
Time Period 1 2 3 4 5 6 7 8 9 10 Site 1 Site 2 Site 3 Site 4 Site 5 Site 6 Site 7 Site 8 Site 9 Site not allocated to the intervention Site is allocated to the intervention
11
A structural multidisciplinary approach to depression management in nursing-home residents: a multicentre, stepped-wedge cluster- randomised trial
A stepped wedge design assessed the effectiveness of the Act in Case of Depression (AiD) approach to the management
Netherlands All five groups received the intervention
Leontjevas, Ruslan et al. A structural multidisciplinary approach to depression management in nursing-home residents: a multicentre, stepped-wedge cluster-randomised trial. The Lancet, Volume 381 , Issue 9885 , 2255 - 2264
The authors concluded that while the AiD approach can reduce depression, screening needs to be addressed in dementia units as these units had lower adherence to screening
12
Provides an alternative to the orthodox RCT approach All participants receive the intervention. Secular trends Evaluate barriers to implementation of the intervention and to iteratively improve implementation in subsequent steps. Preventing contamination between those receiving the intervention and those to receive the intervention may be particularly challenging. Requires frequent data collection.
13
Yes
14
15
Association Between Hospital Penalty Status Under the Hospital Readmission Reduction Program and Readmission Rates for Target and Nontarget Conditions
As part of the Hospital Readmission Reduction Program (HRRP), financial penalties were imposed from Oct 2012, on hospitals with higher than expected readmissions. To assess the impact of the HRRP the investigators used an interrupted time series analysis design to compare trends in readmission rates between hospitals subject to and those not subject to the penalty. Before announcement of the HRRP readmission rates were stable After the HRRP was announced, readmission rates declined more rapidly in hospitals later subject to penalties relative to those not penalized,. After implementation of the HRRP, the rate of change for readmission rates plateaued
Desai NR et al. Association Between Hospital Penalty Status Under the Hospital Readmission Reduction Program and Readmission Rates for Target and Nontarget Conditions. JAMA. 2016 Dec 27;316(24):2647-56.
The investigators concluded that the HRRP was associated with greater reductions in readmission rates in penalized hospitals relative to nonpenalized hospitals.
16
Takes into account underlying trends in the data Can provide a strong quasi-experimental alternative to randomization Can assess effect size, speed and sustainability of the intervention
There can be challenges in identifying comparator groups that provide comparable data It may be difficult to collect sufficient data points before and after the intervention to be able to detect a change in slope Adjusting for patient-level characteristics can be challenging
17
Yes
18
19
The Impact of Green House Adoption on Medicare Spending and Utilization
To understand the impact of the Green House (GH) nursing home model on Medicare spending and utilization. Medicare claims and enrollment data and a resident-level assessment data set to estimate the impact of GH on Medicare acute hospital, other hospital, skilled nursing facility, and hospice spending and utilization. 15 GH intervention nursing homes and 223 matched nursing homes Propensity scoring to weight the data in a way that approximated intervention and comparison groups, with similar organizational characteristics.
Grabowski DC et al The impact of Green House adoption on Medicare spending and utilization. Health services
The investigators conclude that the nursing homes who adopted the GH model did not realize Medicare savings and suggested new approaches to align financial incentives may be required.
20
Not subject to ethical and practical constraints of randomization Can be used in situations where it may not be possible to randomly assign the intervention The design is susceptible to bias and confounding which can be difficult or impossible to mitigate completely Many factors may influence exposure to “intervention” or “control” group. The reliance on existing data means that investigators may not have all needed data for the analyses
21
Yes
22
23
Effects of the Minimum Legal Drinking Age on Alcohol-Related Health Service Use in Hospital Settings in Ontario: A Regression– Discontinuity Approach
To assess the impact of the minimum legal drinking age (MDLA=19) legislation on alcohol related harm. Inpatient and emergency department events from 2002 to 2007, for patients aged 16 to 22
suicides related to alcohol The investigators created age-groups based on the year and month of birth, to create a variable indicating months from the MLDA cut off of 19 years
Callaghan RC et al Effects of the minimum legal drinking age on alcohol-related health service use in hospital settings in Ontario: a regression–discontinuity approach. American journal of public health. 2013 Dec;103(12):2284-91.
The investigators concluded that young adults who gained legal access to alcohol used more hospital care for a variety of alcohol related issues.
Examining the coefficients of the regression models, the investigators found that compared to youth slightly below the MDLA, those just above had 51.8% (p=0.01) more alcohol-associated suicides
24
In spite of a lack of randomization, can yield relatively robust estimates of effects. The design avoids ethical issues related to allocation of participants to intervention or comparison groups. The statistical analysis is dependent on many assumptions, that may not always be clear to stakeholders. Less statistical power than RCTs due to correlation between assignment and treatment variables The design can be susceptible to contamination by other effects related to the same cut-off value.
–
For example, in some settings the minimum alcohol drinking age could be similar to minimum driving age.
25
Yes
26
27
Short Term Impact of Smoke-Free Legislation in England: Retrospective Analysis of Hospital Admissions for Myocardial Infarction
To determine the impact of the introduction of smoke-free legislation enacted in England in 2007 on myocardial infarction (MI) hospital admissions. Hospital admission data were obtained from the NHS hospital episode statistics. Data were obtained for all adults 18+ years admitted July 2002 and Sep 2008 with a diagnosis of MI
Sims M et al Short term impact of smoke-free legislation in England: retrospective analysis of hospital admissions for myocardial infarction BMJ 2010;340 :c2161
The authors concluded that these findings contribute to the evidence base for the effectiveness of smoke free legislation. A small reduction in the number of emergency admissions for myocardial infarction following the implementation of the legislation (-2.4%; 95%CI: -4.06%- -0.66%).
28
Can be used in situations where it may unethical or impractical to manipulate exposure The design is susceptible to bias and confounding which can be difficult or impossible to mitigate completely Exposed group may be difficult to define or may change over time. The reliance of existing data means that the investigators may not have all needed data for the analysis
– e.g. by presenting confidence intervals
29