the Impact of Health Care Innovation June 25, 2017 #ARM17 - - PowerPoint PPT Presentation

www.rti.org

RTI International is a registered trademark and a trade name of Research Triangle Institute.

Using Different Comparison Group Selection Methodologies to Evaluate the Impact of Health Care Innovation

June 25, 2017

Yiyan (Echo) Liu, PhD, and Emily Gillen, PhD

#ARM17 @echo_yliu

Acknowledgment and Disclaimer

This work was supported through a contract from the Centers for Medicare and Medicaid Innovation within the Centers for Medicare & Medicaid Services (HHSM-500-2010-00021I). The contents of this presentation are solely the responsibility of the authors and do not necessarily represent the official views of the U.S. Department of Health and Human Services or any of its agencies.

2

Road Map

• Introduction

– Comparison group selection – Health Care Innovation Awards

• Objective
• Comparison Group Selection

– Matching on the propensity score – Inverse probability of treatment weighting (IPTW) using the propensity

score

– Entropy balancing weighting

• Model Covariates Balance Check
• Difference-in-differences (DinD) Regression Results

3

Comparison Groups in Observational Studies

• Observational studies are increasingly common in the social

sciences.

– Randomization is not always possible.

• Unethical/not feasible
• Evaluations designed after an intervention has been under way (no comparison

group designated)

• Evaluations require a comparison group.

– If the treatment group is selected based on eligibility criteria and the

comparison group is created based on similar, but not eligible, individuals, we could be introducing bias into the evaluation (if eligibility criteria are associated with the outcome).

• Could create even more bias if the treatment group is selected based on

eligibility criteria and the comparison group includes eligible individuals who

• pted out of enrollment (if enrollment is associated with the outcome).

4

Comparison Group Selection

• The conditional probability of receiving a treatment given a set of
• bservable characteristics (Rosenbaum and Rubin,1983)

– A method to reduce bias in comparison group creation

• Using propensity scores (Austin, 2009)

– Stratification

• Divide the sample into strata (e.g., quintiles or deciles), and use propensity

scores to match treatment and comparison observations within strata.

– Covariate adjustment

• Propensity score is used as a covariate in the outcome regression model.

5

Comparison Group Selection (continued)

• Using propensity scores (continued):

– Matching

• Match treatment and comparison group observations on their propensity scores.
• Some examples/decisions to make
• Selection of match: nearest neighbor or greedy
• How many matches: one-to-one or one-to-many
• Replacement: with or without

– Weighting (inverse probability of treatment weighting)

• Individuals in the treatment group receive a weight of 1; the comparison group

receives a weight of the following: 𝑄𝑠𝑝𝑞𝑓𝑜𝑡𝑗𝑢𝑧 𝑇𝑑𝑝𝑠𝑓 1 − 𝑄𝑠𝑝𝑞𝑓𝑜𝑡𝑗𝑢𝑧 𝑇𝑑𝑝𝑠𝑓

• Entropy balancing weighting (Hainmueller, 2012)
• Adjusts inequalities in representation with respect to the first, second, and

possibly higher moments of the covariate distributions.

6

Concerns with Comparison Group

• Achieving balance

– Calculate the absolute standardized differences between the treatment

and comparison groups (before propensity score methodology applied and after).

– Check whether the absolute standardized difference decreases and

whether it achieves acceptable balance.

• One frame of reference: many researchers consider that an absolute

standardized difference ≤ 0.10 indicates acceptable balance (Austin, 2011a).

• Sample size

– Matching (especially 1-to-1) can lead to a loss of observations. If you

don’t have an adequate match in the prospective comparison group, you may need to drop treatment group observations.

• Can be a problem with small samples

7

Health Care Innovation Awards (HCIA)

• HCIA: Up to $1 billion in awards to organizations with creative ideas

to make improvements in delivery systems, health outcomes, or the cost of care.

• Data from one HCIA awardee

– Time span: April 2011–December 2015 – Fee-for-service (FFS) Medicare claims – Treatment group: 6,476 Medicare FFS beneficiaries – Spending and utilization outcomes

• Medicare health care spending (Parts A and B)
• All-cause hospital inpatient admissions
• Emergency department (ED) visits not leading to a hospitalization

8

Objective

• Because no gold standard in applying propensity score methods,

– How do researchers choose which method to use? – What effect does that choice have?

• We use the HCIA awardee data to compare the following:
• 1. Propensity score matching
• 2. Propensity score weighting
• 3. Entropy balancing weighting
• We will apply these methods to construct different comparison

groups, and then look at the innovation effect on three outcomes:

– Medicare spending – Inpatient admissions – ED visits

9

Propensity Score Model Covariates

• Apply the same inclusion and exclusion criteria used for the

treatment group to the initial pool of individuals eligible for the comparison group, to the extent possible

– Geographic region – Age restrictions

• Define model covariates: demographics, health conditions, eligibility

criteria, and payments and utilization in the period prior to enrolling in the intervention

• Run the logistic propensity score model

– With dependent variable: whether the individual is in the treatment group – Include the aforementioned model covariates that might affect the

probability of being treated

10

Propensity Score Model Results

Analysis of Maximum Likelihood Estimates Parameter Estimate Standard Error P Value Intercept

• 0.718

0.117 <0.001 Age

• 0.012

0.002 <0.001 Male

• 0.056

0.026 0.034 White

• 0.833

0.045 <0.001 Disabled 0.033 0.042 0.426 ESRD

• 0.629

0.177 0.000 Number of dual eligible months in the previous calendar year 0.003 0.003 0.439 Number of chronic conditions

• 0.044

0.005 <0.001 Total payments in calendar quarter prior to enrollment 0.000 0.000 0.002 Total payments in second, third, fourth, and fifth calendar quarters prior to enrollment 0.000 0.000 0.001 Number of ED visits in calendar quarter prior to enrollment 0.044 0.025 0.076 Number of ED visits in second, third, fourth, and fifth calendar quarters prior to enrollment 0.053 0.008 <0.001 Number of inpatient stays in calendar quarter prior to enrollment

• 0.021

0.054 0.698 Number of inpatient stays in second, third, fourth, and fifth calendar quarters prior to enrollment

• 0.073

0.026 0.005

11

Propensity Score Matching

• Estimate propensity scores (PSCORE) using logistic regression.
• If caliper matching, set the caliper width or maximum propensity

score distance (e.g., 20% of the standard deviation of the logit of the propensity scores, Austin, 2011b).

– 𝑀𝑃𝐻𝐽𝑈_𝑄𝑇𝐷𝑃𝑆𝐹 = log

𝑄𝑇𝐷𝑃𝑆𝐹 1−𝑄𝑇𝐷𝑃𝑆𝐹

• Within the caliper, we did 1:variable matching (with replacement)

with up to three comparison beneficiaries per treatment individual.

12

Propensity Score Matching Results

Variable Before Matching Standardized Difference After Matching Standardized Difference Treatment Group Comparison Group Treatment Group Comparison Group Mean SD Mean SD Mean SD Mean SD Total payments in calendar quarter prior to enrollment $2,321 $7,363 $1,988 $6,312 0.05 $2,320 $7,363 $2,284 $7,440 0.00 Total payments in second, third, fourth, and fifth calendar quarters prior to enrollment $8,039 $18,720 $7,542 $15,538 0.03 $8,032 $18,713 $8,229 $18,649 0.01 Age 67.17 15.28 71.04 12.27 0.28 67.18 15.27 67.02 14.65 0.01 Percentage male 42.88 49.49 43.32 49.55 0.01 42.89 49.50 43.13 49.53 0.00 Percentage white 89.30 30.91 95.66 20.38 0.24 89.32 30.89 88.83 31.50 0.02 Percentage disabled 35.00 47.70 26.11 43.92 0.19 34.99 47.70 36.02 48.01 0.02 Percentage ESRD 0.59 7.64 0.68 8.21 0.01 0.59 7.64 0.64 7.96 0.01 Number of dual eligible months in the previous calendar year 2.56 4.75 1.90 4.27 0.15 2.55 4.75 2.71 4.88 0.03 Number of chronic conditions 6.12 3.68 6.85 3.68 0.20 6.12 3.68 6.22 3.69 0.03 Number of ED visits in calendar quarter prior to enrollment 0.19 0.72 0.13 0.48 0.10 0.18 0.65 0.17 0.62 0.02 Number of ED visits in second, third, fourth, and fifth calendar quarters prior to enrollment 0.93 2.33 0.65 1.62 0.14 0.92 2.08 0.89 2.21 0.01 Number of inpatient stays in calendar quarter prior to enrollment 0.09 0.38 0.08 0.34 0.04 0.09 0.38 0.09 0.38 0.00 Number of inpatient stays in second, third, fourth, and fifth calendar quarters prior to enrollment 0.29 0.84 0.28 0.76 0.02 0.29 0.84 0.30 0.83 0.00 Number of beneficiaries 6,478 — 85,198 — — 6,477 — 19,431 — — Number of unique beneficiaries 6,478 — 85,198 — — 6,477 — 16,710 — — Number of weighted beneficiaries — — — — — 6,477 — 6,477 — —

13

Propensity Score Weighting

• Estimate propensity scores (PSCORE) using logistic regression
• If doing IPTW, create propensity score weights for each beneficiary

based on their propensity scores (for average treatment effect on the treated [ATT])

– For treatment group: weight = 1 – For comparison group: weight = PSCORE / (1 - PSCORE)

• Replace weight with zero for comparison beneficiaries whose

propensity score is less than the minimum treatment propensity score

• Cap outlier weights (usually at 5)
• Normalize the weights so that both the treatment and comparison

group have a mean weight of 1

14

Propensity Score Weighting Results

Variable Before Weighting Standardized Difference After Weighting Standardized Difference Treatment Group Comparison Group Treatment Group Comparison Group Mean SD Mean SD Mean SD Mean SD Total payments in calendar quarter prior to enrollment $2,321 $7,363 $1,988 $6,312 0.05 $2,321 $7,363 $2,376 $7,891 0.01 Total payments in second, third, fourth, and fifth calendar quarters prior to enrollment $8,039 $18,720 $7,542 $15,538 0.03 $8,039 $18,720 $8,227 $19,313 0.01 Age 67.17 15.28 71.04 12.27 0.28 67.17 15.28 67.17 14.13 0.00 Percentage male 42.88 49.49 43.32 49.55 0.01 42.88 49.49 42.87 49.49 0.00 Percentage white 89.30 30.91 95.66 20.38 0.24 89.30 30.91 89.24 30.99 0.00 Percentage disabled 35.00 47.70 26.11 43.92 0.19 35.00 47.70 35.03 47.71 0.00 Percentage ESRD 0.59 7.64 0.68 8.21 0.01 0.59 7.64 0.59 7.64 0.00 Number of dual eligible months in the previous calendar year 2.56 4.75 1.90 4.27 0.15 2.56 4.75 2.56 4.80 0.00 Number of chronic conditions 6.12 3.68 6.85 3.68 0.20 6.12 3.68 6.14 3.60 0.00 Number of ED visits in calendar quarter prior to enrollment 0.19 0.72 0.13 0.48 0.10 0.19 0.72 0.20 0.81 0.02 Number of ED visits in second, third, fourth, and fifth calendar quarters prior to enrollment 0.93 2.33 0.65 1.62 0.14 0.93 2.33 1.03 3.71 0.03 Number of inpatient stays in calendar quarter prior to enrollment 0.09 0.38 0.08 0.34 0.04 0.09 0.38 0.09 0.40 0.01 Number of inpatient stays in second, third, fourth, and fifth calendar quarters prior to enrollment 0.29 0.84 0.28 0.76 0.02 0.29 0.84 0.30 0.91 0.01 Number of beneficiaries 6,478 — 85,198 — — 6,478 — 85,198 — — Number of unique beneficiaries 6,478 — 85,198 — — 6,478 — 85,198 — — Number of weighted beneficiaries — — — — — 6,478 — 85,198 — —

15

Entropy Balancing Weighting

• Estimate entropy balancing weights using Stata’s ebalance package

(Hainmueller, 2012).

– Syntax: ebalance treat `psvar', gen(ebal) tar(2)

• Replace weight with zero for comparison beneficiaries whose

propensity score is less than the minimum treatment propensity score.

• Cap outlier weights (usually at 5).
• Normalize the weights so that both the treatment and comparison

group have a mean weight of 1.

16

Entropy Balancing Weighting Results

Variable Before Weighting Standardized Difference After Weighting Standardized Difference Treatment Group Comparison Group Treatment Group Comparison Group Mean SD Mean SD Mean SD Mean SD Total payments in calendar quarter prior to enrollment $2,321 $7,363 $1,988 $6,312 0.05 $2,321 $7,363 $2,320 $7,363 0.00 Total payments in second, third, fourth, and fifth calendar quarters prior to enrollment $8,039 $18,719 $7,542 $15,538 0.03 $8,039 $18,719 $8,037 $18,718 0.00 Age 67.17 15.28 71.04 12.27 0.28 67.17 15.28 67.18 15.28 0.00 Percentage male 42.88 49.49 43.32 49.55 0.01 42.88 49.49 42.88 49.49 0.00 Percentage white 89.30 30.91 95.66 20.38 0.24 89.30 30.91 89.31 30.90 0.00 Percentage disabled 35.00 47.70 26.11 43.92 0.19 35.00 47.70 34.98 47.69 0.00 Percentage ESRD 0.59 7.64 0.68 8.21 0.01 0.59 7.64 0.58 7.61 0.00 Number of dual eligible months in the previous calendar year 2.56 4.75 1.90 4.27 0.15 2.56 4.75 2.55 4.75 0.00 Number of chronic conditions 6.12 3.68 6.85 3.68 0.20 6.12 3.68 6.12 3.68 0.00 Number of ED visits in calendar quarter prior to enrollment 0.19 0.72 0.13 0.48 0.10 0.19 0.72 0.19 0.72 0.00 Number of ED visits in second, third, fourth, and fifth calendar quarters prior to enrollment 0.93 2.33 0.65 1.62 0.14 0.93 2.33 0.93 2.33 0.00 Number of inpatient stays in calendar quarter prior to enrollment 0.09 0.38 0.08 0.34 0.04 0.09 0.38 0.09 0.38 0.00 Number of inpatient stays in second, third, fourth, and fifth calendar quarters prior to enrollment 0.29 0.84 0.28 0.76 0.02 0.29 0.84 0.29 0.84 0.00 Number of beneficiaries 6,478 — 85,198 — — 6,478 — 85,198 — — Number of unique beneficiaries 6,478 — 85,198 — — 6,478 — 85,198 — — Number of weighted beneficiaries — — — — — 6,478 — 85,198 — —

17

DinD Regression Specification

18

• Generalized linear model with quarterly fixed effects

𝑧𝑗,𝑢 = 𝛽0 + 𝜈𝐽𝑗 + ෍

𝑢 𝑈

𝛾𝑢𝑅𝑢 + ෍

𝑢 𝑈

𝜾𝒖(𝑅𝑢⋅ 𝐽𝑗,𝑢 ⋅ 𝐸𝑢) + ෍

𝑙

𝜇𝑙𝑌𝑗,𝑢,𝑙 + 𝜁𝑗,𝑢

𝑧𝑗,𝑢 = outcome measure for the ith beneficiary in period t 𝐽𝑗 = 0,1 indicator of the observation in the treatment (innovation) group 𝑅𝑢 = 0,1 indicator of the observation in the tth quarter 𝐸𝑢 = 0,1 indicator of the demonstration (innovation) period 𝑌𝑗,𝑢,𝑙 = a vector of k patient, practice, and/or other characteristics* 𝜁𝑗,𝑢 = regression error term

* All regressions controlled for age, sex, race, disability, ESRD, dual eligibility, number of months of dual eligibility status during the calendar year prior to the innovation, and the number of chronic conditions.

Ordinary Least Squares DinD: Spending under Propensity Score Matching and Weighting

19

• $200
• $100

$0 $100 $200 $300 $400 $500 $600 $700

I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 I 10 I 11

Weight_Estimate Weight_LB Weight_UB Match_Estimate Match_LB Match_UB

Negative Binomial DinD: Inpatient Admissions under Propensity Score Matching and Weighting

20

• 10
• 5

5 10 15 20 25 30

I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 I 10 I 11

Weight_Estimate Weight_LB Weight_UB Match_Estimate Match_LB Match_UB

Negative Binomial DinD: ED Visits under Propensity Score Matching and Weighting

21

• 50
• 40
• 30
• 20
• 10

10 20 30 40

I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 I 9 I 10 I 11

Weight_Estimate Weight_LB Weight_UB Match_Estimate Match_LB Match_UB

Results

• Covariate balance

– Both matching and weighting reduced absolute standardized covariate

differences between the treatment and comparison groups and achieved adequate balance for all model covariates.

• Evaluation results

– All three comparison group methods produced very similar – both in sign and in

magnitude – quarterly DinD regression estimates and standard errors for Medicare spending and inpatient admissions.

– For ED visits, matching and weighting produced different estimates. The estimate

from matching was negative and significant, whereas the estimates from the two weighting methods were not significant.

– Propensity score weighting and entropy balancing weighting produced very

similar results in all three outcomes.

• More matching variations

– Changing the default caliper width in matching from 20% of the standard deviation

• f the logit of the propensity scores to 10% and 30% produced similar results.

22

Conclusion

• In this study of program effects for an HCIA awardee, different

comparison group methodologies produced similar estimates for Medicare spending and inpatient admissions. For ED visits, some differences remain.

• Implications for policy and practice

– Researchers should demonstrate that their evaluation results are robust

to the choice of comparison group methodology.

• Future research

– Add more comparison group selection methods such as other matching

methods.

– Compare the frequentist approach against Bayesian methods.

23

Citations

• Austin, P.C. (2009). The relative ability of different propensity score methods

to balance measured covariates between treated and untreated subjects in

• bservational studies. Med Decis Making; 29:661–677.
• Austin, P.C. (2011a). An introduction to propensity score methods for

reducing the effects of confounding in observational studies. Multivariate Behav Res; 46(3):399-424.

• Austin, P.C. (2011b). Optimal caliper widths for propensity-score matching

when estimating differences in means and differences in proportions in

• bservational studies. Pharm Stat; 10(2):150–161.

http://doi.org/10.1002/pst.433

• Hainmueller, J. (2012). Entropy balancing for causal effects: a multivariate

reweighting method to produce balanced samples in observational studies. Polit Anal; 20:25−46.

• Rosenbaum, P.R. and Rubin, D.B. (1983). The central role of the propensity

score in observational studies for causal effects. Biometrika; 70(1):41-55.

24

Acknowledgments and Contact

• Acknowledgments

– Funded by the Centers for Medicare & Medicaid Services

• Contact information

Yiyan Liu RTI International 307 Waverley Oaks Road, Suite 101 Waltham, MA 02452 www.rti.org yliu@rti.org

25