SLIDE 1 Exploring the Effect of an Advance Letter on Response and Eligibility Rates:
A M
Met eta-Ana nalysis S Stud udy for the he Nationa nal I Immuni unization S n Sur urvey (NIS)
Abera Wouhib1, Jie Zhao2, Meena Khare1 and Vicki Pineau2 AAPOR , Orlando, FL May 18, 2012
“The findings and conclusions in this paper are those of the authors and do not necessarily represent the views of the National Center for Health S tatistics, Centers for Disease Control and Prevention.” 1 National Center for Health S
tatistics, Centers for Disease Control and Prevention
2NOR
C at the University of Chicago
SLIDE 2 Outline
Introduction
Methods
Results
Discussion
SLIDE 3 The National Immunization Survey (NIS)
- sponsored by the National Center for Immunization and Respiratory
Diseases (NCIRD)
- conducted jointly by NCIRD and the National Center for Health Statistics
(NCHS), Centers for Disease Control and Prevention(CDC)
- one of the largest national telephone surveys began data collection since
1994
- monitor childhood vaccination for children 19 and 35 months living in the
United States at the time of the interview
The NIS is conducted as stratified, two phase surveys
- The first phase a list-assisted, random-digit-dialing survey to identify
households (HH) with age-eligible children with stratification at state and local areas to ensure comparable precision among the state and local areas
- The second phase is a mailed survey to providers identified during the
telephone interview to collect provider-reported vaccination histories
Introduction
SLIDE 4
Response rates in telephone surveys, including the NIS, continue to decline over time The NIS uses the Council of American Survey Research Organizations(CASRO ) response rate defined as a product of the resolution rate, screener rate and interview completion rate The NIS eligibility rate, which is defined as percentile of HHs with age eligible children among successfully screened-in HHs, has been also declining recently The differential in decline of CASRO response rates and eligibility rates has been wider among households(HHs) by advance letter mailing status in the past few years
Introduction….cont’d
SLIDE 5 Key Monitoring Statistics and Response Rates
by Advance Letter Status, NIS 2005 -2010
Row Key Indicator 2005 2006 2007 2008 2009 2010 Random-digit-dialed phase AL WAL AL WAL AL WAL AL WAL AL WAL AL WAL 1 Total selected telephone # released 1460066 3005195 1645109 3392721 1469436 3069931 1760771 3950032 1579190 4731439 1711459 5366202 2 Total phone numbers released to CATI 1460066 1134024 1645109 1205815 1469436 1082296 1760771 1456188 1579190 1796841 1711459 2106383 3 Advance letters mailed 1460066 1645109 1469436 1760771 1579190 1711459 (row 3 AL /(row 2 AL + row 2 WAL )) 56.28% 57.70% 57.59% 54.73% 46.48% 44.83% 4 Resolved phone numbers 1095966 2625258 1215413 2981829 1063607 2699406 1238565 3459522 1080840 4147360 1164813 4732912 Resolution rate 75.06% 87.36% 73.88% 87.89% 72.38% 87.93% 70.34% 87.58% 68.44% 87.66% 68.06% 88.20% 5 HH identified 821225 263815 889909 247797 766851 207735 846609 261882 798106 316564 856624 328595 (row 5/row 4) 74.93% 10.05% 73.22% 8.31% 72.10% 7.70% 68.35% 7.57% 73.84% 7.63% 73.54% 6.94% 6 HHs screened for age eligible children 766354 240081 812099 216974 697688 181519 771047 229793 743833 286543 790893 293528 Screening completion rate (row 6/row 5) 93.32% 91.00% 91.26% 87.56% 90.98% 87.38% 91.07% 87.75% 93.20% 90.52% 92.33% 89.33% 7 HHs with no age-eligible children 743098 231412 786730 208383 676987 174413 750074 221088 724726 276737 773168 283848 (row 7/row 6) 96.97% 96.39% 96.88% 96.04% 97.03% 96.09% 97.28% 96.21% 97.43% 96.58% 97.76% 96.70% 8 HHs with age-eligible children 23256 8669 25369 8591 20701 7106 20973 8705 19107 9806 17725 9680 Eligibility rate (row 8/row 6) 3.03% 3.61% 3.12% 3.96% 2.97% 3.91% 2.72% 3.79% 2.57% 3.42% 2.24% 3.30% 9 HHs with with completed RDD interviews 19817 7050 22022 7043 18250 5883 18160 7097 16287 7781 15132 7783 interview completion rate (row 9/row 8) 85.21% 81.32% 86.81% 81.98% 88.16% 82.79% 86.59% 81.53% 85.24% 79.35% 85.37% 80.40% 10 CASRO RR (row 4 * row 6 * row 9) 59.69% 64.65% 58.53% 63.09% 58.06% 63.61% 55.47% 62.65% 54.37% 62.96% 53.64% 63.35%
AL L – househ ehold mai ailed ed an an ad advan ance e let etter er WAL– ho hous useho hold witho hout ut a an a n advance let etter er
SLIDE 6
Trends in Response Rates by Advance Letter Status, NIS 2005 -2010
SLIDE 7
Trends in Eligibility Rates, NIS 2005 - 2010
SLIDE 8 Our objective is to explore the association of an advance letter status and reporting no age eligible children in HHs because of its effect on both CASRO response and eligibility rates CASRO RR= (resolution rate) x (screening rate) x (completion rate)
- Screening Rate = HHs Screened−in
HHs Identified x100
- Interview Completion Rate = HH with completed RDD interviews
HHs with age eligible children x100
Eligibility Rate= HHs with age eligible children
HHs Screened−in x100
HHs identified (n) = HHs screened out + HHs with no age eligible + HHs with age eligible X Y Z 1 = 𝑌
𝑜 + 𝑍 𝑜 + 𝑎 𝑜 = 𝑄 1+𝑄 2+𝑄 3 , where 𝑄 1, 𝑄 2 and 𝑄 3 are proportions
Methods
SLIDE 9 By splitting Y as 𝑍
𝐵𝐵 for HHs with advance letter and 𝑍 𝑋𝐵𝐵 for HHs
without advance letter, we computed the respective proportions 𝑄
𝐵𝐵
and 𝑄
𝑋𝐵𝐵 among identified HHs with and without advance letter
To measure the association of an advance letter status and the reporting
- f no age eligible children, we use a relative risk, say RR = 𝑄𝐵𝐵
𝑄
𝑋𝐵𝐵 ,
- RR > 1 implies direct association
- RR < 1 implies inverse association
- RR ≈ 1 indicates no association
Clearly, RR is a random Variable and for convenience can be transformed in logarithmic form as: 𝜈 𝑗 = log RR = log 𝑄
𝐵𝐵 – log 𝑄 𝑋𝐵𝐵
for computational simplicity The log RR may be estimated from the data as 𝜄 and its variance can be estimated using method of large sample approximation
Methods ….cont’d
SLIDE 10 Let 𝜈 𝑗 be the log RR for each subgroup of interest by applying a methods of meta-analysis, where i=1,2,….k are the subgroups of a study group 𝜈 𝑗 is an effect size assumed to be independently distributed for i=1,2,….k, with an associated variance, 𝜏𝑗
2 estimated by a method of
large sample approximation In meta-analysis, a one-way random-effects model can be stated as:
𝑗 = 𝜄 + 𝜁𝑗 + 𝜉𝑗, 𝜁𝑗 −within study error and 𝜉𝑗 −between study error
2), 𝜉𝑗~𝑂(0, 𝜐2) and 𝜁𝑗 ⟘ 𝜉𝑗 for all i
The assumption of random-effects modeling implies the existence of the between study errors and the parameter 𝜐2
Methods ….cont’d
SLIDE 11 We used the recent three years 2008 - 2010 NIS and RRD sample frame (exchange level) variables received from the Marketing Systems Group (MSG) to create two study groups
- 4 Regions by year
- 5 HH income subgroups by year
The two groups each made up of independent subgroups within study group are:
- 12 subgroups of region by year , k = 12
- 15 subgroups of income by year , k = 15
In the region by year study group, the proportions for subgroup region south and 2008 are estimated as
𝐵𝐵= HHs reported no age eligible children and with AL in South−2008
HHs identified with AL in South−2008
- 𝑄𝑋𝐵𝐵= HHs reported no age eligible children and without AL in South−2008
HHs identified without AL in South−2008
Methods ….cont’d
SLIDE 12 Results
Region-Y ear Subgroup
Region Y ear
µ 𝑗 𝑇𝑇(µ 𝑗) µ 𝑗 [95 % CI of µ i] Wt. in %
SLIDE 13 Results
Income by Y ear Subgroup
Income Y ear
µ 𝑗 𝑇𝑇(µ 𝑗) µ 𝑗 [95 % CI of µ i] Wt. in %
SLIDE 14
In all subgroups, 𝜄 𝑗 is positive and the 95% CI is also positive. This implies a direct association among the two groups, the HHs with mailed advance letter and reporting no age eligible children In all cases the overall estimate, 𝜄 = 0.04 (RR=1.041) and the 95% CI indicates that there is direct (positive) relationship among HHs with mailed advance letter and reporting no age eligible children Among region-year study group, South and West are more likely to report no eligible children higher than 4.1% (the overall mean) and Northeast is more likely to report less than the overall mean Among income-year group HHs with less than $25K annual income are less likely to report no age eligible children than HHs in other income groups
Results…..cont’d
SLIDE 15
Discussion
The association of reporting no age eligible children and advance letter status is not directly related with CASRO response rates, but likely to affect the outcome in general CASRO RR= (resolution rate) x (screening rate) x (completion rate) The differential in reporting no age eligible children by status of advance letter is significant across subgroups by region and income Some households may choose not to participate in the survey once they received advance letter; and perhaps not helping in improving the eligibility rates Further study regarding vaccinations by an advance letter status may be essential to decide on the importance of advance letter
SLIDE 16
Discussion…cont’d
Limitation
We know that advance letter is confounding with listed HH and as result some of our findings could be as a result of characteristics associated with listed HH Some other confounded factors may be existed and affected our conclusion regarding some of the HH characteristics used in this study The landline frame itself has changed substantially over time since cell phone only incidence has increased so much for the household with young children in the past five years Mailing an advance letter doesn’t warrant receiving by the HH Exchange level data was used in the analysis
SLIDE 17
Thank you
Contact: awouhib@ cdc.gov
