Follow-up Strategies for Small Businesses through Embedded - - PowerPoint PPT Presentation

▶
follow up strategies for small
SMART_READER_LITE
LIVE PREVIEW

Follow-up Strategies for Small Businesses through Embedded - - PowerPoint PPT Presentation

Sep 08, 2022 990 likes •1.68k views

Investigating Adaptive Nonresponse Follow-up Strategies for Small Businesses through Embedded Experiments (Plus) Katherine Jenny Thompson Stephen Kaputa Economic Statistical Methods Division Any views expressed are those of the authors and

slide-1
SLIDE 1

Investigating Adaptive Nonresponse Follow-up Strategies for Small Businesses through Embedded Experiments (Plus)

Katherine Jenny Thompson Stephen Kaputa Economic Statistical Methods Division

Any views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.

slide-2
SLIDE 2

Acknowledgments

  • Laura Bechtel
  • Daniel Whitehead
  • Alfred “Dave” Tuttle
  • Jennifer Beck
  • Michael Padgett
  • Eddie Salyers
  • Robert Struble
slide-3
SLIDE 3

Timeline for Our Research

Nonrespondent Subsampling Methods

2014 (and 2016)

slide-4
SLIDE 4

Timeline for Our Research

Nonrespondent Subsampling Methods Contact Strategy for Nonresponse Follow-up (NRFU)

2014 (and 2016) 2015

slide-5
SLIDE 5

Timeline for Our Research

Nonrespondent Subsampling Methods Contact Strategy for Nonresponse Follow-up (NRFU) Adaptive Collection Design

2014 (and 2016) 2015 2016

slide-6
SLIDE 6

Timeline for Our Research

Nonrespondent Subsampling Methods Contact Strategy for Nonresponse Follow-up (NRFU) Adaptive Collection Design

2014 (and 2016) 2015 2016

slide-7
SLIDE 7

Context

  • Research on survey design and data collection

features for 2017 Economic Census

– Survey design – simulation – Data collection – field tests in annual business surveys

  • Adaptive collection design protocols considered

for NRFU

– Nonrespondent subsampling – Targeted allocation

slide-8
SLIDE 8

Timeline for Our Research

Nonrespondent Subsampling Methods Simulation Studies Contact Strategy for Nonresponse Follow-up (NRFU) Field Test (Embedded Experiment) Adaptive Collection Design Field Test (Embedded Experiment)

2014 (and 2016) 2015 2016

slide-9
SLIDE 9

Economic Census

  • Conducted every five years
  • Covers eighteen non-farm sectors
  • Surveys over 4 million establishments
  • Produces industry and geographic estimates

(benchmark measures of the economy)

  • Provides data for sampling frames
slide-10
SLIDE 10

Business Organization Structures (Simplified)

Single Unit (SU)

  • Operate in one primary industry
  • 1 Economic Census questionnaire

Single-Unit Company (SU)

Establishment

slide-11
SLIDE 11

Business Organization Structures (Simplified)

Single Unit (SU)

  • Operate in one primary industry
  • 1 Economic Census questionnaire

Multi Unit (MU)

  • Can operate in more than one industry
  • More than 1 Economic Census

questionnaire

Single-Unit Company (SU)

Establishment

Multi-Unit Company

Establishment 1 Establishment 2 Establishment n 

slide-12
SLIDE 12

Economic Programs

  • Collection Procedures and NRFU

– Designed to obtain respondent data from the largest cases – Small units rarely receive personal contact

  • Multi-unit establishments (Economic Census)

– Company and establishment level NRFU strategies – Post-data collection completeness and coverage procedures

  • Mandated lower bound on the unit response rate
slide-13
SLIDE 13

Consequences

  • Limited research on data collection

strategies for smaller businesses

  • Distinct possibility (prior to our research)

that realized set of respondents were not a random subsample

– Potential for nonresponse bias in survey totals

slide-14
SLIDE 14

Annual Survey of Manufactures (ASM)

  • Alternative to Economic Census in off-

census years (manufacturing sector)

– Similar electronic questionnaire (same items) – Similar editing/imputation procedures

  • Different sampling design

– Census: Stratified SRS-WOR – ASM: Stratified PPS-WOR

slide-15
SLIDE 15

Timeline

Nonrespondent Subsampling Methods Simulation Studies 2014 (and 2016) 2015 2016

slide-16
SLIDE 16

Motivation

  • Find allocation method for nonrespondent

subsampling of Single-unit businesses

  • Maintain mandated unit response rates

– Targeted industry-specific response rates

  • Reduce

– Cost of NRFU – Nonresponse bias

slide-17
SLIDE 17

Nonresponse Subsampling

  • Stratified systematic sample design

– Sorted by measure of size

  • Allocation

– Equal probability (1-in-K) sampling – Optimal Allocation that…

  • minimizes deviation between industry unit response

rates (Min-URR)

  • minimizes deviation between industry sampling

intervals (Min-K)

slide-18
SLIDE 18

Optimal Allocation

  • Formulated as quadratic programs with

linear constraints on overall and industry level response rates

  • Subsampling only applied in designated

subdomains (“Noncertainty Single-units”)

slide-19
SLIDE 19

Relative Bias of the Estimate

Combined ratio estimator with an estimated conversion rate 1-in-2 Nonrespondent Subsample

slide-20
SLIDE 20

Relative Bias of the Estimate

Combined ratio estimator with an estimated conversion rate 1-in-2 Nonrespondent subsample

  • Systematic subsample with 1-in-K

allocation not effective

slide-21
SLIDE 21

Relative Bias of the Estimate Subsampled Subdomains

Combined ratio estimator with an estimated conversion rate 1-in-2 Nonrespondent subsample

  • Slight increase in bias
  • Additional contacts (rounds) not helpful
slide-22
SLIDE 22

Mean Squared Error

Combined ratio estimator with an estimated conversion rate 1-in-2 Nonrespondent subsample

  • Large increase in variance
  • Additional contacts not helpful
slide-23
SLIDE 23

Conclusions

  • Developed clever allocation strategies for

NRFU with potential to decrease nonresponse bias

– Quality cost – increased variance – $ cost – little savings

  • Could offset variance gains with increased

response from subsampled cases

  • Needed new contact strategy for these “hard

to reach” establishments

slide-24
SLIDE 24

Timeline for Our Research

Nonrespondent Subsampling Methods Simulation Studies Contact Strategy for Nonresponse Follow-up (NRFU) Field Test (Embedded Experiment) Adaptive Collection Design Field Test (Embedded Experiment)

2014 (and 2016) 2015 2016

slide-25
SLIDE 25

Motivation

  • Find effective NRFU contact strategy

– Small businesses only – Improve response rates for chronic nonresponders

  • Reduce nonresponse bias  improve estimates
slide-26
SLIDE 26

Embedded Experiment

  • 2014 Annual Survey of Manufactures (ASM)
  • Single-unit establishments
  • Certainty (larger)
  • Noncertainty (smaller) – Domain of interest (target)
  • Split panel design
slide-27
SLIDE 27

Contact Strategies by Treatment Panel

Panel Initial Mail 1st NRFU 2nd NRFU 3Rd NRFU 4th NRFU Control (C) Letter Letter Form Letter Letter Treatment 1 (T1) Letter Letter Certified Letter Form Letter Treatment 2 (T2) Letter Letter Letter/Flyer Form Letter Same Treatment for all panels

slide-28
SLIDE 28

Contact Strategies by Treatment Panel

Panel Initial Mail 1st NRFU 2nd NRFU 3Rd NRFU 4th NRFU Control (C) Letter Letter Form Letter Letter Treatment 1 (T1) Letter Letter Certified Letter Form Letter Treatment 2 (T2) Letter Letter Letter/Flyer Form Letter New treatment in each panel

slide-29
SLIDE 29

Treatment 2: Letter/Flyer

slide-30
SLIDE 30

Analyses Conducted

Statistic/Indicator Examines effects on Proxy Unit Response Rate Unit Response Cox Proportional Hazards Regression Model Parameters Length of Time to Respond Hazard Ratio Probability of Responding Balance Indicator Distance Indicator Representativeness of Respondent Sample Fraction of Missing Information (FMI) Nonresponse bias for specific

  • utcome variables
slide-31
SLIDE 31

Proxy Unit Response Rate

Certainty Cases

T1: Certified Letter at NRFU 2 T2: Letter/Flyer at NRFU 2 C: Control

slide-32
SLIDE 32

Proxy Unit Response Rate

Certainty Cases

T1: Certified Letter at NRFU 2 T2: Letter/Flyer at NRFU 2 C: Control Same

slide-33
SLIDE 33

Proxy Unit Response Rate

Certainty Cases

T1: Certified Letter at NRFU 2 T2: Letter/Flyer at NRFU 2 C: Control Different New NRFU treatment introduced

slide-34
SLIDE 34

Proxy Unit Response Rate

Certainty Cases

  • Letter/flyer slightly more effective than

current procedure

  • Certified letter most effective treatment
slide-35
SLIDE 35

Proxy Unit Response Rate

Certainty Cases Noncertainty Cases

slide-36
SLIDE 36

Proxy Unit Response Rate

Certainty Cases Noncertainty Cases

  • Letter/flyer consistently less effective

than current procedure

  • Certified letter most effective treatment
slide-37
SLIDE 37

Survival Analysis: Cox Proportional Hazards Model

  • Length of time to respond
  • Certainty Units:

Reduced for T2 (Letter/Flyer)

  • Noncertainty Units: Reduced for T1 (Cert Letter)
  • Probability of (eventually) responding
  • Certainty units:

Increases for T2 (Letter/Flyer)

  • Noncertainty Units: Increases for T1 (Cert Letter)
slide-38
SLIDE 38

Fraction of Missing Information (FMI)

  • Bounded between (0,1)
  • Multiple Imputation
  • Proxy Pattern-Mixture (PPM) Model
  • Gamma PPM Model (Andridge and Thompson 2015)
  • Predict outcome variable from frame measure of size (Proxy)
  • Obtained different models for respondents and nonrespondents

(Pattern-Mixture model)

  • Measured sensitivity by range of response mechanisms
  • MAR:missing at random – “Best Case”
  • NMAR:

not missing at random – “Worst Case”

slide-39
SLIDE 39

FMI Results – Noncertainty Units

Strength of Proxy

slide-40
SLIDE 40

FMI Results – Noncertainty Units

Unit Nonresponse Rate (UNR) FMI < UNR

slide-41
SLIDE 41

FMI Results – Noncertainty Units

  • FMI lowest in T1
  • FMI highest for T2
slide-42
SLIDE 42

FMI Results – Noncertainty Units

Difference (spread) indicative of

  • Sensitivity to

response mechanism

  • Strength of proxy
slide-43
SLIDE 43

2014 Experiment Conclusion

  • Certified letter effective
  • Increases response
  • Reduced NR bias
  • Letter/flyer  current procedure
  • No effects on nonresponse
  • No (discernable) effects on NR bias
slide-44
SLIDE 44

Timeline for Our Research

Nonrespondent Subsampling Methods Simulation Studies Contact Strategy for Nonresponse Follow-up (NRFU) Field Test (Embedded Experiment) Adaptive Collection Design Field Test (Embedded Experiment)

2014 (and 2016) 2015 2016

slide-45
SLIDE 45

Motivation

  • Compare two alternative adaptive NRFU

strategies in the field

  • Given new NRFU strategy

Initial Mail Reminder NRFU 1 NRFU 2 NRFU 3 Letter Letter Letter Certified Letter Letter

slide-46
SLIDE 46

Embedded Experiment

  • 2015 Annual Survey of Manufactures (ASM)
  • Single-unit establishments
  • Split panel design
  • Targeted allocation in treatment panel (T = T1+ T2)
  • Simulated nonrespondent subsampling
slide-47
SLIDE 47

2015 ASM Field Experiment

Control Panel (Industries) Nonrespondents Treatment Panel (Industries)

T1 T2

T1 = Systematic subsample with optimized allocation T2 = Complement of T1

Initial Letter Due Date Reminder NRFU 1 - Letter NRFU 2 – Certified Letter NRFU 3 – Final Letter NRFU 2 – Letter NRFU 3 – Final Letter

Single Units

NRFU 2 – Certified Letter NRFU 3 – Final Letter

slide-48
SLIDE 48

2015 ASM Field Experiment

Control Panel (Industries) Nonrespondents Treatment Panel (Industries)

T1 T2

T1 = Systematic subsample with optimized allocation T2 = Complement of T1

Initial Letter Due Date Reminder NRFU 1 - Letter NRFU 2 – Certified Letter NRFU 3 – Final Letter

Single Units

NRFU 2 – Certified Letter NRFU 3 – Final Letter

Paired industries on like characteristics then randomly split into two panels

NRFU 2 – Letter NRFU 3 – Final Letter

slide-49
SLIDE 49

2015 ASM Field Experiment

Control Panel (Industries) Nonrespondents Treatment Panel (Industries)

T1 T2

T1 = Systematic subsample with optimized allocation T2 = Complement of T1

Initial Letter Due Date Reminder NRFU 1 - Letter

Single Units

NRFU 2 – Certified Letter NRFU 3 – Final Letter

Control panel receives standard NRFU ($$)

NRFU 2 – Letter NRFU 3 – Final Letter NRFU 2 – Certified Letter NRFU 3 – Final Letter

slide-50
SLIDE 50

2015 ASM Field Experiment

Control Panel (Industries) Nonrespondents Treatment Panel (Industries)

T1 T2

T1 = Systematic subsample with optimized allocation T2 = Complement of T1

Initial Letter Due Date Reminder NRFU 1 - Letter NRFU 2 – Cert Letter NRFU 3 – Final Letter NRFU 2 – Letter NRFU 3 – Final Letter

Single Units

NRFU 2 – Certified Letter NRFU 3 – Final Letter

Overall 1-in-2 subsample with different subsampling rates by industry (targeted group)

slide-51
SLIDE 51

2015 ASM Field Experiment

Control Panel (Industries) Nonrespondents Treatment Panel (Industries)

T1 T2

T1 = Systematic subsample with optimized allocation T2 = Complement of T1

Initial Letter Due Date Reminder NRFU 1 - Letter

Single Units

NRFU 2 – Certified Letter NRFU 3 – Final Letter

Targeted group receives standard NRFU ($$)

NRFU 2 – Letter NRFU 3 – Final Letter NRFU 2 – Certified Letter NRFU 3 – Final Letter

slide-52
SLIDE 52

2015 ASM Field Experiment

Control Panel (Industries) Nonrespondents Treatment Panel (Industries)

T1 T2

T1 = Systematic subsample with optimized allocation T2 = Complement of T1

Initial Letter Due Date Reminder NRFU 1 - Letter

Single Units

NRFU 2 – Certified Letter NRFU 3 – Final Letter

Complement group receives less expensive NRFU ($)

NRFU 2 – Letter NRFU 3 – Final Letter NRFU 2 – Certified Letter NRFU 3 – Final Letter

slide-53
SLIDE 53

NRFU 2 Treatments (Single Units)

slide-54
SLIDE 54

Evaluation Measures

Measure Description Level Proxy Unit Response Rate Proportion of responding establishments to sampled establishments (unweighted) Panel Quantity Response Rates Proportion of tabulated item value obtained from reported data (weighted) Panel by item Source of Data Item Proportion of responding units that retain reported data after processing Panel by item Fraction of Missing Information (FMI) Measure of effects of potential nonresponse bias

  • n collected items

Panel by item

slide-55
SLIDE 55

Proxy Unit Response Rate

slide-56
SLIDE 56

Proxy Unit Response Rate

Experiment Intervention

slide-57
SLIDE 57

Item Metrics

  • Quantity Response Rates

– Proportion of tabulated item value obtained from reported data – Similar values for all variables, regardless of design

  • Source of Data Item

– Proportion of responding units that retain reported data after processing – Similar values for all variables, regardless of design

slide-58
SLIDE 58

FMI: Targeted Allocation

slide-59
SLIDE 59

FMI: Targeted Allocation

Strength of Proxy

slide-60
SLIDE 60

FMI: Targeted Allocation

Strength of proxy is about the same in both panels (C & T)

slide-61
SLIDE 61

FMI: Targeted Allocation

Strength of proxy is stronger in treatment panel

slide-62
SLIDE 62

FMI: Targeted Allocation

Unit Nonresponse Rate (NRR)

slide-63
SLIDE 63

FMI: Targeted Allocation

The targeted allocation FMI is less than the control FMI for all groups

slide-64
SLIDE 64

Really Final Conclusion

  • Targeted allocation procedure effective

– Maintains unit and item response rates – No detrimental effects on data quality – Reduced cost compared to the uniform follow-up

  • Targeted allocation procedure was implemented in

the 2016 ASM for single unit establishments

  • Planning implementation for the 2017 Economic

Census

slide-65
SLIDE 65

References

Andridge, R.R. and K.J. Thompson. 2015. Assessing nonresponse bias in a business survey: proxy pattern-mixture analysis for skewed data. Annals

  • f Applied Statistics 9(4): 2237–2265.

Thompson, K.J., Kaputa, S., and Bechtel, L. 2018 (accepted). Strategies for subsampling nonrespondents for economic programs. Survey Methodology. Thompson, K.J. and Kaputa, S. 2017. Investigating adaptive nonresponse follow-up strategies for small businesses through embedded

  • experiments. JOS 33(3): 835-856.

Kaputa., S.J. and Thompson, K.J. 2018 (accepted). Adaptive design strategies for nonresponse follow-up in economic surveys. JOS. Kaputa, S.J., Thompson, K.J., and Beck, J.L. 2017. An embedded experiment for adaptive nonresponse follow-up in establishment surveys. Proceedings of the Section on Survey Research Methods, American Statistical Association.

slide-66
SLIDE 66

Thanks!!!

Katherine.J.Thompson@census.gov