Application of data analysis in public auditing of the national - - PowerPoint PPT Presentation

Application of data analysis in public auditing of the national goals attainability: evidence-based policies in contemporary state governance

2020

• 1. Theory of change and theory of transformation:

application for analysis of national goals attainability

Theory of change: evaluation of federal projects

Source: Befani, B., & Mayne, J. (2014). Process Tracing and Contribution Analysis: A Combined Approach to Generative Causal Inference for Impact Evaluation. IDS Bulletin, 45(6), 17–36.

ToC is used:

• on the planning stage:
• to justify the logic of the intervention.
• for the monitoring:
• to analyze intermediate results and to adjust

the intervention.

• for the evaluation:
• to assess the final outcomes of the policy.

Intervention Outcome (intended)

Theory of change is a visualization of causal steps:

STEP 1

THEN THEN THEN

…

ToC includes:

Assumptions: events and conditions that need to occur for the causal link to realize; Risks: the inverse of assumptions; if they occur, the casual link does not hold;

Data on the monitoring of the Federal Projects Reports on the Project’ objectives fullfillment Distinct events of the Federal Project “Assistance for women with children” Preschool facilities with predetermined number of places are created People send their children to the preschool institutions Mothers are employed and get paid Increase in incomes, decrease in poverty

Assumptions Chain process of changes Steps Metrics Sources

Mothers obtain more free time which could be allocated to the work Activities prescribed by the policy Policy output Intermediate

• utcomes

Final outcomes Data on the monitoring of the Federal Projects

• 1. Number of places in preschool

institutions

• 2. Amount of money subsidized to the

private preschools

• 1. Data on the monitoring of the

Federal Projects

• 2. Russian statistical service data
• 3. Data from the Ministry of Labor
• 1. Number of children under 3 years visiting

preschool institutions

• 2. Active employment among women with

children under 3 years Income in households with women having children

The preschool institutions are logistically considerate The price is reasonable

• r free of charge

Women prefer to send children to the institutions rather than take care themselves No discrimination towards women with kids on the labor force market The social payments are still paid after employment The preschool institutions have really been created The policy affects poor households

• 2. Evidence gap maps: Facilitating strategic

accumulation of evidence

Evidence Gap Maps

Visualization of existing evidence of effects of interventions MOTIVATION WHAT IS EGM? Existing studies

• have uneven quality
• require time to find and extract valuable evidence – great

number of different sources, studies are large in size RESULT

• support for evidence-based evaluations of interventions
• easy access to the information about the number of

related studies, level of their quality and main findings

• identification of existing evidence gaps
• facilitation of future research aimed to fill existing

evidence gaps

Source: Snilstveit, B., Vojtkova, M., Bhavsar, A., Stevenson, J., & Gaarder, M. (2016). Evidence & Gap Maps: A tool for promoting evidence informed policy and strategic research agendas. Journal of Clinical Epidemiology, 79, 120–129.

Evidence Gap Map: example

Low quality Medium quality High quality Outcomes State interventions Household income growth Decreasing poverty rate Decreasing unemployment

• 1. Creating new preschool education

facilities

• 2. Social contracts to help job search and

employment

• 3. <…>

<…> <…> <…>

Dujardin et al. (2018) Karabchuk, 2013 Biewen, M. and Waller M. (2007) Blundell and J. van Reenen (2004)

Assessment of evidence quality

№ Criteria 1 2 3 4 5 6 7 Very low quality Low quality Medium quality High quality Very high quality Criterion weight Calculation 1 Study type 4 0.8 3.2 2 Country-specific compliance 2 1.4 2.8 3 Relevance (time context) 1 0.6 0.6 4 Causality 3 1 3 5 Reproducibility 5 0.6 3 6 Conformity of mechanism 3 1.6 4.8 Final score 2.90

Final score 1 - 1,5 Very low quality of evidence Final score 1,5 -2,5 Low quality of evidence Final score 2,5 - 3,5 Medium quality of evidence Final score 3,5 - 4,5 High quality of evidence Final score 4,5 - 5 Very high quality of evidence !"#$% &'()* = ,-./

0 1- ∗ 3-

5 1- – evidence score for the criterion " 3- – weight of criterion "

Closing Evidence Gaps

UK: What Works Network

• collating existing evidence on the

effectiveness of programs and practices

• producing high quality synthesis

reports and systematic reviews in new areas

• assessing the effectiveness of

policies and practices against an agreed set of outcomes

• filling gaps by commissioning new

trials and evaluations

• sharing findings across institutions

in an accessible way US: President Obama’s evidence-based initiatives (2016)

• Building and Using Evidence to

Improve Results

• Investing in Child Care and Early

Learning: support universal preschool through the Preschool for All Initiative

• Fund early education research and

evaluation by supporting the Institute of Education Sciences’ Early Learning Research Network

Challenges in evidence-based policy evaluation

No relevant studies Existing studies have poor design Existing studies are based on unlikely assumptions Existing studies do not comprehend Russian peculiarities

Challenges in evidence-based policy evaluation: data

Unsuitable format Data processing methods change over time No panel dimension Representativity problems Poor regional presence in data

• 3. Policy evaluation:

case of healthcare reforms evaluation

Healthcare dynamics

Source: Federal state statistics service

Model I: pooled OLS

*income, unemployment, poverty, inflation, food consumption by categories, ecology by categories, age composition, crime rate

Evaluation of healthcare spending on mortality, models

!" – healthcare spending #" – regional control variables* $

" – year dummies 14

%&'()*+(,-" = / + 1!-" + 2#-" + 3-"

Model II: panel with trend

%&'()*+(,-" = / + 1!-" + 2#-" + δ$

• " + 3-"

Model III: panel with trend and lag

%&'()*+(,-" = / + θ%&'()*+(,-"67 + 1!-" + 2#-" + δ$

• " + 3-"

Model IV: panel with interactions

%&'()*+(,-" = / + 1!-" + 2#-" + δ$

• " + κ!-"$
• " + 3-"

Model V: panel with trend and lag

%&'()*+(,-" = / + θ%&'()*+(,-"67 + 1!-" + 2#-" + δ$

• " + κ!-"$
• " + 3-"

Source: Own calculation

Results

Oaxaca-Blinder decomposition for policy evaluation

Is it possible to use such a decomposition to uncover effects of certain policy over time?

Source: Own calculations

Decomposition: any structural effect?

Contact information

Daria Tsyplakova, Director of Department for research and methodology, Accounts Chamber of the Russian Federation method@ach.gov.ru Yulia Filippova, Senior Advisor, Accounts Chamber of the Russian Federation Filippova_YA@ach.gov.ru Egor Zhevlenev, Analyst, Accounts Chamber of the Russian Federation e.zhevlenev@cpur.ru