M EASURING OR EVALUATING IMPACT ? In its truest sense impact - - PowerPoint PPT Presentation

MEASURING THE IMPACT OF THE NATIONAL NETWORKS FOR COLLABORATIVE OUTREACH

PROFESSOR JACQUELINE STEVENSON SHEFFIELD HALLAM UNIVERSITY

OVERVIEW OF THE SESSION

 What are we evaluating?  A note of caution....  Our evaluation approach  Some findings to date  Impact – what are the networks aiming to do?

 On what, with what, and on whom?

 Evaluation approaches of the networks

 Issues and considerations

 Where are we now

OUR AIMS AND OBJECTIVES

 Aims:  To conduct an evaluation of NNCOs to assess their value, beyond

existing activity, in terms of enabling co-ordinated, efficient and effective outreach with schools and colleges.

 To work with the networks to help to embed and share good practice

in effective evaluation methodologies relating to outreach activities.

 Objectives:  Undertake a comprehensive mapping of NNCOs  Evaluate:

 The perceived value and benefits of NNCOs  the extent to which they add value beyond existing activity; are engaged

in innovative practice and have engaged with their Local Enterprise Partnership

 The efficacy of networks in achieving intended aims of scheme

 Identify best practice and possibilities for sustainability  To assist the networks in building robust evaluation methods

A NOTE OF CAUTION

 ......to us and to you: what can actually be claimed

through evaluation?

 The episodic assessment of the change in targeted results that

can be attributed to the programme/project intervention

 Or the analysis of inputs and activities to determine their

contribution to results.

 HEFCE 2014: “in straightforward terms, evaluation seeks to

establish how well an activity, project or programme has

• worked. The key question is: has it achieved its intended
• utcomes? A further question is: have there been

unanticipated outcomes?

 So.....impact and outcome evaluation....

IMPACT EVALUATION

 Impact evaluation  is structured to answer the question: “how would outcomes

have changed if the intervention had not been undertaken?”

 Therefore  ideally requires a counterfactual analysis, i.e. “a comparison

between what actually happened and what would have happened in the absence of the intervention”

 In other words,  impact evaluation assesses the changes that can be directly

attributed to a particular intervention (both intended and unintended)

 Is consequently different to outcome evaluation, which

examines whether targets have been achieved

MEASURING OR EVALUATING IMPACT?

 In its truest sense  impact measurement would require using an independent

evaluator, establishing control groups, measuring changes

• ver extended periods of time etc. etc.

 However, evaluation of WP activity is not a rigorous

social science experiment

 Therefore  determining an explicate causal relationship between a

particular activity/intervention and a specific outcome is very difficult/impossible

 So we will not be able to say that ‘because of/without the

NNCOs X,Y or Z definitely did or did not happen...!’

WITH THAT IN MIND...WHAT ARE WE

EVALUATING?

The perceived value and benefits of NNCOs ; The extent to which they: add value beyond existing activity are engaged in innovative practice have engaged with their Local Enterprise Partnership The efficacy of networks in achieving intended aims of

scheme

value, beyond existing activity, in terms of enabling co-

• rdinated, efficient and effective outreach with schools and

colleges.

DATA

 Survey data  Case studies  Workshop events  Work with the Networks

KEY QUESTIONS WE ARE ASKING OF OUR

DATA

 What perceived value and benefits do the networks believe they are able to

• ffer/deliver

That they would not have been able to do without HEFCE? That they would have done anyway?

 To what extend do the networks perceive they add value beyond existing

activity?/ are engaged in innovative practice

Is this a fair and reasonable assessment?

 Have the networks engaged with their Local Enterprise Partnership?  How efficacious (time/money/other resources) do the networks consider

they are/have been in terms of achieving intended aims of scheme?

 Is this a fair and reasonable assessment?  So.......Have the networks offered value, beyond existing activity, in terms of

enabling co-ordinated, efficient and effective outreach with schools and colleges?

SUVEY DATA

 Types of networks; model

• r structure; coverage

 Governance and

management

 Aims and foci  Target groups  Activities and resources  Distribution of formulaic

funding

 Collaboration and

affiliation with other NNCOs

 Gaps, overlaps  Value added?  Key challenges faced and

managed

 Sustainability  Approaches to evaluation

CASE STUDY DATA

 Network lead  Network partners  SPoC  Schools/colleges  Past  Present  Future  Ways of working  Efficacy  Value added?  Key challenges faced and

managed

 Sustainability

FINDINGS: PERCEIVED VALUE AND BENEFIT OF NNCOS

 Build and sustain relationships:  Colleges; Other Outreach Networks; Schools and 6th Form; FECs;

HEIs; Cold spots; Whole city providers; Families/parents; Young people

 Improve practices:  Reduce duplication; Increase collaboration; Better IAG; Share

activities and resources; Better monitoring and tracking; Increase knowledge and understanding; Improve monitoring and evaluation

 Offer greater strategic approach/value:  Clearer focus; more impartial advice; more meaningful inclusion of

partners; specialisms, and targeting

 Offer more/better resources:  SPoC; Website; new outreach activities; new forms of IAG; new

data tools

FINDINGS: POSSIBLE IMPACT MORE BROADLY

 Future funding and sustainability  Better tracking and analysis  Better IAG  Being more cost effective and efficient  But in the end it is all about....  Enhancing progression and enabling fair access  Gaining greater equity, social justice; social mobility  For different groups....  Problematic....

FINDINGS: WITH WHICH TARGET

GROUPS/SETTINGS?

 Age  Adult  Years 9-11; pre-6th form;

6th form

 Location  Work-based; PRU  HEI/FE/6th form  Others involved  Parents/carers  Schools/colleges/HEIs  Social grouping  Low paid/

unskilled/qualified; no previous HE

 Carers  Disabilities  Under-represented gender  Low SES/FSM;  (BME)  Settings (Special; all

state; virtual) PRU; FE; HE

Another aim is to assist the networks in building robust evaluation methods Which leads us to what is being evaluated by the networks.........

DIFFERENT SORTS OF EVALUATION INDICATORS

EVIDENCED ACROSS THE NETWORKS

 Process indicators  dimensions concerned with actions and what needs to be done to

achieve outcomes e.g. ways of doing admissions or giving out IAG, experiences offered e.g. Outreach events

 Outputs indicators  the tangible and intangible products that result from project

activities e.g. Numbers of completed personal statements

 Outcome indicators  the observed effects of the output-specific results e.g. increased

number of applications to HE; changes in confidence

 Impact indicators  attainment of higher level strategic goals; sustained long term

changes e.g. changes in the student demographic profile of HE

IMPACT VS. OUTCOME EVALUATION

• In terms of ‘widening participation’ a NNCO may want to

increase applications to HE of a particular group.

• The activity offered by a network might focus on how to

write a personal statement in order to increase applications

▫ The process evaluation would explore the way in which the

activities were perceived by those involved

▫ The outputs evaluation would measure the numbers of personal

statements written

▫ The outcome evaluation would measure the numbers of applications

made as a consequence

▫ The impact evaluation would determine an increase in applications

to HE from the target group (although the final impact would be acceptances, retention, attainment and success of same group)

▫ However, what might be a measure of outcomes for one target might

be impact for another e.g.

 Outcome: Statements written  Impact: increased applications

 Impact and outcome evaluation both involve the

assessment of intervention effects but at different levels.

 Both impact and outcome evaluation test the causal chain

• f events that has been postulated by the intervention i.e.

 changing knowledge, awareness and confidence in terms of

writing personal statements will lead to a change in application behaviour. Changing application behaviours will lead to a changing student demographic OR

 changing knowledge, awareness and confidence in terms of

writing personal statements will lead to a change in the actual writing of personal statements. Changing the numbers of young people from the target groups writing personal statements will lead to an increase in applications

CONSIDERATIONS FOR US EVALUATING THE

NETWORKS EVALUATION!....

 Approaches  What research/evaluation questions are being asked of the data?  Qualitative vs. Quantitative approaches  Ethical, age appropriate and sensitive evaluation approaches  KPIs - what and why?  Design  What, when and how – evaluation tools  Partnership and collaboration  Data  What data NNCOs have? Who has it? Who has access? How

consistent? what can be systematically claimed from the data?

 Issues  Data gaps; what can actually be measured and claimed....back to

counterfactuals!

WHAT’S MISSING

• Clear research/evaluation questions?

▫ Does changing knowledge, awareness and confidence in terms of

writing personal statements lead to a change in application behaviour.

▫ Does changing application behaviours lead to a changing student

demographic

▫ Does changing knowledge, awareness and confidence in terms of

writing personal statements lead to a change in the actual writing of personal statements.

▫ Does changing the numbers of young people from the target groups

writing personal statements will lead to an increase in applications

• Causal links....

SO WHERE ARE WE NOW?

 Asking questions of the data!  Evaluating:

 The perceived value and benefits of NNCOs ;  the extent to which they add value beyond existing activity; are

engaged in innovative practice and have engaged with their Local Enterprise Partnership

 The efficacy of networks in achieving intended aims of scheme

 Deciding what claims we can make....  Supporting the networks’ evaluation