Planning for CHIP Monitoring and Evaluation September 12, 2012 Dr. - - PowerPoint PPT Presentation

Planning for CHIP Monitoring and Evaluation

September 12, 2012

• Dr. Byron Kennedy, Deputy Director

Anne Kern, Public Health Program Coordinator Monroe County Health Dept., NY Julia Joh Elligers, Director of Assessment, Planning, and Workforce Development NACCHO, Washington, D.C.

Webinar Logistics

• The lines are muted. If you wish to mute/unmute your line to ask/answer a

question, please do the following:

• To unmute your own line, press *7
• To mute your own line, press *6.

2

• Throughout the presentation and during the Q&A session, if you have a question,

please use ReadyTalk’s ‘raise your hand’ feature or use the chat box to indicate you have a question. The facilitator will call your name and ask for your question.

PROJECT REQUIREMENTS & PHAB STANDARDS AND MEASURES: CHIP MEASURES: CHIP MONITORING AND EVALUATION

Project Requirements: CHIP Monitoring and Evaluation

Engage Community Members and LPHS Partners

“Community members must be engaged in a meaningful and substantive way throughout the CHA and CHIP processes, including indicator selection, data collection, data analysis, data presentation and distribution, issue prioritization, CHIP creation, implementation of

4

CHIP, and monitoring of results.” “Partners should be engaged in a strategic way throughout the CHA and CHIP processes, including gaining access to data, mobilizing community members, data collection, data review, issue prioritization, and CHIP implementation.”

Project Requirements: CHIP Monitoring and Evaluation

A CHIP implementation plan that does the following:

• Provides clear, specific, realistic, and action-oriented goals.
• Contains the following:
• Goals, objectives, strategies, and related performance measures for determined priorities in the

short-term (one to two years) and intermediate term (two to four years),

• Realistic timelines for achieving goals and objectives.
• Designation of lead roles in CHIP implementation for LPHS partners, including LHD role.

5

• Designation of lead roles in CHIP implementation for LPHS partners, including LHD role.
• Formal presentation of the role of relevant LPHS partners in implementing the plan and a

demonstration of the organization’s commitment to these roles via letters of support or accountability.

• Emphasis on evidence-based strategies.
• A general plan for sustaining action.

Project Requirements: CHIP Monitoring and Evaluation

Community Health Improvement Process Report: CHIP Overview:

• Describes how community members and LPHS partners were engaged in development of the CHIP,

especially for issue prioritization and strategy development.

• Summarizes the top priorities for action and the process by which these were identified.
• Specifies how your strategy aimed at addressing a social or economic determinant of health/heath

inequity was identified.

• Describes how the CHIP implementation plan was developed.

6

• Includes who was involved in the process.
• Describes the respective roles of all involved in the development process.
• Specifies which strategies are evidence-based, source of the evidence, and how these strategies were

identified and selected. (Note: PHAB uses the terms “evidence-based or promising strategies” in the Standards and Measures Version 1.0)

• Describes how the CHIP and CHIP implementation plan was/will be distributed to the public.
• Describes the plan for monitoring CHIP implementation progress and sustaining action beyond

initial CHIP rollout, including reporting on progress to the public.

• Describes how the QI or Quality Planning technique used in the community health improvement process

was chosen and who was involved in applying the technique.

• Includes final CHA/CHIP Cost Report.

PHAB Requirements: CHIP Monitoring and Evaluation

*Be sure to review the standards listed below to identify the measures and required documentation that PHAB seeks related to developing a CHIP.

Standard 5.2: Conduct a comprehensive planning process resulting in a tribal/state/community health improvement plan

7

PHAB Requirements: CHIP Monitoring and Evaluation

For example… Measure 5.2.2L: Produce a community health improvement plan as a result of the community health improvement process. Required documentation: CHIP dated within the last five years that includes 1a: Community health priorities, measurable objectives, improvement strategies and performance measures with measurable and time-framed targets; 1b. Policy changes needed to accomplish health objectives; c. Individuals and organizations that have accepted responsibility for implementing strategies; 1d. Measurable health

8

and organizations that have accepted responsibility for implementing strategies; 1d. Measurable health

• utcomes or indicators to monitor progress; and 1e. Alignment between the CHIP and the state and national

priorities.

PHAB Requirements: CHIP Implementation

Measure 5.2.4A: Monitor progress on implementation of strategies in the CHIP in collaboration with broad participation from stakeholders and partners* Required documentation: 1. Evaluation reports on progress made in implementing strategies in the CHIP including: 1a. Monitoring of performance measures and

• 1b. Progress related to health improvement indicators [Guidance: Description of progress made on health

indicators as defined in the plan...]; and 2. Revised health improvement plan based on evaluation results [Guidance: …must show that the health improvement plan has been revised based on the evaluation listed in 1

9

[Guidance: …must show that the health improvement plan has been revised based on the evaluation listed in 1 above…] * Not required as part of the CHA/CHIP Project

Learning Objectives

At the completion of the session participants will be able to:

1. Explain the relationship between key elements of the CHIP, quality improvement work, and evaluation. 2. Develop an evaluation plan based on CHIP goals, objectives, outcome, and performance indicators. 3. Understand methods for meaningfully monitoring and reporting progress of CHIP implementation. 4. Describe how monitoring and evaluation can support community and partner engagement in ongoing CHIP implementation.

10

CHIP implementation. 5. Understand how to update the CHIP or work plans based on the results of monitoring and evaluation activities 6. Name the types of community and non-traditional resources that can support monitoring and evaluation activities. 7. Understand the importance of developing evaluation measures during the implementation planning activities. 8. Describe the project and PHAB documentation requirements for CHIP monitoring and evaluation.

Health Action: Priorities for Monroe County Webinar, September 2012

Anne Kern Public Health Program Coordinator Office of Community Health Improvement Byron Kennedy, MD, PhD, MPH Deputy Director of Health Monroe County, NY

11

MISSION

To improve the health of Monroe County residents

VISION

Continuous, measurable improvement in health status in Monroe County

12

13

HEALTH ACTION Process

Assess health status

Process guided by committee-community health experts Collect and compile data Identify health goals based on data Findings published in report cards (healthaction.org)

Data used in report cards

Birth and Death Certificates Registries (diseases, lead poisoning)

14

Registries (diseases, lead poisoning) Surveys/Monitoring systems

• YRBS
• Adult Health Survey

Choose priority goals

Report card presented to community to get feed back on priorities. 1-2 priorities selected for each lifecycle group

14

HEALTH ACTION Process

Define the Leadership

Identify an organization that will lead the community in addressing the priority area

Develop improvement plan

Based on local data Evaluation

Perform the intervention

15

Perform the intervention

Often utilizing grant funding

Evaluate

Use data routinely collected for the report cards, but additional evaluation is often done

15

Examples of Community Health Improvement: Teen Smoking Reduction Immunization Initiatives

16

Immunization Initiatives Lead Poisoning Reduction

16

… Lead Poisoning is Health Problem with a Housing Solution

17

17

Problem Definition

In early 1990s, more than 3,000 childhood lead poisoning cases identified annually. Prevalence more than 35% in several inner-city census tracts. In the City of Rochester, 87% of housing stock built before 1950. Most housing units in the City are rentals.

18

Most housing units in the City are rentals. Lead abatement viewed as too expensive.

18

Next Steps

Vision A community with healthy children free from lead poisoning. Goal Eliminate childhood lead poisoning

19

Eliminate childhood lead poisoning Objective Reduce the number of children who are lead poisoned from 3,000 to zero by 2010.

19

Eliminate lead

Resources Outcomes

MCDPH UR CGR United Way Passionate

Activities

Awareness/ Media Engage new partners Research effective housing policies

Output

Media contacts Media/ Educational materials Lead summit # Attended and who Decreased numbers of houses not passing inspection

Goal

Logic Model

lead poisoning

Passionate Community Partners HUD Funding Funding from

• ther sources

Advocate- policy changes Develop Lead Abatement Training/ Certification Apply for HUD funding Screening Events Housing policy passed and implemented # of houses inspected # Trained in lead abatement HUD funding received #Children screened Increase screening among children Decreased number of children with EBLL

20

Science - Monitor data/latest research to ensure scientific basis for action – Included experts in environmental health Housing - lead safe practices – homeowners, contractors, childcare providers

Coalition to Prevent Lead Poisoning

21

Education and Outreach – community educational activities, media Screening/Professional Education – educate providers, work to increase screening Government relations –translate work of committees into actionable legislative and regulatory initiatives

21

Boundary Networks

Boundary-Spanning Individuals Negotiate the relationship between science and politics. Negotiate the relationship between producers and consumers of information. Provide technical expertise to local communities, particularly when resources are limited.

22

resources are limited. Represent diverse technical areas: health, housing, legal, academic, education, local government, national NGO’s.

22

Sustaining Boundary Networks

Develop and nurture relationships. Commitment to basing decisions on best available evidence/science. Carefully tailored technical assistance requests given interests, expertise, and capacity. Regularly give credit back to experts for their work/efforts.

23

Regularly give feedback to experts about their impact and making a difference.

23

Optimizing Boundary Networks

Gather relevant data (e.g., unpublished data from colleagues/contacts). Perform new analyses with existing data. Interpret and apply data to local conditions (e.g., current policy issue). Communicate findings effectively to target audience (e.g., summary sheet, case study).

24

Avoid unnecessary costs (e.g., new data collection, extensive analyses, consultants).

24

MCDPH monitored main outcomes on an ongoing basis using existing data collection efforts (i.e., lead screening, EBLL and inspections) Other partners completed additional evaluation

Evaluation of Lead Ordinance

Evaluation

25

Evaluation of Lead Ordinance

Funded by Greater Rochester Health Foundation Completed by CGR, University of Rochester City of Rochester inspection data included showed 90% of properties were passing City inspections (demonstrating that landlords were making pro-active improvements to their properties)

25

Children with Elevated Blood Lead Levels >=10 µg/dL

26

Source: Monroe County Department of Public Health

26

Children with Confirmed Blood Lead Levels >=20 µg/dL

27

Source: Monroe County Department of Public Health

27

Children aged 1 & 2 years old screened as a percentage of births the previous two years

28

Source: Monroe County Department of Public Health

28

Monroe County Department of Public Health Housing Inspections

29

Source: Monroe County Department of Public Health

29

Timeline: Childhood EBLL (>=10 µg/dL) Trends & Major Events

3,563 5,680 3,710 3000 4000 5000 6000 1992 NYS Regulation- Screening at age 1&2, and all under age 6 who 2003 City HUD Grant 2004 Monroe County drops level of concern from 20 mg/dl to15 “Lets Make Lead History Summit” HUD funded communications campaign City HUD Grant 2005 Lead Ordinance Passed 2002 United Way designates lead poisoning as #1 policy issue MCDPH and CPLP HUD Grants CGR study to help with targeting high risk housing

30

2,959 2,284 2,046 1,698 1,293 1,179 1,234 1,019 900 675 571 426 363 283 290 222 1000 2000 3000 1977 MCDPH Lead Program Began under age 6 who haven’t been screened Title 10 HUD Funding 1997 MCDPH HUD Grant

2009 MCDPH HUD Grant City HUD Grant

2006 Implementation of City Lead Ordinance MCDPH HUD Grant

2007, 2008 City HUD Grant

2000 Coalition to Prevent Lead Poisoning (CPLP) formed 1999 Lead Free Coalition, HUD Grant

30

Lessons Learned

Data collected routinely can be utilized to mobilize community to action and for evaluation. Important to have passionate community partners (e.g., funding for coalitions, and evaluation). Partnerships with universities helpful in doing more complex analyses.

31

Partnerships with universities helpful in doing more complex analyses.

31

Q & A

32

32

Thank You

33

Planning for CHIP Monitoring and Evaluation

Julia Joh Elligers, Director of Assessment, Planning, and Workforce Development Development NACCHO, Washington, D.C.

Share: Why CHIP evaluation?

What do you see as the benefits of CHIP evaluation?

35

Why CHIP evaluation?

Evaluation can measure how well you implement actions (process evaluation) and whether your actions have an impact (outcome evaluation). When evaluating your CHIP, you are measuring how well you

36

When evaluating your CHIP, you are measuring how well you implemented your plan (process evaluation) and whether your actions improved health in your community (outcome evaluation).

Vision Strategic Issues/Priorities CHA Goals

Community Health Improvement Plan

CHA

37

Outcome Objectives Activities/Tactics Performance Indicators/Measures CHA CHA

Vision Strategic Issues Goals

CHIP

What evaluation aims to answer?

Is the achievement

• f our goals

addressing our strategic issues? Is the achievement of

• ur strategic issues

leading to our vision? Is the achievement of

• ur objectives leading

38

Outcome Objectives Activities/Tactics Performance Measures Are our activities generating the

• utcomes we want

to see?

• ur objectives leading

to our goals? Are we doing what we said we would do?

Vision Strategic Issues Goals

CHIP

What evaluation aims to answer?

39

Outcome Objectives Activities/Tactics Performance Measures Are we doing what we said we would do?

Vision: A community where people are mentally and physically healthy throughout the life cycle. Strategic Issue: How can we ensure that our youth reach their full potential in life? Goal: Reduce the use of marijuana and alcohol use by youth. Objectives: a) Decrease the percentage of youth using marijuana from 20% to 15% by 2014. b) Decrease the percentage of youth drinking alcohol from 50% to 30% by 2014. Outcome Indicators: a) Percentage of middle and high school students indicating that they use marijuana. b) Percentage of middle and high school students indicating that they drink alcohol. Strategies: a) Provide information to youth about the dangers and consequences of using marijuana and alcohol. b) Build the skills of parents and other adults to talk with their children about the dangers and consequences

• f using marijuana and alcohol.

c) Reduce the access of marijuana and alcohol in the community. Tactics: a) Provide marijuana and alcohol awareness programs to youth in middle and high schools. b) Provide workshops for parents and create parent chat groups c) Work with law enforcement to do local vendor compliance checks on alcohol sales to minors. d) Set up a tip line on marijuana sales. Performance Indicators: a) Pre- and post test results of youth participating in awareness programs. b) Number of parents attending workshops. c) Number of parents participating in chat groups. d) Number of vendors who pass alcohol compliance checks. e) Number of calls to the tip line.

Strategies: a) Provide information to youth about the dangers and consequences of using marijuana and alcohol. b) Build the skills of parents and other adults to talk with their children about the dangers and consequences of using marijuana and alcohol. c) Reduce the access of marijuana and alcohol in the community. Tactics: a) Provide marijuana and alcohol awareness programs to youth in middle and high schools. b) Provide workshops for parents and create parent chat groups

Share: Methods for process evaluation

41

b) Provide workshops for parents and create parent chat groups c) Work with law enforcement to do local vendor compliance checks on alcohol sales to minors. d) Set up a tip line on marijuana sales. Performance Indicators: a) Pre- and post test results of youth participating in awareness programs. b) Number of parents attending workshops. c) Number of parents participating in chat groups. d) Number of vendors who pass alcohol compliance checks. e) Number of calls to the tip line.

Example process evaluation methods

• Checklist
• Meeting minutes
• Dashboards
• QI techniques
• Logs
• Documentation
• Observation
• Customer satisfaction inquiry
• Surveys
• Focus groups

42

• Logs
• Conversation with participants
• Steering committee confirmation
• Focus groups
• Interviews

Evaluation Quality Improvement

• There are different methods you can use to evaluate

your CHIP. Quality improvement techniques provide methods for evaluating CHIP activities.

• However, QI may not be the best approach for all

types of evaluation.

Model for Improvement

W hat are we trying to accomplis h? H

• w will we know that a

change is an improvement?

43

• The benefit of QI is that it is cyclical by design.
• QI is also good for studying well-defined processes.
• A limitation of QI is that it isn’t always good at

measuring outcomes that do not have clear numerical values or that have multi-factoral causes.

Act P lan Do S tudy

Vision Strategic Issues Goals

CHIP

What evaluation aims to answer?

Is the achievement of

• ur objectives leading

44

Outcome Objectives Activities/Tactics Performance Measures Are our activities generating the

• utcomes we want

to see?

• ur objectives leading

to our goals?

Vision: A community where people are mentally and physically healthy throughout the life cycle. Strategic Issue: How can we ensure that our youth reach their full potential in life? Goal: Reduce the use of marijuana and alcohol use by youth. Objectives: a) Decrease the percentage of youth using marijuana from 20% to 15% by 2014. b) Decrease the percentage of youth drinking alcohol from 50% to 30% by 2014. Outcome Indicators: a) Percentage of middle and high school students indicating that they use marijuana. b) Percentage of middle and high school students indicating that they drink alcohol. Strategies: a) Provide information to youth about the dangers and consequences of using marijuana and alcohol. b) Build the skills of parents and other adults to talk with their children about the dangers and consequences

• f using marijuana and alcohol.

c) Reduce the access of marijuana and alcohol in the community. Tactics: a) Provide marijuana and alcohol awareness programs to youth in middle and high schools. b) Provide workshops for parents and create parent chat groups c) Work with law enforcement to do local vendor compliance checks on alcohol sales to minors. d) Set up a tip line on marijuana sales. Performance Indicators: a) Pre- and post test results of youth participating in awareness programs. b) Number of parents attending workshops. c) Number of parents participating in chat groups. d) Number of vendors who pass alcohol compliance checks. e) Number of calls to the tip line.

Goal: Reduce the use of marijuana and alcohol use by youth. Objectives: a) Decrease the percentage of youth using marijuana from 20% to 15% by 2014. b) Decrease the percentage of youth drinking alcohol from 50% to 30% by 2014.

Share: Methods for outcome evaluation

46

b) Decrease the percentage of youth drinking alcohol from 50% to 30% by 2014. Outcome Indicators: a) Percentage of middle and high school students indicating that they use marijuana. b) Percentage of middle and high school students indicating that they drink alcohol.

Example outcome evaluation methods

• Assessment data collection methods
• Qualitative methods to measure relationship between activities and
• utcomes

Surveys, focus groups, interviews, ethnographic study

• Statistical modeling to measure relationship between activities and

47

• Statistical modeling to measure relationship between activities and
• utcomes

Cross tabs Multivariate regression Experiments

Vision Strategic Issues Goals

CHIP

What evaluation aims to answer?

Is the achievement

• f our goals

addressing our strategic issues? Is the achievement of

• ur strategic issues

leading to our vision?

48

Outcome Objectives Activities/Tactics Performance Measures

Vision: A community where people are mentally and physically healthy throughout the life cycle. Strategic Issue: How can we ensure that our youth reach their full potential in life? Goal: Reduce the use of marijuana and alcohol use by youth. Objectives: a) Decrease the percentage of youth using marijuana from 20% to 15% by 2014. b) Decrease the percentage of youth drinking alcohol from 50% to 30% by 2014. Outcome Indicators: a) Percentage of middle and high school students indicating that they use marijuana. b) Percentage of middle and high school students indicating that they drink alcohol. Strategies: a) Provide information to youth about the dangers and consequences of using marijuana and alcohol. b) Build the skills of parents and other adults to talk with their children about the dangers and consequences

• f using marijuana and alcohol.

c) Reduce the access of marijuana and alcohol in the community. Tactics: a) Provide marijuana and alcohol awareness programs to youth in middle and high schools. b) Provide workshops for parents and create parent chat groups c) Work with law enforcement to do local vendor compliance checks on alcohol sales to minors. d) Set up a tip line on marijuana sales. Performance Indicators: a) Pre- and post test results of youth participating in awareness programs. b) Number of parents attending workshops. c) Number of parents participating in chat groups. d) Number of vendors who pass alcohol compliance checks. e) Number of calls to the tip line.

Vision: A community where people are mentally and physically healthy throughout the life cycle.

Share: Methods for evaluating strategic issues & visions

50

Strategic Issue: How can we ensure that our youth reach their full potential in life?

Evaluating Strategic Issues & Vision

• In next iteration of the community health improvement process, compare

vision statements and strategic issues. Have things improved in your community?

• Solicit feedback from partners and community residents

51

Creating an Evaluation Plan

• Purpose and focus of evaluation (should be practical, doable, and

reflect evaluation priorities)

• Evaluation questions
• Schematic (e.g. logic model)
• Methods

Remember: You don’t have to evaluate everything. A subcommittee can take on this task. 52

• Workplan & timeline
• Participants
• Resources
• Methods for monitoring results
• How and when information will be applied

this task. Enlist students to help create a plan. Identify a partner who can take ownership of activities

PHAB Documentation

• Evaluation reports that include
• Description of stakeholder and partner participation
• Status with respect to performance measures and health

indicators

• Evaluation-informed revised CHIP

53

• Evaluation-informed revised CHIP

Developing a monitoring & updating system

• Identify roles and responsibilities of partners in monitoring and

updating system

• Identify frequency of monitoring and updating activities
• Identify methods for monitoring and updating
• Documentation

54

• Documentation
• Spreadsheets
• Dashboards
• Balanced Score Cards
• Assessment data collection system

Leveraging your monitoring and evaluation efforts

• To apply for funding
• To engage partners and leaders
• To increase political support
• To engage community residents
• To learn how to continually improve future CHA/CHIP work

55

• To learn how to continually improve future CHA/CHIP work
• To celebrate your accomplishments

Discussion: Revising your CHIP

• Scenario 1: The number of vendors who pass alcohol compliance

checks goes down.

56

Discussion: Revising your CHIP

• Scenario 2: Percentage of middle and high school students indicating

that they use marijuana goes up.

57

Discussion: Revising your CHIP

• Scenario 3: You’ve exceeded your outcome objective to decrease the

percentage of youth drinking alcohol from 50% to 30% by 2014. The percentage of youth drinking alcohol is now 5%.

58

Share: Evaluation Resources

What types of evaluation resources do you have?

59

What types of evaluation resources do you need?

Community Health Improvement Plan

60

Final Thoughts

You don’t have to evaluate everything. Start with something that is doable. Don’t have to be a data person. People who are good at tracking and

• bserving can be valuable evaluation resources.

Remember, evaluation is about telling and learning from your story.

61

Remember, evaluation is about telling and learning from your story.

Questions and Discussion

62

Evaluation Resources

• CDC’s Evaluation Resources:

www.cdc.gov/eval/resources/index.htm

• Evaluation Plan Template:

www.cdc.gov/asthma/.../AppendixF_Evaluation_Plan_Outli ne.doc

• American Evaluation Association Resources:

63

• American Evaluation Association Resources:

www.eval.org/resources.asp

• National Network of Libraries of Medicine Evaluation

Resources: http://nnlm.gov/evaluation/tools/

• Wholey, Hatry, Newcomer, Handbook of Practical Program

Evaluation

• Patton, Utilization-Focused Evaluation

CHIP Resources

• Issue Prioritization Tip Sheet
• CHIP Implementation Plan Template
• 90 Day Implementation Steps Worksheet
• MAPP Action Cycle Resources:

http://www.naccho.org/topics/infrastructure/MAPP/framework/clearinghouse/

64

phase6.cfm

• What Works for Health database: http://www.countyhealthrankings.org/what-

works-for-health

• Take Action Cycle Webinar Series by the County Health Rankings and

Roadmaps project: http://www.countyhealthrankings.org/blog/123

Visit the CHA/CHIP Network to access these resources and more!

Last Word

The next CHA/CHIP training webinar will be on:

‘Collaborating with Partners and Community Members to Change Public and Organizational Policy’

Presenters and Date: Allen Lomax, Victoria Manning & Ken Smith

65

Presenters and Date: Allen Lomax, Victoria Manning & Ken Smith 9/24/12 from 2:30 – 4 PM ET

Please complete the evaluation before logging off the webinar.