Goal Attainment Scaling Clinical Utility and Training Methods Jen - - PowerPoint PPT Presentation

▶
goal attainment scaling
SMART_READER_LITE
LIVE PREVIEW

Goal Attainment Scaling Clinical Utility and Training Methods Jen - - PowerPoint PPT Presentation

Mar 05, 2023 258 likes •740 views

Goal Attainment Scaling Clinical Utility and Training Methods Jen Angeli, Karen Harpster, Amber Sheehan Occupational Therapy and Physical Therapy FINANCIAL DISCLOSURE AACPDM 72 nd Annual Meeting October 9-13, 2018 Speaker Names: Amber Sheehan,

slide-1
SLIDE 1

Occupational Therapy and Physical Therapy

Goal Attainment Scaling

Clinical Utility and Training Methods

Jen Angeli, Karen Harpster, Amber Sheehan

slide-2
SLIDE 2

Occupational Therapy and Physical Therapy

FINANCIAL DISCLOSURE AACPDM 72nd Annual Meeting October 9-13, 2018

Speaker Names: Amber Sheehan, Karen Harpster, Jen Angeli

  • 1. Disclosure of Relevant Financial Relationships

We have no financial relationships to disclose.

  • 2. Disclosure of Off-Label and/or investigative uses:

We will not discuss off label use and/or investigational use in my presentation.

slide-3
SLIDE 3

Occupational Therapy and Physical Therapy

Grab your phone. In the internet browser, type: kahoot.it

slide-4
SLIDE 4

Occupational Therapy and Physical Therapy

https://kahoot.com/welcomeback/ https://create.kahoot.it/login

slide-5
SLIDE 5

Occupational Therapy and Physical Therapy

Learning objectives

  • Synthesize the literature on GAS training

and clinical utility

  • Describe pilot effort to train therapists on

GAS utilization, per best available evidence

  • Propose best practice for GAS

standardization

  • Discuss feasibility of implementation
slide-6
SLIDE 6

Occupational Therapy and Physical Therapy

First, we get on the same page.

1.Goal Attainment Scaling = GAS 2.GAS is a criterion-referenced measure that helps us to quantify a degree to which personal goals are achieved. 3.It was developed in 1968 and was first used in mental health. 4.It is becoming more popular in rehabilitation health, as an index of intervention efficacy.

slide-7
SLIDE 7

Occupational Therapy and Physical Therapy

The GAS parts

  • Document a set of patient-identified goals
  • Identify the construct of change that matters

most to the patient/family

  • Observe/discuss/document current

performance on each goal

  • Agree upon a desired short term outcome
  • Discuss what performance looks like if it is a

little better than the desired short term

  • utcome, or much better than the desired

short term outcome.

slide-8
SLIDE 8

Occupational Therapy and Physical Therapy

For example:

I want to ride my bike to the end of Victory Avenue (0.5 miles).

Performance Interpretation GAS interval Completed MS150 Much better than expected +2 Rode to park for new zipline (1 mile) Better than expected +1 See above Short term goal Rode to Joe’s house for dinner (3 houses down) Progress, but did not achieve goal

  • 1

Cannot ride bike Baseline level of performance

  • 2
slide-9
SLIDE 9

Occupational Therapy and Physical Therapy

Extra parts

  • Weight each goal for importance (patient/caregiver)

and difficulty (therapist)

  • Put it all in a big formula that will calculate a T score.
  • Use the T score or change in T scores to interpret
  • r Minimal Detectable Change = 10 points
slide-10
SLIDE 10

Occupational Therapy and Physical Therapy

What’s nice about GAS

  • It’s patient-centric
  • It provides a single, objective summary of

performance

  • Progress reflects meaningful change in a

prioritized area

slide-11
SLIDE 11

Occupational Therapy and Physical Therapy

What’s not nice about GAS

  • It’s hard to learn.
  • It feels like it’s not clinically feasible.
  • It’s only valid if you follow the rules,

particularly those associated with scaling.

slide-12
SLIDE 12

Occupational Therapy and Physical Therapy

What we learned about GAS in a systematic review

  • 1. Is it being used in pediatric rehab?
  • 2. What’s the rigor of published pediatric

studies employing the GAS?

  • 3. Is GAS a responsive outcome measure?
slide-13
SLIDE 13

Occupational Therapy and Physical Therapy

  • 51 studies in pediatric rehab were included

in our review

  • They weren’t very strong with respect to

rigor and quality

  • Despite this, GAS appeared to detect

meaningful change in ~ 60% of studies

slide-14
SLIDE 14

Occupational Therapy and Physical Therapy

slide-15
SLIDE 15

Occupational Therapy and Physical Therapy

  • Higher Sackett levels = more rigorous study

design

slide-16
SLIDE 16

Occupational Therapy and Physical Therapy

  • 17 quality criteria that group broadly into

categories describing:

– content validity of the scale – reliability of scale construction – reliability of scale rating – additional items related to training and examiner bias

  • Higher quality scores = greater validity in

GAS application

  • See handout for criteria!
slide-17
SLIDE 17

Occupational Therapy and Physical Therapy

Quality summary

  • Of the 51 studies included, less than half (19 studies)

were categorized as Sackett levels I, II, or III.

  • Average quality score = 4.57 points (/17)

Individual studies lost the most points for failure of Goal Attainment Scaling scales to adhere to the criteria of:

  • unidimensionality (1/51)
  • time-specificity (3/51)
  • reliability of scale construction-equidistance of levels (3/51)
  • inter-rater reliability (4/51)
slide-18
SLIDE 18

Occupational Therapy and Physical Therapy

Why does it matter?

  • GAS T scores may inflate (and indicate

progress) simply because of bias introduced from failure to create valid and reliable scales.

slide-19
SLIDE 19

Occupational Therapy and Physical Therapy

What did we do?

slide-20
SLIDE 20

Occupational Therapy and Physical Therapy

Our Goal:

  • To utilize the GAS as an objective
  • utcome measure with a high level of

reliability in clinical practice

– Start with episodic care summer programs

  • Clinic-based GAS session

http://www.acmsoft.com/Services/think-big-start-small.html

slide-21
SLIDE 21

Occupational Therapy and Physical Therapy

https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=hLm-W5SoFqGp_Qb-

  • Z74Bg&q=let%27s+do+it&oq=let%27s+do+it&gs_l=img.3..0l10.18304.28268..28880...0.0..0.105.1091.7j4......2....1..gws-wiz-

img.....0..35i39j0i67.3puL26l830Q#imgrc=FjAo6UDp_7yAjM:&spf=1539226018390

https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=hLm-W5SoFqGp_Qb-

  • Z74Bg&q=Reading+research+articles&oq=Reading+research+articles&gs_l=img.3..35i39j0i24.3052.5474..5884...0.0..0.99.10

78.12......1....1..gws-wiz-img.......0j0i7i30j0i7i5i30j0i8i7i30j0i5i30.4Kmw2Xj2-aY

slide-22
SLIDE 22

Occupational Therapy and Physical Therapy

https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=tby- W8HkDMqf_QbHgLToAg&q=variation&oq=variation&gs_l=img.3..35i39j0l9.2142.2834..3640...0.0..0.102.191.1j1......1.. ..1..gws-wiz-img.0D_0uKckWAo#imgrc=nTk8YeFP9SO0NM:&spf=1539226998417

  • Level of parent and child involvement
  • Video tape GAS performance vs no video
  • Documentation
  • Setting the construct of change
slide-23
SLIDE 23

Occupational Therapy and Physical Therapy

Develop a process

  • Video tape all GAS goal performances
  • Watch videos during team meetings

– Develop documentation (importance, difficulty, level of change, etc)

  • Review goals using GAS checklist
  • Review GAS levels with caregivers

http://www.10minutebiztools.com/Startup-Trial-and-Error.html

slide-24
SLIDE 24

Occupational Therapy and Physical Therapy

GAS in Research: Refine the process even further

  • Independent teams

– COPM/treatment team – GAS team

  • Training to ensure specificity of COPM goal

E.g. “John runs back and forth and puts holes in his socks” “John will sit for 5 minutes to complete a fine motor task”

  • Develop a script to elicit meaningfulness of

goals

slide-25
SLIDE 25

Occupational Therapy and Physical Therapy

What happened next: Therapist Training

  • Developed a training manual, based on current

evidence

– includes procedures and criteria for scale construction, scoring, and interpretation of outcome data

  • Provide formal training

– Introduced the training manual and demonstrate application via case study – Trainees independently review materials and construct a scales based on the same video – Review scale with each trainee using quality criteria to promote adequate interrater reliability

slide-26
SLIDE 26

Occupational Therapy and Physical Therapy

What happened next: GAS Visit via telehealth

  • GAS scales are developed by 2

independent raters based on video taped performance of goals

– GAS scales compared for sameness

  • Child scored on both

– GAS checklist implemented – Feedback given regarding scale construction within 1 week of GAS completion

slide-27
SLIDE 27

Occupational Therapy and Physical Therapy

GAS through telehealth....

  • Volume control
  • Test video call with caregiver

– Ensure they understand how to use technology, camera angle, etc.

  • Make sure the caregiver knows the child

needs to be present for session

slide-28
SLIDE 28

Occupational Therapy and Physical Therapy

GAS through telehealth....

  • Make sure the supplies for the goals are

readily available

  • Making sure the video stays on the child

during goal completion

  • Considering an informal scale to capture how

much assistance the caregiver needs to participate in co-creation of scale

– seems to be impacting validity of the scales we create

slide-29
SLIDE 29

Occupational Therapy and Physical Therapy

  • f doing GAS through

telehealth

  • Materials usually readily available
  • Natural environment
  • Generalization occurs naturally
  • Empowering the caregiver
slide-30
SLIDE 30

Occupational Therapy and Physical Therapy

Criteria for GAS appraisal

  • Eliminated items

– Goals specific to aim of intervention – Pretest score same across all goals/ patients – Specification of follow up time – Adequate therapist training – Report of inter-rater reliability – Report of examples of goals and scaling

  • Added items suggested by other sources
slide-31
SLIDE 31

Occupational Therapy and Physical Therapy

Results thus far

  • 2/5 trained therapists had no prior GAS

background

  • Telehealth goal scaling completed with 6

patients, each by one of 3 therapists

– Average 9.4/15 scaling quality criteria met – After feedback provided, no additional sessions yet to assess potential change in quality – Two trained therapists have not yet had a goal scaling session scheduled

slide-32
SLIDE 32

Occupational Therapy and Physical Therapy

Results thus far, cont.

  • Most easily achieved criteria

– Mutually exclusive goals – All five levels defined

  • similar to the literature
slide-33
SLIDE 33

Occupational Therapy and Physical Therapy

Results thus far, cont.

  • Other easily achieved criteria
  • Scale uni-dimensionality
  • Verification of pre-intervention performance
  • Equidistance of scale levels

– frequent absence in the literature, achieved here due to initial training and study design

slide-34
SLIDE 34

Occupational Therapy and Physical Therapy

Future plans

  • Study design

– Confirm post-test scale score with caregiver

slide-35
SLIDE 35

Occupational Therapy and Physical Therapy

Future plans

  • Study design

– Determine interrater reliability by correlating the ratings of the original goal scaler with a second rater on 20-30% of goals (Krasny- Pacini et al., 2017; King et al., 1999)

slide-36
SLIDE 36

Occupational Therapy and Physical Therapy

Future plans

  • Reporting results

– Adequate example of goals and scaling reported – Results in literature were most commonly stated as:

  • a raw post-intervention T-score (27 of 51 studies)
  • change in T-score (3 of 51)
  • number or % of goals that were achieved (i.e., at or

above the “0” level on the scale; 23 of 51 studies)

  • effect size (4 of 51)
slide-37
SLIDE 37

Occupational Therapy and Physical Therapy

Zip Coat

  • Goal: Mom would like Peter to be able

to zip his coat independently

  • Construct of Change: Level
  • f independence
slide-38
SLIDE 38

Occupational Therapy and Physical Therapy

Let's Practice

slide-39
SLIDE 39

Occupational Therapy and Physical Therapy

Zip Coat

  • Goal: Mom would like Peter to be able

to zip his coat independently

2:​ 1:​ 0:​

  • 1:​
  • 2: Needs help with engaging zipper and stabilizing coat to pull up zipper
slide-40
SLIDE 40

Occupational Therapy and Physical Therapy

Zip Coat

  • Goal: Mom would like Peter to be able

to zip his coat independently

2:​ 1:​ 0:​

  • 1: Stabilize coat and pull up zipper with coat on the table
  • 2: Needs help with engaging zipper andstabilizing coat to pull up

zipper

slide-41
SLIDE 41

Occupational Therapy and Physical Therapy

Zip Coat

  • Goal: Mom would like Peter to be able

to zip his coat independently

2:​ 1:​ 0: Stabilize coat and pull zipper up with coat on

  • 1: Stabilize coat and pull up zipper with coat on the table
  • 2: Needs help with engaging zipper andstabilizing coat to pull up zipper
slide-42
SLIDE 42

Occupational Therapy and Physical Therapy

Zip Coat

  • Goal: Mom would like Peter to be able

to zip his coat independently

2:​ 1: Engage and pull up zipper with coat on the table 0: Stabilize coat and pull zipper up with coat on

  • 1: Stabilize coat and pull up zipper with coat on the table
  • 2: Needs help with engaging zipper andstabilizing coat to pull up zipper
slide-43
SLIDE 43

Occupational Therapy and Physical Therapy

Zip Coat

  • Goal: Mom would like Peter to be able

to zip his coat independently

2: Engaging and pull up zipper while the coat is on​ 1: Engage and pull up zipper with coat on the table 0: Stabilize coat and pull zipper up with coat on

  • 1: Stabilize coat and pull up zipper with coat on the table
  • 2: Needs help with engaging zipper andstabilizing coat to pull up zipper
slide-44
SLIDE 44

Occupational Therapy and Physical Therapy

Jennifer Angeli, DPT, PhD

Physical Therapist, Cincinnati Children’s Hospital Medical Center Assistant Professor, Rehabilitation Sciences, University of Cincinnati

513.252.5722 Jenny.Angeli@cchmc.org Amber Sheehan, OTR/L

Occupational Therapist, Cincinnati Children’s Hospital Medical Center

513.636.8526 Amber.Sheehan@cchmc.org Karen Harpster, OTD, PhD

Occupational Therapist, Cincinnati Children’s Hospital Medical Center Assistant Professor, Rehabilitation Sciences, University of Cincinnati

513.803.3604 Karen.Harpster@cchmc.org

slide-45
SLIDE 45

Occupational Therapy and Physical Therapy

References

  • Grant, M., & Ponsford, J. (2014). Goal attainment scaling in brain injury rehabilitation: strengths,

limitations and recommendations for future applications. [KH, SS]. Neuropsychol Rehabil, 24(5), 661-

  • 677. doi: 10.1080/09602011.2014.901228
  • Harpster, K., Sheehan, A., Foster, E. A., Leffler, E., Schwab, S. M., & Angeli, J. M. (2018). The

methodological application of goal attainment scaling in pediatric rehabilitation research: a systematic

  • review. Disability and rehabilitation, 1-10.
  • Kiresuk T.J. & Sherman R.E. (1968). Goal Attainment Scaling: A general method for

evaluating comprehensive community mental health programs. Community Mental Health Journal, 4(6), 443–453.​

  • Kiresuk, T.J., Smith, A., & Cardillo, J.E. 1994. Goal Attainment Scaling: Applications, Theory, and
  • Measurement. Hillsdale, NJ: Lawrence Erlbaum Associates.
  • Mailloux, Z., May-Benson, T. A., Summers, C. A., Miller, L. J., Brett-Green, B., Burke, J. P., et al. (2007).

The Issue Is—Goal attainment scaling as a measure of meaningful outcomes for children with sensory integration disorders. American Journal of Occupational Therapy, 61, 254–259.​

  • McDougall, J. & Wright, V. (2009). The ICF-CY and Goal Attainment Scaling: Benefits of their combined

use for pediatric practice. Disability and Rehabilitation, 31 (16): 1362–1372.

  • Ruble, L., McGrew, J.H., & Toland, M.D. (2012). Goal Attainment Scaling as an outcome measure in

randomized controlled trials of psychosocial interventions in autism. J Autism Dev Disord, 42(9): 1974– 1983

  • Schaaf, R.C. et al. (2013). An intervention for sensory difficulties in children with autism: A randomized
  • trial. J Autism Dev Disord, online.
slide-46
SLIDE 46

Occupational Therapy and Physical Therapy