Human Factors Methods to Support the Experts Review of Automatically - - PowerPoint PPT Presentation
Patient Safety through Intelligent Procedures in medication Human Factors Methods to Support the Experts Review of Automatically Detected Adverse Drug Events Nicolas LEROY Brian BJRN , Adrian BACEANU, Marie-Catherine BEUSCART-ZEPHIR 1
MIE 2009 – Sarajevo, August 31 2009
1
Patient Safety through Intelligent Procedures in medication
Nicolas LEROY Brian BJØRN , Adrian BACEANU, Marie-Catherine BEUSCART-ZEPHIR
MIE 2009 – Sarajevo, August 31 2009
2
Consortium: 13 partners
1/ Hospitals
France, Denmark With / without CPOE
2/ Industry:
Oracle, IBM, Medasys (CPOE editors) Vidal (pharmaceutical Kbase)
3/ Academic teams
Data & Semantic mining, Decision Support Systems,
Human Factors Engineering
Duration: 40 months (Jan 08 April 2011)
MIE 2009 – Sarajevo, August 31 2009
3
Rule generation
Data mining
MIE 2009 – Sarajevo, August 31 2009
4
Rule generation Validation of the rule
Knowledge based
drugs
chart review
Validation in the clinical context
Data mining
MIE 2009 – Sarajevo, August 31 2009
5
Validation of the rule
Knowledge based
drugs
chart review
Validation in the clinical context
Integration in a CDSS Module
Detection of dangerous situation Feed back to clinician
Rule generation
Data mining
MIE 2009 – Sarajevo, August 31 2009
6
Rule generation Validation of the rule
Knowledge based
drugs
chart review
Validation in the clinical context
Data mining
Integration in a CDSS Module
Detection of dangerous situation Feed back to clinician Expected benefit : Help clinicians to improve the patient safety Potential Risk : Over alerting
MIE 2009 – Sarajevo, August 31 2009
7
Rule creation Validation of the rule
Knowledge based
drugs
chart review
Validation in the clinical context
Data mining
Integration in a CDSS Module
Detection of dangerous situation Feed back to clinician
MIE 2009 – Sarajevo, August 31 2009
8
Rule creation Validation of the rule
Knowledge based
drugs
Data mining semantic mining
Integration in a CDSS Module
Detection of dangerous situation Feed back to clinician chart review
Validation in the clinical context
The Expert review requires :
explained the observed abnormality
MIE 2009 – Sarajevo, August 31 2009
9
Rule creation Validation of the rule
Knowledge based
drugs
Data mining semantic mining
Integration in a CDSS Module
Detection of dangerous situation Feed back to clinician chart review
Validation in the clinical context
Our objective:
evaluation and the improvement of the PSIP rules
process
MIE 2009 – Sarajevo, August 31 2009
10
MIE 2009 – Sarajevo, August 31 2009
11
Chart review
Consultation
MIE 2009 – Sarajevo, August 31 2009
12
Chart review
Consultation
MIE 2009 – Sarajevo, August 31 2009
13
Chart review
Consultation
Rule evaluation in the clinical context
Analysis
rule Evaluation
relevance
MIE 2009 – Sarajevo, August 31 2009
14
Chart review
Consultation
Rule evaluation in the clinical context
Analysis
rule Evaluation
relevance Agree with the rule ?
MIE 2009 – Sarajevo, August 31 2009
15
Chart review
Consultation
Rule evaluation in the clinical context
Analysis
rule Evaluation
relevance Agree with the rule ? Yes Yes
validated
MIE 2009 – Sarajevo, August 31 2009
16
Chart review
Consultation
Rule evaluation in the clinical context
Analysis
rule Evaluation
relevance Agree with the rule ? Yes Yes
validated
No
Not validated (this rule could not explain the case under review)
MIE 2009 – Sarajevo, August 31 2009
17
Chart review
Consultation
Rule evaluation in the clinical context
Analysis
rule Evaluation
relevance Agree with the rule ? Yes Yes
validated
No do not know
Uncertain Not validated
MIE 2009 – Sarajevo, August 31 2009
18
Chart review
Consultation
Rule evaluation in the clinical context
Analysis
rule Evaluation
relevance Agree with the rule ? Yes Yes
validated
No do not know
Uncertain Not validated Think aloud methodology
MIE 2009 – Sarajevo, August 31 2009
19
“think-aloud”.
allows to track all the experts’ actions with the Expert Explorer application
verbalizations are typewritten and coded.
MIE 2009 – Sarajevo, August 31 2009
20
MIE 2009 – Sarajevo, August 31 2009
21
MIE 2009 – Sarajevo, August 31 2009
22
MIE 2009 – Sarajevo, August 31 2009
23
Independently of the clinical context, Do you agree with the rule ?
MIE 2009 – Sarajevo, August 31 2009
24
Independently of the clinical context, Do you agree with the rule ? Expert 1 Expert 2
No 2% Do not know 12% Yes 86% Yes 81% Do not know 19%
MIE 2009 – Sarajevo, August 31 2009
25
Do you think that the rule applies to the case under review ?
MIE 2009 – Sarajevo, August 31 2009
26
Expert 1 Expert 2 Do you think that the rule applies to the case under review ?
No 48% Yes 40 % Do not know 12% No 51% Yes 16 % Do not know 33%
MIE 2009 – Sarajevo, August 31 2009
27
Expert 1 Expert 2 Do you think that the rule applies to the case under review ?
No 48% Yes 40 % Do not know 12% No 51% Yes 16 % Do not know 33%
Jha 2008 Hwang 2008
94% 79%
MIE 2009 – Sarajevo, August 31 2009
28
Expert 1 Expert 2 Do you think that the rule applies to the case under review ?
No 48% Yes 40 % Do not know 12% No 51% Yes 16 % Do not know 33%
Jha 2008 Hwang 2008
94% 79%
MIE 2009 – Sarajevo, August 31 2009
29
Independent
context Depending
context Effect
Cause Problems with Data
MIE 2009 – Sarajevo, August 31 2009
30
Independent
context Depending
context Effect
Cause Problems with Data
MIE 2009 – Sarajevo, August 31 2009
31
Independent
context Depending
context Effect
Cause Problems with Data
MIE 2009 – Sarajevo, August 31 2009
32
Independent
context Depending
context Effect
Cause Problems with Data
MIE 2009 – Sarajevo, August 31 2009
33
Independent
context Depending
context Effect
Cause Problems with Data
MIE 2009 – Sarajevo, August 31 2009
34
Independent
context Depending
context Effect
Cause Problems with Data
MIE 2009 – Sarajevo, August 31 2009
35
MIE 2009 – Sarajevo, August 31 2009
36
Effect is not « important » enough to be significant
Independent
context Depending
context
Effect explained by patient characteristics
Effect
Normal reaction
About the lab results of a patient detected as abnormal « Yes, but it is a very very tiny effect, it’s just over the superior bound » Problems with Data
MIE 2009 – Sarajevo, August 31 2009
37
Effect is not « important » enough to be significant
Independent
context Depending
context
Effect explained by patient characteristics
Effect
Normal reaction
About the lab results of a patient detected as abnormal « Yes, but it is a very very tiny effect, it’s just over the superior bound » Problems with Data
MIE 2009 – Sarajevo, August 31 2009
38
Effect is not « important » enough to be significant
Independent
context Depending
context
Effect explained by patient characteristics
Effect
Normal reaction
Effect explained by other dugs Conditions level not significant
Cause (about a thrombopenia effect) « No, it’s the Innohep that would cause the thrombopenia. It’s a much better candidate than the beta-lactams»
MIE 2009 – Sarajevo, August 31 2009
39
Effect is not « important » enough to be significant
Independent
context Depending
context
Effect explained by patient characteristics
Effect
Normal reaction
Effect explained by other dugs Conditions level not significant
Cause (about a thrombopenia effect) « No, it’s the Innohep that would cause the thrombopenia. It’s a much better candidate than the beta-lactams»
MIE 2009 – Sarajevo, August 31 2009
40
Effect is not « important » enough to be significant
Independent
context Depending
context
Effect explained by patient characteristics
Effect
Normal reaction
Effect explained by other dugs Conditions level not significant
Cause
Problem with administration route Effect disappear without change in drugs Problem with the delay administration -effect
« The thrombocytosis appears on day 12 while the metronidazole treatment is given on days 1, 2 and 3, so the delay is too important. »
MIE 2009 – Sarajevo, August 31 2009
41
Effect is not « important » enough to be significant
Independent
context Depending
context
Effect explained by patient characteristics
Effect
Normal reaction
Effect explained by other dugs Conditions level not significant
Cause
Problem with administration route Effect disappear without change in drugs Problem with the delay administration -effect
« The thrombocytosis appears on day 12 while the metronidazole treatment is given on days 1, 2 and 3, so the delay is too important. »
MIE 2009 – Sarajevo, August 31 2009
42
Effect is not « important » enough to be significant
Independent
context Depending
context
Effect explained by patient characteristics
Effect
Normal reaction
Effect explained by other dugs Conditions level not significant
Cause
Problem with administration route Effect disappear without change in drugs Problem with the delay administration -effect
Problems with Data
Technical problem with data Clinical data are missing about outcome Clinical data are missing about causes
« We don’t have the home treatment »
MIE 2009 – Sarajevo, August 31 2009
43
Effect is not « important » enough to be significant
Independent
context Depending
context
Effect explained by patient characteristics
Effect
Normal reaction
Effect explained by other dugs Conditions level not significant
Cause
Problem with administration route Effect disappear without change in drugs Problem with the delay administration -effect
Problems with Data
Technical problem with data Clinical data are missing about outcome Clinical data are missing about causes
« We don’t have the home treatment »
MIE 2009 – Sarajevo, August 31 2009
44
– Experts = future users of the system – Expert review of past hospitalization allows :
contextualize them
– Based on common chart review methods used in other similar project – Addition of the think aloud method
– The flexibility of statistical procedures used to generate the rules allows to fix most of the problems detected by the think aloud protocol
MIE 2009 – Sarajevo, August 31 2009
45
Effect is not « important » enough to be significant Effect explained by other dugs
Parameters used in the rule Judgment in the clinical context Data
Technical problem with data Conditions level not significant Problem with administration route Effect disappear without change in drugs Problem with the delay administration -effect Effect explained by patient characteristics Clinical data are missing about outcome Clinical data are missing about causes
Cause Effect
Normal reaction
OK OK
MIE 2009 – Sarajevo, August 31 2009
46
– The think aloud methodology were useful but time consuming – We built a questionnaire based on the coding scheme.
– The questionnaire has been successfully tested on a sample of stays and will be integrated in the Expert Explorer – It will support the continuous evaluation and improvement of the rules by the end users
the rules
– This questionnaire may also be generalized to projects using similar evaluation methodology
MIE 2009 – Sarajevo, August 31 2009
47