AUTOMATED VALIDATION OF CLINICAL INCIDENT TYPES J.GUPTA, - - PowerPoint PPT Presentation

AUTOMATED VALIDATION OF CLINICAL INCIDENT TYPES

J.GUPTA, I.KORINSKA, J.PATRICK SCHOOL OF INFORMATION TECHNOLOGY SYDNEY UNIVERSITY HEALTH INFORMATICS CONFERENCE, BRISBANE 5 AUG 2015

MOTIVATIONS

• Improve patient safety and quality of healthcare service
• Over million clinical incidents in one state and under 2% are

scrutinised

• Current performance extraction processes
• labour intensive,
• there are inefficiencies in the software used and
• classification models not statistically tested
• Urgent need for innovation in classifying IIMS datasets and
• improve software architecture

OBJECTIVES

• Determine best performing statistical text classifiers (STC)
• For multiclasses (13,12) clinical incident types – going beyond 2
• Demonstrate methods of improve specificity and sensitivity of the

classification models e.g.

• impact of balanced v/s unbalanced design
• size (number of reports)
• classifier’s effect (Clinician v/s Expert)

OVERVIEW - SOFTWARE ARCHITECTURE & DATA WAREHOUSE

Incident Information Management System (IIMS) 1 AA 2 AV 3 BHP 4 BBP 5 CM 6 DOC 7 FALL 12 Classes 8 HAI 13 Classes 9 MED 10 NUT 11 PATH 12 PC 13 PU Data pool: 7 Hospitals datasets Period:2004 - 2008

RESEARCH DESIGN

Experiment Clinical Incident types/N. Fields Classifier’s effect Size/Balance effect

1a 13* 14 fields§ Clinician 5448** Unbalanced 1b 12^ 10 fields§§ Clinician 5148^^ Unbalanced 2a 12^

“

Clinician 1200~ Balanced 2b 12^

“

Expert 1200~ Balanced

Algorithms/Statistical Classifiers Used:

Naïve Bayes (NB), Naïve Bayes Multinomial (NBM), J48, and Support Vector Machine using radial basis function (SVM_RBF)

* AA, AV, BHP, BBP, CM, DOC, FALL, HAI, MED, NUT, PATH, PU, PC ^ AA, AV, BHP, BBP, CM, DOC, FALL, HAI, MED, NUT, PATH, PU, **500, 500, 500, 500, 500, 500, 500, 361, 500, 250, 306, 500,30 = 5448 ^^500, 500, 500, 500, 500, 500, 500, 361, 500, 250, 306, 500 = 5148 ~100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100 , 100 = 1200

§ 10 categorical and 4 free text = 14 fields §§ 6 categorical and 4 free text = 10 fields

TOOLS AND MEASURES

• WEKA: Classifiers
• Naïve Bayes (NB) & Naïve Bayes Multinomial (NBM)
• Decision Trees (J48) &
• Support Vector Machine with radial basis kernel function (SVM_RBF)
• Standard accuracy measures calculated were:
• Percentage correctly classified, Recall
• Precision, F-measure, Kappa statistic
• Area under curve (AUC) of receiver operating characteristics (ROC)
• Confusion Matrix analysis

1A Recall rate : DT and SVM 2A Recall Rate : DT and SVM

1A Recall rate : NB and NBM 2A Recall Rate : NB and NBM

EXP1: A & B

Algorithms DT NB NBM SVM_RBF CIT 13 12 13 12 13 12 13 12 Accuracy [%] 73.66 75.54 69.71 71.86 78.29 80.44 79.06 68.89 Kappa statistic 0.71 0.73 0.67 0.69 0.76 0.79 0.77 0.66 Precision 0.74 0.74 0.71 0.71 0.79 0.72 0.79 0.79 AUC 0.89 0.89 0.90 0.90 0.96 0.91 0.89 0.89

EXP2: A & B

Algorithms DT NB NBM SVM_RBF

Expert Clinician Expert Clinician Expert Clinician Expert Clinician

Accuracy [%]

70.17 65.91 70.08 69.60 81.32 79.58 54.92 41.12

Kappa statistic

0.68 0.63 0.67 0.67 0.80 0.78 0.51 0.33

Precision

0.70 0.66 0.71 0.71 0.81 0.8 0.69 0.63

AUC

0.89 0.85 0.89 0.91 0.97 0.96 0.41 0.66

RESULTS: EXPERIMENT 1 & 2 - PRECISION SCORE

KEY FINDINGS & WHAT NEXT

• Using Classifiers for Multiclass datasets like Clinical Incident Types in IIMS is achievable
• Confusion matrix is useful in improving the classifiers performance
• Standard measures of performance used in this study are adequate to determine changes
• NBM classifier works well with Clinical Incident Types
• Large dataset can be processed with high accuracy and minimum human resources
• Balanced design and reduction in sample size improves but model's respond to efficiency differently
• Explore application of automation and software architecture improvement
• Explore Improving performance of the classifier further
• Explore application on real time data to drive change in quality of service in healthcare