[PPT] - Effective plots to assess bias and precision in method comparison PowerPoint Presentation

SLIDE 1

Effective plots to assess bias

and precision in method comparison studies

Bern, November, 2016

Patrick Taffé, PhD Institute of Social and Preventive Medicine (IUMSP) University of Lausanne, Switzerland Patrick.Taffe@chuv.ch

SLIDE 2

2

Outline

Bland & Altman’s limits of agreement method (1986)
Extension to proportional bias and heteroscedasticity (1999)
A new methodology to quantify bias and precision
Illustration with a simulated example

SLIDE 3

3

Statistical methods for assessing agreement between two ...

www.ncbi.nlm.nih.gov/pubmed/2868172 by JM Bland - 1986 -

Cited by 35451 -

methods ? Ex: blood pressure

SLIDE 4

4

Bland & Altman (1986) : They wanted a measure of agreement which was easy to estimate and to interpret for a measurement on an individual patient. An obvious starting point was a plot of the differences versus the mean

f the measurements by the two methods :

SLIDE 5

5

The bias (differential bias) between the two measurement methods is estimated by the mean difference :

mean bias

SLIDE 6

6

If the differences are normally distributed, we would expect about 95%

f the differences to lie between the mean +- 1.96*SD, the so called

limits of agreement (LoA) (Bland & Altman, 1986):

SLIDE 7

7

The decision about what is acceptable agreement is a clinical one:

We can see that the blood pressure machine (S) may give values between 55mmHg above the sphygmomanometer (J) reading to 22mmHg below it, => such differences would be unacceptable for clinical purposes

SLIDE 8

8

However, these estimates are meaningful only if we can assume bias and variability are uniform throughout the range of measurement, assumptions which can be checked graphically:

=> assumptions approximatively met

SLIDE 9

9

In some cases the variability of the measurements increases with the magnitude of the latent trait (heteroscedasticity), as well as with the mean difference (proportional bias):

Plasma volume expressed in percentage of normal value: as measured by Nadler and Hurley

2

2 6 10 14 18 difference: Nadler-Hurley 60 80 100 120 140 average: (Nadler+Hurley)/2

Plasma volume data (Bland & Altman, 1999)

SLIDE 10

10

In this case, a linear regression of the differences on the averages can be estimated along with the LoA (Bland & Altman, 1999):

Plasma volume expressed in percentage of normal value: as measured by Nadler and Hurley

2

2 6 10 14 18 difference: Nadler-Hurley 60 80 100 120 140 average: (Nadler+Hurley)/2 difference linear prediction upper 95% LoA lower 95% LoA

Plasma volume data (Bland & Altman, 1999)

SLIDE 11

11

In that case, the LoA are more difficult to interpret (width not constant), and more importantly, there are settings where Bland & Altman’s plots are misleading !

Indeed, we will show that when variances of the measurement errors of the two methods are different, Bland and Altman’s plots may be misleading5

2

2 6 10 14 18 difference: Nadler-Hurley 60 80 100 120 140 average: (Nadler+Hurley)/2 difference linear prediction upper 95% LoA lower 95% LoA

Plasma volume data (Bland & Altman, 1999)

SLIDE 12

12
20
10

10 20 difference: y1-y2 10 20 30 40 50 average: (y1+y2)/2

mu_Y1=23.77 mu_Y2=24.05 sig2_y1=60.90 sig2_y2=84.75 sig2_e1=3.53 sig2_e2=30.44 biais = 0.28

LoA

Simulated examples where the regression line shows an upward or a downward trend but there is no biasJ

20
10

10 20 difference: y1-y2 10 20 30 average: (y1+y2)/2 difference Linear prediction upper2 lower2 zero

mu_Y1=10.33 mu_Y2=10.28 sig2_y1=34.68 sig2_y2=69.12 sig2_e1=0.95 sig2_e2=36.41 biais = -0.04

LoA

20
10

10 20 difference: y1-y2

10

10 20 30 average: (y1+y2)/2 difference Linear prediction upper2 lower2 zero

mu_Y1=9.61 mu_Y2=9.99 sig2_y1=60.56 sig2_y2=34.69 sig2_e1=26.25 sig2_e2=1.00 biais = 0.38

LoA

SLIDE 13

13
r a zero slope and there is a biasJ
20
10

10 20 difference: y1-y2 10 20 30 40 50 average: (y1+y2)/2

mu_Y1=23.74 mu_Y2=23.73 sig2_y1=101.30 sig2_y2=97.90 sig2_e1=3.47 sig2_e2=23.93 biais = -0.01

LoA

20
10

10 20 difference: y1-y2 10 20 30 40 50 average: (y1+y2)/2

mu_Y1=25.99 mu_Y2=26.05 sig2_y1=93.75 sig2_y2=92.50 sig2_e1=5.02 sig2_e2=25.73 biais = 0.06

LoA

SLIDE 14

14

Accepted September 2015, online October 2016

Therefore, the goal of my presentation is to introduce a new methodology for the evaluation of the agreement between two methods

f measurement, where the first is the and the other

the to be evaluated:

Effective plots to assess bias and precision in method comparison studies

Patrick Taffé Institute for Social and Preventive Medicine, University of Lausanne, Switzerland Patrick.Taffe@chuv.ch

SLIDE 15

15

More specifically, the objectives of this new methodology are to

identify and quantify the amounts of differential and proportional

biases,

develop a method of recalibration in order to correct the bias of the

new measurement method,

and compare its precision with that of the reference standard.

The methodology requires several measurements by the reference standard and possibly only one by the new method for each individual. It is applicable in all circumstances with or without differential and/or proportional biases and when the measurement errors are either homoscedastic or heteroscedastic.

SLIDE 16

16

Get ready !

SLIDE 17

17

2 The measurement error model 2.1 Formulation of the model 6

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7 7 7

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

8 .

5
σ
*

#

9 9 #(

#

=
#
=

*

7

99 7

(9*
+*

#

ε

7 ε 9 #7

SLIDE 18

18

: (

#

7 #

. ' 5

ε

σ θ

7

7 7

. ' 5

ε

σ θ

( +* (9

* (

#

θ

7

θ

+*

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7 7 7

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

8 .

5
σ

SLIDE 19

19

; 7

7

3 α =

7

# β =

* #

#

α

#

β 9 ( 9

< ( (9
(

7

σ

: (9(

≡

. 9-5

" 7 # * 9

( 6

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 20

20

" ( 9*

#

9
: 9 *( *+ 4
9 9 *

*

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 21

21

)* .73#=5 99(-+

< 9 (

+ > !

4>
" (( * 9*4

* (9

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 22

22

2.2 Estimation of the model : *

7

9
+
4
#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 23

23

< *

9

9 * 9 9 .!?,5

6
7

@ . A 5

=

y

7

7

7 7

. A 5 . 5 . A 5 . 5

= ∫ ∫

y y

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 24

24

: * >

# #

α

#

β 9 9 %! ( 9 !?, @

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 25

25

B #

@ α

B #

@ β

9 9 * 6

B

B # #

@ @ @ . #5

α

β = + −

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 26

26

( (C 9 * . #5 D9E

2

2 4 bias 20 40 60 80 y1 and y2 10 20 30 40 BLUP of x y2 Reference standard y1 New method Bias

Bias plot

Bias = 20% Bias = 10%

SLIDE 27

27

2.3 Recalibration of the new method <( 9 * *996

B

B B # # # #

@ @ . 5F

α

β = −

)*

7

B

#

* ( *

SLIDE 28

28

2 4 6 8 10

standard deviation of the measurement errors

10 20 30 40 BLUP of x estimated value true value (reference standard y2) estimated value true value (new method recalibrated y1_corr)

Precision plot after recalibration

" (9+ (

#

@ @ . ' 5

ε

σ θ

7

7

@ @ . ' 5

ε

σ θ ( @

* *

DE6

SLIDE 29

29

2.4 Why Bland and Altman’s plot may be misleading (

# 7

=

− ( (

# 7

. 5 F 7

=

+ 9*

!
The problem is that the regression line may show a positive or negative

slope when there is no bias or have a zero slope in the presence of a bias.

<
6
α

β ε = + +

99-
2

2 6 10 14 18 difference: Nadler-Hurley 60 80 100 120 140 average: (Nadler+Hurley)/2 difference linear prediction upper 95% LoA lower 95% LoA

Plasma volume data (Bland & Altman, 1999)

SLIDE 30

30

%!(9* 6

#

7

7 # # 7 7 7

. ' 5 (.

5

3 . ' 5

ε

ε

σ β ε σ β = ⇔ = θ θ

9* ( (

> 9

G

#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 31

31

3 A simulation study

(

99 * * (*

C#33(
9* #3 #= ( 9
*#9 *

SLIDE 32

32

;* *6

7

# # #

H #7

8

.3.#I3# 5 5

ε

ε = − + +

7

7 7 7

8

.3.7I37 5 5

ε

ε = +

8

J#3 H3K

−
*

##33 =

9( * 8 J#3 #=K

−
< * 94H9#7
< ( #

7

SLIDE 33

33

< L ! - * 96

40
20

20 40 difference: y1-y2 10 20 30 40 50 60 average: (y1+y2)/2

LoA

SLIDE 34

34

* 9 * 77 ( 9 96

2

2 4 bias 20 40 60 80 y1 and y2 10 20 30 40 BLUP of x y2 Reference standard y1 New method Bias

Bias plot

SLIDE 35

35

*

9420=/=M:NJ410#'4300K.(4H5
9##//=M:NJ#30'#7/K.(#75
#

7 # # # # # #

8

.3 . ' 55

ε

α β ε ε σ = + + θ

7

7 7 7 7 7

8

.3 . ' 55

ε

ε ε σ = + θ

7

8 .

5
σ

SLIDE 36

36

< ( 9 * * #38#=9 #9 * 6

2

4 6 8 10

standard deviation of the measurement errors

10 20 30 40 BLUP of x estimated value true value (reference standard y2) estimated value true value (new method recalibrated y1_corr)

Precision plot after recalibration

true value estimated value

SLIDE 37

37

; *(C 96

20

40 60 measurement method 10 20 30 40 BLUP of x y2 Fitted values y1 Fitted values y1_corr Fitted values

Comparison of the methods

SLIDE 38

38

"L! 9 99 9( 96

30
20
10

10 20 difference: y1-y2 10 20 30 40 50 average: (y1+y2)/2

LoA for the recalibrated method

SLIDE 39

39

In summary, We have developed a new methodology to assess the bias and precision of a new measurement method relative to the reference standard, which does not have the shortcomings of Bland and Altman’s LoA methodology. It is, however, in spirit of the original paper in the sense that new graphical representations of the bias and of the performance of the method to be evaluated are proposed. In addition, we have shown a very simple way to recalibrate the new method to be able to use it in place of the more complex and costly reference standard.

SLIDE 40

40

biasplot: A Stata package to effective plots to assess bias and precision in method comparison studies

,+<O

:,((?(!*C ,+<P (

+,

?( +P

Q+
Q+P
<"

?( <*P

☺

SLIDE 41

41

Thank you for your attention ☺ ☺ ☺ ☺