Federal Institute for Vaccines and Biomedicines Statistical - - PowerPoint PPT Presentation

▶
federal institute for vaccines and biomedicines
SMART_READER_LITE
LIVE PREVIEW

Federal Institute for Vaccines and Biomedicines Statistical - - PowerPoint PPT Presentation

Oct 19, 2022 272 likes •512 views

Federal Institute for Vaccines and Biomedicines Statistical Considerations in Setting Acceptance Criteria Statistical Considerations in Setting Acceptance Criteria 1. Reference intervals (and outlier tests) 2. Tolerance intervals 3. Process

slide-1
SLIDE 1

Federal Institute for Vaccines and Biomedicines

09.09.2011 Kay-Martin Hanschmann 1

1. Reference intervals (and outlier tests) 2. Tolerance intervals 3. Process performance indices, 6 sigmas, … 4. Summary Statistical Considerations in Setting Acceptance Criteria Statistical Considerations in Setting Acceptance Criteria

Disclaimer

The views expressed here are those

  • f the author and may not necessary

reflect those of the German Regulatory Authorities

slide-2
SLIDE 2

Federal Institute for Vaccines and Biomedicines

09.09.2011 2

Reference Intervals Reference Intervals

  • Mean ±

2 standard deviations (Mean ± 3 s)

  • Easy to implement
  • Data should be symmetrically (ideal: normally) distribu
  • Otherwise: Transformation of the data
  • Reliable, unbiased estimation of mean and

standard deviation necessary

Kay-Martin Hanschmann

slide-3
SLIDE 3

Federal Institute for Vaccines and Biomedicines

09.09.2011 3

1-2-3-σ-Rule

Reference Intervals include a certain amount of the data (assumption: normal distribution)

95% of the data can be found in the interval [Mean - 2 s, Mean + 2s] Or: With 95% probability a value lies in this interval

  • 1. Reference Intervals

Interval Probability Mean ± … Within Outside … 1 s 0.683 0.317 … 2 s 0.954 0.046 … 3 s 0.998 0.003 … 1.64 s 0.900 0.100 … 1.96 s 0.950 0.050 … 2.58 s 0.990 0.010

Kay-Martin Hanschmann

slide-4
SLIDE 4

Federal Institute for Vaccines and Biomedicines

measured Percent

  • 100

100 200 300 5 10 15 20

09.09.2011 4 Kay-Martin Hanschmann

Mean ± 2 SD covers about 95% of the data Mean ± 3 SD covers about 99% of the data

  • 1. Reference Intervals

Mean ± 6 SD covers…? – possibly too much!

slide-5
SLIDE 5

Federal Institute for Vaccines and Biomedicines

measured Percent

50 100 150 200 250 300 350 400 450 500 5 10 15 20 25

09.09.2011 5 Kay-Martin Hanschmann

  • 1. Reference Intervals

Skewed distributions / non-normal distributed data:

Transformation of data (→ skewed specification)

Specification 39.8 – 251.2 (Mean ± 2s, of logarithmised data)

slide-6
SLIDE 6

Federal Institute for Vaccines and Biomedicines

09.09.2011 6 Kay-Martin Hanschmann

  • 1. Reference Intervals

How much data needed to set up specifications?

N=10,000 N=100 N=12

95%-Confidence Interval [99.3 – 100.5] [29.6 – 30.4]

measured Percent

50 100 150 200 5 10 15 20

95%-Confidence Interval [95.0 – 106.8] [26.3 – 34.8]

measured Percent

50 100 150 200 5 10 15 20

95%-Confidence Interval [76.6 – 115.8] [21.9 – 52.5]

measured Percent

50 100 150 200 5 10 15 20

slide-7
SLIDE 7

Federal Institute for Vaccines and Biomedicines

09.09.2011 7 Kay-Martin Hanschmann

  • 1. Reference Intervals

Example Reference Interval I

Mean +/-2s as “warning” limits, Mean +/-3s as “intervention” limits (validated with 10 samples)

# batches tested

2 4 6 8 10 12 14 16 18 20

  • 60
  • 40
  • 20

20 40 60 80

measured mean +/- 2s mean +/- 3s

VALIDATION “bad“ batches (sub-potent, not safe, …) “bad“ batches (sub-potent, not safe, …) “grey area” “grey area”

slide-8
SLIDE 8

Federal Institute for Vaccines and Biomedicines

09.09.2011 8 Kay-Martin Hanschmann

  • 1. Reference Intervals

# batches tested

2 4 6 8 10 12 14 16 18 20

  • 60
  • 40
  • 20

20 40 60 80

measured mean +/- 2s mean +/- 3s

VALIDATION

Example Reference Interval II

Mean +/-2s as “warning” limits, Mean +/-3s as “intervention” limits (validated with 10 samples)

slide-9
SLIDE 9

Federal Institute for Vaccines and Biomedicines

09.09.2011 9 Kay-Martin Hanschmann

  • 1. Outlier tests

Outliers and outlier tests

Is it an outlier? – Or does it belong to the population?

It depends on how much information we have…

Torere, Bay of Plenty, New Zealand

slide-10
SLIDE 10

Federal Institute for Vaccines and Biomedicines

09.09.2011 10 Kay-Martin Hanschmann

  • 1. Outlier tests

There are several outlier tests… but be careful using them!

1.30 1.35 1.40 1.45 1.50 1.55 1.60 1.65

  • utlier?

1.30 1.35 1.40 1.45 1.50 1.55 1.60 1.65 1.30 1.35 1.40 1.45 1.50 1.55 1.60 1.65

  • utlier?
  • utlier?

1 2 3 4

  • utlier

according Dixon’s test

slide-11
SLIDE 11

Federal Institute for Vaccines and Biomedicines

09.09.2011 11 Kay-Martin Hanschmann

Tolerance Intervals Tolerance Intervals

Intervals that cover percentiles of the population with a certain probability

Non-parametric TI – percentiles:

  • Example: [p0.05

– p0.95 ] might serve as TI for the mean 90% of the population

  • Example: Smallest –

largest observation [y(1) – y(n) ] might serve as TI for whole population

  • BUT: To cover actually 90% of the population with [y(1)

– y(n) ], N=19 measurements are necessary

  • AND: To cover actually 90% of the population with [y(1)

– y(n) ] with 95% probability, N=46 measurements are necessary

slide-12
SLIDE 12

Federal Institute for Vaccines and Biomedicines

09.09.2011 12 Kay-Martin Hanschmann

  • 2. Tolerance Intervals

TI (2–sided) for normal distributed data:

Guttman (1970), Rasch (1996)

slide-13
SLIDE 13

Federal Institute for Vaccines and Biomedicines

09.09.2011 13 Kay-Martin Hanschmann

  • 2. Tolerance Intervals

TI (2–sided) for normal distributed data:

Howe (1969)

slide-14
SLIDE 14

Federal Institute for Vaccines and Biomedicines

09.09.2011 14 Kay-Martin Hanschmann

  • 2. Tolerance Intervals

# batches tested

2 4 6 8 10 12 14 16 18 20

  • 60
  • 40
  • 20

20 40 60 80

measured Tolerance Interval

slide-15
SLIDE 15

Federal Institute for Vaccines and Biomedicines

09.09.2011 15 Kay-Martin Hanschmann

  • 2. Tolerance Intervals

# batches tested

2 4 6 8 10 12 14 16 18 20

  • 60
  • 40
  • 20

20 40 60 80

measured Tolerance Interval

Dynamic Tolerance Intervals

What about early OOS results? May belong to population –

  • r may be OOS!

# batches tested

2 4 6 8 10 12 14 16 18 20

  • 60
  • 40
  • 20

20 40 60 80

measured Tolerance Interval

slide-16
SLIDE 16

Federal Institute for Vaccines and Biomedicines

09.09.2011 16 Kay-Martin Hanschmann

  • 2. Tolerance Intervals

Dynamic Tolerance Intervals

TI widens, when data shows a trend – may lead to undesired effects

# batches tested

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

  • 60
  • 40
  • 20

20 40 60 80 100 120

measured Tolerance Interval

slide-17
SLIDE 17

Federal Institute for Vaccines and Biomedicines

09.09.2011 17 Kay-Martin Hanschmann

  • 2. Tolerance Intervals

Fixed Tolerance Intervals

With a sufficient amount of validation data the TI will be more reliable

# batches tested

2 4 6 8 10 12 14 16 18 20

  • 60
  • 40
  • 20

20 40 60 80

measured Tolerance Interval

VALIDATION

slide-18
SLIDE 18

Federal Institute for Vaccines and Biomedicines

09.09.2011 18 Kay-Martin Hanschmann

  • 2. Tolerance Intervals

Two-Step Approach

# batches tested

2 4 6 8 10 12 14 16 18 20

  • 60
  • 40
  • 20

20 40 60 80

measured Tolerance Interval

VALIDATION RE-VALIDATION

slide-19
SLIDE 19

Federal Institute for Vaccines and Biomedicines

09.09.2011 19 Kay-Martin Hanschmann

Process performance indices, 6 sigmas, … Process performance indices, 6 sigmas, …

  • PPI: Similar to reference limits (mean ±

3SD x Ppu)

  • Advantage to be product specific
  • BUT: Wouldn’t such limits be to wide (and could

include e.g. OOS batches)?

  • At least: Limits must exclude range, where sub-

potency, non-safety, … starts

  • Thus: 6 sigmas are not recommended (for normal
  • r similar distributed data these would include

anything)

slide-20
SLIDE 20

Federal Institute for Vaccines and Biomedicines

09.09.2011 20 Kay-Martin Hanschmann

Crucial: Reliable estimation of the specification limits

  • Too narrow intervals could result in too many re-tests and / or falsely

rejected batches

  • Too wide intervals could lead to falsely released batches (sub-potent,

safety concerns, …)

Summary Summary

  • Specification based on what we have observed so far (might be few),

thus future results may represent what we missed in validation

  • Samples used for validation were too homogeneous (too narrow

specifications) – bad luck?

  • Samples used for validation had high variability (new processes,

untrained personnel, too wide specifications)

  • Validation performed with few samples, repeatedly tested (should

be avoided!)

slide-21
SLIDE 21

Federal Institute for Vaccines and Biomedicines

09.09.2011 21 Kay-Martin Hanschmann

  • For a complete new product it will be difficult to set up reliable

specification limits with only 3 batches/tests

  • risk of miss-specification might be high, especially if variability
  • f parameter of interest is expected to be high; a re-validation

should be planned (n=8-12 batches) → 2-step-approach

Summary II Summary II

  • No universal tool available –

parameter / product dependent

  • Aim to obtain high sensitivity to detect critical batches (sub-

potent, safety risks, …) and

  • To obtain high specificity in order to avoid false negative

results and to limit unnecessary re-tests

slide-22
SLIDE 22

Federal Institute for Vaccines and Biomedicines

09.09.2011 22

Thank you for your attention 

Kay-Martin Hanschmann