Quality Assurance of Monitoring/Surveillance Data Takanori UKENA, - - PowerPoint PPT Presentation

June 12, 2013

Quality Assurance of Monitoring/Surveillance Data

Takanori UKENA, Ph.D.

takanori_ukena@nm.maff.go.jp

Ministry of Agriculture Forestry and Fisheries Food Safety and Consumer Affairs Bureau

1

MAFF

Contents

Importance of Quality Assurance

• 1. Laboratory Management, ISO/IEC 17025
• 2. Use of Validated Analytical method
• 3. Internal Quality Control and Proficiency

Testing

• 4. Measurement Uncertainty
• 5. Evaluation of Analytical Results

Experience of MAFF Japan

2

MAFF

Importance of Quality Assurance

3

MAFF

What is Quality?

“degree to which a set of inherent characteristics fulfils requirement". ⇒ In case of analysis “delivery of reliable data/results within an agreed span of time under agreed conditions, at agreed costs, and with necessary aftercare". . The “agreed conditions” should include a specification as to the precision and accuracy of the data

4

(ISO 9000)

MAFF

• Quality

Management

• Quality

Assurance

• Quality Control

(QC)

5

MAFF

Quality Management

(ISO/IEC 17025:2005)

Quality management:

coordinated activities to direct and control an organization with regard to quality

6

MAFF

(ISO 9000:2005) The laboratory shall establish, implement and maintain a management system appropriate to the scope of its activities.

Quality management system:

management system to direct and control an organization with regard to quality

Note : generally includes establishment of the quality policy, and quality

• bjective, quality planning, quality control, quality assurance, and quality

improvement.

Quality Assurance

Quality assurance:

All those planned and systematic actions necessary to provide adequate confidence that analytical results will satisfy given requirements for quality.

GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009)

7

MAFF

part of quality management focused on providing confidence that quality requirement will be fulfilled

(ISO 9000:2005)

Quality Control

Quality Control: part of quality management focused on fulfilling quality requirement

8

MAFF

(ISO 9000:2005)

for chemical analysis laboratories ⇒ Internal Quality Control and Proficiency testing.

Why quality assurance is needed?

• Prevention of quality

problems

Risk manager need to make decision based on data

• btained from other persons, other laboratories,
• ther countries.
• Basis of cconfidence in the data
• International requirement

9

MAFF

What is reliable analytical results?

• Use of validated method

If you don’t use validated methods, only you will know how good your numbers are.

• By a laboratory with Quality Management

Internal quality control and proficiency testing

Fit for purpose

Data produced by a measurement process enables a user to make technically and administratively correct decisions for a stated purpose.

10

• Back ground information/data

MAFF

Use of analytical results

• Conformity assessment

Import/export control

• Risk management activities

Occurrence data for estimation of exposure Evaluation of effectiveness of risk management options

• International activities

GEMS/Food, Codex, JECFA, JMPR, etc.

11

MAFF

Inadequate data (example)

The data from Brazil and Uruguay could not be used as they had been obtained by analytical methods with high LODs, ranging from 5 µg/kg to 50 µg/kg, which are inadequate to detect and measure ochratoxin A at 0.94 µg/kg and 0.19 µg/kg, the weighted mean concen-trations found in Europe and the USA in cereals and cereal products, respectively. 5.3 Distribution (JECFA 47, 2001)

12

MAFF

Codex guidelines for Import and Export Control testing laboratories

• Compliance with the general criteria in

ISO/IEC 17025

• Participation in appropriate proficiency

testing schemes

• Use of validated method
• Internal quality control procedures

Guidelines for the Assessment of the Competence of Testing Laboratories Involved in the Import and Export Control of Food (CAC/GL 27-1997)

13

MAFF

Food Control Laboratory Management: Recommendations

• International Harmonized Protocol for the Proficiency

Testing Pure & Appl. Chem., 65 (1993) 2132-2144

• Protocol for the Design, Conduct and Interpretation of

Method Performance Studies

Pure & Appl. Chem., 67 (1995) 331-343

• Harmonized Guidelines for Internal Quality Control

Pure & Appl. Chem., 67 (1995) 649-666 Food Control Laboratory Management: Recommendations (CAC/GL 28-1995, Rev.1-1997)

Protocols and Guidelines adopted in Codex

14

MAFF

Guidelines for Settling Disputes over Analytical (Test) Results

the following information should be shared between competent authorities of the importing and exporting country

(CAC/GL 70-2009)

• validation status of the analytical method used
• raw data
• internal quality assurance/control

(control charts, sequence of analysis, blank data, recovery data, etc.)

• performance in relevant proficiency testing or

collaborative studies;

• official accreditation status of the laboratories.

15

MAFF

Topic １

Laboratory Management, ISO/IEC 17025

16

MAFF

Why is ISO/IEC 17025 important?

• Requirements of Codex guidelines
• ISO an International standard / international

recognition Benefit of accreditation for laboratory

• A marketing advantage for laboratory
• ften required for international trade
• A bench mark for performance

In depth assessment Required periodic reassessment An indicator of technical competence

17

MAFF

Requirements of ISO/IEC 17025

Management (15)

• Organization
• Management system
• Document control
• Review of requests/contracts
• Subcontracting of tests
• Purchasing services/suppliers
• Service to the customer
• Complaints
• Control of non-conforming tests
• Improvement
• Corrective action
• Preventive action
• Control of records
• Internal audits
• Management review

Technical (10)

• General
• Personnel
• Accommodation and environmental

conditions

• Test method and method validation
• Equipment
• Measurement traceability
• Sampling
• Handling of test and calibration

items

• Assuring quality of test and

calibration results

• Reporting the results

18

MAFF

What is Accreditation of ISO/IEC 17025?

• Formal recognition by an authoritative body of the competence of a

laboratory to carry out specific tests or types of tests in a reliable, credible and accurate manner

• Evidence of Quality, Credibility and Reliability
• technical competence of staff
• validity and appropriateness of test methods
• suitability, calibration and maintenance of equipment
• appropriate handling of test materials
• quality control procedure, e.g., use of CRM, Proficiency testing, etc.
• Actual performance and ethical behavior is critical.

Not a guarantee

19

MAFF

Accredited to each analytical method each laboratory

Need to confirm :

• For what method the laboratory is accredited?
• Which laboratory is accredited?

If the organization has blanch laboratories.

Notes for Accreditation of ISO/IEC 17025

20

MAFF

ILAC, APLAC

ILAC: International Laboratory Accreditation Cooperation The international organization on laboratory accreditation with membership of accreditation bodies and affiliated organizations. Maintain a Mutual Recognition Agreement (MRA) among accreditation bodies to facilitate trade and to eliminate the need for duplicate accreditation. Assist developing accreditation programs APLAC: Asia Pacific Laboratory Accreditation Cooperation

21

MAFF

ILAC-MRA, APLAC-MRA

• A laboratory accredited by one MRA partner has

equivalent competence to laboratory accredited by the

• ther MRA partner(s)

https://ilac.org/home.html http://www.aplac.org/home.html

• Accreditation body shall comply with ISO/IEC 17011

and that its accredited facilities are in compliance with ISO/IEC 17025 (laboratories), ISO 15189 (medical laboratories) and ISO/IEC 17020 (inspection bodies).

22

MAFF

“Tested once, accepted everywhere”

Topic 2

Use of Validated Analytical method

23

MAFF

Chemical Analysis

Qualitative

• the presence or absence of a particular substance, but

not the mass or concentration.

• Identification of the substance

e.g., structure elucidation of natural product

Quantitative

• determination of the absolute or relative abundance

(concentration) of particular substance(s) present in a sample.

24

MAFF

Analytical method for Risk Management

Analytical Methods

• Whenever
• Whoever (analytical specialist)

can follow the method procedure and obtain similar results for the same sample.

(e.g., variability of results fall in expected interval).

25

MAFF

Validated method

“If you don’t use validated methods, only you will know how good your numbers are.” AOACI

Validated Test Method: An accepted test method for which validation studies have been completed to determine the accuracy and reliability of this method for a specific purpose.

Reference: ICCVAM Guidelines for the nomination and submission of new, revised and alternative

Guidelines on Analytical Terminology (CAC/GL 72-2009)

26

MAFF

• Validated method is a method tested in many

laboratories on the same samples, showing that the method is rugged enough to produce comparable results in different labs, with different operators.

Method Validation Study

Collaboration study, Ring-test- trial

27

MAFF

Validation Study Protocol and Guidelines

• AOAC International

AOAC Official Methods of Analysis (2002). Interlaboratory Collaborative Study, Appendix D: Guidelines for Collaborative Study Procedures To Validate Characteristics of a Method of Analysis http://www.aoac.org/vmeth/guidelines.htm

• ISO 5725-2:1994

Accuracy (trueness and precision) of measurement methods and results -- Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method

• IUPAC

Protocol for the Design, Conduct and Interpretation of Method- Performance Studies. Pure & Appl. Chem., 67(2), 331-343 (1995)

28

http://www.iupac.org/publications/pac/67/2/0331/

MAFF

Comparison of Protocol for the study

* test statistics and application procedure are different between the protocols . 29

MAFF

General Criteria in Codex Procedure Manual

Single-laboratory Validated Method

single-laboratory validated methods must fulfill the following criteria:

(i)the method is validated according to an

internationally recognized protocol (e.g. IUPAC Guidelines) (ii) the use of the method is embedded in a quality system in compliance with the ISO/IEC 17025

30

MAFF

The method should be complemented with information on accuracy demonstrated for instance with:

–regular participation in proficiency schemes, where

available;

–calibration using certified reference materials, where

applicable;

–recovery studies performed at the expected

concentration of the analytes;

–verification of result with other validated method where

available.

31

(Codex Procedural Manual)

MAFF

• Harmonized Guidelines for Single-Laboratory Validation
• f Methods of Analysis

IUPAC/ISO/AOAC, 2002

Guidelines on Single-Laboratory Validation

Performance characteristics

• Applicability
• Selectivity
• Calibration and linearity
• Trueness, Precision
• Recovery
• Limit of Detection, Limit of Quantification
• Sensitivity
• Ruggedness

etc.

32

MAFF

Recommended methods in Codex

Recommended Methods of Analysis and Sampling (CODEX STAN 234-1999) General Methods of Analysis for Contaminants (CODEX STAN 228-2001)

33

MAFF

Criteria approach

• Identify method performance characteristics based
• n existing method validation data.

and establish criteria for evaluating acceptable method of analysis ⇒ laboratory can choose a method meeting criteria

(flexibility)

34

MAFF

Guidelines for Criteria approach

<Codex Procedural Manual> Working Instructions for the Implememtation of the Criteria Approach in Codex Guidelines for Establishing Numeric Values for Method Criteria and/or Assessing Methods for Compliance Thereof

35

MAFF

Examples for numeric values for the criteria

* The sR should be calculated from the Horwitz / Thompson equation 36

MAFF

Terms related to variability of analytical results

37

MAFF

Terms related to variability of test results

Accuracy Trueness Precision

Repeatability Reproducibility Measurement Uncertainty Intermediate precision

38

MAFF

Distribution of results for repeated analysis

m

A C

µ: True value m: mean X-axis: concentration Y-axis: frequency

B D

µ µ µ µ m m m

39

MAFF

accuracy, trueness, precision

accuracy trueness precision ○ ○ ○ × ○ × × × ○

µ (true value); m (mean); σ (standard deviation); c (m+s)

m m

σ

m

s

40

MAFF

A B C

Accuracy

The closeness of agreement between a test result or measurement result and a reference value.

• When applied to a test method, the term accuracy

refers to a combination of trueness and precision.

Guidelines on Analytical Terminology (CAC/GL 72-2009)

41

MAFF

A reference value is usually provided with reference to a) a certified reference material; b) a reference measurement procedure; c) a comparison of measurement standards.

Trueness

The closeness of agreement between the average

• f an infinite number of replicate measured

quantity values and a reference quantity value.

• Measurement trueness is inversely related to

systematic measurement error, but is not related to random measurement error.

• Measurement accuracy should not be used for

‘measurement trueness’ and vice versa.

Guidelines on Analytical Terminology (CAC/GL 72-2009)

42

MAFF

Precision

The closeness of agreement between independent test/measurement results obtained under stipulated conditions.

• depends only on the distribution of random errors and does

not relate to the true value or to the specified value.

• usually expressed in terms of imprecision and computed as

a standard deviation of the test results. Less precision is reflected by a larger standard deviation.

• Quantitative measures of precision depend critically on the

stipulated conditions. Repeatability and reproducibility conditions are particular sets of extreme conditions.

Guidelines on Analytical Terminology (CAC/GL 72-2009)

43

MAFF

Repeatability

Precision under repeatability conditions.

Repeatability conditions:

Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in the same test or measuring facility by the same operator using the same equipment within short intervals of time.

Guidelines on Analytical Terminology (CAC/GL 72-2009)

44

MAFF

Reproducibility

Precision under reproducibility conditions.

Reproducibility conditions:

Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in different test or measurement facilities with different operators using different equipment.

Guidelines on Analytical Terminology (CAC/GL 72-2009)

45

MAFF

Relative standard deviation

RSDｒ : Repeatability relative standard deviation RSDR : Reproducibility relative standard deviation Relative standard deviation (%) = standard deviation (SD) / mean ×100

same as CV (coefficient of variation)

1

2 2 1

  + +   n ) μ x ( ) μ x ( SD

n



46

MAFF

Horwitz equation

RSDR （%） ＝ 2 C -0.1505

RSDR （%）: reproducibility relative standard deviation C：concentration ratio

if concentration ratio =100 %, then C = 1

1 %, C = 0.01 A ppm C = A×10-6

C .

• R

RSD

10 log 5 1

2 (%) 

Horwitz equation approximation of Horwitz equation

47

MAFF

Graphical expression of Horwitz equation

48

MAFF

• 200
• 150
• 100
• 50

50 100 150 200 相対標準偏差 [%]

1 ppm 1 ppb 1 ppt 1% 16% 45% 128% 4%

RSDR

Horwitz/Thompson equation

RSDR （％）＝

Thompson, M., Analyst, 125, 385-386 (2000)

22 C-0.5 2 C-0.1505 C < 1.2×10-7 C > 0.138 1.2×10-7 ≤ C ≤ 0.138

RSDR （%）: Reproducibility relative standard deviation C：concentration ratios

49

MAFF

Thompson, M., Analyst, 125, 385-386 (2000)

• 30
• 20
• 10

10 20 30

Concentration

Relative standard deviatio

1 ppt 1 ppb 1%

16% 22%

1 ppm 0.1ppm

Graphical expression of Horwitz/Thompson equation

50

MAFF

HorRat Value

• HorRat(R) = RSDR/PRSDR HorRat(r) =RSDr/PRSDR

RSDr, RSDR: observed relative standard deviation (sr/C, sR/C) PRSDR: predicted relative standard deviation calculated from Horwitz/Thompson

• Normal range of HorRat

The ratio of the reproducibility relative standard deviation to that calculated from the Horwitz equation,

Guidelines on Analytical Terminology (CAC/GL 72-2009)

51

MAFF

Limit of Detection (LOD) and Limit of Quantification (LOQ)

52

MAFF

What is Limit of Detection?

Limit of Detection (LOD) :

The true net concentration or amount of the analyte in the material to be analyzed which will lead, with probability (1-β), to the conclusion that the concentration or amount of the analyte in the analyzed material is larger than that in the blank material.

Critical Value (Lc) :

The value of the net concentration or amount the exceeding of which leads, for a given error probability α, to the decision that the concentration or amount of the analyte in the analyzed material is larger than that in the blank material.

53

MAFF

Guidelines on Analytical Terminology (CAC/GL 72-2009)

LOD and Critical Value (Lc) Lc

Results of blank α: Type I error (false positive) β: Type II error (false negative) Results of LOD concentration sample

LOD Sample of which concentration > LOD will lead to the decision that the concentration of the material is larger than that in the blank material with probability larger than 1-β If results > Lc, then they will lead to the decision that it is not blank, detected with probability larger than 1- α. If results are lower than Lc, they are N.D. (could not decided to be detected)

54

β

MAFF

The limit of detection LOD is estimated by, LOD ≈ 2×t1-αν σo [where α = β],

—Where t1-αν : Student's-t, based on ν degrees of freedom for a one- sided confidence interval of 1-α σo : the standard deviation of the true value (expectation).

LOD = 3.29 σo, when the uncertainty in the mean (expected) value of the blank is negligible, α = β = 0.05

• Usually estimated by using variance of sample (SD of True value

is not known.)

• LOD depends on probability (α and β)
• Calculate t-value depending on number of data

(LOD should not always equals to 3.29×s (SD. observed))

55

MAFF

α=0.05: Degree of Freedom = 1 ⇒ t 0.95,1 = 6.31 (2×t-value=12.62) Degree of Freedom = 10 ⇒ t 0.95,9 = 1.81 (2×t-value=3.62) Degree of Freedom = ∞ ⇒ t 0.95,∞= 1.64 (2×t-value=3.29)

t-distribution, df=１ t-distribution, df=3 t-distribution, df =10 Standard normal distribution

t-Distribution and Standard Normal Distribution

56

MAFF

Limit of Quantification

Limit of Detection (LOD) :

A method performance characteristic generally expressed in terms of the signal or measurement (true) value that will produce estimates having a specified relative standard deviation (RSD), commonly 10% (or 6%). LOQ is estimated by:

LOQ = kQ ×σQ , —Where kQ = 1/RSDQ

If σ is known and constant (SD of the estimated quantity is independent of concentration) , and σQ = σo : substitution of kQ = 10% ⇒ LOQ = (10×σQ) = 10 σo （3.04 ×LOD, α = β = 0.05）

57

MAFF

Guidelines on Analytical Terminology (CAC/GL 72-2009)

Example of Method Criteria at Different Concentrations

58

(Codex Procedural Manual)

MAFF

Topic 3 Internal Quality Control and Proficiency testing

59

MAFF

3-1 Internal Quality Control (IQC)

60

MAFF

Harmonized Guidelines for Internal Quality Control in Analytical Chemistry Laboratories, Pure & Appl. Chem. 67 (1995) 649-666. Introduction Definition Quality assurance practices and internal quality control Internal quality control procedure IQC and within-run precision Control materials in IQC Recommendations Conclusions

Guidelines on internal Quality Control

61

MAFF

Definition

Internal Quality Control

Internal Quality Control： Set of procedures undertaken by laboratory staff for the continuous monitoring of operation and the results of measurements in order to decide whether results are reliable enough to be released.

62

MAFF

Control materials in IQC

Control materials are characterized substances that are inserted into the run alongside the test materials and subjected to exactly the same treatment.

• appropriate concentration of the analyte
• same matrix in terms of bulk composition, including minor

constituents

• similar physical form
• stable and possible to divide the material into effectively

identical portions for analysis

63

MAFF

Certified reference material (CRM):

Reference material accompanied by documentation issued by an authoritative body and providing one or more specified property values with associated uncertainties and traceability, using valid procedures

Documentation is given in the form of a “certificate” (ISO guide 30:1992). Procedures for the production and certification of certified reference materials are given, e.g. in ISO Guide 34 and ISO Guide 35.

Use of Certified reference material (1)

GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009)

64

MAFF

Metrological Traceability:

Property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the stated measurement uncertainty.

Metrological traceability of a measurement result does not ensure that the measurement uncertainty is adequate for a given purpose or that there is an absence of mistakes.

Metrological Traceability

Guidelines on Analytical Terminology (CAC/GL 72-2009)

65

MAFF

Comparability of measurement result, for quantities of given kind. (metrologically traceable to the same reference)

Metrological Traceability (2)

Definition of unit, SI base Primary national standard Reference standards Working standards Measurement result

BIPM

Primary national standard (other countries)

End user National Metrology Institute, Designated Institute Calibration

• rganization

Competent authority, Industry, Academia

Larger Measurement Uncertainty in lower stage

66

MAFF

Use of Certified reference material (2)

• COMAR (International Database)

http://www.comar.bam.de/en/

• Purchase from reagents manufacturer or

reagents sales company

• Comparison of the measurement results with

the certified value

67

MAFF

Comparison of the measurement results with the certified value

CRM m m

C C   Δ

http://www.erm-crm.org/ERM_products/application_notes/Pages/index.aspx

2 2 Δ CRM m

u u u + 

Δm：absolute difference between mean measured valued and certified value Cm: mean measured value CCRM : Certified Value UΔ： expanded measurement uncertainty um: measurement uncertainty uCRM : uncertainty of the certified value

If Δm ≤ UΔ then there is no significant difference between the measurement result and certified value.

Δ Δ

2u U 

68

MAFF

Constraint on the use of CRM

⇒ Preparation of House reference material

e.g. materials in proficiency testing, spiked control materials, etc.

• For the majority of analysis there is no

closely matching CRM available.

• Cost to stock every relevant CRMs
• Not applicable to unstable materials
• Availability of sufficient amounts for IQC to

use over extended periods

69

MAFF

If the use of reference material is not practical, limited check on bias is possible by a test of recovery.

Recovery checks

A test portion of the test sample spiked with a known amount of the analyte is analyzed alongside the original test material. applicable unstable analyte/matrices, ad hoc analysis

70

MAFF

Blank determination

an analysis of a sample without the analyte, or an analysis without a sample, i.e. going through all steps of the procedure with the reagents only.

1.In many analyses sample results are calculated by

subtracting blank readings from sample readings.

2.Blank readings can be excellent monitors in quality

control of reagents, analytical processes, and proficiency.

• 3. They can be used to estimate several types of method

detection limits.

71

MAFF

Insertion of control materials for IQC (1)

Recommendation

1.Short (e.g. n<20) frequent runs of similar materials

• 2. Longer (e.g. n>20) frequent runs of similar materials
• at least once per run
• analyze in duplicate at least half of the test materials selected

at random

• insert at least one blank determination
• at an approximate frequency of one per ten test materials at

least once per run

• analyze in duplicate at minimum of five test materials selected at

random

• insert one blank determination per ten test materials

72

MAFF

• 3. Frequent runs containing similar materials but with a wide

range of analyte concentrations

• 4. ad hoc analysis (statistical control is not applicable)

Insertion of control materials for IQC (2)

• at an approximate frequency of one per ten test materials at

least once per run.

• at least two concentration levels, one close to the median level
• f typical test materials, and the other is approximately at the

upper or lower of decile as appropriate.

• duplicate a minimum of five test materials.
• insert one procedural blank per ten test materials.
• duplicate analysis on all of the test materials
• recovery tests or use of formulated control material
• blank determination

73

MAFF

Use of Control Charts

74

Control Chart (Shewhart Chart):

A statistical tool used to monitor process stability and

• control. One of the Seven Basic Tools of Quality Control.

MAFF

• 1. Understanding current and past process

performance and its degree of consistency

• 2. Establishing a "state of statistical control" by

identifying and removing causes of unnatural (or "special cause") variation so as to achieve a consistent and predictable level of process quality

• ver time;

Common Control Charts

75

chart R

• X

chart s

• X

chart R

• X

MAFF

How to calculate/plot X-Bar and R chart (1)

76

MAFF

77

: The grand mean of all subgroup averages

X

X : The mean for each subgroup

n X X X X

n

+ + +  

2 1

n : The number of measurements within a subgroup (here, n=2)

k X X X X

k

+ + +  

2 1 k : The number of subgroups (here, k=9) : Calculate the following statistics

How to calculate/plot X-Bar and R chart (2)

MAFF

78

R : The average of the ranges for all subgroups

k R R R R

k

+ + +  

2 1

Ri : The individual range for each subgroup k : The number of subgroups (here, k=9)

How to calculate/plot X-Bar and R chart (3)

MAFF

79

Upper control limit (UCL):

R A X UCL

2

+ 

: Calculation of Upper and Lower control limit

Lower control limit (LCL):

R D UCL

4



For For R chart: X chart:

R A X LCL

2

  R D LCL

3



For For R chart: X chart:

How to calculate/plot X-Bar and R chart (4)

MAFF

80

Constants for Calculating Limits for X-Bar and R charts

How to calculate/plot X-Bar and R chart (5)

http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc321.htm

MAFF

81

MAFF

How to calculate/plot X-Bar and R chart (6)

Time line (Day/Run)

82

Example of X-Bar and s chart

MAFF

Time line (Day/Run)

3-2 Proficiency testing

83

MAFF

Proficiency testing (PT) : evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons

External Program for Quality Control (Proficiency testing)

84 (ISO/IEC 17043:2010)

• Participating laboratories analyze same sample distributed

from proficiency testing provider.

• Reported results were compared to the assigned value and

statistically evaluated.

• Participants can confirm their results.

MAFF

How to participate in PT program?

• Proficiency testing provider

Internationally renowned: Fera(FAPAS, FEPAS), AOACI, AOCS, etc. Domestic (In case of Japan): JSAC, FDSC, JAB, etc.

• Participation

Through internet, agency, etc. Check schedule of individual proficiency testing round.

85

MAFF

example ：FAPAS

• Round: Nutritional components, Food ingredients
• contaminants, Pesticides, Veterinary drug residues,

Food additives, etc.

• Participating laboratories analyze distributed sample

using their method. Report the results and the method information through website.

• Results were Statistically evaluated and z-score were

returned to participating laboratories.

|z|≦2, 2＜|z|＜3, |z|≧3

Example of Proficiency Testing Scheme

86

MAFF

Result of PT

ｚ = (x - X) / σp

x: the participant’s reported result X：the assigned value

e.g., robust mean after excluding results that are clearly spurious and outliers by statistical evaluation.

σp ：the target value for standard deviation

e.g., using RSDR from collaborative study, calculation from Horwitz equotion etc.

Usually shown in z-score defined as follows:

87

MAFF

z-Scores for Patulin (80.7 g/L) in Apple Purée Test Material From: FAPAS PROTOCOL FOR THE ORGANISATION AND ANALYSIS OF DATA SIXTH EDITION, 2002

Graphical expression of z-Scores (example)

88

MAFF

How to interpret a result of PT?

• Checking bias of the results
• Review of method procedure and management system of

laboratory by periodical participation

• Information on methods used by other laboratories

Importance of Participating PT

• A result of |z| >2 is not specially rare.
• Is that a usual operation for the laboratory?

z-Scores “|z| ≤ 2” is “satisfactory” ?

89

MAFF

Topic 4 Measurement Uncertainty

90

MAFF

Definition

Non-negative parameter characterizing the dispersion

• f the values being attributed to a measurand, based
• n the information used.

Measurement Uncertainty in Codex (1)

Expanded measurement uncertainty: Measurement uncertainty:

product of a combined standard measurement uncertainty and a factor larger than the number

• ne

Guidelines on Analytical Terminology (CAC/GL 72-2009)

91

MAFF

• Estimated range in which the true value would

be found

• Showing reliability of the results
• Estimated by experimental data and statistical

evaluation

What is Measurement Uncertainty?

92

MAFF

Analytical results and its variability

• Analytical results of repeated analyses of the

same sample can be found in a normal distribution.

• In case of chemical quantitative analysis,

reproducibility, between laboratory variation, depends on concentration of analyte, independent of variety of food or analyte.

93

MAFF

68% 95% 99%

68% of the observations have values within the range of µ±1σ 95% of the observations have values within the range of µ±2σ

Normal Distribution

Mean : µ SD : σ

+σ +2σ +3σ +4σ − σ − 2σ − 3σ − 4σ µ

94

MAFF

Measurement Uncertainty and Bias

(Value we want to know)

(unknown) Population distribution (unknown)

True Value

Sample dist stribution

• n

Mean

Bias: difference between the result and the true value Measurement Uncertainty : Estimated range in which the true value would be found.

X-axis: Concentration Y-axis: Density

95

MAFF

Measurement Uncertainty in Codex (2)

Guidelines on Measurement Uncertainty (CAC/GL 54-2004)

Measurement Uncertainty have to be estimated in Codex.

• One of the requirements of the ISO/IEC 17025:2005

The measurement uncertainty of a result shall be estimated and then made available if requested.

• Guidelines CAC/GL 27-1997 require laboratories

involved in the import/export of foods to comply with general criteria in ISO/IEC 17025.

96

MAFF

Measurement Uncertainty in Codex (3)

Reporting the results in a form of “a ± U”

Guidelines on Measurement Uncertainty (CAC/GL 54-2004)

—where a: The best estimate of the true value of the concentration

• f the measurand (the analytical result)

u: The Standard uncertainty U: The expanded uncertainty (usually equal to 2u). The range “a ± 2u” represents a 95% level of confidence in which the true value would be found.

97

MAFF

Guidelines on Measurement Uncertainty (CAC/GL 54-2004)

• 2. Top-down approach
• Use data from collaborative trials, proficiency studies,

validation studies or intra-laboratory quality control samples,

1.Bottom-up approach

• Identification of uncertainty sources
• Estimation of the size of the uncertainty component

associated with each potential source.

• Calculation of combined uncertainty

Estimating Measurement Uncertainty

98

MAFF

Typical Value of Expanded Uncertainty

CAC/GL 54-2004 GUIDELINES ON MEASUREMENT UNCERTAINTY 99

MAFF

MU in Compliance Decisions

Maximum Level

Result ±U above ML Result > ML but ML within U Result < ML but ML within U Result ±U < ML

＋U －U

Guidelines on Measurement Uncertainty (CAC/GL 54-2004) Guidelines on Estimation of Uncertainty of Results (CAC/GL 59-2006)

Situation ii） iii） iv） i） Result

10

MAFF

Topic 5 Evaluation of Analytical Results

10 1

MAFF

Information from Laboratory

Require all relevant data

(especially when using 3rd Party laboratory)

• validation status of the methods of analysis used, sample

handling, preparation procedures.

• raw data (including spectral data, calculations, chemical

standards used);

• results of repeat analysis;
• internal quality control (control charts, blank data, recovery

data, uncertainty data, use of RMs);

• performance in proficiency testing
• official accreditation status of the laboratory

10 2

MAFF

Typical Problem for the Reported Results (Examples)

• mistakes

e.g., sampling number, classification of commodity

• Ambiguous data

no formula for calculations of LOD, LOQ, recovery, etc. no information on recovery correction

• Inappropriate significant figures

103

MAFF

Check

• Analytical work was correctly done as specified?
• Sample treatment was appropriate ?
• Validated method used? Verified for the commodity?
• LOD and LOQ were properly calculated?
• LOQ was lower than expected occurrence level?
• Calibration was appropriate?

–reliable standard reagents –linearity of calibration：R2≧0.99 ?

• Recovery corrected?
• Significant figures were properly calculated?

10 4

MAFF

Recovery

• Recovery/recovery factors:

Proportion of the amount of analyte, present in, added to or present in and added to the analytical portion of the test material, which is presented for measurement.

Guidelines on Analytical Terminology (CAC/GL 72-2009)

105

MAFF

Cf : fortified concentration, analytical results of spiked sample Cu : unfortified concentration, analytical results of un-spiked sample CA : added concentration

Food /Feed

Reagents spiked

Measurement

Results

106

A u f

C C C (%) Recovery Marginal  

MAFF

U A f

C C C Recovery % Total + 

107

Criteria for Recovery in Codex

(Codex Procedural Manual) 107

MAFF

Recovery Correction

All data, when reported, should

(a)be clearly identified as to whether or not a recovery correction

has been applied and

(b) if a recovery correction has been applied, the amount of the

correction and the method by which it was derived should be included with the report. This will promote direct comparability of data sets. Correction functions should be established on the basis of appropriate statistical considerations, documented, archived and available to the client.

Harmonized IUPAC Guidelines for the Use of Recovery Information in Analytical Measurement (CAC/GL 37-2001)

108

MAFF

Recovery check as IQC

IQC control charts for recovery should be established during method validation and used in all routine analysis. Runs giving recovery values outside the control range should be considered for re-analysis in the context of acceptable variation, or the results reported as semi- quantitative.

Harmonized IUPAC Guidelines for the Use of Recovery Information in Analytical Measurement (CAC/GL 37-2001)

109

MAFF

Significant Figures(1)

• Addition and Subtraction

–The result should have as many decimal places as the

measured number with the smallest number of decimal places.

(e.g.) 2.54 cm + 15.75 cm = 18.29 cm 1.46 mg + 12.7 mg = 14.2 mg (e.g.) 3.58 kg – 0.779 kg = 2.80 kg

1.46

+ 12.7 14.16 2

110

MAFF

Significant Figures(2)

• Multiplication and Division

–The result should have as many significant figures as the

measured number with the smallest number of significant figures. e.g. 1.42 mg/kg × 15 kg = 21 mg 100 m ÷ 12 s = 8.3 m/s

Divided by 4 pieces 8.43cm ÷ 4 = 2.11 cm

1.42

× 15 710 142 2130

111

MAFF

Exact numbers, such as the number of people in a room, have an infinite number

• f significant figures.

Rounding of numbers (1)

ISO 80000-1: 2009

Rounding range： 0.1 Rounding range：10

112

MAFF

The rounding shall always be carried out in only one step. e.g. 12.254 should be rounded to 12,3 and not first to 12,25 and then to 12,2.

Rule A: The even multiple Rule B: The greater in magnitude

ISO 80000-1: 2009 (ISO 31-0 1992)、JIS Z 8401:1999

If there are two successive integral multiples equally near the given number, two different rules are in use.

113

MAFF

Rounding of numbers (2)

Experience of MAFF Japan

114

MAFF

Roles of MAFF and MHLW for food safety

Processing, Distribution Primary Production Processed Food Fresh produce Raw material

Food Chain

MHLW (enforcement)

Setting MRLs

Food Inspection, etc.

MHLW (enforcement) Import quarantine Exporting Country MAFF (enforcement)

Fertilizer, Feed, Veterinary drug the use of Pesticide

Cooperation

MAFF (recommendation)

Improvement of production/processing methods of domestically produced food, etc.

Im food

Consumer Farm Dome produ Food 115

MAFF

【Contents】 ・Identification of a food safety problem ・Establishment of a risk profile ・Ranking of the hazard for risk management priority ・Establishment of risk assessment policy ・Consideration of the result of the risk assessment ・ Evaluation of risk management options ・ Implementation of management decision ・Risk communication etc

Standard Operating Procedure (SOP) for MAFF & MHLW

Publication：25 Aug 2005, rev 2006, 2012

116

MAFF

Example of priority list of Chemical hazards for food safety (MAFF 2006, revised in 2010)

• Metals

–Arsenic –Cadmium –Lead –Methyl mercury

• Mycotoxins

–Aflatoxins –Zearalenone –T2-toxin

HT-2 toxin

• Marine toxins

–Paralytic shellfish poisoning toxins –Diarrhetic shellfish poisoning toxins –Ciguatera poisoning toxins –Domoic acid –Brevetoxin

• Process contaminants

–Acrylamide –Polycyclic aromatic hydrocarbons –Furan –3-monochloropropane-1,2-diol

(3-MCPD) 3-MCPD fatty acid esters

etc.

117

MAFF

Guideline for the evaluation and publication of analytical results of surveillance/ monitoring (2006, MAFF)

Purpose

• To Guide MAFF officials for assessing the quality
• f analytical reports from internal/private

laboratories, and for relevant considerations for their publication

• To provide analytical laboratories with the format

for reporting the result of analysis

118

MAFF

Guideline for assessing the result of surveillance/ monitoring (MAFF)

Format for reporting the result of analysis by Laboratories Laboratory Lab Name, Address, Person in charge A . Analysis

• 1. Analyte / Matrix

2 . Laboratory Sample Date of receipt, Condition for storage of laboratory sample and

replicate sample (temperature etc.)

3.Sample Preparation

Procedure for preparation of a portion for the measurement from Laboratory sample

119

MAFF

• 4. Information of analytical method

Standard operating procedure of the analytical method

• 5. Information of method validation

Multi Lab / Single Lab / Other

Matricies used for validation

• 6. LOD, LOQ

Value, Calculation method

• 7. Calibration

Used standard reagent, Purity, Protocol for preparation of

standard solution, Linear range

120

MAFF

• 8. Recovery

Matrix, Analyte, Concentration, Number of replication, Value of recovery rate and RSDr, Raw data or corrected by recovery

• 9. Measurement Uncertainty

Value of measurement uncertainty Method for estimation

• 10. Analytical results

Number of tested samples, Value of measurement of each sample

121

MAFF

B Quality Control

• 1. Internal

in accordance with GLP guideline by MHLW frequency

• 2. Proficiency testing

provider, matrix, analyte, z-score ※ must participate in the last two years (same or similar matrix and analyte)

• 3. ISO/IEC17025 Accreditation

122

MAFF

C Sampling (only when the laboratory itself conducted sampling) Sampling method and it’s reference Sample size for each step in the sampling procedure

123

MAFF

Before establishing guidelines

• Large variety but small sample number for one

commodity e.g. only 5 samples for one commodity

• No attention to quality of analytical result

Poor requirement of internal quality control, proficiency testing, precision/recovery data for the used method

• Priority to low financial cost

124

MAFF

Change in design of surveillance in MAFF (1)

After establishing guidelines

• Statistic based sample number

e.g. enough to obtain occurrence data on 95 percentile value with 95% confidence level

• Use of validated method
• Requirement of internal/external quality control
• method validation/verification data
• participation to proficiency testing program
• ISO/IEC 17025 accreditation
• Priority to data quality

125

MAFF

Change in design of surveillance in MAFF (2)

Challenges for enhancing quality assurance in Japan

• Continued capacity building of risk assessors/

managers

• Enhancement of quality assurance of official

and private laboratories and research institutes

• Collaboration for analytical method validation
• Supply of standard reagents and certified

reference materials

126

MAFF

Thank you for your attention!

127

MAFF