Trends

April 2007

Dr. Westgard attended the recent CLSI workshop to learn about forthcoming standards on risk assessment, quality, and (E?)QC. It seems the rumors of the death of Options 1, 2, and 3 may have been exaggerated...

Increased influence of ISO

Update on EQC Option #4

The good news

CLSI sponsored a conference in Baltimore on April 20 to consider “Facing the Challenge: Practical approaches to evaluate analytical performance.” This conference described and discussed the CLSI Evaluation Protocols (EP) documents for method validation and quality control. Most of these documents deal with method validation, but there are documents under development that are intended to help laboratories customize their QC systems to meet their individual needs.

Here is the list of approved (A) documents that were under discussion:

• EP5-A2. Precision performance of quantitative measurement methods;
  
• EP6-A. Evaluating linearity of quantitative measurement procedures;
  
• EP7-A2. Interference testing;
  
• EP9-A2. Method comparison and bias estimation using patient samples;
  
• EP10-A3. Quantitative clinical laboratory measurement procedures;
  
• EP12-A. Qualitative test performance;
  
• EP14-A2. Evaluation of matrix effects;
  
• EP15-A2. Verification for precision and trueness;
  
• EP17-A. Limits of detection;
  
• EP18-A. Quality management for unit use testing;
  
• EP21-A. Total analytical error.
  
• EP22. Presentation of Manufacturer’s Risk Information
  
• EP23. Laboratory QC Protocols Based on Manufacturer’s Risk Mitigation Information and the Laboratory Environment

Note that documents EP5 and EP18 are currently under revision and that the EP22 and EP23 are the new guidelines that are expected to be completed within a year or two.

Perhaps the most valuable part of the conference was the systematic review of the intended applications of these documents, with some attention to the numbers of specimens, important factors in the experimental design, and critical aspects of the statistical analysis of the data. These documents clearly should have widespread application in US laboratories, yet it is unknown how many of the 36,000 laboratories that perform non-waived testing actually have these documents. The cost of the documents, typically $120 each for nonmembers, would be an issue in many laboratories.

Increased influence of ISO

CLSI’s global interests are becoming more evident in these EP documents, as ISO concepts and terminology are being adopted and included. That is obvious in the title of EP15-A2 with its emphasis on “trueness.” But it is more extensive in the EP5 document that is under revision, both in concepts and terminology. For example, the ISO concepts of “repeatability” and “reproducibility” are being introduced. The term “within-lab precision” - apparently a term equating repeatability - will replace what was called “total precision” in earlier CLSI documents. The ISO concept of reproducibility extends precision to consider between-lab effects, which might be estimated via peer-comparison, proficiency testing (PT), or external quality assessment EQA) programs. To be more consistent with ISO 15189 “Particular requirements for quality and competence of medical laboratories”, the new EP5 will consider more levels of controls (3-5) to support the development of “precision profiles,” which is another recommendation from ISO. The next revision of EP5 (and also EP9) will focus solely on manufacturers and their use for establishment of claims.

As EP5 and EP9 evolve into guidelines exclusively for manufacturers, EP15 will become the main guideline for laboratory users. Laboratories will use EP15 for verification of manufacturers’ claims for precision and trueness. These changes, or more specific intended uses, accommodate the need for more specimens and more extensive testing by manufacturers while reducing the numbers to a minimum for laboratory user protocols.

Another example of a protocol aimed at manufacturers is EP21 (Total Analytical Error), which seems to be widely supported by FDA for use with “waived” methods. EP21 requires 120 patient specimens to estimate total analytical error for each decision level; given 2 or 3 decision levels for most tests, specimen numbers will often be as high as 240 to 360, which is feasible for only for manufacturer, not ordinary laboratory users. Thus, EP21 will be used mainly by manufacturers and mainly for FDA submissions for waived tests.

It is interesting that “total analytical error” is not an ISO term, nor an ISO concept, though in practice it bears some similarities to the ISO “uncertainty of measurement.” FDA’s support for the use of total error in clearance of waived tests suggests that this concept has a valuable, practical application. FDA statisticians even discussed evaluating total error against a goal such as an Allowable Total Error, which might be defined by the CLIA criteria for acceptable performance in proficiency testing. It is also interesting to see that total error is not used by FDA for performance claims in moderate and high complexity tests, where precision and trueness are the key parameters. Even the FDA is uncertain about what terms and concepts apply to different classes of tests...

Update on EQC Option #4

You may recall two years ago, at the CLSI workshop “QC for the Future,” CLSI launched a project to support the CLIA recommendations for “equivalent quality testing” (or “Equivalent QC” or EQC), which would allow laboratories to reduce the amount of QC, under certain conditions, from 2 levels a day to 2 levels per week or even 2 levels per month. At that time, there was concern from both manufacturers and laboratory scientists about the suitability of these reductions in QC, as well as the scientific validity of the related CMS protocols for user validation of EQC. Three different EQC options have been described in Appendix C of the State Operations Manual:

• Option 1, for analytic systems with internal controls that monitor the whole analytic process, where the CMS validation protocol requires a laboratory to test external and internal controls for a 10 day period, and if no problems are detected, the laboratory can reduce QC frequency to once per month;
  
• Option 2, for analytic systems with internal controls that monitor only parts of the analytic process, where the CMS validation protocol requires a laboratory to test external and internal controls for a 30 day period, and if no problems are detected, the laboratory can reduce QC frequency to once per week;
  
• Options 3, for analytic systems with no internal controls, where the CMS protocol requires a laboratory to test external and internal controls for a 60 day period, and if no problems are detected, the laboratory can reduce QC frequency to once per week.

At the conclusion of the 2005 “QC for the Future” meeting, CLSI announced the appointment of the EP-22 subcommittee, whose original charge was to develop an “Option 4” for EQC. The purpose of Option 4 was to recreate a mechanism for FDA approval of a manufacturer’s QC directions that fulfilled the CLIA QC requirements, therefore allowing laboratories to follow those directions without any further validation. Remember that the original CLIA rules included a provision for the FDA to approve or “clear” a manufacturer’s QC recommendation, after which the laboratory could simply follow the manufacturer’s QC directions. Remember also that the QC clearance provision was postponed for some 10 years and then disappeared altogether in the Final CLIA Rule of 2003. CMS argued that QC clearance was no longer needed because of the improvements in analytical systems.

During the time that the FDA QC clearance provision was postponed, CMS also had to postpone implementation of certain more stringent QC requirements for moderate and high complexity methods, and therefore allowed laboratories to run a minimum of two levels of control per day and still be in compliance with the CLIA QC requirement. With publication of the Final Rule, CMS allowed laboratories to reduce QC even further by implementing “equivalent quality testing”, which was then translated in the SOM to mean “equivalent QC” procedures. And, since FDA wasn’t able to provide QC clearance, CMS decided that laboratories themselves should validate the reduced frequency of QC by performing some simple protocols. Unfortunately, the validation protocols recommended by CMS are not scientifically valid, e.g., the protocol for option 1 calls for monitoring the stability of a method for 10 days, and if no problems are detected, concludes that the method is stable for 30 days and control can be run once per month. Another basic problem is that just running external controls doesn’t assure that medically important errors would be detected; to assure error detection, the external QC procedure must be properly designed to have the sensitivity to detect critically sized errors, which depend on the quality required for the test and the precision and bias observed for the method.

Now, after two years, the project to develop EQC Option 4 has expanded to 3 subcommittees that will produce a coordinated set of documents based on risk assessment. The original EP-18 document that described an error matrix for assessing possible problems with a test system is now being revised to provide manufacturers with guidance on tools and techniques for risk assessment and mitigation. The EP-22 document now has a new title and focus: “Presentation of Manufacturer’s Risk Information.” EP-23 will now provide guidelines for “Laboratory QC Protocols Based on Manufacturer’s Risk Mitigation Information and the Laboratory Environment.” This title, too, has changed, by only one letter but nonetheless significantly. Before, the focus was on "Laboratory EQC Protocols...[emphasis added]" - now it is just "Laboratory QC Protocols...[no more 'E']" Clearly, the scope of this guidance document has been expanded.

The strategy is no longer concerned with developing a protocol to validate the frequency of QC. Now it will provide a general strategy for laboratories to customize all aspects of their QC system. This strategy depends on manufacturers identifying all the potential problems with their analytic systems (EP18), eliminating those problems when possible, and then providing the user with information on the “residual risks” (EP22). The user should then deal with those residual risks by putting in place the proper control mechanisms in the laboratory (EP23). This sounds good in principle, but will it work in practice?

• This strategy puts the burden of proof on the user, who is unlikely to have a good understanding of risk analysis, and possibly not even a good understanding of QC procedures and systems. This assumption of user knowledge and skills is not reasonable for the small laboratories and POLs who are the intended users of these documents.
  
• This strategy also assumes that this target audience of small laboratories and POLs would have access to these documents, which, one can argue, is unlikely. Given the expense of CLSI documents, typically $120 each, most small laboratories and POLS will not have these documents, and it is also unlikely they will be widely available even in laboratories of moderate and large hospitals;
  
• How will laboratories know the residual risk after customization of their QC system?
  
• Can the laboratory simply pass on the residual risk to the physician user and patient consumer?
  
• Will the laboratory have to characterize that residual risk and disclose the amount to the physician user and patient consumer? If so, here’s where the ISO concept of measurement uncertainty becomes applicable, which makes me wonder if this is just a strategy to introduce measurement uncertainty?
  
• Finally, CMS is making no commitment to eliminate options 1, 2, and 3, even when, if, and after EP-18, 22, and 23 documents are approved. Therefore, it is quite possible that options 1-2 will remain, even though the validation protocols are not scientifically valid. That would leave many small laboratories with EQC option 1 or 2 being the simplest and easiest way to reduce the frequency of QC.

So, after two years of waiting, we probably wait another one to two years to see the outcome of these new EP guidelines. Meanwhile, CMS will have allowed implementation of the existing EQC options in the 20,000 labs inspected by the state government agencies, thereby establishing and supporting a bad practice that will be difficult to change once with the new EP guidance arrives.

What is most surprising is that CMS’s flawed EQC approach now seems to have become more acceptable not only to manufacturers and laboratory users who are interested in simplifying and reducing QC, but also to the professional organizations who want to maintain their “deemed status” and market share for inspections and accreditation. CLSI itself faces its own challenge to assert quality leadership while being burdened by the need to support CLIA/CMS in the US marketplace.

And now, the good news!

I was very impressed by the FDA guidance to manufacturers on the proper uses of the EP documents and, particularly, the fine points about proper data analysis. The FDA’s use and application of EP21-A (Total Analytical Error) for waived tests seems to me a more critical assessment of test quality than what is being done for non-waived tests (where there is no application of total error). Also, the guidance on estimating total error using EP21-A recommends the use of 120 specimens at each clinical decision interval of interest and recommends the proper handling of replicate measurements. For example, the experimental protocol goes back to EP9 which specifies duplicate measurements by both the new method and the comparative method. However, the EP21estimation of total error should be based on single measurements by the test method, thus the guidance is to plot the individual replicates from the test method against the average of the replicates for the comparative method. It was also acknowledged that there may be cases where more replicates are needed by the comparative method in order to minimize its contribution to the direct estimation of total error.

Just to clarify the “direct” estimation of total error, this refers to calculating the differences between the test values and the comparative value, then plotting those differences on a Bland-Atlman plot or a “mountain plot” (See Krouwer and Monti, Eur J. Clin Chem Biochem 1995;33:525-7) to display the actual distribution of differences. Total error is then estimated as the 2.5 and 97.5 percentiles of the distribution, after critical examination of possible outliers. In clinical labs where the number of patient comparison samples are much lower, typically 20 to 40, it is necessary to estimate the total error from a calculation of bias and SD, e.g., bias + 2SD, or bias + 3SD.

Hopefully, FDA’s use of total analytical error will someday extend to the clearance of all tests, thereby providing a more critical assessment of quality, rather than just verifying a manufacturer’s claims for precision and trueness. It will be interesting to see what happens when total analytical error collides with ISO’s precision, trueness, and uncertainty. Given the practical value of total error as demonstrated by the FDA’s application with waived tests, will the FDA change-over to measurement uncertainty for making decisions on waived tests? If so, will the FDA accept the GUM bottom-up model for estimation of measurement uncertainty, or prefer a top-down approach that might be more consistent with total error? I know that few people will find this confrontation of analytical concepts exciting, but it is one of the small pleasures I look forward to as the QC for the Future emerges.

James O. Westgard, PhD, is a professor emritus of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.