Essays

January 2008

Recent audioconference explained how EQC, despite its dubious scientific validity, is no longer educational, but policy. The audioconference brought up the possibility once more that the FDA will provide a QC clearance option for manufacturers. (Hint: Don't get your hopes up...)

• What's the real deal?
• What's a laboratory to do?
• What could the FDA do?
• From control to compliance
• From compliant to complicit

[PLEASE NOTE: AS OF JANUARY 2016, EQC BECAME ILLEGAL WITHIN THE US. LABS USING EQC MUST SWITCH TO USING IQCP.]

On December 6, 2007, AACC provided a teleconference on “New Developments in CLIA and QC”, which focused on changes in the CLIA regulations that are being implemented as of January 2008. Prior to that conference, we discussed some possible issues with the new “equivalent QC” (EQC) procedures that can now officially be used in US laboratories. Please see the earlier discussion on Equivocal QC: Coming soon to a laboratory near you for background information.

In that teleconference, there was one point that surprised me. There was a statement that the FDA was expected to provide a review of a manufacturer’s recommendations for “Alternative QC” (AQC). From slide #11, here are the exact words:

“Manufacturers may validate an alternative QC proposal; If FDA agrees that the validation shows that the alternative is equivalent to traditional QC, then; Labs may use the alternative QC instead of the CLIA-mandated QC.”

My understanding has been that FDA review was certainly part of the original CLSI initiative for EQC Option #4, but that objective was no longer on the table once the focus changed from EQC to AQC. However, my information was second-hand, based on occasional conversations with people who were involved with the CLSI committees.

What’s the real deal?

To clarify this point, I followed up with Dr. James Nichols and Ms. Judy Yost, who were the presenters of the teleconference.

Dr. Nichols indicated that he was providing a historical perspective on the original intention of the CLSI EP-22 committee. Ms. Yost indicated that this indeed was the original plan, but that she didn’t believe that the FDA has agreed to do this.

That still leaves the issue unresolved, so I contacted Dr. Steve Gutman at the FDA. He indicated that the FDA would be interested in undertaking this role IF scientifically valid protocols could be developed by CLSI. But he regarded the emerging EP guidelines as too complex and unvalidated for use by FDA. Thus, FDA currently cannot play that role. Dr. Gutman did indicate, however, that the FDA has not given up hope that more scientific guidelines may emerge in the future.

So, my earlier assessment stands. Outside of the regulatory bubble, in the reality of today's laboratory, FDA review or clearance of QC is not going to happen. It isn't possible right now because the EP committees have not come up with quantitative guidelines that the FDA considers scientifically valid. Moreover, it is not likely to happen in the future because of the inherently qualitative and subjective nature of the risk management approach that has been adopted by the EP-22 and EP-23 committees.

What’s a laboratory to do?

EQC allows a laboratory to reduce external QC on the assumption that a manufacturer’s internal QC (electronic measures, physical checks, internal controls) are effective in identifying any problems with the testing process. In the absence of any review by FDA, this is ultimately a matter of trusting the manufacturer (and assuming the manufacturer did a good job in the design and development of the test system).

To implement EQC, each laboratory, specifically the laboratory director, assumes the legal and ethical responsibility of deciding whether or not the manufacturer has minimized the risk of bad test results. That’s obviously a difficult problem since, as the FDA's admits, there aren’t sufficient scientific guidelines to do this! Nevertheless, the government (CMS with CDC and FDA complicity) seems comfortable passing on the risk of EQC to laboratory directors. Our regulators take it on faith that individual laboratory managers know more than the FDA about judging instruments and methods.

CMS has defined so-called “validation protocols” for laboratories to use when implementing EQC. The existence of such protocols should resolve FDA’s needs for explicit experimental guidelines - if the protocols were valid. Unfortunately and obviously, these protocols are not scientific. Remember EQC "Option 1" uses a 10-day test period to prove that a method is stable for 30 days. This sham protocol is supposed to legitimize the reduction of QC to two levels once per month. But by its silence and inaction, the FDA actually confirms that these CMS protocols are not scientifically sufficient (otherwise FDA could apply that approach themselves).

Despite CMS's eariler admission that, "We blew it", despite the absolute lack of any scientific basis for the validation protocols, and despite the well-voiced objections of many in the laboratory community, laboratories are now free to implement EQC. In deciding whether or not to implement EQC, it would be better for laboratories to apply a more rigorous validation protocol and a more objective QC design process than that recommended by CMS. (See CLIA Final Rules: Appropriate QC Procedures for further information).

What could the FDA do?

I’ve never quite understood FDA’s position on QC clearance, even though this issue has been under discussion since 1992 when the first “final” CLIA rules were published. Remember the part of the rules where CMS delayed implementation of certain QC requirements until the FDA could implement a QC clearance process. [See QC Clearance Postponed Again & Again & Again & Again.] The FDA has had over 15 years to work on QC clearance, but it evidently couldn’t figure out how to do it, didn’t want to do it, or wasn't able to devote any time and effort to the issue. I suspect it’s a matter of resources, priorities, and interests. Clearance of new drugs must have a much higher priority than clearance of laboratory tests and QC procedures.

Clearance of laboratory tests basically comes down to “truth in labeling.” A manufacturer makes claims for certain performance characteristics, such as precision, accuracy, reportable range, detection limit, reference limits, and possible interferences. The FDA reviews a manufacturer’s experimental data to confirm the data supports the manufacturer’s claims. The experimental data are collected following “consensus” protocols developed by CLSI, such as the following:

• EP5-A2. Evaluation of Precision Performance for Quantitative Methods.
• EP9-A2. Method Comparison and Bias Estimation Using Patient Samples.
• EP6-A. Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach.
• EP17-A. Protocol for Determination of Limits of Detection and Limits of Quantification.
• EP7-A2. Interference Testing in Clinical Chemistry.
• C28-A2. How to Define and Determine Reference Intervals in the Clinical Laboratory.

Although these CLSI protocols are accepted by both manufacturers and the FDA, there is no requirement for the FDA to use consensus protocols when it reviews manufacturer’s claims in the 510k approval process. The FDA can issue its own guidance for manufacturers whenever it wants. Waiting for CLSI to develop acceptable protocols is just an sign that FDA isn't seriously interested in providing QC clearance.

Given FDA's apparent need for an existing CLSI document, I have suggested that FDA (and anyone else, for that matter) consider the guidance in C24-A3 “Statistical Quality Control for Quantitative Measurement Procedures: Principles and Definitions.” This CLSI standard describes how a QC procedure can be selected on the basis of the quality required for a test and the precision and accuracy observed for a method. C24-A3 recommends the calculation of a Sigma-metric:

Sigma = (TE_a-bias)/CV where TE_a represents the quality requirement for a test in the form of an allowable total error, and bias and CV represent the inaccuracy and imprecision of the method).

The Sigma-value is then used to select appropriate control rules and number of control measurements using a graphical tool called the Sigma-metrics QC Selection Tool.

The FDA could use the manufacturer’s claimed precision and bias (which is already part of a manufacturer’s 510k submission) along with the CLIA criterion for acceptable performance in proficiency testing (available in the CLIA regulations) to assess quality on the Sigma scale. The resulting Sigma-metric could then be used to identify the right control rules and right numbers of control measurements, i.e., the right QC procedure, which should be used whenever the stability of the measurement procedure is to be tested. Information from a manufacturer’s risk assessment studies could then provide guidance for “whenever” external QC should be performed.

One important feature, and potential difficulty, is the need to define the quality required for a test. The existing “criteria for acceptable performance in proficiency testing” in the CLIA regulations may not be right. They were developed almost 20 years ago and the quality needed for laboratory tests today may be more demanding. Also, the criteria only cover about 80 tests and don’t include important analytes such as HbA1c and PSA. Therefore, time and effort are needed to update the quality requirements for laboratory tests, or other sources of information about quality goals should be utilized, e.g., the Ricos databank on biologic variability which covers over 300 different quantities.

Alternatively, manufacturers could make claims for the quality that their methods are intended to achieve. Better yet, the FDA could require that manufacturers make claims for the measurement uncertainty, which would actually be consistent with global requirements for characterizing method performance and managing the quality of laboratory tests, e.g., ISO 15189. The quality required or claimed is essential information if we are to properly manage and assure the quality of laboratory tests. The FDA should at least make it known that such information is needed by manufacturers as well as laboratories if quality is to become a quantitative and measurable characteristic of laboratory tests.

From control to compliance

One striking part of the teleconference was the question/answer period. It seemed that laboratories were most concerned about CMS taking a hard-line position on proficiency testing and the provision that laboratories are not allowed to retest or compare PT results. There was little concern about the validity of the EQC protocols. If it’s legal, apparently laboratories will do it. Compliance has won out over quality.

With widespread adoption of EQC, a new low standard for laboratory QC will be established, which will be difficult to raise in the future. We already know that the 1992 CLIA minimum of two levels of QC per day became the standard QC of laboratories today. Even when EP22 and EP23 guidelines for AQC become available, why would laboratories adopt a more complex risk assessment approach when it takes more time and effort than EQC? EQC lowers the bar for QC, almost to the ground. And doing just a little QC can be more dangerous than none at all, because it makes it look like quality is still being controlled and gives us false confidence. The probable outcome of EQC is that the quality of laboratory tests will suffer, but we may never know - there won’t be enough data available to evaluate the resulting variance of laboratory tests.

From compliant to complicit

We know better! Professional organizations, such as AACC and CAP, know much better! But nobody wants to take a stand against these obviously bad, wrong, indefensible EQC procedures. By not objecting to these regulations, by doing nothing, the laboratory community has gone beyond being compliant. It is now complicit in their adoption and implementation.

Have laboratory standards fallen so far that we no longer care about what’s scientifically valid, but only whether or not we can get away with it? We’ll soon know. As of 2008, Laboratories inspected by CMS have the green light to implement EQC. Deemed providers (other accreditation organizations approved by the government) are also starting to give the green light to the laboratories they inspect and accredit. They know better, but they have to adopt some form of EQC to maintain their deemed status and also to be competitive in the inspection/accreditation business. If your competitor gains business by loosening standards, market forces will compel you to loosen your standards, too.

Nobody seems to care, but we should. We should represent the interests of our patients because they are not able to do so themselves. The only people in healthcare who really understand test quality are those of us in the laboratory and we know many things can and do go wrong in the testing process. Assuring the quality of our testing is certainly a challenge in today’s healthcare, but if we don't look out for the patients and their test results, no one else can. Our efforts are needed to assure the quality of laboratory testing and to improve the quality of healthcare.

James O. Westgard, PhD, is a professor emeritus of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.