Trends

Back in 2005, CLSI held a conference on "Quality Control for the Future." At that time, the future - specifically the future described by CLSI at the time - seemed like it was going to happen on an accelerated schedule. But that future has taken a long road to reach us, and there were some major breakdowns along the way.

QC For the Future, Six Years Later

James O. Westgard, PhD
April 2011

We have now marked the 6th anniversary of CLSI’s conference on “Quality Control for the Future” that was held in Baltimore, MD, on March 18, 2005.  The proceedings of that conference were published in the October 2005 issue of LABMEDICINE [1].  The idea for that conference came from CMS and AdvaMed, which were joined by AACC, ASCLS, and CDC to organize a program that was hosted by CLSI.  Additional sponsors included ACLA, AMT, ASCP, ASM, CAP, CLMA, COLA, and JCAHO.

The focus of this conference was a proposal for the development of an Option 4 for Equivalent QC (EQC).  CMS’s original guidelines for EQC (published in the State Operations Manual in 2004) had led to much debate in laboratories and in industry over the options for reducing the frequency of QC from the original CLIA requirement for 2 levels per day to possibly 2 levels per week or even 2 levels per month.  At this conference, the chair of the CLSI Evaluation Protocols Area Committee introduced a proposal for a 4th option for EQC, as follows [2]:

“First, the manufacturer performs a risk assessment of their product using risk assessment standards (e.g., ISO 14197, CLSI EP18) and tools (eg, failure modes and effects analysis, fault tree analysis).  This risk assessment would focus primarily on conditions that would lead to erroneous test results, for example, operator error or damaged reagents.

“After identifying the potential risks, the manufacturer determines if the risks are controlled by internal failsafe mechanisms or internal quality control.  Remaining risks may need to be controlled by an external QC protocol.  The objective is to mitigate all risks through a combination of failsafe mechanisms and quality control, reducing risks to minimal or negligible levels, or at least detecting the risk as well as current traditional QC practices.  The proposal alternative should be equivalent or better than traditional QC.  Therefore, taking the example of damaged reagents, the alternative QC must be able to identify the damaged reagents as well as traditional QC.

“Once the risks have been identified and the failsafes and quality checks needed to mitigate the risks have been identified, the next step is to prove that those failsafes and quality checks actually work.  Looking again at the damaged reagents example, the manufacturer would need to simulate damaged reagent conditions, and prove that the failsafe or alternative QC did indeed detect the error condition and prevent the reporting of a false result. That proof data would then be submitted to FDA for review and approval.

“As stated above, the CLSI consensus process for document development will be utilized in order to ensure that the views of laboratory professionals, government officials, and industry are all included.  The definition of “scientifically rigorous validation” will be decided by a CLSI subcommittee that included representatives from all 3 of these constituencies.  Once developed, the document will then proceed through CLSI’s process that provides an opportunity for all of CLSI’s members to provide comments and vote to accept or reject the document.”

That document, CLSI EP23 “Laboratory Quality Control Based on Risk Management,” will soon be available to the laboratory community.  While the exact wording of the final document is not yet available to the public, the basic contents have been discussed publicly and it is clear there have been significant compromises from the approach described in the original proposal and the emerging guideline.

First, the FDA will not review and approve a manufacturer’s risk analysis studies to validate claims for alternative control mechanisms.  While it is not clear whether that outcome is due to FDA’s reluctance to take on additional regulatory review or the manufacturers’ reluctance to have their claims for alternative QC reviewed, it is clear that FDA will not review QC claims as part of the 501k approval for new analytic methods and devices!  That leaves laboratories with the responsibility to decide whether or not a manufacturer’s built-in controls are sufficient to assure the intended quality of test results, in accordance with the CMS/CLIA guidance that the laboratory director is ultimately responsible for QC.   

Second, given a laboratory’s responsibilities to validate alternative QC mechanisms, it is not clear whether or not a manufacturer’s risk analysis studies will be available to the laboratory.  During the CLSI development process, after it became clear that FDA review would not be happening, another CLSI committee (EP22) worked on a document to provide manufacturers with guidance on how to present their risk analysis information to medical laboratories.  After much work and the release of a proposed version of that document (EP22P  Presentation of a Manufacturer's Risk Mitigation Information for Users of In Vitro Diagnostic Devices), there was opposition from manufacturers to having even voluntary guidance on how to present risk information.  That document was killed (technically, the committee voted not to continue working on the document so it is possible in the future this document could be resurrected).  Nonetheless, manufacturers have no requirement to provide risk analysis information to laboratories and, even if they were willing to do so, there is no guidance on how to make that information available in an understandable way.

Third, the medical laboratory is now faced with having to learn risk analysis in order to develop QC Plans and validate alternate QC mechanisms.  While it is possible for the laboratory to request risk analysis information from the manufacturer, the laboratory will still need to know the risk analysis process and methodology in order to make sense of such studies.  Manufacturers seem to be hoping that the laboratory will trust them to have performed proper validations for their control mechanisms, even though they seem reluctant to provide the information to prove that their built-in controls work.

Was the Future worth the wait?

Do you think that the “QC of the Future” in the form of Risk Management QC Plans really satisfies CLSI’s own requirement for “rigorous scientific validation”?   If it does, why isn’t FDA willing and able to review and approve a manufacturer’s claims for alternate QC mechanisms? If it does, why aren’t manufacturers willing to provide the validation information to laboratories in a standard format to support their implementation of QC Plans?  Are laboratories just supposed to trust that manufacturers will do this right and believe that any and all control mechanisms perform equally and perfectly?

With publication of EP23 expected by mid-year, the future for 'QC for the Future' will be in the hands of medical laboratories.  The main outcome of the CLSI process is an endorsement of risk analysis as an approach for developing laboratory QC Plans, but medical laboratories will need to make an investment in learning the process and implementing a practical methodology.  The starting point should be the Joint Commission's methodology for Proactive Risk Reduction because it is well documented and has been applied in many institutions accredited by JC.  Adaptation will be necessary to focus on detection as the primary option for mitigating risks.  That adaptation should incorporate Six Sigma concepts and metrics to make risk analysis a more quantitative and objective process where residual risk can be expressed in terms of a residual defect rate, or defects per millions (DPM), or the number of potentially harmful test results that will be produced by the laboratory.  Those are practical measures of residual risk that have real meaning to people in medical laboratories and to the patients they serve.

References

1. “QC for the Future: A report from the CLSI conference.” LABMEDICINE 2005;36 (Oct);609-640.
2. Ochs L. QC for the Future: CLSI standard development and Option 4 proposal.  LABMEDICINE 2005;36 (Oct);639-640.