Method Validation

CAP just made changes to their 2020 checklist on test verification. Are those changes more stringent or less stringent? Are we going futher on the road to defining Quality down?

Diluting Quality Standards: Method Validation, Verification, and (now) Confirmation

James O. Westgard and Sten A. Westgard
September 2020

CAP added an intriguing new item to its checklist for Verification of Test Performance Specifications-FDA Cleared/Approved Tests [1].

It seems that CMS discovered that the laboratory requirement to verify a manufacturer’s performance claims is being done mainly by the manufacturers’ representatives, not the laboratory’s own personnel. To address this issue, CAP is adding the following statement to the checklist:

If an FDA-cleared or approved method was verified by someone other than the laboratory’s personnel (eg, manufacturer’s representative), the laboratory must ensure that the verification correlates with its in-house test performance by showing confirmation of performance specifications by laboratory personnel testing known specimens.”

In explaining how to do this, it was stated that “lab personnel could take known patient samples of known concentration and run those and see if they can obtain the results that the manufacturer representative obtained, and then document those results.”

So, if we understand this correctly, the lab personnel can confirm a method’s performance claims by showing they get the same results when they repeat patient samples previously run by the manufacturer’s representative. While getting the same results may provide some evidence about the repeatability (or precision) of the method, those results might still be inaccurate and wrong. How can getting the same wrong results provide confirmation of test performance specifications?

A Slippery Slope

Laboratory quality practices have been changing for the last 30 years, ever since the CLIA Final Rules were published in 1990. Those changes in general have been in the direction of quality compliance, rather than quality assurance. Still, it is hard to believe that it is now accepted practice to have the manufacturer’s representative perform the laboratory’s verification studies. We can understand the reason – not enough time available in the lab, but it is an extremely short sighted practice. It is through method validation and verification studies that the laboratory learns about the performance of the method and has the chance to study possible problems before they happen in real-time on real patients. It is an essential part of learning about the behavior of new methods, as well as planning and preparing the quality control procedures and practices that are needed during routine operation.

Dependence on the manufacturer’s guidance may also explain the poor QC practices that are being seen in US laboratories today, as documented by a study published in 2018 [2]. This was a survey of SQC practices for chemistry and immunochemistry systems in 21 major academic laboratories and showed that most laboratories used 2 SD control limits, even for high volume multi-test analytic systems. It is well known that 2 SD limits can lead to high levels of false rejections, e.g., approximately 10% when analyzing 2 levels of control per run which is the CLIA requirement. Obviously, something is wrong and most likely laboratories have found ways to inflate the SDs being used to set the control limits, perhaps using bottle values determined across many labs, peer-group SDs determined for a group of laboratories, or perhaps some recommendation for medically acceptable SDs that are larger than the method’s analytical SD.

Then to compensate for the high false rejection rates, whenever there is a control problem the laboratory just repeats the control until it is in, rather than trouble-shooting the method. Maybe the laboratory repeats the control several times if necessary to get it in. Then all is well, except for the patient test results, which may be in error.

The Bigger Problem with Confirmation

Confirmation is consistent with current trends towards compliance, as well as consensus opinions of best practices, rather than objective, scientifically based practices. Confirmation is perhaps the next step in reducing a laboratory’s responsibility for the quality of its testing processes and its responsibility for correct patient test results. We might now expect that CLSI will start developing “confirmation” protocols to further simplify laboratory practices for ensuring quality.

CLSI already has 2 different types of protocols for method validation and method verification. The method validation protocols are intended for manufacturers use in documenting their performance claims to FDA, e.g., EP5 for precision claims and EP9 for accuracy or bias claims. For laboratory verification of those claims, there is another document, EP15, that provides a simpler experimental approach that requires less data. The current EP15-A3 protocol calls for 5 days with 5 replicates per day to verify within run precision and laboratory repeatability. The 25 measurements in an EP15 experiment can also be used to make an estimate of bias for that test material.

It is likely that this EP15 protocol is now being used by manufacturer’s personnel to verify instrument claims. Perhaps it can be further simplified, e.g., duplicates per day for 5 days to provide a confirmation protocol. However, it would be necessary to provide laboratories with some support for the data analysis that involves ANOVA calculations followed by calculations of verification intervals. Less data means the verification results having higher uncertainty and are less reliable for confirming method performance. This would be considerably better than just repeating known patient samples and confirming those results are the same as obtained by a manufacturer’s representative. But how simple is simple enough for the laboratory to actually perform its function and requirement to ensure the quality of the tests it produces?

Already a Problem for SQC

A step in this direction of diluting down our quality standards and practices has already been taken with the introduction of risk management and IQCPs (Individual Quality Control Plans). While an IQCP allows laboratories extreme flexibility in applying various control mechanisms [3], it is not a reproducible process. Different people (or teams) dealing with the same method will most likely come out with different IQCPs. Risk management, as applied in an IQCP, is an arbitrary, non-quantitative process that can readily lead to inadequate QC.

A better approach would be to implement a risk-based SQC strategy as part of a Total QC Plan [4]. This is a more quantitative process that develops the QC plan on the basis of the documented Sigma performance of the testing process. The Sigma-metric is the best indicator of risk because it is calculated from the quality required for the test and the precision and bias observed for the method [5]. Following the “road map” from the CLSI EP24-Ed4 document [6], a practical process using simple graphic tools can be implemented for planning risk-based SQC procedures [7-9].

How Low is too Low?

There actually is a case that has demonstrates the problem of setting quality standards too low. That is the FDA’s initial set of requirements for Emergency Use Authorizations of tests for COVID-19. Allowing for the use of “contrived” samples was perhaps necessary in the first few weeks of the pandemic, and while a requirement for using real patient samples was later imposed, some tests have never been properly validated by either the manufacturer or the laboratory. With laboratories still having supply chain problems, the natural process of gravitating to better methods may not be possible in many laboratories. What once were minimally acceptable tests are no longer acceptable, but are still in use because of necessity.

Once quality standards are reduced, such as for the validation of tests for COVID19, it is difficult to raise those validation/verification/confirmation standards to a new and acceptable level. For SQC, it is even worse. Compliance with CLIA requires only the analysis of two levels of control every 24 hours, therefore that’s what manufacturers recommend, therefore that’s what many laboratories do, without any consideration of the quality required for intended medical use and the observed precision and bias of the method in the laboratory.

What’s the point?

Laboratories need simple scientifically valid protocols that are practical experimentally as well as practical for data analysis and calculations. If method performance confirmation is now becoming an accepted function, laboratories need a new protocol to confirm the manufacturer’s installer’s verification of the manufacturer’s validation claims. Just repeating the analysis of some of the samples analyzed by the manufacturer’s installer is not good enough. Perhaps this new confirmation protocol can be a simpler version of the EP15 protocol, along with some support for the ANOVA calculations for precision and some guidance on how to use those results for confirmation of bias. Let’s commit to a scientifically sound approach, an experiment that is practical to perform in the laboratory, and data analysis that can be performed by laboratory personnel, in order to maintain a standard of quality that is appropriate for patient care.

References

1. VN Newitt. New checklist hones lab’s verification and validation requirements. CAP TODAY, July 2020.
2. Rosenbaum MW, Flood JG, Melanson SEF, et al. Quality control practices for chemistry and immunochemistry in a cohort of 21 large academic medical centers. Am J Clin Pathol 2018;150:96-104.
3. CDC, SMA, US Department of Health and Human Services. Developing an IQCP. A step-by-step guide. wwwn.cdc.gov/CLIA/Resources/IQCP/
4. Westgard SA, Bayat H, Westgard JO. Selecting a risk-based SQC procedure for a HbA1c Total QC Plan. J Diabetes Sci Technol. 2018;12:780-785.
5. Westgard JO. Six Sigma Risk Analysis. Designing analytic QC plans for the medical laboratory. Madison WI:Westgard QC, Inc., 2011.
6. CLSI C24-Ed4. Statistical quality control for quantitative measurement procedures: principles and definitions. 4th edition. Wayne PA: Clinical and Laboratory Standards Institute, 2016.
7. Bayat H, Westgard SA, Westgard JO. Planning risk-based SQC strategies: Practical tools to support the new CLSI CwrEd4 guidance. J Appl Lab Med 2017;2:211-221.
8. Westgard JO, Bayat H, Westgard SA. Planning risk-based SQC schedules for bracketed operation of continuous production analyzers. Clin Chem 2018;64:289-296.
9. Westgard JO, Westgard SA. Establishing evidence-based statistical quality control practices. Am J Clin Pathol 2019;151:364-370.