Method Validation
In 2007, Method Validation Skills are Still Vital
A recent "false epidemic" makes it clear that rigorous statistical skills are not optional in the new era of instrument diagnostics.
January 2007
A couple of recent encounters have made me aware that healthcare laboratories are losing their capabilities for validating analytical methods. In recent discussions with laboratory scientists, I discovered:
- One laboratory whose standard practice for method validation was to hire an outside consultant;
- Another laboratory that depended on the instrument manufacturers to provide the method validation data as part of their installation services;
- One laboratory scientist who was being assigned a method validation study but had NO training or experience in doing such studies.
Remember that CLIA requires that US healthcare laboratories validate the precision, accuracy, reportable range, and reference intervals for all “non-waived” tests. In addition, those tests that are of high complexity require validation of their analytical sensitivity, analytical specificity, and any other performance characteristic required for test performance.
Lack of Method Validation Leads to a False Epidemic
The seriousness of the method validation issue is perhaps best characterized by a January 22, 2007, New York Times article by Gina Kolate “Faith in quick test leads to epidemic that wasn’t.”
“For months, nearly everyone involved thought the medical center had had a huge whooping cough outbreak, with extensive ramifications. Nearly 1,000 health care workers at the hospital… were given a preliminary test and furloughed from work until their results were in; 142 people… were told they appeared to have the disease; and thousands were given antibiotics and a vaccine for protection. Hospital beds were taken out of commission, including some intensive care.
“Then, about eight months later, health care workers were dumbfounded to receive an e-mail message from the hospital administration informing them that the whole thing was a false alarm.
“Not a single case of whooping cough was confirmed with the definitive test, growing the bacterium, Bordetella pertussis, in the laboratory…. Now, as they look back on the episode, epidemiologists and infectious disease specialist say the problem was that they placed too much faith in a quick and highly sensitive molecular test that led them astray.
“….Many of the new molecular tests are quick but technically demanding, and each laboratory may do them in its own way. These tests, called ‘home brews’, are not commercially available, and there are no good estimates of their error rates.
According to Dr. Katrina Kretsinger, who is a medical epidemiologist at CDC, there may be 100 different PCR protocols and methods being used throughout the country. These tests are widely used when a whooping cough epidemic is suspected because they provide quick answers, whereas the definitive method of growing the bacteria can take weeks. Unfortunately, these quick “home brew” tests have not been adequately evaluated, even though the regulations require that highly complex tests receive a thorough evaluation.
In fact, the problem with “home brew” tests is well recognized. The FDA has been working for some time to develop guidelines for ASRs (Analyte Specific Reagents, in their terminology). An initial guidance document was released in late 2006 and the comment period was just extended until March 5, 2007 (according to the January 2007 issue of Clinical Lab News).
What’s wrong?
One problem is the lack of education and training for laboratory scientists. Only a few CLS curricula have specific coursework in method validation concepts and practices. Most laboratory scientists, including Clinical Chemists, learn these skills “on-the-job.” It is interesting to note that the longest running workshop at AACC is the Method Evaluation program taught by Neill Carey, Carl Garber, and Dave Koch. This workshop was honored on its 30th anniversary at the July 2006 meeting in Chicago, which attests to the ongoing need for training in this area.
That problem is much more severe for scientists working in molecular diagnostics! They often lack the fundamental experience in routine clinical chemistry methods where the method evaluation concepts and practices have been developed and are widely applied. Without an understanding of the basic concepts, it is difficult to develop appropriate practices for new measurements and technology.
Another problem is that the inspection/accreditation organizations don’t necessarily evaluate the resources and skills of the laboratory in this area. As long as there is a folder that contains data and some statistical calculations, it is generally accepted that the method validation requirements have been satisfied. Also likely, the inspectors themselves have not have been well-trained in this area and may lack the practical skills to identify the problem and help the laboratory with a solution.
Finally, there are those damned statistics that get in the way of a practical understanding of experimental results. As evidence of this problem, Clinical Chemistry (the International Journal of Molecular Diagnostics and Laboratory Medicine) recommends that authors utilize the Bland-Altman approach (difference plot with t-test statistics) for analyzing method comparison data, in spite of the fact that regression techniques are usually much more informative, particularly in identifying proportional analytical errors that invalidate the error estimates from t-test analysis. Evidently, laboratory scientists are not sophisticated enough to understand and utilize regression analysis correctly. That again speaks to the inadequacy of our education and training programs and the lack of proper guidance in the validation of molecular tests, even by a leading international journal.
What to do?
I tell my CLS students that they have a professional responsibility for the quality of the testing in their laboratory. They must learn how to manage quality because they can’t assume or depend on others to know how to do this. It’s actually an up-hill fight to make them understand that quality is still an important issue, particularly when one observes the complacency in the laboratory community, professional organizations, inspection/accrediation agencies, and even manufacturers
I tell them that we must start with the basics, with simple quantitative methods, and learn the principles and practices in those cases. Then, they must extend those principles and practices to the new tests and technology that they will encounter in the future.
The same applies to the healthcare laboratory in general. A laboratory must acquire or develop the personnel who have the necessary skills to evaluate simple quantitative methods and are able to extend those principles and practices to new tests and technology. That will require highly skilled laboratory scientists who can investigate the critical characteristics of new tests and technology, develop the experimental procedures and appropriate data analysis, determine the requirements for clinical quality, and reliably assess the suitability for routine clinical applications.
While it is true that new tests and technology may require new approaches for quality management, those approaches will likely develop as an evolution rather than a revolution of current principles and practices of metrology - the science of measurements. In the field of chemistry, those principles and practices are best represented by analytical chemistry, not biology or biochemistry or molecular chemistry. In the field of laboratory medicine, the principles and practices of metrology are best represented by the known principles and best practices in clinical chemistry. That's why molecular "home-brew" tests, which are very dependent on recipe and technique, are likely to have problems with quality. That's also why the scientists who develop these tests may not have the necessary training and experience in quality management.
Conclusion
It's "back to basics" time again. Managing the quality of laboratory testing begins with method validation. It's still important, in fact, it may be more important than ever in the age of molecular diagnostics.
James O. Westgard, PhD, is a professor emritus of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.