Basic Method Validation
Determining Inaccuracy
Bias, Accuracy, Inaccuracy, call it what you will. Sometimes your method generates different results than other methods, for example, systematically shifting values high or low. Laboratories must determine if bias exists in their methods, and estimate the severity of that bias.
- Review of Method Validation Plan
- What's the Problem?
- Guidance on bias from CMS, State Operations Manual, CAP, and CLSI
- Using the t-test, linear regression, and comparison plots
- What's the point?
- Where to go from here?
- References
Often people talk about their “cholesterol counts” as if the numbers are absolute and inviolable. But the number is not perfect and the reliability of the “count” depends on both the precision and accuracy of the measurement procedure. Accuracy is very important because there are national guidelines for the use and interpretation of a cholesterol test, e.g., a value of 200 mg/dL or less is good and does not require any treatment, whereas a value of 240 mg/dL or higher requires additional follow up by the physician and possible treatment. This assumes that a person can have their cholesterol measured anywhere by any method and get a result that can be interpreted uniformly throughout the country.
For cholesterol, this “gray zone” or “clinical decision interval” is 20% [100*(240-200)/200], which would seem to allow for rather large measurement errors. However, it is important to recognize that an individual subject has a biologic CV of about 6.0%. An individual whose true homeostatic set point is 200 mg/dL would actually be expected to vary between 176 mg/dL and 224 mg/dL (95% confidence interval). When biologic variability is accounted for, the clinical decision interval is actually quite stringent. CLIA actually sets an allowable total error (TEa) of 10% as the acceptability criterion for the combined effect of precision and accuracy in a single test result.
Likewise, there are national guidelines for interpretation of the HbA1c test. The American Diabetes Association recommends that a patent’s value should be 7.0 %Hb or less, and that a change in treatment should be considered if the value is 8.0 %Hb or higher. For accuracy or bias, the NGSP requirement is that the 95% confidence interval for patient differences should be contained within +/- 1.0 %Hb, which suggests that the allowable total error should be 14.3 % [1.0%Hb/7.0%Hb*100], without allowing for individual biologic variation. It is therefore puzzling that the CAP has recently expanded its HbA1c grading criterion from 15% to 20% in proficiency testing surveys.
Review of Method Validation Plan
Accuracy studies include the comparison of methods experiment, as well as recovery and interference experiments. CLIA requires laboratory to perform a comparison of methods experiment for all non-waived methods. Recovery and interference studies are required only for methods that have not been cleared by the FDA or methods that have been modified by the laboratory. The method validation plan calls for interference and recovery studies to be performed in the initial phase, if necessary. The comparison of methods experiment is performed in the final phase, along with the day-to-day replication experiment. Therefore, we will consider the comparison of methods experiment first and the recovery and interference experiments later.
