Tools, Technologies and Training for Healthcare Laboratories
  2. Home
  3. CLIA & Quality
  4. CLIA Final Rule
  5. Part VII: "Equivalent" QC Procedures

CLIA Final Rule

Part VII: "Equivalent" QC Procedures

CLIA Final Rule:
"Equivalent" Quality Control Practices

In the very near future, CMS will release the Interpretive Guidelines to the Final CLIA Rule. Those guidelines will contain a dangerous new practice called "equivalent quality control procedures." Dr. Westgard explains what this new procedure is, and why it is fatally flawed.

November 2003 - Updated January 2004

A recent presentation on the CLIA Final Regulations revealed a new approach for establishing "equivalent QC procedures" [1].

One might ask why there is a need for equivalent QC procedures when the Final CLIA Rule has reduced QC to essentially 2 controls per 24 hours for most tests. After more than ten years of CLIA, the most frequent deficiencies still have to do with QC: some labs still don't run 2 levels per day, don't follow manufacturer's instructions, or don't have any QA program. Hard to imagine, isn't it? And the objective of "equivalent QC" appears to be a further reduction of QC to 2 controls once a week or even 2 controls once a month. As you might expect, I have some serious concerns about "equivalent QC."

The official CMS position on "equivalent QC" is found in the Appendix C of the CLIA Rules, which provides "interpretive guidelines" for laboratory inspectors and is also known as the State Inspectors Manual [2]. These interpretive guidelines are important because they directly influence how laboratory inspectors will apply the CLIA regulations when they inspect your laboratory.

What are the Final CLIA QC Requirements?

493.1256 Standard: Control procedures. [3]

(a) For each test system, the laboratory is responsible for having control procedures that monitor the accuracy and precision of the complete analytic process.

(b) The laboratory must establish the number, type, and frequency of testing control materials using, if applicable, the performance specifications verified or established by the laboratory as specified in 493.1253(b)(3).

(c) The control procedures must

1. Detect immediate errors that occur due to test system failure, adverse environmental conditions, and operator performance.

2. Monitor over time the accuracy and precision of test performance that may be influenced by changes in test system performance and environmental conditions, and variance in operator performance;

(d) Unless CMS approves a procedure, specified in Appendix C of the State Operations Manual (CMS Pub.7) that provides equivalent quality testing, the laboratory must

1. Perform control procedures as defined in this section unless otherwise specified in the additional specialty and subspecialty requirements at 493-1261 through 493.1278.

2. For each test system, perform control procedures using the number and frequency specified by the manufacturer or established by the laboratory when they meet or exceed the requirements in paragraph (d3) of this section.

3. At least once each day patient specimens are assayed or examined perform the following for

i. Each quantitative procedure, include two control materials of different concentrations.

The bottom line for QC is that laboratories must analyze two control materials of different concentrations at least every day or 24 hour period unless CMS approves a procedure that provides equivalent quality testing.

What does "equivalent" mean in the CLIA regulations?

Here's where the information in the State Operations Manual becomes important because CMS's interpretative of equivalency is not the same as what we would expect for the scientific comparison of two QC procedures. "Equivalent" QC means "reduced" QC! Equivalent QC allows a laboratory to reduce QC from the minimum requirement of testing 2 controls per day to testing only 2 controls per week or only 2 controls per month. Going from little to almost nothing!

A quick cursory review of this concept of equivalency should raise concerns about how "immediate error detection"can be achieved if controls are analyzed only once per week or once per month. It gives a whole new meaning to the word "immediate", but our patients probably already know that the healthcare system has a different concept of time than they do. The government's concept of time is even more extended, with immediate meaning 24 hours at the shortest and possibly as long as a week or a month for the new equivalent QC procedures.

What does "equivalent" mean for quality testing and QC?

The purpose of QC is to detect changes in performance, i.e., changes in precision and accuracy that affect the correctness of laboratory tests. Correct test results are critical for correct patient treatment decisions. Therefore, the bottom line for QC is to protect patients from mistreatment, i.e., to assure patient safety. Patient safety is the number 1 goal in healthcare today and QC is an important part of patient safety!

In fact, safety can help us understand QC. Consider the fire alarm, which is universally used for protecting people's safety in both public and private buildings. Fire alarms can be triggered manually or automatically, but it is the automatic mode that is now required by all building codes. The automatic mode makes use of a smoke detector, which functions very much like the error detector or QC procedure used to monitor analytical testing processes.

Ideally, the fire alarm should never sound unless there truly is a fire. We've all experienced false alarms and know how easily a few false alarms can compromise the response to a true alarm, therefore the ideal performance is to have no false alarms. On the other hand, the fire alarm system should be very sensitive for detecting a real fire. It should have a 100% chance of going off when a real fire exists.

Likewise, the performance of a QC procedure can be described by two similar characteristics [4]:

  • Probability for error detection (Ped), which describes the probability of getting a rejection signal when there is a change in the precision or accuracy of the analytical method;
  • Probability for false rejection (Pfr), which describes the probability of getting a rejection signal when there is no change in method performance, except for the inherent imprecision of the method.

Ideally, Ped should be high, near 1.00 to provide a 100% chance of detecting a medically important analytical error. Ideally, Pfr should be low, near 0.00 to provide a 0.0% chance of false rejections that would otherwise waste time and effort and slow the reporting of patient test results.

These two characteristics can also be described in units of time, called Average Run Lengths for false rejection and error detection [5]. ARLfr describes how long it takes to observe a rejection signal when the method performs under stable operating conditions, and ARLed describes how long it takes to detect a problem when something goes wrong with the method. Detection time should be fast when there is a problem, whereas false alarms should be few and far between.

How do you demonstrate equivalency?

Lsn79f1Given that there are two characteristics of interest, the issue of equivalency should consider both dimensions. In a simple manner, we might compare the individual characteristics, one to another, or compare some combined outcome of the individual characteristics. The issue can be considered by looking at the two rectangles shown in the accompanying figure. Are they equivalent? On one hand, both the heights and widths are different. On the other hand, the areas might be very close.

In demonstrating the equivalency of QC procedures, we have a similar problem. We might simplify the comparison by agreeing that the error detection characteristic is more important than the false rejection characteristic. No one will care if a there is a low rate of false alarms if the detector isn't sensitive enough to pickup a real fire. Thus, error detection could be considered the primary characteristic. Of course, false alarms may cause a serious difficulty because people start to disregard the alarm signal, so both characteristics are important in the long run.

Given QC procedure A that has a Ped of 0.90 and QC procedure B that has a Ped of 0.09, which is better? The error detection capability of A is ten times better than B, therefore, A would be preferred.

Given QC procedure A that has an ARLed of 1.1 and QC procedure B that has an ARLed of 4.4, which is better? QC procedure A will detect a problem in 1/4th of the time required by QC procedure B, therefore A would be preferred.

Given QC procedure A that has a Pfr of 0.01 and QC procedure B that has a Pfr of 0.05, which is better? If the method is working perfectly, then A is better because it has a lower probability of false alarms and will minimize the time and effort wasted responding to those false alarms. However, if a problem occurs with the method, we don't know the error detection capabilities of either A or B, therefore we can't make any judgment about which will assure correct test results. We have no way to judging equivalency in the absence of information about error detection.

How is equivalency established via CMS guidelines?

The CMS guidelines allow a laboratory to analyze 2 external controls per day for a period of 10 to 60 consecutive days to evaluate equivalent QC. If the internal and external control results are acceptable during this entire period of testing, the laboratory may reduce the external control testing interval.

What characteristic of the control procedure is being tested here? Error detection or false rejection? Obviously, the evaluation considers a period where stable operation is expected. How could the lack of alarms under error-free operation demonstrate that errors can be detected? How can equivalency be established without testing the error detection capability of the QC procedure? The answer is, that it can't! The prescribed experiment is fatally flawed because it doesn't test the right characteristic, therefore, it can't lead to a correct conclusion.

What tests systems are eligible for equivalent QC procedures?

There is some saving grace in that not all test systems are eligible for equivalent QC. Those NOT eligible include tests with an extraction phase, molecular amplification procedures, thin layer chromatography, electrophoretic procedures, and the specialty and subspecialty test that have specific requirements given in 493.1261 through 493.1278.

Furthermore, the reduction to monthly or weekly QC depends on whether the test systems utilize "internal/procedure controls" and the analytic components that they monitor. Three cases are identified as options for equivalent QC:

  • Test systems with internal or procedural controls that monitor all analytic testing components.
  • Test systems with internal or procedural controls that monitor a portion of the analytic testing components.
  • Test systems without internal or procedural controls.

Based on 10 days of testing for test systems with internal or procedural controls that monitor all analytic testing components, QC can be reduced to one time per month. The obvious question is how ten days data shows equivalency for monthly QC, but as noted earlier, there's nothing rational or scientific about this.

For test systems with internal or procedural controls that monitor a portion of the analytic testing components, 30 days of testing without any QC problems will allow QC reduction to once a week.

For test systems without internal or procedural controls, it takes 60 days of testing without any QC problems to change from daily to weekly QC.

What's the bottom line?

Even the CMS is not willing to fully endorse the practice of "equivalent QC" without reservation! As they state in section D5445:

NOTE: Since the purpose of control testing is to detect immediate errors and monitor performance over time, increasing the interval between control testing (i.e. weekly, or monthly) will require a more extensive evaluation of patient test results when control failure occurs (see 493.1282). The director must consider the laboratory's clinical and legal responsibility for providing accurate and reliable patient test results versus the cost implications of reducing the quality control testing frequency. [CMS emphasis, not ours]

So while the CMS has explained what "equivalent QC" is, they are not willing to state that it is clinically or legally defensible. If you follow this procedure, basically you're on your own.

I suspect that "equivalent QC" was probably a poor choice of words. There isn't any way that a QC procedure performed only weekly or monthly can provide the error detection capability that is equivalent to one performed daily. "Alternative QC" would have been a better term if the intention was for the government to demonstrate some flexibility to accommodate current and future changes in measurement technology.

These guidelines for equivalent QC aren't the answer. Both the concept and experimental approach for "equivalent QC" are fatally flawed and will further endanger patient safety. The ultimate danger is that laboratory tests will become "garbage in - garbage out" and lose all usefulness for medical care.

What to do? In principle, it's very simple! We need to continue to exercise professional responsibility and do a good job of providing the quality test results that our patients need and deserve. In practice, that means selecting QC procedures based on the quality required for the test and the precision and accuracy observed for the method [6]. This website provides a lot of information that will help you do that.

Remember, just because it's possible to get away with something, it doesn't mean we have to take the easy way out. We can and should do what is right to provide the quality we would want if the patient were a member of our own family. No laboratory is required to implement "equivalent QC" and we should not adopt this fatally flawed practice!

If manufacturers want to change from accepted traditional QC practices, then they will need to improve the stability of their methods and/or improve their QC technology, then demonstrate the performance characteristics to the FDA to obtain approval for new QC instructions. That was the intention of the original CLIA provision for QC clearance, and though the Final CLIA Rule removed the requirement for manufacturers to submit their QC instructions for FDA approval, it remains the best way for manufacturers to voluntarily gain approval of alternative QC procedures.

References

  1. AACC Audioconference: CLIA QC Standards Part II: The New CMS Interpretive Guidelines. October 29, 2003.
  2. CMS State Inspectors Manual. (Appendix C is available at http://cms.hhs.gov/manuals/Downloads/som107ap_c_lab.pdf ) Accessed 3/31/09.
  3. CLIA Final Rules. U.S. Department of Health and Human Services. Medicare, Medicaid and CLIA Programs: Laboratory requirements relating to quality systems and certain personnel qualifications. Final rule. Fed Regist 2003;68:3640-714
  4. Westgard JO, Groth T, Aropnsson T, Falk H, deVerdier C-H. Performance characteristics of rules for internal quality control: Probabilities for false rejection and error detection. Clin Chem 1977;23:1857-1867.
  5. Westgard JO, Fallon KD, Mansouri S. Validation of iQM active process control technology. Point of Care 2003;2:1-7.
  6. Westgard JO. Internal quality control: planning and implementation. Ann Clin Biochem 2003;40:593-611.