CLIA Final Rule

An updated contemplation of the newly proposed 2024 CLIA acceptability criteria / acceptance limits (ALs). Here's a trick question: When is a goal not a goal?

Contemplating the Consequences of New CLIA Goals

Sten Westgard, MS
March 2019, Updated October 2022

As I read through the changes to CLIA’s proficiency testing criteria, I am reminded of the 1929 painting, La Trahison des images (“The Tyranny of Images”), by Rene Magritte. The famous Surrealist painted a pipe along with the words, “Ceci n’est pas une pipe” (“This is not a pipe”).

https://www.renemagritte.org/the-treachery-of-images.jsp

More specifically, the phrase in the new regulations that invokes the surrealist Magritte for me is the twice-repeated assertion:
“Criteria for acceptable performance is meant for PT scoring only and not intended to be used to set acceptability criteria for a laboratory's verification or establishment of performance specifications.”

So in the new CLIA regulations, acceptable does NOT equal acceptability.

More technically, the new acceptance limits (AL's) for PT are not acceptablity criteria for methods.

That begs the question, what are they?

The goals that aren’t goals?

I am sympathetic to the plight of the regulator. If the new goals are used as performance specifications, they will have a significant impact to laboratories and the marketplace. The new tightened goals, if used to determine Sigma-metrics, or used to set individual performance specifications, or used to set performance goals for new instruments, will create new winners and losers in the diagnostic space. Given the time we are in, regulators are understandably reluctant to be seen as interfering with the free functioning of the marketplace. Further, if these new goals are usedr as individual laboratory performance specifications, and if this causes many laboratories to reject methods, or declare more runs out-of-control, or even worse, fail PT more often, there will be fewer laboratories that can continue to run tests.

This plight is not unique to laboratory regulation. James Reason, the safety expert, has described what is called “The Regulator’s Dilemma”: regulators, as tough as they may pretend they are, ultimately are invested in the viability of the industry they regulate. Simply put, inspectors cannot destroy the industry that they inspect. If , because of much tighter goals, CMS/CLIA inspectors shut down a large number of laboratories the disruption to healthcare would potentially be catastrophic, and even if patients weren’t denied care, the regulatory actions would draw significant ire, protest and lobbying. Therefore, inspectors need regulations that are tight enough to eliminate a certain percentage of “bad” laboratories, but they have to allow the vast majority of labs to stay in business.

Evidence of this approach can be seen in the discussion of how the new goals were determined. Simulations of new goals were generated using existing PT survey data, with an eye on the “miss rates”:

“While narrowing limits may increase miss rates per challenge, we do not expect a high unsuccessful rate based on the data simulations provided by the PT programs. We expect the rates of unsatisfactory events would be low based on the simulation data, and that the rates of unsuccessful events (two consecutive or two out of three testing events being unsatisfactory) would be even lower; therefore, we believe it is reasonable to propose tighter limits given current analytic accuracy. We used all data available to us to minimize the negative consequences of the proposed changes (for example, too many unsuccessful performances) to acceptance limits, including simulations provided by PT programs.”

Thus, the goals being proposed are trying to keep 98-99% of laboratories in compliance, and at most, trying to reject about 1-2 % of the current performing laboratories. This is hardly a radical improvement. The expansion of regulated analytes and the tightening of goals merely keeps CLIA relevant, rather than antiquated, in the realm of laboratory testing.

Another reason to view the new proposed goals as authentic performance specifications is to examine how they were changed. Primarily, the new proposed goals were tightened to conform more closely to the biological goals :

“There are 53 analytes (existing or proposed) for which we are proposing a percentage-based AL, for which biological variability data were published.... Where there were such data, we used AL to get as close to, or below, an accuracy goal for the test that was based on biological variability data. Then we simulated several percentage-based ALs to see if their results would have passed or failed at each simulation…. Of the 53 analytes, 34 of the proposed ALs were tighter than or equal to biological variability limits. For 19 analytes, the limits we are proposing are looser (greater) than the limits required to meet accuracy based upon biological variability. For these 19 analytes, using ALs based upon biological variability would be untenable because the current analytical accuracy for such testing would not be expected to be able to meet such limits.”

The biological variability-based goals to which CLIA refers are goals explicitly meant to be performance specifications for laboratories, and are used as such worldwide. If CMS/CLIA are aligning the CLIA goals with biological goals, they are making these PT goals closer to “real” performance specifications.

CLIA has built the new acceptance limits on a foundation provided by the acceptability framework of biological variation. They even explicity acknowledge this debt: "Over the years, there have been many proposed criteria for establishing goals for analytical performance. The various possible approaches werer reviewed an a heirarchy was established based on a 1999 consesus conference [ed. Stockholm]. There strategies were reconsidered at the 2014 European Federation of Clinical Chemistry and Laboratory Medicine Strategic Conference in Milan."

Those conferences, those heirarchies, were built to provide acceptability criteria, performance specifications, not "acceptance limits."

Clearly, the “this goal is not a goal” statement is absurd on its face, a regulatory sleight of hand, and an abdication of responsibility by CMS. They want to regulate labs, but not regulate the analytical performance of labs. Really? Isn’t providing analytically reliable test results that the primary reason we have labs in the first place? Not just to move specimens around, but to test them and, to produce results that are accurate and appropriate for patient care. We see a lot of regulations surrounding analytical quality, but there seems ot be a failure of will to truly set a benchmark for the quality of the core competency of the laboratory.

Now, a valid argument against a direct imposition of performance specifications is that every laboratory should establish their own individual goals. Every laboratory has their own unique patient population, clinical needs, interpretative needs, etc. It’s not really possible to create a single, top-down set of performance specifications that is appropriate for all laboratories and all clinical care situations. So each laboratory should make their own choice. In practice, however, labs don’t engage in circumspection and meditation to generate their own specific goals for targets for quality. They adopt recommendations from the scientific literature and regulatory/accreditation requirements, such as the CLIA PT goals. Someone else’s goals. The fact that they are judged by the CLIA PT goals makes those goals the most relevant performance specifications to use for laboratory testing in the US.

Regardless of the semantics and intention of the new proposed goals for CLIA, the reality is that, “if it looks like a goal, smells like a goal, is used by inspectors like a goal: IT’S A GOAL”. Just as the CLIA goals of 1992 were adopted as analytical performance specifications for laboratories throughout the country and even worldwide, the ultimate use of these new CLIA goals will be as validation and verification goals and as individual performance specifications, in addition to their official designated purpose as PT scoring goals. The committees may not have intended the new Acceptance Limits to be used for other purposes, but they would be naïve to believe that they won’t be used in this way.

A goal by any other name, even by any other intention, is still a goal.