Sigma Metric Analysis

After GAMECHANGE, we were flooded with requests by laboratories who worried that they were going to fail the  new CLIA 2025 PT rules. Volunteered data allowed us to provide some benchmarking and understand who was in trouble and who wasn't. An anonymous example shows what methods are safe and what are in jeopardy.

Two Immunoassay Analyzers Prepare (or Panic?) for CLIA 2025 PT

October 2024
Sten Westgard, MS

The new CLIA PT rules went into legal effect in July 2024, but the Proficiency Testing (PT) programs don't have to implement them in their surveys until January 1st, 2025. That provides another half year of anxiety, or respite, depending on your perspective.

Since the airing of our GAMECHANGE webinar, which is still available on-demand, we've worked with dozens of laboratories to get a glimpse of the January 1st future. Using Sigma-metrics, we've estimated the quality of laboratory assays and compared them against the key benchmark: 2.3 Sigma.

Why 2.3 Sigma and not 3 Sigma, the traditional minimum allowed quality? Because CLIA 2025, while tightening the acceptability criteria, still relies on a score of 80%. That is, a lab can fail 20% of the survey and still pass PT. Therefore, the key metric with an assay is 2.3. If your assay is failing 20% of the time to achieve the results within the CLIA 2025 criteria, that's the danger zone.

Calculating Sigma metrics from routine laboratory performance data.

For this example, we're going to not only keep the laboratory anonymous, but also the instrument and the control material. So this is a completely abstract set of data. The Sigma metric was calculated based on the routine QC data, in most cases over 100 data points (more than 3 months of performance) for the imprecision, and the observed mean was compared against the expected mean to estimate bias. Of course, CLIA 2025 goals were used to set the allowable total error (TEa).

[If you need a refresher on Sigma-metrics, check that out here. In this current section of the website, there are hundreds of real-world examples to review.]

 The PT challenges here are the low end of  Vitamin B12, mid-level Free T4, and most of the Folate range. Out of 8 analytes, there are vulnerablities in 3 of them. Depending on where the PT specimens are aimed, it's possible to get lucky and miss being tested in those vulnerable ranges.

From a QC perspective, there are assays where few Westgard Rules are needed, but more assays where the laboratory should consider more rules. If the current lab QC practices are lax (wide ranges, single rule), then January 1st will be a rude awakening.

The second instrument does better:

The only real trouble is the Folate assay. This is similar across both instruments, indicating this may be more of a manufacturer's design issue than the lab's performance issue. Unless the laboratory itself is found at fault in how they operate the Folate method, it could be that the Folate method, as currently designed, is not up to the standards that CLIA have now set.

Graphic Display of performance by Normalized Method Decision Chart (NMEDx)

How does this lab compare to other US labs?

Again, through the work of GAMECHANGE volunteers, we've been able to benchmark multiple labs and instruments from every major diagnostic manufacturer.

Here we can see an anonymous "Sigma Heat Map" of other peer instruments and labs. You can decode this heat map this way. You want to see Green, Dark Blue, and Blue; that represents the percentage of the assessed menu that achieved 6, 5, or 4 Sigma performance. The Yellow represents 3 Sigma performance (minimum acceptable performance analytically, but passable for PT). The Red represents the percentage of the menu that falls below 3 Sigma, and thus represent unacceptable performance as well as heightened risk for PT failure come January 1st.

In this snippet of the graph, the 6 columns represent similar instruments and labs. Our anonymous instruments are the 1st column and 3rd column from the left of the 6 peers. Thus, we would say the laboratory is performing better than most of its peers. The percentage of Six Sigma performance in the group seems fairly stable, but the peers of this laboratory have some larger proportions of red and yellow.

And what's the column all the way on the left? That's a competitor, a next generation IA analyzer, and it bests all of the instruments from this manufacturer. For that instrument on the left, January 1st 2025 brings with it no new anxiety, no new danger of PT failures. The 6 on the left have some worries. But believe us, there are far worse instruments out there in US laboratories...

Want to see more of this? Curious to know where you stand with CLIA 2025?

We continue to work with laboratories across the US to assess and benchmark their performance against CLIA's new PT rules. Email This email address is being protected from spambots. You need JavaScript enabled to view it. if you're interested in joining this program. If you submit your performance data, you'll get back a custom sigma assessment, as well as the full graphs comparing your performance to the US diagnostic landscape.