Sigma Metric Analysis

Continuing in our series of CLIA 2025 check-ups, we take a look at an example cobas e801 in the US. Can these IA assays hit the new goals?

IA on the cobas e801 in the US - Can it hit CLIA 2025 targets?

November 2024
Sten Westgard, MS

The new CLIA PT rules went into legal effect in July 2024, but the Proficiency Testing (PT) programs don't have to implement them in their surveys until January 1st, 2025. That provides another half year of anxiety, or respite, depending on your perspective.

Since the airing of our GAMECHANGE webinar, (which is still available on-demand), we've worked with dozens of laboratories to get a glimpse of the January 1st 2025 future. Using Sigma-metrics, we've estimated the quality of laboratory assays and compared them against the key benchmark: 2.3 Sigma.

Why 2.3 Sigma and not 3 Sigma, the traditional minimum allowed quality? Because CLIA 2025, while tightening the acceptability criteria, still relies on a score of 80% for passing success. That is, a lab can fail 20% of the survey and still pass PT. Therefore, the key metric with an assay is 2.3 Sigma, which is the equivalent of a 20% failure rate. If your assay is failing 20% of the time to achieve the results within the CLIA 2025 criteria, that's the danger zone.

Calculating Sigma metrics from routine laboratory performance data.

For all the examples in this series, we're going to keep the laboratory anonymous, but to give it more relevance to our readers, we'll name a name. This is an Roche cobas e801 instrument we're benchmarking, from a laboratory in the US. Using data on cumluative imprecision and cumulative bias against the peer group, we can extract the data needed to benchmark performance on the Six Sigma scale.

[If you need a refresher on Sigma-metrics, check that out here. In this current section of the website, there are hundreds of real-world examples to review.]

Graphic Display of performance by Normalized Method Decision Chart (NMEDx)

42% of the performance measured here is 3 Sigma, 2 Sigma, or less than 2 Sigma. That's not good.

The four concerning analytes here are LH, Folate, FSH, and TT4, all of which have multiple levels in the 2 Sigma or worse zones. beta-Hcg, FT4, Progesterone, and Vitamin B12 each have one level in the danger zone, while their other two levels are higher. For those assays, it's possible to get lucky in the PT survey.

How does this lab compare to other US labs?

Again, through the work of GAMECHANGE volunteers, we've been able to benchmark multiple labs and instruments from every major diagnostic manufacturer.  

In the center cluster of columns, those represent Roche IA performance, with this example displayed on the 4th from the left. To the right you can see a small cluster of competitor instruments. To the left, you see another group of instruments from yet another diagnostic competitor to Roche. We actually have a LOT more data for that particular diagnostic manufacturer, and we're seeing just the section of the worst performance we've benchmarked. 

But with that glance, Roche seems to have an advantage over the left, but not the right of its major competitors.

Notice also that there isn't as much green in the columns as you have seen with other Sigma heat maps. Remember, this is IA, not CC. The new CLIA 2025 goals are harder on IA than CC.

Want to see more of this? Curious to know where YOU stand with CLIA 2025?

We continue to work with laboratories across the US to assess and benchmark their performance against CLIA's new PT rules. We've many more performance examples (of Siemens and Roche and Abbott and QuidelOrtho), and those are available to participants in our national benchmarking program. Email This email address is being protected from spambots. You need JavaScript enabled to view it. if you're interested in joining this program. If you submit your performance data, you'll get back a custom sigma assessment, FREE, as well as the full graphs comparing your performance to the rest of the US diagnostic landscape.