Advanced Quality Management / Six Sigma

As more and more papers are published about Six Sigma metrics, here are the secrets to getting better papers and better outcomes.

Six Secrets for better Sigma Metric Studies

Part One: Evaluate, but Commit. Weigh the options, but make a decision.

November 2020
Sten Westgard, MS

Two decades ago, the field of Sigma metric studies didn’t exist in laboratory medicine. It wasn’t until David Nevelainen's et al[1] groundbreaking paper that Six Sigma was even mentioned in connection with quality indicators in the lab.

My, how things have changed in the past 20 years.

At first, probably the only publications about Six Sigma and Sigma metrics came from either "the" Westgard and the much less important “a” Westgard[2-6]. But as they years have rolled by, more and more labs and lab professionals paid attention and began looking at their own processes and publishing their findings and their experiences. Today, in 2020, I can’t keep up with all the papers that are coming out with Sigma metrics. Nor can I keep up with all the review requests. But as a frequent reviewer of Sigma metric submissions, I have noticed a number of weaknesses in Sigma metric studies. Things that diminish the impact of a paper, but also things that would reduce the utility of Sigma metrics being implemented in the laboratory. The same drawbacks that may prevent a Sigma metric paper from being published can easily lead a laboratory to fail in their implementation of Sigma metrics.

Thus, here is a quick set of tips on some of the important issues.

Secret 1: Choose a set of Total Allowable Error (TEa) Goals

There are many different TEa goals out there to choose from. There is also an vigorous debate about which set of goals is “the best.” One key thing that came out of the 2015 Milan Consensus statement[7] is the admission that No Single Set of TEa goals is going to fit all tests. In fact, the Milan Consensus encourages laboratories to make use of all types of goals where they are appropriate – a recommendation we have attempted to follow through our list of Consolidated goals (8] and our Sigma VP goals [9]. While there is an prevailing opinion among some reviewers that the European Federation of Laboratory Medicine (EFLM) biological variation-derived goals[10] are the most scientific, there’s a corresponding inability of today’s instrumentation to achieve those goals. If your most evidence-based goal is impossible to achieve, you haven’t forged a new path, you’ve driven into a ditch.

Here’s the secret: as long as we have professional laboratorians with a bit of time on their hands, there is going to be a debate on which goals are most important. Further, the continuous evolution of medical care and treatment, alongside the relentless improvements in instrumentation and engineering, mean that there will be new tests constantly introduced, as well as new uses of existing tests, and better precision of existing tests leading to increased ability to track smaller changes of clinical status. We should always expect changes for analytical performance specifications.

I see a lot of papers where the Sigma metrics are evaluated by multiple sets of TEa goals in an attempt to straddle the debate without taking a side. But that results in a paper that has no endpoint. You’ve outlined two recipes to bake the cake but you haven’t put anything in the oven. Certainly you can make analyses with different sets of goals, but at the end of the day, what is your lab actually going to do? If you are in the USA, your laboratory is bound by CLIA and CAP goals, so implement your QC design based on metrics derived from those goals. Similarly, laboratories in China should plan their QC around the NCCL goals, etc Don't ignore your own reality. Face it.

Secret 2: Choose a medical decision level

Many laboratories that attempt to start with Sigma metrics face the same implementation challenge: if you run two control levels (or three), will you may very well get two Sigma metrics (or three). Does that mean you have to have different control rules for each control level?

That way madness lies.

It’s not a surprise that you might get different performance at different levels (if the performance was the same at all levels, you wouldn’t need to run two or three control levels). If you think about controls being placed at the low end of a range, it’s entirely expected that the precision down there is different than the precision found at the high end of the range. This secret is a similar challenge to the first one.

You can certainly calculate Sigma metrics at multiple levels, but the practical reality is that you can only implement one QC design for all the levels of any particular test. You’re not going to be able to run full “Westgard Rules” on your low control, with 4 to 6 control measurements, while on the high control, you’re only doing 3 SD, with just 1 control. I haven’t seen the software that could even attempt to implement such a strategy.

What we encourage instead, and this I hope is common sense, is that you plan your QC around the most important medical decision level.  At what level do the most critical diagnoses occur? Where is there a cutoff that divides sick patients from healthy patients? Wherever that decision is being made, that’s where you want to know the performance, that’s where you want calculate the Sigma metric, and that’s where you want to optimize your QC rules and control measurements.

We have a set of recommendations, some a little dated[11], and others more contemporary [9]. Again, the most interesting part of these decision levels is what is appropriate for your laboratory and your patients. If you’re running a dialysis clinic, your calcium decision levels are going to be different than we what typically expect for “normal” patients. If you’re implementing a tight glycemic control protocol, your decisions on glucose are definitely different than usual.

If you can show that you have evaluated the possibilities, and committed to a choice, and then those results are worth sharing. Otherwise, you’re stalled at a fork in the road and you’re running out of gas waiting to decide.

That’s the first installment of this Six Sigma Secrets series. Stay tuned for part 2 next month.

References

1. Nevelainen D, Berte L, Kraft C, Leigh E, Picaso L, Morgan T. Evaluating laboratory performance on quality indicators with the six sigma scale. Arch Pathol Lab Med 2000 Apr;124(4):516-9.
2. Westgard JO. Six Sigma Quality Design, 2nd Edition. Westgard QC, Madison, Wisconsin. 2006.
3. Westgard JO, Darcy T. The truth about quality: medical usefulness and analytical reliability of laboratory tests. Clin Chim Acta. 2004 Aug 2;346(1):3-11. doi: 10.1016/j.cccn.2003.12.034. PMID: 15234630
4. Westgard JO, Westgard SA. The quality of laboratory testing today: an assessment of sigma metrics for analytic quality using performance data from proficiency testing surveys and the CLIA criteria for acceptable performance. Am J Clin Pathol. 2006 Mar;125(3):343-54. PMID: 16613337
5. Westgard JO, Westgard SA. Assessing quality on the Sigma scale from proficiency testing and external quality assessment surveys. Clin Chem Lab Med. 2015 Sep 1;53(10):1531-5. doi: 10.1515/cclm-2014-1241. PMID: 25719323
6. Hens K, Berth M, Armbruster D, Westgard S. Sigma metrics used to assess analytical quality of clinical chemistry assays: importance of the allowable total error (TEa) target. Clin Chem Lab Med. 2014 Jul;52(7):973-80. doi: 10.1515/cclm-2013-1090. PMID: 24615486
7. Sverre Sandberg, Callum G Fraser, Andrea Rita Horvath, Rob Jansen, Graham Jones, Wytze Oosterhuis, Per Hyltoft Petersen, Heinz Schimmel, Ken Sikaris, Mauro Panteghini Defining analytical performance specifications: Consensus Statement from the 1st Strategic Conference of the European Federation of Clinical Chemistry and Laboratory Medicine. Clin Chem Lab Med 2015 May;53(6):833-5. doi: 10.1515/cclm-2015-0067.
8. Consolidated Comparison of Chemistry Performance Specifications, Westgard QC. https://www.westgard.com/consolidated-goals-chemistry.htm
9. Westgard Sigma Verification of Performance Program goals. https://www.westgard.com/sigma-verification-of-performance-program.htm
10. EFLM Biological Variation Database. https://biologicalvariation.eu/
11. Statland BA Medical Decision Levels https://www.westgard.com/decision.htm