Advanced Quality Management / Six Sigma

Want World Class Quality? You need to ask for it by name. But getting the right Sigma-metrics requires more than just requiring Sigma-metrics in your tender or RFP. You need to specify more than just Sigma-metrics. You need to specify the goal, the level, the time period... well, we'll tell you how it can be done.

Seven Tips to Better Sigma in ‘17

Gentle Advice for Adding Six Sigma to the Tender process

Sten Westgard, MS
January 2017

Adding Six Sigma to your list of vendor demands seems like an easy, simple thing to do. And since we’re well into the second decade of the 21st century, it’s long overdue that we ask for a proof of test quality in our bidding process. After all, if you run the tender process and don’t ask for a specification for quality, aren’t you exposing yourself to the danger that you will get the cheapest instrument that sacrificed quality to reach the lowest price? When we look at goods and services, don’t we often find that the cheapest price comes at the cost of sacrificing quality? You can get very cheap food with a very fast turnaround time at McDonald’s, but we all know that the quality of that food is not healthy for us.

So here's a way you can add quality into your tender, bidding process, and/or request-for-proposal. Ask for Sigma-metrics:

1. Specify the goals. The analytical Six Sigma is not quite so straightforward as other Sigma-metrics (for pre-analytical and post-analytical processes). There isn’t a universally-accepted performance specification for tests, or even a global consensus on what analytical goals should be achieved. In some parts of the world, “Ricos goals” are popular, even though they may be so tight for some analytes that high sigma-metrics are unachievable. In Germany and countries that follow the Rilibak model, there are a different set of goals. In the US, of course, CLIA goals are the law of the land. This lack of consensus on goals from country to country means that you can’t assume what goals are being used by your vendors when they submit Sigma-metrics. Instead, you must specify what goals should be used. For instance, in the US, you should mandate that the metrics be calculated using CLIA and CAP goals (there’s no use in seeing how well the instrument can be benchmarked against German Rilibak goals – you’re going to be operating it here in the US under CLIA regulations).
   
   
2. Specify the decision level (s) for each test. We all know that analytical method performance varies along the span of the working range. A test typically has poorer performance near the lower end of its range. We also observe that when we run two or three controls on a test method, the performance at each level may be different. If we would use each level as a way to evaluate Sigma-metrics, we could potentially get 2-3 Sigma-metrics per test, and they could be different in significant ways. Do we want to ask for 2-3 different Sigma-metrics per test? Or is there a way to focus on just one critical decision level instead? When laboratories determine their own Sigma-metrics, we strongly recommend that laboratories calculated the Sigma-metric at or near the most important decision level. This is advice that should extend to the bidding process. When you ask for vendors to provide you Sigma-metrics, make sure that data is coming from the relevant part of the reportable range. It would be easy to supply very high metrics by just using the data from the higher controls. That might generate impressive metrics, but it might also be entirely irrelevant to the clinical use of the test. It may be far more useful for individual laboratories to specify their own decision levels and demand the Sigma-metric performance at those levels. Particularly for laboratories that serve distinct patient populations, these decision levels will not be the same as other laboratories.
   
   
3. Specify the time period for determination of the imprecision. We all know how to get great imprecision: use a within-run estimate. If you use the CV determined over a very short period of time, within a single run, you cut out a lot of the variation from day to day, operator to operator, calibration to calibration, reagent lot to reagent lot, etc. But best practice, as recommended by CLSI EP5 and CLSI C24 version 3, for determining imprecision is not a short-term study, but something more like at least a month of data, and ideally 3-6 months of cumulative performance data. Get a realistic estimate of imprecision from the vendor, not something so optimistic and contrived that you’ll never actually experience this performance in your own routine operations.
   
   
4. Specify that the "raw" data be made available to you. It is important to be able to double-check the math. If you have the imprecision and bias data provided, you can confirm that the vendor performed the sigma-metric calculations correctly. [I know you will be shocked to hear this, but sometimes manufacturers make mistakes in their calculations of Sigma-metrics.]
   
   
5. Specify that imprecision data be obtained from independent, third-party controls. It has even reached the stage of ISO 15189 recommendation that labs should be using controls that are not from the manufacturer. The problem with manufacturer-created controls is that they can be “tweaked” or adjusted along with the reagent lot, and therefore the controls won’t pick up any changes in performance – and give you a false sense of security. You must have an objective view of analytical performance.
   
   
6. Seek out published papers of instrument evaluation and performance independent of the vendor. This is part of your due diligence. The manufacturer can provide their Sigma-metrics, but does that agree with what the scientific literature says? Again, make sure you’re not being given a “sweetened” version of the performance data.
   
   
7. (Optional) Try to get the performance data from one of the site visits or reference laboratories. Often the diagnostic vendor will take you on a tour of another customer laboratory. That’s a real-world laboratory, so if they have their performance data, that may provide you a more accurate picture of performance than what the manufacturer will provide to you in the bid application. Obviously, the site visit isn’t going to be one of the worst labs, so there may still be some better-than-average performance on those Sigma-metrics, but at least you will know that you have been to the lab, seen how they work, and then you can match up the Sigma-metrics with what you have seen.

Some of this may seem like common sense. But there are two main goals when asking for Sigma-metrics: not only to get the right data from one of the vendors, but to get a standardized apples-to-apples comparison of all the vendors.

Some of you recognize the skepticism, even some cynicism, implicit in these recommendations. Yes, there are manufacturers out there who will “nudge” their Sigma-metrics to look better. And yes, if you don’t tell them the rules, they will make up their own. They have every incentive to look as good as they can, but make sure you see them as they really are.

Your patients are relying on you.