Questions

In August 2021, the webinar, Sigma Stronger, covered the latest developments in quality benchmarking and updates to the design of QC procedures. After a lively question and answer session, all the questions that couldn't be answered live were saved for later reply. Here are all the questions and answers.

Q and A from the Sigma Stronger Webinar

Sten Westgard, MS
September 2021

These questions were from the audience participating in the Sigma Strong webinar in August 2021.

Question: For Sigma Metrics calculation, do you recommend to derive the 'bias' from the difference between the EQA assigned value and the Lab's value? If not, what would you suggest?

There are multiple ways to determine bias. Ideally, we would like to know the difference between our measurement and a reference method or reference material. That would tell us the “true” bias. This ideal is difficult to reach, with reference methods and reference methods often too expensive, too distant from the lab, or in some cases, non-existent (for methods without standards). EQA/PT surveys serve the role of monitoring bias over the long term, some of them are in fact “accuracy-based” and include a reference value/assigned value. Most EQA/PT surveys, however, are consensus-based, and you can compare only to the most common value of your peers or of another group. One of the drawbacks of many EQA/PT surveys is that they occur only a few times a year. In the US, some PT surveys only come twice a year. It’s less than ideal to have to wait 6 months to find out the size of your bias. That’s why peer group programs may offer a better, more practical, determination of bias. The peer group mean is updated much more frequently, is more accessible, and is built on a lot more data than the EQA/PT means. In situations where the reference ideal, EQA/PT, and peer groups are not available or practical, an assayed control can provide at least a target or assigned mean, against which you can compare your observed mean. This is less desirable, but at least it can give you some estimate of bias.

Question: Regarding the 'bias' again; if you suggest to use 'peer group' data, using, for example, %CV and difference between my mean and peer group mean as bias from a QC vendor's interlaboratory comparison could be acceptable?

Yes

Question: If an assay is determined as a six sigma assay how many times do controls need to be run?once?

A Six Sigma assay, using the Parvin QC Frequency model, could reduce its QC frequency to once per 1,000 samples. Bear in mind, that does not mean running 1 control per 1,000. It means running all the controls that are necessary for QC – which for CLIA means at least 2 controls. But remember there are many other reasons controls need to be run – and the principle reason for QC frequency tends to be regulatory compliance. If you are a low volume laboratory that runs 100 tests a day, even if you have a Six Sigma assay, you cannot run QC only once per 10 days. You will need to run QC every day that you run patients, not only because of regulatory compliance, but also since the manufacturer instructions include daily start-up, calibration, and other activities that “break” the stability of the run. The QC frequency recommendations for 5 and 6 Sigma primarily benefit the very high volume laboratories that run thousands of specimens per day.

Question: What can be the reason when 2 out of 3 analyzers running positive 1 level pos bias after new reagent lot implemented?

There are an infinite number of reasons why any instrument might be out of control on any particular day. Remember that even when the same instrument is purchased multiple times for a laboratory, each instrument is unique and individual. So the fact that 2 similar instruments in your laboratory are "in" control, that does not guarantee that a 3rd similar instrument will not be "out".

Question: How often do we need to evaluate an assay to check whether the sigma metric is maintained?

The Sigma metric can be checked as often as you want, but recalculating after every QC point is overkill. After a significant change in the performance – either due to maintenance or a major out-of-control event – it is commonsense to re-establish performance specifications such as mean, SD, and also therefore, the Sigma-metric. If the instrument is humming along quite stable, however, then perhaps reviewing the Sigma-metric every month (at the same time you perform your monthly QC review), or every quarter is a good option. Just because you check your Sigma-metric doesn’t mean you are going to have to change your QC. If you have an assay at 6, 7, 8, then back to 6 Sigma, you will still have the same QC recommendation (1:3s with 2 controls). As long as the Sigma-metric fluctuates within the same category (6,5,4,3), the QC doesn’t have to change. Only when the Sigma-metric changes significantly (from 3 to 4, from 5 to 4, etc.) do you need to change how you do QC.

Question: Can you comment on when QC is commonly run? For instance, immediately after daily maintenance? once a shift? Can different concentrations be run at different times, instead of together at the same time?

There are many approaches to the placement of controls within a run. One approach is to run the controls immediately after all the start-up activities. This has two attractions: 1. The method is at its best, therefore you are most likely to get good results that say you are “in”. 2. It makes sense to test the instrument right away and make sure you are “in” before you run any or any significant number of patients.

Another approach is to space the control measurements throughout the run, so that there is more continuous monitoring of the performance. If you run all your controls up front at the beginning of the day, you don’t know what’s happening the rest of the day. If you instead space the controls so there are some in the morning, some mid-day, some in the evening, you can catch errors – if they occur – more quickly and with fewer patients impacted. This may mean running a “low” control in the morning, a “high” control mid-day, and the “normal” control at night. Any kind of scheduling of the levels is possible.

In years past, Westgard recommended a Multistage approach to QC, where you designed a “startup QC” procedure, with more control rules and more stringent rules, and after that, you switched to a more lenient “monitor QC” with fewer measurements and a wider single rule. In practice, most laboratories devote only enough effort to design a single approach to QC, and then they cross their fingers that the impact of errors, when they happen, will be minimal.

Question: I understand the use of sigma for establishing and running QC. Is there an application for anticipating or determining sigma as part of new method evaluation? And if so, how might this be approached?

The Sigma-metric is ideally suited to aiding in the verification and validation of a new method. If you are bringing a new method into the laboratory, it’s obvious that you want to identify and exclude assays that are 3 Sigma and lower. And if you establish your Sigma-metric during verification and validation, you will also know how you will need to design and implement QC for the new method.

The validation studies for imprecision CLSI EP5 and bias CLSI EP9 allow you to gather all the information necessary to characterize the performance on the Six Sigma scale. So as you complete the studies for validation, you automatically have all the information you need to determine the Sigma-metric at the same time. (Well, almost – you need the performance specification or TEa in order to calculate the Sigma metric. That quality goal is often included in validation studies but not always.)

Thanks to all who participated and sent in questions.

Don't see the question you'd like to see answered? Send in the question here.