Basic QC Practices

There's a great temptation to lean on the means and ranges provided by the manufacturer. But the ease of using someone else's work can obscure when real errors are taking place.

Double-Blind QC: How Manufacturer Means and Ranges lose sight of real errors

December 2023
Sten Westgard, MS

[Scene: Shoulder of a highway, two cars. One of them has flashing lights. The other car has a police officer standing outside it, talking sternly to the driver.]

Officer: Do you know how fast you were going?

Driver: I was driving 30 mph.

Officer: No, I clocked you at 120 mph in a 25 zone.

Driver: But my speedometer said I was only driving 30 mph.

Officer: I don’t care what it said, you were going 120.

Driver: But I was using car manufacturer miles. They tell me for every 2 real mph, I’m actually only going 1 mph. So instead of 120 I must have been going 60.

Officer: You’re still over the speed limit.

Driver: But I was also using the manufacturer setting for miles per hour. They tell me that they set my starting speed at -30, instead of zero. So you thought I was going 60, but my car was telling me I was only going 30.

Officer: Even if that were real, which it isn't, you would still have been going over the limit.

Driver: That’s okay, officer. I’m just going to go back and drive this stretch of highway over again. I’m pretty sure I’ll drive under the speed limit when I repeat it.

Does this seem crazy? Would you accept a car that didn’t tell you the actual speed you’re driving?

But that’s what a lot of laboratories do every day when they use manufacturer ranges and manufacturer means. They operate with ranges too wide, with means offset from their true mean, and because of that they don’t truly know whether they are in or out-of-control.
This situation gets exacerbated by the use of manufacturer controls paired with the instrument. When the manufacturer provides both the instruments, reagents, and controls. Since those controls are often provided “for free,” there is a strong incentive for the manufacturer to provide advice that keeps all the values “in.”

In a recent consultation with a laboratory, I checked the ranges that the manufacturer was recommending vs. the actual performance that was observed by the laboratory. 75% of the time, the manufacturer was recommending wider limits than warranted by the actual laboratory performance. And sometimes those limits were much, much, much wider:

For one analyte, the manufacturer was actually suggesting limits that were 12 times the lab's actual SD. With limits so wide, this lab should have been essentially outlier-free.

Nevertheless, this laboratory struggled with their own ranges and the suggested ranges, resulting, in a single month, for only 28 mostly biochemistry methods:

• 78 outliers
• 106 repeats or additional controls
• 11 lot changes
• 22 mean reassignments
• 73 data point deletions

When you work that out over approximately 1,736 controls, that might not seem so bad, but when you think about it spread out over a single month:

• 2.5 outliers a day
• 3.42 repeats or additional controls day
• A lot change about once every 3 days
• A readjustment of the mean almost every day
• 2.35 data points deleted every day

It should be noted that this analytical system is one where the technology and logistics end up generating frequent changes in lots. That can exacerbate an already brittle system.

And remember that’s only in the chemistry section. If we extrapolate that across more tests and more sections, well, that’s a depressing thought.

 Here’s how the problem with manufacturer mean and ranges can manifest:

This looks beautiful doesn't it, but of course there seem to be a lot of values accumulating on one side of the mean.

When you plot with actual lab performance, the picture changes:

In this case, the manufacturer chart misses a big error (a 1:3s violation) and generates a false rejection (an 8:x violation from runs 13-21) with the shifting of the values below the artificial mean. Manufacturer means and ranges both miss real errors and generate false errors.

This reinforces the message from ISO 15189: labs are strongly encouraged to use independent controls. Another strong point: using your own laboratory's mean and SD is the only way to truly see what's happening to your methods.