Trends

The Theranos Scandal is an object lesson for the laboratory industry. It's not only an example of what NOT to do, but also an example of what we risk when we don't demand proof of quality, or what we might suffer if we don't prove the quality we're delivering. We don't want to be Theranosed. And we certainly don't want to be Theranosing others...

Thuffering Theranos!
How to make sure you're not being Theranosed.
And how to prove you're not Theranosing your clients, customers, clinicians and patients

Sten Westgard, MS
June 2016

It seems a week cannot go by without more bad news for the former darling startup:

• There’s the release of "lightly redacted" inspection report, which reveals finally some damning details about the performance of the Edison machine.
• The CMS response to Theranos’ response, detailing how 43 out of 45 times, Theranos didn’t provide enough evidence to show it was making real corrections.  And the threatened banning of Elizabeth Holmes from running any laboratory in the US for two years.
• The “retirement” of President and COO Sunny Balwani
• The voiding of two entire years worth of results reported to patients by Edison machines an unprecedented recall in the history of diagnostics.
• The subsequent filing of class action lawsuits against Theranos for problems that may have been caused or missed by those voided results.
• The opening of Federal and SEC investigations into whether or not Theranos misled Federal officials and/or investors, thereby possibly committing fraud.
• Forbes revision of Elizabeth Holmes' worth from $4.5 billion to zero.
• The severance of the deal between Walgreens and Theranos, cutting off Theranos from its biggest source of revenue and customers.
• The announcement that Jennifer Lawrence  has been attached to star as Elizabeth Holmes in a movie adaptation of this sad story, regardless of the fact that we still don’t know
• Finally the announcement that Elizabeth Holmes will appear at the AACC conference in Philadelphia to present her data and answer questions about the technology.

The problems are so notorious that "to Theranos" has now been turned into a verb equivalent "to Enron."

There’s so much that’s happened that it’s hard to know where to start. Indeed, most of the stories have been covered by other news outlets already, and by real journalists. About the only additional insight we can add here is a closer reading of the lightly redacted inspection report. Because buried in that are some performance details that no one else seems to have noticed.

But before we do that, let's admit that it's more than a little embarrassing that it got this far. That a company that provided no shred of proof of its performance was nevertheless able to market itself to hundreds of millions of dollars of venture capitalist funding and a $9 billion valuation and the Time 100, etc. The laboratory industry remained largely silent when we should have been vocal, challenging, and skeptical.  Instead, we let a startup bully and bluster its way into prominence, and at the same time, claim that all of us in the the laboratory world and diagnostics industry were unnecessary, antiquated, and soon to be driven into obsolescence.

Let’s start with the QC failure rates. The inspection report details that there were significant out-of-control results for many tests, sometimes up to 87% of QC results were out more than 2 SD!!

Knowing these failure rates, we can easily convert those into Six Sigma benchmarks, using the simple short-term scale (a table look up based on the DPM/defect rate)

Recall that the minimum performance required for stable operation is 3 Sigma. Given that, and the inability of any Edison methods to achieve even close to 3 Sigma, we are hardly surprised by the voiding of two years of test results.  Indeed, based on those Sigma benchmarks, you might even predict the unreliability of the results.

But the QC failure rates aren’t the only hidden nuggets in the report. There are also some details on the reported imprecision rates of some other tests.

The report notes that the imprecision for Vitamin B12 at level 1 was 52.5% and level 3 was 48.5%. Already that doesn’t sound good, right? But not put it into context: the “Ricos goal” – a desirable specification for allowable total error of 30%, as well as a specification for allowable imprecision of 7.5%. So the Theranos method exceeds the recommended imprecision by more than 600%!  The imprecision alone is nearly double the allowable total error (even if we accounted for bias, we are in terrible shape).

The inspection note also notes the imprecision for Vitamin D at level 1 was 63.8% and at level 3 was 26.4%.  Again, this doesn’t sound good, but just how bad is it? For that we can consult a paper from 2011:

Adie Viljoen, Dhruy K Singh, Ken Farrinton, Patrick J. Twomey,  Analytical Quality Goals for 25-Vitamin D Based on Biologic Variation, Journal of Clinical Laboratory Analysis 25 : 130–133 (2011)

This paper notes that the most forgiving (minimum) performance specification for imprecision should be 9% and the most forgiving (minimum) specification for allowable total error is 32.2%.

We can further calculate analytical Sigma-metrics by dividing the allowable total error by the imprecision observed. We don't know anything about bias, so we have to assume it's zero for the moment. Even with this optimistic assumption, the Sigma-metrics are terrible:

Thus, the Theranos methods consistently fail to meet even the most lenient goals for precision, and on the Sigma-metric  scale, these methods are achieving mostly below 1 Sigma. Again, it’s not a shock that these test results are being voided.

I also hope it’s not necessary to state that this level of performance is significantly worse than what is provided and observed  in traditional diagnostic platforms. Indeed, many “Traditional” POC devices are also far better in precision and performance than Edison.

Theranos' performance raises many more questions – Why was Theranos hiding this data? Did they understand how bad this performance was? What specifications were they setting for performance and why was this egregious performance considered acceptable?

 At the AACC session, questions from the attending scientists are encouraged, and I hope key questions will be asked about long-term imprecision  observed in the methods, comparison of these results against traditional laboratory instruments (for example, what is the bias between Edison and the Siemens instruments that Theranos has installed and running at the Newark facility?). If we know CV and bias, we can calculate analytical Sigma-metrics and determine the current performance of their methods – and whether or not this company deserves another chance to deliver test results to patients.

But let's not pretend that Theranos is alone in some of these practices, even in the diagnostics industry. There are instrument managers who prefer to hype their quality rather than provide hard facts. There are labs that build a facade of quality but hide an inner core of corrupted quality.

• If you're dealing with a vendor, demand proof of quality
• Proof of quality means more than an assertion by the company that of course they provide high quality. You need data to prove it, preferably long-term imprecision, and bias measured against a reference method, peer group or EQA/PT group, and then that needs to be benchmarked against a qualiy goal.
• Quality data needs to be objective and open - don't just accept it if a company claims world class or Six Sigma performance, read the data that lies underneath those metrics. I've seen examples of a major manufacturer choosing extremely wide goals from one country and then demonstrating Sigma-metrics based on those goals in a country with significantly tighter goals. This is misleading - and if a company is willing to mislead you on their Sigma-metrics, they're probably willing to mislead you on about anything else.
• Don't just accept the first set of data provided to you by a manufacturer - that might be their optimized, best customer. Ask for multiple examples. Ask to contact the customers themselves and make sure the data is really from that lab. Even better, ask for something like a peer group report from one of the customer (for example, a Bio-Rad Unity Peer group Monthly Comparison Report, as well as its equivalent from other programs from Randox, etc). That data is hard to fake since it's from a third party, and those reports contain all the data necessary to benchmark the instrument performance on the Sigma scale

So those are just a few ways we can avoid being Theranosed by our vendors. But then turn around and make sure you're not committing the same sins with your clients, clinicians and patients.

• Don't simply claim quality. Provide proof of it, preferably quantitative proof. Sigma-metrics are an excellent way to do this.
• Don't artificially boost your Sigma-metrics by choosing overly ideal estimates of bias or imprecision, or by choosing quality goals that are wider than appropriate.
• Make quality part of your investments in operations - don't just by the low bid, the cheapest methods, the cheapest controls - if you don't make quality part of your purchasing process, you will find out that quality isn't a part of the results you deliver.

When we conduct ourselves this way, holding outselves to a higher standard than mere compliance, we will prevent any more Theranos upstarts from gaining footholds in our industry. Innovation has an important role to play in the diagnostics industry, but fraud does not.