Questions
AACC Expert Access Questions & Answers - Six Sigma
On March 5th, 2002, Dr. Westgard was the guest at the AACC's Expert Access Online. The topic was Six Sigma.
We thank all of you who submitted questions to the forum at that time. If you still have questions, please email us now and we will try to respond.
Intra- and inter-individual biological variation for some analytes will cause one to have results that do not meet 5 or 6 sigma criteria. Also, 5 or 6 sigma thresholds exceeds clinical needs/expectations in some situations. For example, precision goals for cholesterol by some providers are 5%; if there was no bias in the test, the total error would be ~10%, satisfying CLIA and the MD, but not 6 sigma. I'm not convinced that industrial specs for manufacturing processes are applicable or reasonable stds for clinical lab testing on biologically variant humans. Seattle, WA
One issue raised here is how to set the tolerance limit, which could be done based on biologic variability, clinical decision intervals, or analytical total error limits. In 1999, a consensus conference on "Strategies to Set Global Analytical Quality Specifications in Laboratory Medicine" recommended a hierarchy of goal-setting models, in which the highest and preferred model considers the evaluation of analytical performance on clinical outcomes in specific clinical settings. The 2nd model in the hierarchy considers the evaluation of analytical performance in general using data based on biological variation or data based on the analysis of clinicians' opinions. The 3rd model is published professional recommendations and the 4th model is performance goals set by regulatory bodies. The proceedings were published in Scand J Clin Lab Invest, vol 59, no. 7, November 1999. In this context, the preferred approach is to establish a quality requirement for a specific clinical situation. Biologic variation goals and proficiency testing requirements are lower in the hierarchy. One way of interpreting the clinical outcome of a test is to use a clinical decision interval defined from the test interpretation guidelines. For example, for cholesterol a decision interval can be defined from 200 mg/dL to 240 mg/dL, based on NCEP clinical interpretation guidelines. That interval should encompass both preanalytical and analytical variability. Individual biologic variability is accounted for as a preanalytical component of variation. The calculations are more complicated than for a total error criterion, but can be carried out with an electronic spreadsheet or computer program. With this approach, the maximum allowable imprecision (for N=2 QC procedures) is a CV from 2.4% to 2.7% IF BIAS IS ZERO. [See chapter 11 in Six Sigma Quality Design and Control for a more detailed explanation.] The precision goal is therefore considerably more demanding than the 5% CV referenced in the question.
The second issue about whether or not 6 sigma concepts and goals apply in medicine and laboratory testing will probably be decided by the customer, in this case, the corporate customer who pays a large portion of the cost of healthcare. It will be very difficult to defend current laboratory performance goals. For example, where did the recommendation for 5% CV for cholesterol come from and how is it justified? What is the expected DPM with a 5% CV? If only 95% of the results meet the quality requirement, then 5.0% of results are in error by greater than 10% and the DPM is expected to be 50,000 for cholesterol testing. Industrial benchmarks make it obvious that this is very poor quality. It is important for laboratories to think about Six Sigma concepts and metrics for this very reason. Corporate customers are beginning to understand the Sigma performance metrics and will start to expect that kind of information from healthcare organizations. Laboratories are fortunate in that many tests, particularly highly automated analyses, will actually meet the 6-sigma goal.
At what point does cost/test overide quality of the analytical result (i.e. hemoglobin a1c on automated chemistry analyzers vs HPLC analysis or homocystine on these same analyzers vs HPLC)? Boca Raton, Fl
The cost vs quality tradeoff is always a difficult issue. I think it is important to make an assessment of the quality needed in quantitative terms. If you have 5-sigma performance, that test is controllable; if you have 4-sigma performance, that test will take more time and effort to monitor performance and assure quality; if you have 3-sigma performance, the test won't be very reliable. The lower the sigma metric, the stronger the argument for improving the quality of that test. One advantage of determining the sigma metrics for your tests is to be able to establish the priorities for improvement. The lower the sigma metric, the higher the priority for improvement.
What is the future of Six Sigma Quality Management? Lima, Ohio
Because Six Sigma is an evolution of TQM, there is a foundation to build on in many healthcare organizations. With the emphasis on measures of performance, Six Sigma should be a "natural" fit for the evolution of healthcare quality systems. That said, Six Sigma QM is just beginning to be applied in healthcare. You will hear about more healthcare applications in the next year, but it will take another two or three years before it becomes more widely applied. In industry, it has taken about five years for Six Sigma to develop momentum and become well established. It will probably take at least five years for healthcare to get to where other industries are now.
When developing reference ranges for an analyte, what is the minimum number of samples and measurements which to start? What is the smallest sigma we should shoot for? Davis, CA
Not sure I have any answer for this based on Six Sigma concepts and metrics. The Six Sigma concept and metric relates more directly to the performance needed for clinical usefulness of a test. That performance might depend on the interval between a reference limit and a cutoff point, in which case the quality of the estimate of reference range is important. I think the IFCC guidelines for estimating reference intervals call for about 120 samples per group.
TEa is given. SD of an automated method can hardly be reduced. Bias is heavily dependent on reference materials (calibrators). Then…the precision improvement, if needed for diagnostic, prognostic or treatment purposes, How can be achieved?. Is sigma metric only an indicator of the status of the art? Badajoz, Spain
TEa may be constant in a specified marketplace, e.g., CLIA requirements for the USA. However, there may be significant differences in the TEa requirements in other countries and other external quality assessment schemes. The sigma metric for an analytical testing process certainly is very dependent on the imprecision observed for the method. The advantage of automated technology is that the method SDs should be small and the sigmas should be high. The available precision depends very much on the manufacturer of the analytical system, so in that sense, the sigma metric does reflect the state of the art. However, it also depends on how well the analytical system is maintained and operated in a given laboratory setting, since both the imprecision and bias may be affected by laboratory operation. If improvements in performance are needed, they often depend on the manufacturer, BUT laboratories can help drive those improvements by the way they set their "purchase specifications" for new systems.
How long do you think it would take for a laboratory to start a pilot Six Sigma project? How long before they would see results? Norwalk, CT
The time for completion of a Six Sigma project depends very much on the project that is selected and the skills and training of the project team. If the project crosses section or departmental boundaries, then it becomes more complicated. There is a big advantage to focus an initial project on the assessment of analytical quality of
laboratory tests because the information is readily available, the project involves mainly laboratory people, and quality planning tools and methodology are available. We took this approach when we began implementing TQM over a decade ago (see Koch et al, Selection of medically useful quality-control procedures for individual tests done in a multitest analytical system, Clin Chem 1990;36:230-233). While that project took several months at that time, it led to the development of the tools and methodology that make it possible to do a similar project today in a matter of two or three weeks. It is always good to start with a relatively simple project that will yield useful results quickly.
Is there a reference book specific for quality control in a forensic[crime] laboratory? (This lab is not CAP,JCAHO, or SAMSA accredited). Do you have any reference material that you can suggest? (I am very familar with clinical tox labs but not a strictly forensic lab. We do not perform any work related drug screens but do perform impairment testing: ethyl alcohol and volatile testing as well as screening urine and blood for drugs, and identifying them (confirms via GC-MS). Stillwater, MN
Unfortunately, I don't know what books are available for forensic laboratory applications. What I do know is that the quality of that testing had better be good or it will be picked apart in court. The courtroom is one area where sigma metrics may be quite useful to communicate the quality of analytical testing process to a jury. Think about the analogy with Firestone tire production and airline baggage handling and the problem you would have if your laboratory testing process were demonstrated to be worse than 5 sigma. The technical barrier to understanding test
performance disappears when it can be benchmarked against other industries. If I were the lawyer, I would ask you to state the quality requirement, precision, and accuracy of your testing method. If you don't know, you're already in trouble. If you do know, then I've got my calculator ready and will next ask you about the sigma
metric for your process in comparison with other benchmarks. So, my main advice for forensic laboratories, get ready to deal with Six Sigma concepts and metrics.
In Africa, where no importance is attaach to laboratory science, I want to know how we can improve the performances as one of the six sigma quality of clinical laboratory sciences? Kaduna, Nigeria
I'm not sure things are really that much different! These days laboratory testing seems to mainly be of interest from a cost perspective, not from a quality perspective. Cost has certainly taken priority over quality with laboratory managers in the US. But there still is an opportunity to deal with quality as part of one's professional interest and responsibility. I do see Six Sigma as a new opportunity because the sigma metrics can be more easily communicated to managers and business people. Laboratory test quality can be more understandable to non-technical people if presented in the context of Six Sigma. The starting point is to first know what quality you're producing in your own laboratory so you can set your own priorities and agenda for improving test performance.
Could you please provide more explanation about the 1.5s shift that you detail early in your slides? I don't understand the reasoning behind the shift and whether it is artificial or purposeful. Harrisburg, PA
According to Harry and Schroder, it is common in production processes to observe small shifts over time. These shifts are difficult to detect and also difficult to correct, although they often self-correct over time. By achieving 6 sigma performance, a process can actually tolerate shifts up to 1.5s without producing many defects. So the idea is that if you have good enough performance, you don't need to worry about small changes from run to run.
Is it your recommendation that we use six sigma standards for not only QC, but also performance improvement plans (such as preanalytical processes?) Harrisburg, PA
The Six Sigma metrics can be used with any process, any outcome that can be measured or counted. In many QI projects, we measure error rates or defect rates to describe how often the outcome is not in compliance with the standard for performance. The Six Sigma concept, metric, and performance goals are all applicable. In commercial laboratories where Six Sigma has already been adopted, much of the focus is on error reduction in preanalytical processes. In fact, Six Sigma is probably being used more in improvement of preanalytical processes than analytical processes.
Dr. Westgard, Thank you for your presentation. One of the biggest lab challenges today is compliance in POCT programs. Since so many departments participate in the testing and reporting of results, any QC/QA improvements would be helpful. How can Six Sigma be utilized to improve POCT programs? Syracuse, NY
One useful application is to determine the sigma metric for the performance of your POCT analyzer and to relate that to the QC that is being used. If there's any place that needs 6 sigma performance, it's the POCT area. That level of performance would mean that minimal QC with simple rules and wide limits should be sufficient. For manufacturers, the goal of 6 sigma performance should help them set product specifications that will lead to reliable tests and products. An example application of the Six Sigma approach is provided for cardiac markers in NPT in chapter 13 of the Six Sigma Quality Design and Control book.
I am aware of 6 Sigma being used by a large commercial laboratory and multi-site facilities. Does the 6 Sigma program most benefit large volume operations? Bedford, MA
I think Six Sigma will benefit all laboratories, regardless of size. Commercial laboratories have gotten into Six Sigma earlier because it is better known in business and industry right now. They also stand to benefit more from improvements just because of the volume factor. As hospital start implementing Six Sigma, it will become better known in clinical laboratories.
Are there any Six Sigma benchmarks available for monitoring performance improvements in the laboratory? Do you anticipate more information in the near future? Long Island, NY
The only published benchmarks for laboratory processes so far is the paper by Nevalainen that is referenced in slide 12. There's more information in that paper than that extracted for the slide, so it would be worth looking at that paper more closely. It would seem to be a natural to start using sigma metrics in outcomes studies, particularly any study that measures compliance with some stated standard of performance. For example, the Pathology Q-Probes program could readily make use of sigma-metrics. Individual laboratories may be reluctant to publish their own performance figures, just because those figures may not be as good as they thought. So I would look to programs like Q-Probes to take a leadership role in the benchmarking of laboratory processes.
What areas of the Clinical Laboratory do see benefiting the most from the Six Sigma process? Our Healthcare facility recently has adopted the Six Sigma process and many of us are anxious to put the concepts to work in order to see the benefits. Westerville, OH
Personally, I think that the big benefits will come by establishing the performance of pre-analytical and post-analytical processes, such as patient ID, order entry, completeness of orders, add-on orders, receipt of results, etc. Up till now, there hasn't been any effective way to communicate the terrible performance occurring in some of the pre- and post-analytical processes. Common thinking in healthcare is that error rates of a couple percent are okay. Once physicians and patients understand the sigma metrics, those kinds of error rates won't be tolerable any longer. Taking on those issues and projects, however, will be a lot of work and take a lot of time. We need to remember to take care of our own processes before being too critical of others. That also means applying Six Sigma to analytical testing processes and QC. You will get some real insights into the quality of your own testing processes from doing that.
In your opinion, where has Six Sigma Quality Management been most successfully implemented? Miami, FL
There are many "success stories" that have been documented in business and industry with companies such as General Electric. GE has applied Six Sigma widely in its medical products business units and many of the skilled Six Sigma "Master Black Belts" that are working in other organizations today have gotten their training and experience with GE. I'm more concerned about where Six Sigma is needed most! And the answer to that is HEALTHCARE organizations. The most successful area of application should be healthcare.
Standardization of troponin I has presented a great challenge to the scientific community. Is there any role for 6 Sigma in improving harmonization and standardization in the troponin realm? Greenville, NC
This is a good question for which I have no good answer, except to say that the use of sigma metrics will make it easier to compare the performance of different methods. Currently, a tronponin result is interpreted against the cutoff established for each individual method. Those cutoff points and the method's detection limit can be used to define a clinical quality requirement and assess the sigma metric of the method (again I refer you to chapter 13 in the Six Sigma Quality Design and Control book for more details). The first impact of this kind of assessment will be to highlight the precision performance and lead to improvements in that area. After that, I would expect the improvements would also consider the systematic errors or accuracy of the methods, which should contribute to resolving the calibration and harmonization issues.
Hi, Dr. Westgard. What advice do you have for us in veterinary medicine whereby some assays are still in first or second generation of development (especially endocrine assays) and the CV's and bias do not enable us to meet the total allowable error goals that we would most desire? Our clinicians are expecting the tests to be more accurate and precise than the current methodology can give us and the manufacturers don't seem to be responding (less extensive market than in humans may not justify more R & D). It is frustrating - we are doing more nonstatisatical QC (correlation with other testing, testing in duplicate, using skilled technicians who review the data periodically), but still not easy to ensure that a 'quality result' is reported! Scotland, U.K.
I think the quantitative framework and Six Sigma metrics and goals should still be useful to characterize performance and communicate that information to both manufacturers and clinicians. The more quantitative you can be in documenting the need for improvement, the stronger is your argument to the manufacturer. For clinicians, the information should be helpful so they don't over-interpret test data. It would be possible to actually tailor the interpretation guidelines for a test on the basis of the performance available from the method. Concerning the veterinary marketplace, some manufacturers are starting to recognize the payment and reimbursement for tests may be better in veterinary medicine and that marketplace may attract new business.