Questions

Some questions and answers on trends, "Westgard Rules", manufacturer ranges, significant figures, quality requirements, and more. It's a real grab bag.

If you don't see the question that you'd like to see answered, please send it in to This email address is being protected from spambots. You need JavaScript enabled to view it.

This Section is Exclusively Sponsored by Technopath Clinical Diagnostics

• What's a trend?
• How to apply the R_4s rule?
• Manufacturer Ranges: Use them or lose them?
• Significant Figures
• Finding quality requirements for unregulated analytes
• Standard deviation ratios; verifying a new lot of reagents
• Where to get bias
• Patient-based QC and Average of Normals

What's a trend?

This question comes from Robbie Keith of Summit Laboratory

We are in the process of evaluating our QC program. Our techs monitor Levy-Jennings charts for shifts and trends weekly. We would like to know what you consider to define a shift or trend (e.g. how many points are required increasing or decreasing to define a trend?)

Consider control rules such as 4_1s, 10_mean, etc., as good indicators of shifts and trends. The number of observations needed increases as the limit approaches the mean of the control material in order to keep the false rejections down. Minimum number of consecutive observations above or below the mean should probably be set as 6. There are some recommendations, particularly in the Germany, to use 7 above or below the mean, or 7 trending consecutively in one direction.

Applying the R_4s rule: within one run or two?

This question comes from Marty Baecker at Scott & White Hospital, Temple, Texas:

I am presently using the multirule system and am very pleased with it. I do have a question about the R_4s rule. Is it just to be applied within a run or is it also to be applied over two consecutive runs within one control material?

We recommend that the R_4s rule be used only within a run in order to maintain its relation to random error. If used across runs, it would be possible that a systematic change be misinterpreted as an increase in the imprecision of the method.

This is discussed in the section on FAQs on page 160 in the Basic QC
Practices book. It's also mentioned in the previously posted FAQs on Basic QC.

Manufacturer Ranges: Use them or lose them?

This question comes from a Dr. Rajiv in India:

Do we need to establish our own values on the same controls or can we use the ranges supplied by the manufacturer?

It is best to determine your own values for the mean and SD of the control materials. Manufacturer's limits will tend to be very broad to encompass the performance of their systems operating in many different laboratories. Optimal error detection depends on comparing your QC results with the range of results expected for your individual instrument.

If we cannot use the manufacturer's mean and SD values then how do we guarantee the accuracy of the assay?

Initially, the accuracy of your method is assess by method validation experiments, particularly the comparison of methods experiment.When the method is put into service, you also establish your own means and SDs for control materials, which become the baseline for monitoring systematic changes or errors. The objective of routine QC is to identify any systematic changes and hopefully eliminate the causes, thus returning the method to its stable state of operation. During routine service, you should also use proficiency testing results to monitor whether this baseline remains stable or changes. When there are changes in the source or lot numbers of control materials, it is important to carefully establish the means and SDs for the new materials while the old materials are still in use. It should also be useful to compare your observed means on control materials with the manufacturer's recommended means. All this information needs to be considered in monitoring the accuracy of a method.

Check out our Method Validation section

What about the systematic/proportional bias that can be present?

The calibration function and materials used are critical. Any changes of calibration materials need to be carefully evaluated by the above procedures.

Significant Digits

From Marit Okland in Norway:

This question is about digits in measurements. For example, if you have one digit in the result from the patient, how many digits shall you use in the results of the control material? And how many digits in the facit, and in the mean of the control material results? Shall the mean have just as many digit as the facit?

In general, it is nice to have one more significant figure for the control measurements vs the patient test result. A histogram of the control measurements should give you a minimum of 5 to 7 intervals. If you are only seeing one or two different results, you need another significant figure.

The biggest limitation from "rounding" comes when applying multirule procedures because the different control limits can't be set appropriately. In these case, use of a single limit rule is probably better if the extra significant figure can't be obtained.

Concerning the mean, I would generally calculate the mean to one more significant figure than the measurements themselves.

Finding quality requirements for unregulated analytes

From Lucie Fritz, a graduate student at MCVH

I am interested in setting a total allowable error limit in a new test (homocysteine) in setting up and evaluating a methodology in the lab. It currently is not being included in any proficiency testing regimens I can find, and I see no CLIA requirements for it, either. How can I set some initial total error for evaluation of the new method?

It is always difficult to establish the quality requirement for new and more specialized tests, such as homocysteine. In this particular case, I would guess that the initial allowable total error for PT surveys will be in the 20 to 25% range. The reasoning behind this figure is that similar analytes that are regulated by CLIA have 20% or 25% total allowable errors. In general, if you can't find a specific CLIA requirement, finding a general quality requirement for the category of tests (hematology, toxicology, etc.) is a good back-up plan.

While it is easiest to work with a total error type of requirement, it may sometimes be better to start with a clinical "decision interval" type of requirement; however, that means you also have to find information about the within-subject biological variation and use the more complicated clinical quality-planning model for translating that requirement to operating specifications. This clinical requirement can be determined as the change in test results that would be judged to be medically important, i.e., the change that would cause the physician to make a change in his diagnosis or treatment. Recommendations in the literature on the interpretation of test results, or clinical practice guidelines in hospitals, may also provide this information.

Standard deviation ratios; verifying a new lot of reagents

From Jose Carlos Basques, Labtest Diagnostica, Brazil:

I would like to know the use of the standard deviation of the estimates in method comparison experiment. I was participant of the Workshop: Concepts & Practices in the Evaluation of Laboratory Methods given by Dr. David Koch, Neill Carey, and Carl Garber and they informed that the ratio standard deviation of comparative method/standard deviation of the estimates should be more than 7(seven). Could you explain the rationale of this statement?

The correlation coeffient is related to the SDs of the patient population and the analytical method by the expression 1 - 1/(SD_pop/SD_analytical)2. When the ratio SD_pop/SD_analytical is 7, that term squared is 49, and the reciprocal is about 0.02, thus the correlation coefficient would be 0.98, which would indicate a wide analytical range. Therefore, you can use either criterion - a ratio of 7 or an r of 0.98 - as an indicator of a wide analytical range. In practice, r is readily available and is more commonly used to assess the validity of the analytical range for use of simple regression analysis.

In the lesson Method Validation: The Inner, Hidden, Deeper, Secret Meaning, Dr Westgard states: "laboratory regulations in the USA require that performance for a new method be verified prior to reporting patient results. Precision and accuracy are specifically identified, along with analytical sensitivity, analytical specificity, reportable range, reference values, and any other applicable characteristic." I and many colleagues in Brazil agree with these statements because the laboratory unknowns the performance of the new method, but there is one disagreement among us: what characteristics should we verify in a new lot of reagents? Many of us agree that we should verify the within-run imprecision and the bias, and validate the lot comparing the results with the ones we have got previously. However, others think that a more extensive evaluation should be done. Therefore, we would like to know what characteristics do you evaluate for verifying a new lot of reagents?

The amount of work needed to check out new reagents probably depends on the particular test and method. My general concern is usually for systematic error, therefore most often would make that assessment. However, for certain methods, it may also be expected that a change in the reagent may affect the random error and it would also be reasonable to check the imprecision of the method. Further checking might be necessary in methods where the detection limit is critical and experience shows that changes in reagents may also result in changes in the detection limit.

Where to get bias

From Smyth County Community Hospital, Marion, Virginia

I am still a little confused as to what should be used for bias on the OPSPEC Charts and with QC validator. Is it as simple as using the SDI result from peer group comparisons? Do we need to use internal comparison data to known standards or linearity products?

Concerning the appropriate estimate of bias to use when planning a QC procedure, it will depend on what information is available on your method. If it is a new method, then it is likely that you will have data from a comparison of methods experiment that can be used to estimate bias. If the method is in routine use, you will have more recent results from a PT survey that may better represent current method performance. Or, you may have internal comparisons that are ongoing between two methods that measure the same analyte in your laboratory.

In using any of these estimates, it is important to enter the bias figure in percent, i.e., the percent at the medical decision level of interest when calculated from comparison of methods data, or the average SDI converted to concentration units, then divided by the average concentration of the group of survey specimens.

"Patient-Based QC" and Average of Normals

From an instrument designer at Chiron Diagnostics.

I am looking for information on something now called patient-based, patient-focused or patient-centered quality control. In a past life it may have been known as patient histograms. Is this becoming the "new trend" for an assay's performance evaluation? How is it used? What is evaluated? Should it be part of an analyzer' s QC feature package? If so, do you have any information on the data evaluation?

I believe the term "patient-focused QC" was used by Dr. George Klee in his presentation at the AACC meeting this summer. I don't think there is any established definition of this term, however, it does have a good ring to it and will likely be picked by others. A summary of Dr. Klee's presentation appeared in a recent issue of Clinical Laboratory News (probably October) and should also be available at the AACC website. It was titled "Patient-Focused Quality Control: Is it time to rethink traditional QC?" and was written by Sue Auxter.

I think there are two issues that need to be addressed to provide patient focused statistical QC procedures. One is to design statistical QC procedures to detect medically important errors. This is also called for in the NCCLS revised draft on Internal Quality Control (C24A2) that is currently out to the membership for vote. One approach for doing this is provided by our QC Validator computer program.

Another issue is the use of patient data for QC purposes. In that context, the Average of Normals (AoN) technique would be one way to monitor the mean of a patient population. Again it would be desirable to design the AoN procedure to monitor medically important errors. We've done some work on this in the past (see Clin Chem 1996;42:1683-1688) and are continuing to develop an automatic design for AoN QC procedures, in addition to providing two-stage QC designs for traditional statistical QC procedures.

Implementation of AoN procedures tends to be a problem in many clinical laboratories, probably due to the difficulties with their existing information systems. The technique should be useful, particularly in high volume hospital and commercial laboratories. However, it must be properly planned on the basis of the relative sizes of the population and analytical SDs for the test of interest. Tests that are well-controlled physiologically, such as sodium and calcium, could be monitored with a relatively low number of patient specimens, less than 100, whereas tests such as cholesterol may require a couple hundred specimens. Some ideas about the numbers of specimens needed can be found in the 1996 clin chem paper. Of course, these numbers will change with different instrument precision and different patient populations, which is why the AoN procedure needs to be carefully designed if it is to be useful.

For anyone who would like to implement an AoN procedure, we may be able to help support your applications with our next generation QC Validator program or through direct consultation.