Navigating method evaluation in clinical laboratories

It’s a familiar scenario in clinical laboratories: An instrument is aging and needs replacement, a physician is requesting a new test in-house, or a more advanced methodology has become available, and the laboratory is tasked with deploying it. In any case, before implementing a new instrument or test, laboratories need to evaluate each method carefully. This practice not only follows regulatory and accreditation requirements but also assesses how much error and what type of error is present in the test method when compared to the comparative or gold standard method.

Two terms are used often when it comes to method evaluation — analytical validation and analytical verification. These terms refer to evaluating Food and Drug Administration (FDA)-modified/laboratory developed tests (LDTs) and FDA-approved tests, respectively. Method evaluation requirements vary depending on the complexity of the test — non-FDA-approved tests (LDTs) require more studies to evaluate them than FDA-approved tests. Method evaluation for FDA-approved waived tests is not required but often is performed as good laboratory practice. Regardless of the type of test, most often the evaluation starts with outlining the plan and predetermining performance goals for each analyte, followed by evaluation experiments to allow data collection. At the end of the evaluation, the lab determines the acceptability of each method for use in patient care is determined.

WHY IS PREDEFINING PERFORMANCE GOALS IMPORTANT?

Developing a detailed method evaluation plan with predetermined acceptability criteria is an important step in method verification or validation as it ensures that the specific test meets the quality goals needed for patient care. Performance goals are generally defined in terms of allowable total error (ATE), and they dictate the performance characteristics required to pass the method evaluation. ATE goals can be expressed in percentages or concentration units and are specific for an analyte and their intended use. There are several resources and guidelines that can be used when defining ATE for a clinical test: clinical outcome studies, biological variation databases, professional organizations, regulatory agencies, proficiency testing organizers, and state-of-the-art models for the specific method. These sources differ in the magnitude of total error allowed for each analyte, laboratories should choose ATE objectively and appropriately to match their analytical system.

WHAT STUDIES ARE REQUIRED IN THE METHOD EVALUATION?

After defining performance goals, a plan for the individual studies describing the number of samples, timeline for data collection, and acceptability criteria for each study should be outlined. For FDA-approved tests, verification of performance specification includes precision, accuracy, and reportable range studies. Reference range verification also should be part of the evaluation process and can be done by confirming the manufacturer’s reference interval, or by evaluating the appropriateness of the currently used reference range if the test is already offered in the clinical laboratory.

LDTs need the same basic studies as FDA-approved tests, but they also require an analytical sensitivity study (how low can a method accurately and precisely detect an analyte of interest) and analytical specificity experiments (to learn what possible entities can interfere with the measurement). As best practice, any changes made to an FDA-approved test also should be addressed in the validation. For example, if the approved patient population for an assay is adults 18 years or older, and a lab wants to perform the test on pediatric patients, comparison samples from the pediatric population should be included in the evaluation.

Interestingly, if the method is FDA-approved on a specific analyzer, but the laboratory is using an analyzer that the method was not FDA-approved for, then the test falls into the non-FDA-approved/FDA-modified category. The same applies to specimen types such as body fluids. If the test was FDA approved on blood, but the laboratory wants to use the test on a body fluid that has not been FDA approved, then the laboratory needs to show analytical sensitivity and specificity in addition to other studies. There are many other evaluation studies that the laboratory can consider performing, such as carryover stability of the analyte over time, or a dilution study to extend the analytical measuring range.

What are some examples of acceptable criteria?

These acceptable criteria are based on those set at the University of Wisconsin Health and Emory University at Grady Hospital Systems in Atlanta and are based on professional experience. There are several Clinical and Laboratory Standards Institute guidelines that are helpful in outlining studies for method evaluation. Manufacturer claims offer another approach to decide whether the method being evaluated is performing as expected. Accuracy and precision studies can be evaluated separately but ultimately should be evaluated together by estimating total analytical error (combining precision and accuracy) at various medical decision levels and comparing the result to ATE.

WHAT ARE SOLUTIONS TO ISSUES THAT OCCUR DURING METHOD EVALUATION?

Laboratories often run into the issue of not meeting the performance goals during a method evaluation. Below are some helpful solutions to this issue.

Precision day to day: Look for outliers, repeat the precision study, select different quality control (QC) materials, or compare the coefficient of variation (CV) from the precision study to the current QC performance (if applicable).

Accuracy study: Look for outliers on a Bland Altman plot, recalibrate both assays (if applicable), or change the reagent lots. If high concentration specimens are unable to be obtained to reach the high end of the analytical measurement range (AMR), create samples by spiking with known materials or use historical proficiency testing samples. The correlation coefficient (r) does not provide information about the accuracy of the new method but does help in deciding which linear regression approach should be used to obtain slope and y-intercept. An r > 0.975 permits one to use the common least squares regression, whereas an r < 0.975 dictates that one should use Deming or Passing-Bablok regression instead.

Reportable range study: Use saline or other diluent to lower the observed range if you are unable to meet the measurement within 10%, use a different kit of the linearity material or a different calibrator lot, or use a patient sample with a high concentration and then serially dilute the specimen to obtain multiple concentrations over the range. If none of the above alternatives are available, truncating the AMR is also an option. It is important to note that truncating or shrinking the AMR within the approved range is not considered a modification to an FDA-approved test.

Evaluation of new methods is a necessary and important process for clinical laboratories to perform so that the testing used by the laboratory meets quality goals and can be safely used to support patient care. Knowing when to verify versus validate these new methods will keep your lab on the right track as it supports patients and providers.

Kornelia Galior, PhD, DABCC, is assistant professor in the department of pathology and laboratory medicine at Emory University School of Medicine. +Email: [email protected]

David Koch, PhD, DABCC, FADLM, is director of clinical chemistry, toxicology, and point-of-care testing at Grady Health System and associate professor at Emory University School of Medicine. +Email: [email protected]

Jill Palmer, MT(ASCP), is director of laboratory services UPH-Meriter, UW Health University Hospital Core Lab in Madison, Wisconsin. +Email: [email protected]