QIBA Newsletter

In My Opinion

Quantitative Imaging Biomarker Metrology: Ongoing Efforts to Develop Recommendations for Assessing the Technical Performance of Multiparametric Imaging

By ERICH HUANG, PhD  

In 2015, the QIBA Metrology Working Group published a series of six papers providing recommendations on methodology for the assessment of the technical performance of quantitative imaging. Topics covered included proper terminology, statistical methods for assessing metrics such as bias, repeatability, or reproducibility of an individual Quantitative Imaging Biomarker (QIB), and techniques to compare these metrics for two or more QIBs. Although some of the concepts in these papers extend readily to multiple QIBs, other aspects of the technical performance of multiparametric quantitative imaging involve different methodologies.

Statistical techniques to estimate technical performance aspects such as limits of agreement or reproducibility must account for correlations between the QIBs. The specification of the computational procedure to combine the QIBs into a numerical score for diagnosis, prognosis, treatment selection, or response assessment, as well as the selection of which QIBs to use, requires proper use of model construction and validation techniques in the statistics and machine learning literature.

To develop an analogous series of papers for multiparametric quantitative imaging, the QIBA Multiparametric Metrology Task Force was formed in 2018. Statisticians, computer scientists, imaging specialists, and clinicians, all with extensive experience in medical imaging, compiled recommendations based on a review of the statistics and machine learning literature for three use cases.

The first use case involves characterizing changes in the physiology through longitudinal changes in multiple QIBs. The second involves combining QIBs via a statistical or machine learning model to classify patients according to phenotypes, namely observable physiological characteristics. The third involves combining QIBs in a similar manner to forecast the risk of an outcome of interest such as disease progression, recurrence, or death. These frameworks can also be applied to radiomics, namely analyses involving data-driven, high-throughput measurements.

The authors applied these methods to both simulated and real data. The simulation studies provided insight into the behavior and performance of these methods under different scenarios (e.g., level of bias or reproducibility in QIB measurements; level of signal in predicting the outcome or phenotype; or number of cases available for model development). Notable findings from the simulation studies include the significant underestimates of the performance of the model in predicting an outcome or phenotype when the QIBs contain measurement error, whereas no such effect is observed when the QIBs are biased.

The result of this effort will be a series of five papers: an overview, one paper for each of the three use cases, and a fifth one on the application of these concepts to radiomics. This series is expected to be published in early 2022.

Analysis Tools and Techniques

The Role of “Ground Truth” in QIBA: Past and Future

By NANCY OBUCHOWSKI, PhD

Throughout my tenure with QIBA there has been active debate about the need for ground truth in QIBA’s Profiles. These debates stem from the difficulty in ascertaining ground truth for some imaging biomarkers, and often lead to proposed solutions that lack biological and statistical rigor, such as defining ground truth from the consensus of imaging methods used in a study. A “Ground Truth Task Force” was thus formed in December 2020 by QIBA leadership to assess the role of ground truth in QIBA’s Profiles. The Task Force members, acknowledged at the end of this article, met three times during January and February 2021.

Following Sullivan et al [2015], the Task Force first decided not to use the term “ground truth,” but rather “true value.” The Task Force then established a hierarchy of terms that included “true value,” “reference value,” “biomarker measurements,” etc. to delineate the relationships between these terms (see Figure). The Task Force made the following observations regarding this hierarchy:

• The True Value, although not always easily observed, is assumed to exist.
• Reference Value and Comparison Value are each defined relative to a True Value. The pyramid levels below these may not have a relationship to a true value.
• Reference Values have negligible bias and imprecision, such that they can be considered highly concordant with the True Value [Sullivan et al, 2015].
• Biomarkers fall under Comparison Values where there is a clear connection to truth. Practically, though, a True or Reference Value may not be available for all biomarkers. When available, measurement bias can be estimated relative to the True or Reference Value.
• When comparing a biomarker to another Comparison Value, agreement can be assessed, but the bias of the biomarker cannot be estimated.
• Standardization of imaging methods relative to the True Value is possible with biomarkers but not with Clinically Acceptable Measurements and Markers. The latter should still be standardized against a state-of-the-art Clinically Acceptable Measurement or Marker, when available.
• Expert Opinion and Casual Observation are typically not quantitative in nature and thus are not the focus of QIBA.

The ability to characterize and quantify the bias of a measurement relative to a True or Reference Value is important to quantitative imaging. This is possible with biomarkers, but not Clinically Acceptable measures, nor imaging Markers. Performance metrics currently used by QIBA, such as bias and linearity, as well as quantification of true change over time, are not applicable to Markers.

Given limited resources and the fundamental differences in performance metrics for biomarkers and Markers, the Task Force recommended that QIBA prioritize Profiles involving biomarkers over Profiles involving Markers. QIBA has focused on biomarkers in the past and has established a respected and trusted methodology; in the opinion of the Task Force, this focus is well justified.

Reference:
Sullivan et al, ‘Metrology Standards for Quantitative Imaging Biomarkers’, Radiology 2015

Task Force Members:
Andrew Buckler, MS
Jana Delfino, PhD
Alexander Guimaraes, MD, PhD
Tim Hall, PhD
P. David Mozley, MD
Nancy Obuchowski, PhD (Chair)
Nicholas Petrick, PhD
Gudrun Zahlmann, PhD

New QIBA Chair

RSNA/QIBA is pleased to announce that Timothy J. Hall, PhD, assumed the role of QIBA Chair, beginning October 1, 2021. Gudrun Zahlmann, PhD, will continue to serve as Vice Chair, and RSNA and QIBA leadership will begin the search to identify a second Vice Chair.

QIBA Resources:

• QIBA News 
• QIBA Webpage
• QIBA Wiki
• QIBA Biomarker Committees
• QIBA Organization Chart
• QIBA LinkedIn page

Please contact QIBA@rsna.org for more information. We welcome your participation.

*Please note that the list of references has been migrated to EndNote.
*To obtain access to the RSNA EndNote citations, please send an email request to: qiba@rsna.org.

The list of references showcases articles that mention QIBA, quantitative imaging, or quantitative imaging biomarkers. In most cases, these are articles published by QIBA members or relate to a research project undertaken by QIBA members that may have received special recognition.

New submissions are welcome and may be directed to QIBA@rsna.org. 

Join us in Chicago for RSNA 2021!

For more information:  https://www.rsna.org/annual-meeting