QIBA Newsletter
QIBA Newsletter October 2021 • Volume 13, Number 3: Quantitative Imaging Biomarker Metrology: Ongoing Efforts to Develop Recommendations for Assessing the Technical Performance of Multiparametric Imaging
-
Quantitative Imaging Biomarker Metrology: Ongoing Efforts to Develop Recommendations for Assessing the Technical Performance of Multiparametric Imaging
By ERICH HUANG, PhD
-
The Role of “Ground Truth” in QIBA: Past and Future
By NANCY OBUCHOWSKI, PhD
Improve the value and practicality of quantitative imaging biomarkers by reducing variability across devices, sites, patients and time.
QIBA CONNECTIONS
QIBA Wiki
QIBA CONTACT
Contact Us
Timothy J. Hall, PhD
QIBA Chair
In My Opinion
Quantitative Imaging Biomarker Metrology: Ongoing Efforts to Develop Recommendations for Assessing the Technical Performance of Multiparametric Imaging
By ERICH HUANG, PhD
In 2015, the QIBA Metrology Working Group published a series of six papers providing recommendations on methodology for the assessment of the technical performance of quantitative imaging. Topics covered included proper terminology, statistical methods for assessing metrics such as bias, repeatability, or reproducibility of an individual Quantitative Imaging Biomarker (QIB), and techniques to compare these metrics for two or more QIBs. Although some of the concepts in these papers extend readily to multiple QIBs, other aspects of the technical performance of multiparametric quantitative imaging involve different methodologies.
Statistical techniques to estimate technical performance aspects such as limits of agreement or reproducibility must account for correlations between the QIBs. The specification of the computational procedure to combine the QIBs into a numerical score for diagnosis, prognosis, treatment selection, or response assessment, as well as the selection of which QIBs to use, requires proper use of model construction and validation techniques in the statistics and machine learning literature.
To develop an analogous series of papers for multiparametric quantitative imaging, the QIBA Multiparametric Metrology Task Force was formed in 2018. Statisticians, computer scientists, imaging specialists, and clinicians, all with extensive experience in medical imaging, compiled recommendations based on a review of the statistics and machine learning literature for three use cases.
The first use case involves characterizing changes in the physiology through longitudinal changes in multiple QIBs. The second involves combining QIBs via a statistical or machine learning model to classify patients according to phenotypes, namely observable physiological characteristics. The third involves combining QIBs in a similar manner to forecast the risk of an outcome of interest such as disease progression, recurrence, or death. These frameworks can also be applied to radiomics, namely analyses involving data-driven, high-throughput measurements.
The authors applied these methods to both simulated and real data. The simulation studies provided insight into the behavior and performance of these methods under different scenarios (e.g., level of bias or reproducibility in QIB measurements; level of signal in predicting the outcome or phenotype; or number of cases available for model development). Notable findings from the simulation studies include the significant underestimates of the performance of the model in predicting an outcome or phenotype when the QIBs contain measurement error, whereas no such effect is observed when the QIBs are biased.
The result of this effort will be a series of five papers: an overview, one paper for each of the three use cases, and a fifth one on the application of these concepts to radiomics. This series is expected to be published in early 2022.
Analysis Tools and Techniques
The Role of “Ground Truth” in QIBA: Past and Future
By NANCY OBUCHOWSKI, PhD
Throughout my tenure with QIBA there has been active debate about the need for ground truth in QIBA’s Profiles. These debates stem from the difficulty in ascertaining ground truth for some imaging biomarkers, and often lead to proposed solutions that lack biological and statistical rigor, such as defining ground truth from the consensus of imaging methods used in a study. A “Ground Truth Task Force” was thus formed in December 2020 by QIBA leadership to assess the role of ground truth in QIBA’s Profiles. The Task Force members, acknowledged at the end of this article, met three times during January and February 2021.
Following Sullivan et al [2015], the Task Force first decided not to use the term “ground truth,” but rather “true value.” The Task Force then established a hierarchy of terms that included “true value,” “reference value,” “biomarker measurements,” etc. to delineate the relationships between these terms (see Figure). The Task Force made the following observations regarding this hierarchy:
- The True Value, although not always easily observed, is assumed to exist.
- Reference Value and Comparison Value are each defined relative to a True Value. The pyramid levels below these may not have a relationship to a true value.
- Reference Values have negligible bias and imprecision, such that they can be considered highly concordant with the True Value [Sullivan et al, 2015].
- Biomarkers fall under Comparison Values where there is a clear connection to truth. Practically, though, a True or Reference Value may not be available for all biomarkers. When available, measurement bias can be estimated relative to the True or Reference Value.
- When comparing a biomarker to another Comparison Value, agreement can be assessed, but the bias of the biomarker cannot be estimated.
- Standardization of imaging methods relative to the True Value is possible with biomarkers but not with Clinically Acceptable Measurements and Markers. The latter should still be standardized against a state-of-the-art Clinically Acceptable Measurement or Marker, when available.
- Expert Opinion and Casual Observation are typically not quantitative in nature and thus are not the focus of QIBA.
The ability to characterize and quantify the bias of a measurement relative to a True or Reference Value is important to quantitative imaging. This is possible with biomarkers, but not Clinically Acceptable measures, nor imaging Markers. Performance metrics currently used by QIBA, such as bias and linearity, as well as quantification of true change over time, are not applicable to Markers.
Given limited resources and the fundamental differences in performance metrics for biomarkers and Markers, the Task Force recommended that QIBA prioritize Profiles involving biomarkers over Profiles involving Markers. QIBA has focused on biomarkers in the past and has established a respected and trusted methodology; in the opinion of the Task Force, this focus is well justified.
Hierarchy of values starting with truth, followed by measurements of decreasing levels of certainty, with a brief description of each level on the left and examples on the right.
Reference:
Sullivan et al, ‘Metrology Standards for Quantitative Imaging Biomarkers’, Radiology 2015
Task Force Members:
Andrew Buckler, MS
Jana Delfino, PhD
Alexander Guimaraes, MD, PhD
Tim Hall, PhD
P. David Mozley, MD
Nancy Obuchowski, PhD (Chair)
Nicholas Petrick, PhD
Gudrun Zahlmann, PhD
Nancy Obuchowski, PhD, is Vice-Chairman of Quantitative Health Sciences at the Cleveland Clinic and Professor of Medicine at the Cleveland Clinic Lerner College of Medicine of Case Western Reserve University. She is a Fellow of the American Statistical Association. Her research interests include study design and statistical analysis methods for imaging screening and diagnostic tests and imaging biomarkers. She is a member of the QIBA Steering Committee.
QIBA Leadership Announcements
Thank you to Dr. Guimaraes
RSNA and QIBA leadership would like to extend their sincere appreciation to Alexander R. Guimaraes, MD, PhD, for his service, dedication, and leadership as QIBA Vice Chair and then QIBA Chair over the past four years. Dr. Guimaraes stepped down as QIBA Chair at the end of September.
Since joining QIBA in its very early years, Dr. Guimaraes has served on several QIBA committees, including a term as Co-chair of the MR Coordinating Committee from May 2015 – January 2017.
New QIBA Chair
RSNA/QIBA is pleased to announce that Timothy J. Hall, PhD, assumed the role of QIBA Chair, beginning October 1, 2021. Gudrun Zahlmann, PhD, will continue to serve as Vice Chair, and RSNA and QIBA leadership will begin the search to identify a second Vice Chair.
QIBA Activities
QIBA Biomarker Committees are open to all interested persons. Meeting summaries, the QIBA Newsletter and other documents are available on the QIBA website RSNA.ORG/QIBA and wiki http://qibawiki.rsna.org/.
QIBA Resources:
- QIBA News
- QIBA Webpage
- QIBA Wiki
- QIBA Biomarker Committees
- QIBA Organization Chart
- QIBA LinkedIn page
Please contact QIBA@rsna.org for more information. We welcome your participation.
*Please note that the list of references has been migrated to EndNote.
*To obtain access to the RSNA EndNote citations, please send an email request to: qiba@rsna.org.
The list of references showcases articles that mention QIBA, quantitative imaging, or quantitative imaging biomarkers. In most cases, these are articles published by QIBA members or relate to a research project undertaken by QIBA members that may have received special recognition.
New submissions are welcome and may be directed to QIBA@rsna.org.
Join us in Chicago for RSNA 2021!
For more information: https://www.rsna.org/annual-meeting