QIBA Newsletter

In My Opinion

DWI Profile: Experience and Lessons Learned in Validation of Site Conformance and the Quest for Simple yet Informative Technical Evaluation Procedures

By DARIYA I. MALYARENKO, PhD and THOMAS L. CHENEVERT, PhD  

QIBA Profiles are designed to provide a structured roadmap for imaging centers, scanner manufacturers, radiologists, and other stakeholders that seek use of quantitative image readouts for disease characterization and/or to monitor response to treatment. Despite clear goals and specifications, one faces many challenging aspects to validate conformance to QIBA roadmaps.

As detailed in Profiles, multiple “actors” (people, scanners, and software) are required to function in concert to achieve stated numerical claims, yet procedures to validate performance of each actor can be difficult to define, let alone implement in practice. Currently, an array of “checklists” to be completed by site personnel serves as an attestation of full or partial conformance, but validation of “self-attestation” checklist responses can be difficult. As a more tangible first step, the DWI Biomarker Committee focused on the one actor that can be objectively validated – the MRI scanner.

Toward this end, the DWI BC provided four essential elements:

1. description of suitable DWI phantom(s)
2. detailed DWI scan protocol
3. target performance specifications, and
4. analysis software to assess performance derived from DWI DICOM

Since water mobility (or ADC) is sensitive to temperature, the DWI Profile describes use of ice water-based phantoms that serve to both determine and control temperature, thus providing a “true value” so that absolute ADC bias of an MRI system is measurable. Simple ice water-based phantoms have been used to certify MRI scanner performance for site qualification and periodic QA in multiple clinical trials utilizing DWI in breast, prostate, and brain.

DWI phantom designs and protocols were adapted to specific trial objectives, but phantom data and experience gleaned from these trials certainly contributed to development of the QIBA DWI Profile. The next generation DWI phantom contains polyvinylpyrrolidone (PVP) materials for an array of ADC values to assess MRI system linearity, though still uses ice water for temperature control. PVP phantom preparation, scan protocol, analysis procedures and performance benchmarks were tailored for QIBA and are detailed in the current “QIBA DWI MRI Profile Conformance Testing” documentation available on the QIBA Wiki.

These procedures were evaluated by a handful of academic centers around the globe to test performance of their MRI systems within the context of QIBA DWI specifications. The exercise was certainly informative to all parties and provided QIBA DWI BC valuable feedback to further improve its Profile and streamline technical conformance validation steps. Moreover, DICOM datasets from these tests were shared with QIBA DWI BC for independent analysis to validate that specified technical benchmarks were met and consistent with the site’s results.

Motivated by the common feedback theme for “process simplification,” DWI BC members modified the PVP phantom analysis software for greater automation and DWI QC Report generation, with an example shown below. Key features built into the analysis workflow include robust handling of MRI manufacturer-specific DWI DICOM series conventions and image sort orders, and summary of performance measures relative to QIBA-specific benchmarks.

We used this workflow for subsequent validation of commercial DWI QC software (“qCal”) described in the accompanying article by Kevin Miller in this QIBA Newsletter issue. The numerical consistency of the qCal output versus QIBA tools was checked for representative DWI phantom DICOM examples from the dominant MRI vendors. It is hoped that this user-friendly commercial product will further streamline site conformance validation and provide real-time feedback for system performance based on DWI phantom evaluation.

Analysis Tools and Techniques

New QIBA Conformance Certification Service Supports Critical DWI Profile and Biomarker Claims for Breast, Liver, Brain, and Prostate

By KEVIN MILLER, MSE

DWI and derived quantities, such as ADC, are being incorporated into clinical trials as potential diagnostic or therapy response biomarkers. As presented in Dr. Nancy Obuchowski’s article in the October 2021 QIBA Newsletter, “The ability to characterize and quantify the bias of a measurement relative to a True or Reference Value is important to quantitative imaging [and] this is possible with biomarkers.” In support of these needs, QIBA’s DWI Biomarker Committee, co-chaired by Drs. Malyarenko and Boss, has developed a Profile —currently at “Consensus Stage” in the Profile development process — to quantify ADC biomarker performance thresholds, address deficiencies, and capture best practices in DWI acquisitions1.

However, from a practical perspective, MR users have been set back by the lack of a validated analysis tool, especially for multi-site and longitudinal studies. Drs. Malyarenko and Chenevert of the University of Michigan developed analysis tools based on the Profile to quantify site acquisition performance. These tools and analyses have resulted in improvements in the quality, repeatability, and consistency of scans with respect to specified DWI phantoms; however, the tools were not designed as commercial software products to be broadly disseminated.

In parallel with that work, CaliberMRI, Inc., a company focused on the development of quantitative MRI technologies and products, developed a quality control platform for MRI scans, known as qCal-MRTM, with the objective of providing a highly automated tool for evaluating the critical scan performance parameters with respect to reference value phantoms. The cloud-based software is designed to read DW-MRI DICOM data of the QIBA DWI phantom. The DWI DICOM is generated using multiple b-value acquisitions, where “b-value” is a factor representing the degree of diffusion weighting, based on the amplitude, timing, and duration of applied gradient pulses.

The resultant qCal-MR analysis report provides:

1. visualization of DWI and derived ADC maps
2. highly automated, efficient and consistent identification of ROIs
3. statistical analyses of ROIs and VOIs applied to DWI and ADC maps
4. tabulation of summary statistics of defined performance metrics including SNR, bias and random error and ADC linearity, and
5. analyses of calculated performance as compared to NMR-verified reference values for the phantom.

CaliberMRI’s “QIBA” DWI phantom and a screenshot of a qCal-MRTM visualization are provided in Figures 1 and 2, respectively.

qCal-MRTM has been in use at many sites around the world for more than a year now. The cloud-based toolset has been maintained and advanced during that time, to the benefit of users. As an example, an MR-readable thermometer is now incorporated into the QIBA DWI phantom, and qCal-MRTM was expanded to provide automated readout and temperature correction for ADC calculations. In addition, qCal-MRTM was updated to incorporate NMR data generated by NIST for room temperature calibration of DWI scans, simplifying the phantom setup and preparation for quality control scans.

QIBA leadership recognized the relevance of the new commercial platform to the MRI DW committee’s efforts and established that validation of qCal-MRTM with respect to the Profile and the UM analysis tools and establishment of CaliberMRI as a scanner QC analysis service provider could yield a sustainable process for accelerating the adaptation of the DWI Profile in the medical imaging community. To that end, during 2021, qCal-MRTM was rigorously evaluated against UM benchmark cases. These baseline DICOMs, generated using scanners from Philips, GE, and Siemens at different sites with various staff members, provide a repeatable basis for validation and verification.

In December 2021, QIBA entered into an agreement with CaliberMRI to provide services to client sites for the certification of site conformance to the processes defined in the Profile for DWI MR. For more information on DWI Profile Conformance Certification, please email the author (kmiller@qmri.com) or visit CaliberMRI’s website (calibermri.com, or qmri.com).

Reference:
Boss, Malyarenko, et al; QIBA DWI Profile Consensus Version; 2019-Dec-20.