How Well Can AI-Based CAD Models Identify Alzheimer's Disease?
Researchers explore reasons why AI sometimes yields false-negative diagnoses
The gold standard for diagnosing Alzheimer’s disease (AD) involves amyloid PET and cerebrospinal fluid (CSF) biomarker analysis using the amyloid-tau-neurodegeneration (ATN) classification system.
But AI computer-aided diagnosis (CAD) models are increasingly being used in AD diagnosis, too. Although this technology already yields important benefits and has the potential to be refined even further, it can lead to false negatives, according to research recently published in Radiology: Artificial Intelligence.
Researchers found false-negative classification in a subgroup of patients that had atypical structural MRI patterns and clinical measures. These factors fundamentally limited the diagnostic capability of the CAD models, said one of the study’s authors, Kun Zhao, PhD, a principal investigator at the School of Artificial Intelligence, Beijing University of Posts and Telecommunications.
“With advances in computer science and technology, numerous studies have used structural MRI-based CAD models to identify AD from normal controls,” Dr. Zhao said. “Despite using different models in their respective studies, these studies reported similar levels of accuracy.”
In his view, this consistency raises many important questions, including why these patients are misclassified and whether the misclassified participants overlap across different models. Dr. Zhao emphasized the need for a greater understanding of the specific mechanisms behind their misclassification and how those insights can help improve the existing diagnostic framework.
Putting CAD's Diagnostic Capabilities to the Test
In the retrospective study, Dr. Zhao and colleagues assessed 3,258 baseline structural MRI scans from five multisite datasets and two multi-disease datasets collected between September 2005 and December 2019.
A total of 1,391 individuals with AD were included in the study. Of those, 205 had other neurodegenerative diseases and 1,662 were healthy controls.
The investigators used the 3D nested hierarchical transformer (3DNesT) model as well as other CAD techniques to classify AD using 10-fold cross-validation and cross-dataset validation. Then they analyzed the subgroup that had been misclassified by CAD by comparing clinical and neuroimaging biomarkers using independent t tests with Bonferroni correction.
“We first identified participants who were consistently misclassified across multiple CAD models,” Dr. Zhao said. “We then focused on the subgroup of patients with a clinical diagnosis of AD whose scans were falsely classified as healthy controls by the models.”
The false-negative subgroup (n=223) had minimal atrophy and performed better on the Mini-Mental State Examination compared with the true-positive subgroup.
“Despite having pathological features—such as amyloid-β and tau burden—comparable to those of correctly classified AD patients, these individuals exhibited remarkably preserved brain structure and cognitive performance,” Dr. Zhao said.
“Their cognitive scores and gray matter volumes were not only better than those of typical AD patients, but in some cases even surpassed those of individuals with mild cognitive impairment and some healthy controls,” he explained.
This finding was unexpected, he said, because it suggests that a CAD model’s “error” can point to a biologically meaningful phenomenon: a subgroup of patients with high resilience to AD pathology.
According to Dr. Zhao, the findings highlight an important consideration for clinical practice: A negative result from an MRI-based CAD model does not rule out AD.
“The subgroup of patients we identified—those with high brain resilience—may present with cognitive complaints but show minimal atrophy on MRI, leading the model to falsely classify them as healthy,” he said. “In reality, these individuals harbor significant AD pathology (amyloid-β and tau) but are protected by mechanisms that preserve brain structure and function.”
For clinicians, this means that patients with concerning cognitive symptoms, but unremarkable MRIs may still have AD and could benefit from further biomarker testing, such as CSF or PET, Dr. Zhao noted.
“More importantly, understanding the biological mechanisms that confer this resilience could inspire future therapeutic strategies aimed not just at removing pathology but at enhancing the brain's ability to withstand it, potentially improving the quality of life for all AD patients,” he said.
“Understanding the biological mechanisms that confer this resilience could inspire future therapeutic strategies aimed not just at removing pathology but at enhancing the brain's ability to withstand it, potentially improving the quality of life for all AD patients.”
— KUN ZHAO, PHD
A Challenge Not Yet Solved
Ilya M. Nasrallah, MD, PhD, associate professor of radiology at the University of Pennsylvania in Philadelphia, authored a commentary on the study.
“The diagnosis of AD is challenging,” he said. “High confidence in diagnosis requires imaging with PET, which is expensive and not uniformly available in every region, or lumbar puncture assays, which are invasive, to evaluate for the presence of amyloid and tau protein.”
“However, most patients with memory symptoms will have a brain MRI in part to evaluate for any non-neurodegenerative and treatable cause of the cognitive symptoms,” Dr. Nasrallah continued. “Brain MRI also provides a window into the presence of Alzheimer’s disease, as Alzheimer’s disease changes the shape of the brain in particular ways.”
Dr. Nasrallah emphasized that these changes could vary in different individuals, so there isn’t a “one size fits all” approach. He noted that there are powerful new AI methods that are flexible to these patterns of variation.
“Improving our ability to screen patients for potential Alzheimer’s disease via MRI, which is broadly available, can allow for faster diagnoses and more appropriate utilization of confirmatory testing, which would benefit both patients and the healthcare system,” he said. “Current AI tools can do an excellent job in determining whether AD is present in relatively advanced cases.”
Dr. Nasrallah said that the article by Dr. Zhao and colleagues compares various methods to show that they likely all base their predictions on similar features, as they tend to make similar kinds of errors.
“However, due to the variation in how AD changes brain shape, we probably need to train these models on much larger datasets,” he said. “The authors found that the models were not as good at diagnosing AD when they were tested on data that differed from the data used to train them. This emphasizes that although we have made great progress with new artificial intelligence techniques, the challenge is not yet solved.”
For More Information
Access the Radiology: Artificial Intelligence study, “Structural MRI-based Computer-aided Diagnosis Models for Alzheimer Disease: Insights into Misclassifications and Diagnostic Limitations,” and the related commentary, “Let’s Agree to Be Wrong: Transforming Alzheimer Disease Diagnosis.”
Read previous RSNA News stories on AI and Alzheimer disease diagnosis: