Model Predicts Lung Nodule Risk with Fewer Data
Deep learning approach matched clinician performance in lung cancer screening CT, with a fraction of the usual dataset
An AI-based algorithm that estimates lung nodule malignancy risk reached clinician-level accuracy with as little as 20% of the training data, according to a study published in Radiology: Artificial Intelligence.
“Our findings suggest that robust, generalizable AI tools can be developed without massive datasets,” said lead author Bogdan Obreja, MSc, a PhD candidate at Radboud University Medical Center (RUMC) in Nijmegen, the Netherlands.
Lung nodules are common findings on cancer screening with low-dose CT, yet only a small percentage end up progressing to cancer. A reliable AI-assisted risk assessment tool has the potential to reduce unnecessary follow-up work and prioritize patients who need timely intervention.
“A key consideration when developing these algorithms is the amount of data needed to effectively train the system while maintaining robust performance,” Obreja said.
Conventional wisdom dictates that more data equals better results. Obreja and his colleagues evaluated how the amount of training data affects the performance of a specific AI model designed to predict malignancy risk in pulmonary nodules detected on lung cancer screening CT.
The team trained the model on 16,077 annotated nodules from the National Lung Screening Trial (NLST) and conducted external testing using data from the Danish Lung Cancer Screening Trial.
The algorithm was just as good at predicting whether lung nodules were cancerous when it used at least 80% of its training data as when it used all the data. Even when trained on only 20% of the data, it still performed nearly as well as 11 clinicians.
“Most nodule malignancy risk estimation algorithms reported in the literature are trained on large datasets exceeding 10,000 nodules and demonstrate strong performance. However, our findings suggest that such large volumes of data may not be necessary to make a strong model,” Obreja said.
Obreja attributed the algorithm’s performance in part to a deep learning architecture well-suited for nodule malignancy risk estimation. The model combined 2D and 3D convolutional neural networks, allowing it to capture both detailed slice-level and volumetric information.
“Combining 2D and 3D models improves overall performance and enhances robustness, reducing variability across predictions,” Obreja said.
A Boost for Smaller Teams and Better Data Diversity
The findings may be good news for smaller research groups and companies that lack the resources of larger enterprises.
“For lung cancer screening, resources may be better spent on collecting and annotating diverse datasets, such as scans from different populations, scanners and rare or challenging cases, rather than accumulating large volumes of similar data,” Obreja said.
He and his colleagues emphasized that the optimal amount and type of data is highly task-dependent, and that more data will continue to be advantageous in many applications.
“We believe that beyond a certain point, gains in performance from additional data, especially if they consist mainly of common cases, become increasingly limited,” said study senior author Colin Jacobs, PhD, associate professor and research group leader at RUMC’s Department of Medical Imaging. “However, incorporating more diverse data, particularly in rare cases like atypical cystic nodules or airway nodules, is still likely to improve model performance.”
Dr. Jacobs said future work should emphasize broader external validation, greater data diversity and methods to manage clinical variability. As part of the Lung Nodule Analysis 2025 (LUNA 25) challenge, the researchers have recently released a dataset of 6,163 nodules, including 555 malignant and 5,608 benign, from the NLST dataset used in the study, along with a baseline algorithm like the one presented in the study.
“We hope that research groups will apply and evaluate models on their own lung cancer screening datasets,” Obreja said. “This will help determine whether the performance observed in our study generalizes across different populations and settings, and across both similar and alternative model architectures.”
“We believe that beyond a certain point, gains in performance from additional data, especially if they consist mainly of common cases, become increasingly limited. However, incorporating more diverse data, particularly in rare cases like atypical cystic nodules or airway nodules, is still likely to improve model performance.”
— COLIN JACOBS, PHD
AI Accuracy Depends on More Than the Algorithm Alone
The model’s reliance on quality imaging features that helped separate the nodules into two groups enabled it to achieve the Bayes error rate, which is the theoretical minimum possible error given the inherent noise in the data, according to Kellie J. Archer, PhD, professor and chair of the Division of Biostatistics at the Ohio State University College of Public Health in Columbus.
In a commentary accompanying the study, Dr. Archer commended the model’s use of training data that spanned the full variability of the clinical application. She also cautioned against overgeneralizing the study’s findings.
“It is important to emphasize that these results held for this particular AI model,” Dr. Archer said. “They may not generalize across other models because performance is highly dependent on the measured features.”
To illustrate the kind of variability that adds strength to the model, she noted that the 16,077 nodules used for training came from 22 different scanners across four manufacturers, and that researchers recruited participants from 33 sites. “Although the training dataset did not match the racial distribution of the U.S. population, all major racial groups were represented, and there was good balance with respect to gender,” Dr. Archer said.
In addition, Dr. Archer noted that the use of two different cohorts for training and testing ensured a fair estimate of how well the deep learning AI algorithm will generalize to new patients.
“The quality of the data, having good features and making sure that you have good representation among the participants in your study are all really essential components of developing strong AI models,” Dr. Archer said.
For More Information
Access the Radiology: Artificial Intelligence article, “Characterizing the Impact of Training Data on Generalizability: Application in Deep Learning to Estimate Lung Nodule Malignancy Risk,” and the related commentary, “How Much Training Data Does AI Need?”
Read previous RSNA News stories on lung cancer: