AI Challenges Fuel Innovation Beyond the Leaderboard
Competitions bring together global experts and produce datasets, algorithms and collaborations that endure
For more than a decade, RSNA’s AI Challenges have drawn thousands of participants from around the world to tackle some of medical imaging’s most difficult problems. Along the way, the competitions have generated uniquely valuable datasets, open-source algorithms and collaborations that continue to advance radiology AI long after the winners are announced.
Those lasting benefits stem in part from the diversity of both the sources of data and the competitors themselves. Radiology sites from around the globe have contributed de-identified imaging studies and associated clinical data to enable the assembly of datasets for training and testing AI models.
While the challenges focus on issues related to radiology, participants have included everyone from practicing radiologists and trainees to data scientists, computer engineers and AI specialists with no prior connection to medical imaging. Hosted on Kaggle, the competitions form a global community that organizers say has become one of the program’s greatest strengths.
“The RSNA AI Challenges help bring important radiology questions to the broader data science community,” said Errol Colak, MD, FRCPC, an abdominal radiologist at Unity Health Toronto and associate professor at the University of Toronto. “Many participants might never have considered applying their expertise to medical imaging, but these competitions present them with a well-defined, clinically meaningful issue to tackle. That expands the community of people using AI to address questions that matter for patient care.”
Dr. Colak has helped organize several RSNA AI Challenges, including the Abdominal Trauma Detection Challenge in 2023. “Opportunities like these bring together talented people from diverse backgrounds, not only to compete, but also to learn from one another and collaborate. That exchange of ideas is incredibly valuable for radiology because it helps move the field forward in ways that extend beyond any single competition,” he said.
“Another lasting benefit is that the challenges generate large, carefully curated, multinational datasets with expert annotations. These datasets remain available to researchers long after each competition ends,” Dr. Colak added. “The top-performing algorithms are also released as open-source code with detailed explanations, which allows the community to study different approaches, learn from them and build on that work. In that sense, the impact of each challenge extends well beyond the leaderboard.”
Dr. Colak explained that these datasets are often notable for being among the first or most substantial publicly available datasets in their respective areas. They provide important foundations for future model development, benchmarking, and collaboration across the radiology AI community.
A Challenge Sparks an AI Career
As an early participant in the AI challenges, Felipe Kitamura, MD, PhD, a neuroradiologist and chief medical officer at Eden in Palo Alto, CA, has shared many of the experiences Dr. Colak has observed. He, too, appreciates how the innovation continues after the competition closes and the winners are announced.
For his first exposure to the RSNA AI challenges, the then-resident and colleagues from Brazil partnered with a team of computer scientists and AI engineers from Universidade Federal de Goiás in Brazil.
Dr. Kitamura describes RSNA’s 2017 Pediatric Bone Age Challenge as an exhilarating experience that combined earnest research and learning with fun gamification features like medals, badges and a real-time leader board tracking competitors’ progress.
The group went on to claim third place out of 700-plus entries but, importantly, the work didn’t stop there. According to Dr. Kitamura, the momentum, motivated by his team’s performance, carried back to Universidade Federal de São Paulo in Brazil to continue conducting AI research.
“One thing that we did was to validate our model on a local dataset from our hospital here in Brazil,” he recalled. “We know AI will not necessarily ‘work’ in places different than those where it was trained, such as in one country versus another.”
Because the dataset for the Pediatric Bone Age challenge originated in the United States, Dr. Kitamura’s group wanted to determine whether the model would generalize to the Brazilian population. Their testing produced favorable results.
“We kept doing work in AI with the dataset, but we also continued AI research in many other areas,” he said. “The competition was also good in that sense, because it opened doors to collaborate with other international universities.”
Beyond fostering international collaborations, the challenges continue to influence the field after the competitions end through publications, subgroup analyses and other opportunities to discuss the results of a challenge and share lessons learned, Dr. Kitamura noted.
Just as importantly, they help cultivate the people who will drive the field forward. Beyond attracting the global talent pool that Dr. Colak has seen returning for the contests challenge after challenge, the competitions have helped shape careers, including Dr. Kitamura’s own journey into radiology AI.
He credits visibility from the Pediatric Bone Age Challenge for landing him a job leading the AI team within a large radiology practice shortly thereafter. Most recently, he has been named chief medical officer at Eden, a software development firm. He also sits on RSNA’s AI Committee and has served on planning task forces for the latest challenges.
Bridging the Gap to Clinical Practice
While the challenges have helped launch careers and advance research, questions remain about how quickly those innovations can make their way into routine clinical practice.
Like Dr. Kitamura, Ian Pan, MD, a radiology resident at Brigham and Women’s Hospital in Boston, took an early interest in the potential of AI in radiology. That interest led him to become deeply involved in the AI challenges, earning a first-place win in the Pneumonia Detection Challenge in 2018, a second-place finish in the Pulmonary Embolism Detection Challenge in 2020 and sixth-place honors in the Cervical Spine Fracture AI Challenge in 2022.
Despite his personal enthusiasm for the competitions and the radiology community’s general support of AI, both Dr. Pan and Dr. Kitamura expressed some disappointment that greater strides haven’t been made at this point. “There are a lot of interesting solutions and approaches that come out of these challenges,” Dr. Pan acknowledged. “For me, it’s always like ‘well, let’s see how it does in the real world,’ but there are so many barriers to implementation.”
Part of the problem, he suggested, is that the algorithms being developed are targeted at a particular finding—often urgent in nature—like pulmonary embolism. “Commercially available products have been used largely as a kind of triage system to prioritize a positive finding so that the radiologist can read it sooner,” he observed.
“It's interesting to reflect upon these prior challenges, which were more narrow in their scope, and then think about the future,” Dr. Pan said. “How do we expand beyond that and think about how we can create challenges that will help us move towards more a general purpose, like a radiology AI assistant, rather than kind of ‘triage’ algorithm that only identifies a few specific findings?”
That evolution is beginning to take shape in different ways. The upcoming RSNA Knee Abnormality Detection Challenge expands the model toward more comprehensive tasks involving multiple abnormalities, while a planned abdominal triage challenge reflects RSNA’s move toward benchmark-focused challenges designed to support durable, standardized evaluation beyond a single competition.
Despite implementation hurdles, Dr. Pan believes the challenges remain an important driver of innovation in radiology AI. “One of the nice things about these competitions is they get a bunch of very talented people who otherwise would not be working on radiology-related challenges to focus on a particular task and come up with some novel solutions,” Dr. Pan emphasized. “I think they do have influence on the future and evolution of the field.”
For More Information
Learn more about the 2026 RSNA Knee Abnormality Detection Challenge.
Access the Radiology: Artificial Intelligence study, “Deep Learning for Pulmonary Embolism Detection: Tackling the RSNA 2020 AI Challenge.”
Explore the RSNA Annotated Library of AI Systems (ATLAS) to learn how radiology AI systems perform beyond the competition setting.
Read previous RSNA News stories on RSNA AI Challenges: