In Fourth Year, RSNA AI Challenge Propels Winners to Greater Heights
Datasets lead to research, spinoff businesses and improved patient care
Neuroradiologist Felipe Kitamura’s first foray into a machine learning (ML) competition three years ago appeared to be headed to an inglorious conclusion. Dr. Kitamura, then a neuroradiology fellow at Universidade Federal de São Paulo in Brazil, had teamed up with engineers from Brazil’s Federal University of Goiás for the first RSNA AI Challenge in 2017. The deadline was only two weeks away and he and his teammates had not come up with a workable solution for determining pediatric bone age, the theme for the challenge.
“We tried a lot of the standard models that were available, but they just didn’t work well for us,” Dr. Kitamura said.
With time running out, the Goiás engineers hatched an idea: build a customized neural network architecture that was only 1% of the size of the neural networks that other teams were using. The algorithm worked well enough to earn the team a third-place finish in RSNA’s first AI Challenge.
“We were not expecting to have that kind of result, so we were quite excited,” Dr. Kitamura said.
The experience fired Dr. Kitamura’s enthusiasm for AI. He notched top 10 finishes at other AI competitions, including the 2018 RSNA Pneumonia Detection Challenge, and then became a member of the RSNA AI Challenge organizing committee. He also was hired as head of AI at DASA, one of the largest medical diagnostics companies in the world.
“The first RSNA Challenge gave me a visibility that I was not expecting and changed my career in ways that I wouldn’t have imagined,” he said. “I would say that 80% of my time now is spent working on ML projects and 20% is working as a radiologist at the university."
AI Challenge Competitors Thrive
Dr. Kitamura’s experience demonstrates how the AI Challenge, now in its fourth year, has not only been a boon to research but also a life-changing experience for many competitors. Some have gone to top finishes in other AI competitions. Others have launched businesses.
And one, Ian Pan, MD, a medicine intern at University Hospitals Cleveland Medical Center, even made history. Dr. Pan recently became the first physician to achieve Grandmaster status on Kaggle, the data science platform that hosts RSNA AI challenges. The extremely rare distinction is awarded to people who win gold medals in ML competitions, both as part of a team and as an individual.
“It’s one of my biggest achievements, partly because I never thought it would happen so quickly,” he said.
Just two years ago, Dr. Pan was a medical student at Brown University in Providence, RI, when he and his teammate Alexandre Cadrin-Chênevert, MD, a radiologist in Québec, won first place in the 2018 RSNA Pneumonia Detection Challenge. The win launched a string of successes for Dr. Pan in competitions both inside and outside of radiology.
“Winning the Pneumonia Detection Challenge gave me confidence in my ability to compete with other individuals and showed that the process that Alex and I created for building these models is something that can compete with models from the best data scientists,” Dr. Pan said.
As he prepares to enter a radiology residency in Boston next year, Dr. Pan continues to fit AI research into his busy life. He has developed algorithms for thyroid nodule detection and bone age assessment, and his entry in RSNA’s 2019 Intracranial Hemorrhage AI Challenge was eventually deployed at his hospital. He also consults for MD.ai, a company focused on medical image annotation.
Dr. Pan typically runs experiments on his computer overnight and reviews the results in the morning. He then makes adjustments to his code and runs the experiments again during the day while he is at the hospital. He credits his connection with radiology in helping him craft effective AI tools.
“My approach in these competitions is to first understand the data,” he said. “Coming from a medical imaging background makes it easier for me to do that.”
“There’s a lot of noise in the industry today with so many people doing AI. It’s been very helpful to be a winner in a contest like the RSNA Bone Age Challenge to help our work stand out above that background noise.”
ALEXANDER BILBILY, MD
Building a Business, Improving Patient Care
Radiologists Alexander Bilbily, MD, and Mark Cicero, MD, winners of the RSNA 2017 Pediatric Bone Age AI Challenge, echo that sentiment.
“The fact that we can speak the languages of radiology and AI and marry the two fields together is an advantage,” Dr. Cicero said. “We’re able to more easily identify areas where AI solutions will have an impact and where they won’t.”
The two radiologists bonded in medical school over a common interest in AI. They formed a company called 16Bit AI — a reference to the 16-bit pixels in medical imaging — to develop and market AI solutions.
For the 2017 RSNA Pediatric Bone Age AI Challenge, the pair exhaustively recorded the results of their experiments and built hypotheses around those results to improve the AI’s architecture. The team captured first place in the competition. Their methodical approach resulted in an algorithm that was generalizable to different populations, a critical element of a successful AI application for medicine.
“Our algorithm has become almost indispensable in doing these bone age assessments,” Dr. Cicero said. “It’s being used hundreds of times per day worldwide. We’re actually in the process of obtaining regulatory approval and it’s really propelled our company to global recognition.”
“There’s a lot of noise in the industry today with so many people doing AI,” Dr. Bilbily added. “It’s been very helpful to be a winner in a contest like the RSNA Bone Age Challenge to help our work stand out above that background noise.”
The two radiologists are working to grow their company, an effort that recently got a boost when they won a $1.3 million grant to work on a COVID19 AI for clinical decision support. The grant allowed them to hire additional staff to sustain the company in the short term while keeping their big picture goals in sight.
“In the long term, we’ve identified a few key opportunities to not only take our business to the next stage of size and reach, but also to impact health care on an international level,” Dr. Bilbily said. “That’s really why we’re doing this — to create devices that improve the health of people.”
In 2019, first place in the RSNA Intracranial Hemorrhage Detection and Classification AI Challenge, was captured by the team SeuTao, led by Sen Wang, MD. In the challenge, teams were required to develop algorithms capable of identifying and classifying subtypes of hemorrhages on head CT scans. The data set, which comprised more than 25,000 head CT scans contributed by several research institutions, was the first multiplanar dataset used in an RSNA AI Challenge.
AI Challenge Becoming Prestigious
The post-competition success of competitors is not lost on challenge organizers like Safwan S. Halabi, MD, associate professor of radiology at Stanford University School of Medicine in California, chair of the RSNA Machine Learning Learning Data Standards Subcommittee of the Radiology Informatics Committee (RIC) and a member of the Machine Learning Steering Subcommittee of RIC. Luciano Prevedello, MD, serves as chair of the Machine Learning Steering Subcommittee and John Mongan, MD, PhD, serves as vice-chair.
“We’re seeing more and more participants every year and they are actually coveting that ability to say they achieved top 10 status in the competitions,” he said. “The challenge will continue to attract people because the high-quality data is there.”
“Putting together such huge and well-curated data sets is a merit of RSNA,” Dr. Kitamura said. “This is essential for these kinds of AI challenges because algorithms are only as good as the data that feeds them.”
The increasing participation from both physicians and those outside of health care is giving the RSNA AI Challenge an outsized impact that Dr. Halabi likened to that of the space program.
“All of the industries around building a space shuttle or sending people to the moon resulted in the creation of everything from the graphing calculator to entirely new methods of communication,” he said. “The offfshoots that come out of these programs really drive innovation.”
That innovation will ultimately benefit both radiologists and patients.
“There are definitely more and more people who, instead of being fearful of AI taking over their physician jobs in the future, are embracing it and becoming part of the process,” Dr. Halabi said. “At the end of the day, these challenges will provide better tools for physicians to reach as many people as possible with the most accurate and precise diagnostic abilities.”
For More Information
Learn more about the RSNA 2020 AI Challenge and view winning entries from past competitions at RSNA.org/AI-Challenge.
Read previous RSNA News stories on AI:
Radiology Informatics Experts Help Shape the Future of Artificial Intelligence
RSNA at the Forefront of AI Innovation
RSNA 2020 AI Challenge Focuses on Pulmonary Embolism
In 2020, the RSNA/Society of Thoracic Radiology (STR) Pulmonary Embolism Detection Challenge, tasks participants with creating AI algorithms for detecting and characterizing pulmonary embolism.
The 2020 AI Challenge organizing team of data scientists and radiologists includes: Robyn Ball, PhD, Errol Colak, MD, Stephen Hobbs, MD, Jayashree Kalpathy-Cramer, PhD, Felipe Kitamura, MD, Matthew Lungren, MD, John Mongan, MD, PhD, Luciano Prevedello, MD, George Shih, MD, Anouk Stein, MD, and Carol Wu, MD.
RSNA and STR created a massive dataset of more than 12,000 CT scans collected from five international research centers. The dataset was labeled with detailed clinical annotations by a group of more than 80 expert thoracic radiologists.
“We have international collaborators who are donating their data for the world to use,” said Safwan Halabi, MD, who has served as an organizer for the AI Challenge since its inception. “That’s really the catalyst that will help bring these tools to developing nations that will be validated for physicians and effective in day-to-day practice.”
The top 10 performing competitors will be recognized during RSNA 2020 with $30,000 in total prize money. The winning algorithms are freely accessible and will be posted online.