MEDNAX Radiology Solutions tests AI model in largest independent radiology validation exercise to date

November 20, 2019 | by Brian Baker
MEDNAX Radiology Solutions tests AI model in largest independent radiology validation exercise to date

MEDNAX Radiology Solutions, providing radiology interpretations at over 2,050 partner facilities nationwide, allows radiology AI model testing at scale. The depth and diversity of imaging studies on the MEDNAX Imaging Platform (MIP) allow rapid and robust multi-site validation studies, such as the one recently performed by Qure.ai on their algorithm qER for detection and worklist prioritization of intercranial hemorrhage on head CT scans.

The qER algorithm was validated against radiologist findings on more than 300,000 head CT scans; a representative study of a two-week cohort yielded an impressive area under the curve (AUC), a metric used to evaluate AI models, of 0.96.  Deployed in a radiologist reading environment such as MIP, qER can prioritize those critical studies on the radiologist’s worklist to accelerate time-to-care of emergent conditions.

Removing barriers to real world implementation

Many radiology practices talk about visionary AI, but there are only a few implementing AI into their workflows. Meanwhile, developers are rapidly churning out new models – which often do not perform as well as expected when tested in the real world.

Models trained and tested on limited datasets often fail to generalize to a new device type, hospital or patient population. Deidentified radiology images or condition-specific studies are not commonly available.

Together with my colleague, Imad Nijim, CIO at MEDNAX Radiology Solutions, we led the development of the AI model validation functionality within MIP. MIP enables comparison of an algorithm’s performance against actual radiologist analyses, which is critical before an algorithm can be used in production  

“MEDNAX Radiology Solutions and its affiliates, including vRad, are well positioned to help developers validate AI on large, diverse datasets,” said Nijim. MEDNAX Radiology Solutions can test the performance of AI models in practical, real-world settings, running those models in the background on routine and stat imaging without affecting the radiologist’s workflow or study turnaround time.

This unique capability attracts many AI innovators such as Qure.ai to MEDNAX Radiology Solutions for model development, model validation and model testing. The scale at which MIP operates also tests the developer’s engineering capabilities. It is no small feat to have a model process studies in real time at the rate they are generated in a large teleradiology platform. Passing 300,000 studies through Qure.ai’s model demonstrates the robustness of the solution.

How it works: The validation process

MEDNAX Radiology Solutions added qER to MIP processes and monitored qER’s performance. Natural language processing algorithms were used to correlate results between algorithm output and radiologist reports. In addition, the team was able to compute AUC, sensitivity and specificity. A sample of cases where the AI results disagreed with the radiology reports were manually double checked by diagnostic radiologists. The comparison with manually reviewed studies yielded an impressive AUC of 0.96. “An AUC of 0.8 to 0.9 is considered excellent, while 0.9+ is considered outstanding,” said Nijim.

“Working with MEDNAX Radiology Solutions is like working with hundreds of clinical practices in the United States because of the diversity in image types, geography and device settings from which the studies originate,” said Pooja Rao, PhD, Research and Development Head at Qure.ai.

“This validation paves the way for deployment in real time across MEDNAX Radiology’s wide network, where any benefits from a prioritization model are multiplied severalfold,” added Ricardo C. Cury, MD, FACR, Chief Medical Officer at MEDNAX Radiology Solutions.

MEDNAX Radiology Solutions and its affiliates make up one of the largest radiology practice groups in the world and have become AI innovators with access to large volumes of patient studies through its proprietary imaging platform. By working with AI partners from across the US and abroad, MEDNAX Radiology Solutions can improve the quality of AI that is available to radiologists. For more information on MEDNAX Radiology Solutions’ services and partners, visit https://www.mednax.com/radai/.

 

Sample Images Identified by the qER algorithm with ICH:
qer algorithm with ICHradiology ai

 

About the author

Brian Baker
vRad Director of Software Engineering

Mr. Baker is committed to enhancing patient care through process improvement and technological innovation. He leads the vRad team responsible for the industry’s most advanced radiological workflow system, including the design, development and integration of leading-edge AI applications. He promotes a collaborative culture as the foundation of these systems that help radiologists provide diagnostic insights more efficiently and effectively every day.