Achieving Real-World Success with AI

“If we told clinicians, ‘we will use advanced math to help you improve care,’ they would probably be fine with it. But the term ‘artificial intelligence’ raises natural skepticism about what it really means.” – Mark D. Weisman, MD, MBA, Chief Medical Information Officer, TidalHealth

“First, do no harm” is a promise many of us make when becoming clinicians. To me it means that, for as long as I practice medicine, I must carefully weigh the risks and benefits of my decisions for patients. This principle underpins a healthy skepticism from some clinicians when learning about new approaches and technologies, such as artificial intelligence (AI), that claim to improve patient care.

In my conversations with other clinicians, I sometimes encounter a wariness that AI could one day replace us, which could negatively affect quality patient care. This is why trust and evidence are essential to incorporating AI into medical practice. But that trust and evidence stems from rich involvement from clinicians themselves.

TidalHealth, a health system serving the Delmarva Peninsula, a peninsula between the Chesapeake Bay and the Atlantic Ocean shared by the states of Delaware, Maryland and Virginia, is successfully integrating AI into its clinical workflows. The health system has taken a proactive, thoughtful and – most importantly – ongoing journey to adoption.^[1] Two critical leaders in TidalHealth’s journey are John Jordan, PharmD, BCPS, Senior Director of Pharmacy Services, and Mark D. Weisman, MD, MBA, Chief Medical Information Officer.

Based on my experience with TidalHealth, the day-to-day challenges I experience as a practicing radiologist, the feedback I receive from my fellow clinicians and the research that Watson Health has conducted on the topic, I believe there are three guiding principles hospitals can follow as they begin to build up their AI capabilities.

Trustworthy AI begins with projects that address real-world challenges and enhance clinical processes.

“AI will establish more trust with physicians when it solves the challenges we’re actually struggling with. For example, when I see a patient for abdominal pain in the office, I need to find if they’ve had endoscopies, colonoscopies, MRIs, lab work, and any other parts of the medical record that can help me diagnose that abdominal pain. If there’s an AI tool that searches multiple data bases and pulls those records for me into one spot, that’s the type of problem I need help with.” – Dr. Weisman

For example, integrating an AI-powered clinical decision support tool into the right part of TidalHealth clinicians’ EMR workflow has made the tool easier to use and increased the number of searches ten-fold. Time studies also revealed that 73 percent of AI-assisted searches at the health system provided answers in less than one minute, saving up to three minutes over other search methods.

Emerging research also backs up TidalHealth’s real-world results. One recent study^[2] found that AI can help radiologists in one of their most important, but time consuming, manual tasks – finding and extracting all relevant information within a patient’s EMR during an exam. In the study, clinicians manually created a list of 1,385 key medical terms they would use to find relevant patient information. Researchers used natural language processing, a form of AI, to expand that list to nearly 30,000 by suggesting clinically or semantically relevant terms. Radiologists reviewed the broader list of terms and validated 97.32 percent of terms as useful and relevant.

Clinician involvement in the development and training of AI is imperative.

“As an early adopter, I was part of development efforts around IBM Micromedex with Watson. The results were not always great in the early days of testing. But using feedback from clinicians, that algorithm learned over time. I watched it get better and better with every iteration.” – Dr. Jordan

Not all AI algorithms are created equal. Clinicians want assurance that they are using well-trained, mature, non-biased AI. Using a balanced scorecard during development can help teams keep an eye on the big-picture strategy while also helping them follow the right processes. And it’s important to ask the right questions up front, such as: How was the algorithm developed and trained to date? What kind of diversity was in the data set? Who published the data and looked for bias? What about security protocols?

Data scientists, clinicians and researchers at Watson Health Imaging are exploring new ways to train AI, and there are some promising results. For example, one study^[3] used a two-stage pipeline to detect pulmonary embolus (PE) with a model that used sparsely annotated chest CT images with a high level of performance (AUC .94 validation set, AUC .85 test set). Input from radiologists was incredibly important to this work and other development efforts. When we developed IBM Patient Synopsis, for example, we worked with nearly 100 radiologists to determine the information they would find helpful at the time of interpretation.

Clinicians will embrace and celebrate AI when they see for themselves how it can augment their capabilities.

“Think about all the manual tasks physicians are being asked to do these days – like capturing quality indicators, keeping up with the latest evidence, looking through databases to pull records together for a patient – imagine if we didn’t have to do all that alone? AI is something that can help us do more than we did in the past. It’s here to help us, not to replace us.” – Dr. Jordan

Clinician involvement in developing and training AI algorithms is valuable, but finding accurately annotated, timely and diverse data sets can be challenging. When we look at the field of radiology, for example, identifying large datasets of medical images acquired in diverse settings, from a variety of equipment manufacturers that have been expertly annotated by clinicians of varied training and experience, creates a high bar.

Additionally, clinicians often provide rich information, but that data often resides in unstructured clinical notes. This means that key areas of documentation – such as a problem list in the EMR, an admission note or a discharge summary – may not reflect this wealth of insights.

Furthermore, it’s been said that “diseases don’t read textbooks.” In just the chest, for instance, think about the myriad ways in which a pulmonary nodule, emphysema, or a pulmonary embolus might appear on a CT, which might have been acquired at thick or thin slices, with or without contrast, or early or late in enhancement.

One recent study^[4] indicates that AI, in the form of automated text analysis, can help clinicians detect relevant information and discrepancies held within unstructured and annotated notes. Using 1,323 unstructured clinical notes annotated by 46 clinicians to create 5,372 annotated notes, an AI algorithm detected disease concepts with 0.94 precision and detected discrepancies in the information with 0.98 precision. An application of these types of tools within EMRs could lead to more accurate, complete, and updated patient records.

AI may never replace clinicians, but it might be more accurate to predict that clinicians that use AI could one day replace clinicians that don’t. “It has to be a partnership – the clinician and AI, working hand-in-hand,” Dr. Jordan said. “If we do that, these tools will make us better.”

For more coverage based on industry expert insights and research, subscribe to the Diagnostic Imaging e-Newsletter here.

References:

[1]TidalHealth results from EMR integration from a 2020 case study, “IBM Micromedex: Improving efficiency and care with AI” by Tina Moen, PharmD.

[2] Automatic Patient Data Summarization for Radiologists. Jadhav A et al. Presented at the 2019 RSNA Conference

[3] Pi-PE: A Pipeline for Pulmonary Embolism Detection Using Sparsely Annotated 3D CT Images. Rajan D. et al. Proc Machine Learning for Health NeurIPS Workshop, PMLR, 2020: 116 220-232

[4] “Improving the Path from Diagnoses to Documentation: A Cognitive Review Tool for Clinical Notes and Administrative Records.” Guo Y et al. AMIA Annu Symp Proc. 2018: 518-526. eCollection 2018.