AI Improves Radiologist Performance in Pinpointing COVID-19 Pneumonia

Amid the continuing conversation in radiology of whether artificial intelligence (AI) tools can help providers identify patients with COVID-19 pneumonia, research reveals a new AI model can help radiologists distinguish these patients.

Published in Radiology, investigators, led by Harrison X. Bai, M.D., from the department of diagnostic imaging at Rhode Island Hospital, reported their AI tool improved radiologist accuracy, sensitivity, and specificity in identifying patients with COVID-19 pneumonia.

“It has been hypothesized that COVID-19 infection is difficult to contain because of its potential transmission from asymptomatic carriers. A major hurdle in controlling the current pandemic is making out subtle radiological differences between COVID-19 and pneumonia of other etiology,” Bai’s team wrote. “To capture and properly manage all cases of COVID-19, it is essential to develop testing methods that accurately recognize the disease as distinct from other causes of pneumonia on chest CT.”

To create an algorithm that reached this goal, Bai and his colleagues examined chest CT images from 521 patients who tested positive for COVID-19. These individuals were treated in either China or the United States between January 2020 and April 2020. They also evaluated imaging findings from 665 patients, also treated in China or the United States between Jan. 1, 2017 and Dec. 31, 2019, who did not have COVID-19 infection.

In developing their deep learning model, they used 16,000 CT slices to train the tool to distinguish between slices with and without pneumonia-like findings (both COVID-19 and otherwise). The training set included 830 patients, the validation set included another 237 patients, and the test set included 119 patients.

Six radiologists, blinded to the scan diagnoses, conducted two reviews, providing scores for either COVID-19 or another pneumonia etiology. For the second round to review, conducted a day later without any feedback provided on the first round, the slice were shuffled, and the radiologists reviewed them with the help of the AI tool.

According to the study results, the AI tool outperformed the outcomes provided by the radiologist. The tool was more accurate that the provider alone (96 percent vs. 85 percent), more sensitive (95 percent vs. 79 percent), and more specific (96 percent vs. 88 percent). But, when radiologists conducted the second review with AI assistance, their performance improved – their accuracy rose to 90 percent vs 85 percent. In addition, their sensitivity increased to 88 percent vs. 79 percent, and their specificity rose to 91 percent vs. 88 percent.

These results, the investigators said, point to the support role that AI tools can play during this pandemic. As a method to bolster radiologist performance, they can provide patient care benefit. This can be particularly helpful because differentiating between COVID-19 pneumonia and pneumonia of other origins has proven difficult, and incorrectly interpreting an imaging study can lead to ineffective or inefficient patient management, the team explained.

“When compared to a radiologist-only approach, AI augmentation significantly improved radiologists’ performance in distinguishing COVID-19 from pneumonia of other etiology yielding higher measures of accuracy, sensitivity, and specificity,” Bai’s team wrote. “The results that we present suggest that integrating AI into radiologists’ routine workflow has potential to improve diagnostic outcomes related to COVID-19.”