{Reference Type}: Journal Article
{Title}: Assessing Laterality Errors in Radiology: Comparing Generative AI and Natural Language Processing.
{Author}: Kathait AS;Garza-Frias E;Sikka T;Schultz TJ;Bizzo B;Kalra MK;Dreyer KJ;
{Journal}: J Am Coll Radiol
{Volume}: 0
{Issue}: 0
{Year}: 2024 Jul 1
{Factor}: 6.24
{DOI}: 10.1016/j.jacr.2024.06.014
{Abstract}: <B>OBJECTIVE: </B>We compared the performance of generative AI (G-AI, ATARI) and natural language processing (NLP) tools for identifying laterality errors in radiology reports and images.<BR><B>METHODS: </B>We used an NLP-based (mPower) tool to identify radiology reports flagged for laterality errors in its QA Dashboard. The NLP model detects and highlights laterality mismatches in radiology reports. From an initial pool of 1124 radiology reports flagged by the NLP for laterality errors, we selected and evaluated 898 reports that encompassed radiography, CT, MRI, and ultrasound modalities to ensure comprehensive coverage. A radiologist reviewed each radiology report to assess if the flagged laterality errors were present (reporting error - true positive) or absent (NLP error - false positive). Next, we applied ATARI to 237 radiology reports and images with consecutive NLP true positive (118 reports) and false positive (119 reports) laterality errors. We estimated accuracy of NLP and G-AI tools to identify overall and modality-wise laterality errors.<BR><B>RESULTS: </B>Among the 898 NLP-flagged laterality errors, 64% (574/898) had NLP errors and 36% (324/898) were reporting errors. The text query ATARI feature correctly identified the absence of laterality mismatch (NLP false positives) with a 97.4% accuracy (115/118 reports; 95% CI = 96.5% - 98.3%). Combined Vision and text query resulted in 98.3% accuracy (116/118 reports/images; 95% CI = 97.6% - 99.0%) query alone had a 98.3% accuracy (116/118 images; 95% CI = 97.6% - 99.0%).<BR><B>CONCLUSIONS: </B>The generative AI-empowered ATARI prototype outperformed the assessed NLP tool for determining true and false laterality errors in radiology reports while enabling an image-based laterality determination. Underlying errors in ATARI text query in complex radiology reports emphasize the need for further improvement in the technology.