AI Monitoring AI with LLMs: The American College of Radiology's Imaging AI Registry

The authors aim to describe a technical workflow for scalable automated monitoring of artificial intelligence (AI) in radiology through the Assess-AI registry. They developed and optimized large language model (LLM) prompts to extract clinically relevant findings from radiology reports, achieving high agreement with reference standards for specific use cases. While the workflow demonstrates feasibility for report-finding extraction, the independent accuracy and clinical utility of the approach remain to be established.