Human Judgment and the Limits of Artificial Intelligence for Automated Rank Order Lists in Diagnostic Radiology Residency Selection

The authors aimed to evaluate the reliability of large language models (LLMs) in generating rank order lists (ROLs) for diagnostic radiology residency selection based solely on pre-interview application data compared to including human interview scores. The study found that LLMs showed low agreement with the final ROL when using application-only data, while including human input significantly improved alignment. Additionally, LLM-generated rankings exhibited systematic biases against certain applicant subgroups, indicating that AI should not replace human judgment in this context.