November 18, 2026 · Journal of the American College of Surgeons · DOI: 10.1097/XCS.0000000000002297

Using Artificial Intelligence to Evaluate the Quality of Narrative Feedback in Surgical Entrustable Professional Activities.

Listen to this summary

This study asked whether fine-tuned large language models could reproduce expert ratings of narrative feedback quality in surgical EPA micro-assessments, and whether a simplified rubric (mQuAL) would improve consistency. Across 1,509 comments from a general surgery residency, encoder and generative models closely matched expert scores for suggestion and connection, with encoder models outperforming generative ones for evidence (accuracy 0.93 vs 0.77). Simplifying evidence scoring to a binary rubric reduced rater disagreement from 27.8% to 8.5% and improved model performance, supporting AI-assisted feedback quality assessment in surgical training.

Michel S Kabbash, Victoria J Nolan, Sasank S Desaraju, Jeremy A Balch, Benjamin Shickel, Jennifer H Fieber, George A Sarosi, Tyler J Loftus

This is one of 33,000+ journals available on OSLR. Try it free for 14 days.

Free 14-day trial. 33,000+ journals. Cancel anytime.

14-day free trial. No commitment.

“

"Oslr has become part of my weekly routine on my day off. The clinical relevance of the summaries is outstanding — I'd rate it 9/10. Being able to consume research hands-free is a huge advantage for busy physicians."

Dr. Jennifer Thompson

Dr. Jennifer Thompson

Portland, OR

Stay current without falling behind

33,000+ journals. 3-minute audio summaries. Free for 14 days.

Download on the App StoreGet it on Google Play