Can a Large Language Model Grounded in Text-Based Agency-Specific Prehospital Protocols Provide Accurate Care Recommendations?

The authors aimed to evaluate the accuracy of a retrieval-augmented generation (RAG)-based large language model (LLM) in providing care recommendations for prehospital emergency scenarios based on specific emergency medical services (EMS) protocols. The study found that the LLM achieved a 75% accuracy rate in recommending appropriate patient care actions across various clinical scenarios, with some notable missed actions, particularly in pediatric cases. Overall, the findings suggest that while the LLM can generate grounded responses, there are areas for improvement in its clinical accuracy.