Confidence-Accuracy Alignment in Cardiology Knowledge: Comparing Medical-Specific and General-Purpose Large Language Models Using ACCSAP

The authors investigate whether medical-specific large language models (LLMs) provide improved clinical reliability compared to general-purpose models in cardiology knowledge assessment. Using a standardized benchmark, they found that general-purpose models like Gemini and ChatGPT outperformed the medical-specific model MedGemma in diagnostic accuracy, although all models exhibited poor confidence calibration. The study concludes that while general-purpose LLMs may excel in complex clinical reasoning, their self-reported confidence is not a reliable indicator of correctness, suggesting a need for clinician oversight in their use.