This study evaluates the comparative performance of trained machine learning models, commercial off-the-shelf (COTS) large language models (LLMs), and physicians in predicting 30-day mortality and readmission using discharge summaries from the MIMIC-IV dataset. While machine learning models demonstrated superior performance compared to both LLMs and human evaluators, none of the approaches achieved clinical-grade reliability, defined by us as a Matthews Correlation Coefficient > 0.8. Interestingly, all methods performed better at predicting mortality (3% incidence) than readmission (21.1% incidence), suggesting that the signal-to-noise ratio may be higher when predicting mortality. The universal difficulty both machines and humans encountered in these prediction tasks indicates fundamental challenges in forecasting post-discharge outcomes from discharge summaries alone. These findings highlight both the potential and current limitations of artificial intelligence in clinical prediction tasks, while emphasizing the inherent complexity of such predictions regardless of the approach used.

The authors have declared no competing interest.

This study did not receive any funding

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Physionet. MIMIC-IV [Internet]. Available from: https://physionet.org/content/mimiciv/3.1/

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes