Excited to present our work at MLHC 2025 at held at Mayo Clinic on Saturday, Aug 16! 🏥 Thank you to my collaborators Akshay Swaminathan, @jason-fries.bsky.social, Jenelle Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung, Jesutofunmi A. Omiye, Mehr Kashyap, Nigam Shah

📢 How factual are LLMs in healthcare? We’re excited to release FactEHR — a new benchmark to evaluate factuality in clinical notes. As generative AI enters the clinic, we need rigorous, source-grounded tools to measure what these models get right — and what they don’t. 🏥 🤖

Was AC for one of the papers. Went to metareview and noticed that two reviews were basically paraphrases of each other (down to ordering of weaknesses) and LLM generated. Noticed the paper was also weirdly well written garbage. Then I investigated the deadbeat reviewers, realized they don't exist.

Really nice work, earlier this year we found similar results in addition to the fact that, clinical LLMs are more sensitive to changes in instruction phrasings compared to their general domain counterparts. arxiv.org/abs/2407.09429