Interesting chat about hallucination in patient information dialogues. When we ask domain experts to check statements such as "X increases liklihood of Y", response is often "depends on context" or "we dont know, need more experiments". Does this make statement a hallucination?

New blog: Hard to Change Poor Research Culture Research culture is very important but also very hard to change. I suspect this is one reason why it is so difficult to get people to do more rigorous and meaningful experiments. ehudreiter.com/2025/11/24/h...

Aberdeen CS is hiring a new lecturer for its "Joint Institute" with South China Normal University. Basically you would be based and do research in Aberdeen, but would be expected to go to China a few times a year and teach at SCNU. Closing 28 Nov www.abdn.ac.uk/jobs/vacanci...

I'm disturbing reports about chatbots encouraging children to kill themselves. such as www.bbc.co.uk/news/article... . Shame that the AI Safety community in general, and the @AISecurityInst in particular, seem to have little interest in this, very disappointing...

New blog: Understanding what users want from NLG When building an NLG system, it really helps to understand what users want; this came up several times at the recent INLG conference. I discuss some of our work in this space, and give a few suggestions. ehudreiter.com/2025/11/06/u...

I'm trying to understand OpenAI's healthbench. "HealthBench: Evaluating Large Language Models Towards Improved Human Health" doesnt say much about the BM(eg, very few examples). Are there other papers? I dont care how well model X performs, I want to judge if I can trust the BM

Just back from INLG. Nice event as always, but I am concerned that it is losing its uniqueness. Maybe for 2026 Ill suggest some special tracks which are interesting to INLG community but not ARR types (eg, user requirements/eval, non-LLM techniques).

New blog: Most common uses of AI in Healthcare Data on usage of AI in healthcare suggests that most common uses in 2025 are probably (A) giving personalised health information to patients and (B) helping clinicians write documents. ehudreiter.com/2025/10/21/m...

One of my main goals for 2025-26 is to help my 6 senior PhD students submit their PhDs before I retire. Glad to say that Nicolay Babakov has now done so, with viva scheduled for Dec. Other five students seem to be on track, which is encouraging.

Somewhat frustrated yesterday to once again read ACL paper which did all sorts of complex things (including the usual results tables showing best approach) on garbage data. With minimal ack of this in limitations. Most fundamental rule of CS is Garbage In, Garbage Out

New blog: Good diagrams for research papers Ive seen a number of diagrams recently which are too complicated and difficult to understand. I explain some of the problems I see and give advice. ehudreiter.com/2025/10/08/g...

Really interesting paper on real-world evaluation in IR. I should learn more about eval in IR, its not something Ive ever properly looked at dl.acm.org/doi/10.1145/...

Several people have asked me recently if I will still be able to contribute to research projects after I retire in summer 2026. Absolutely! I will have emeritus statius, and am very hapy to remain involved in research projects at Aberdeen amd elsewhere.

Aberdeen CS is hiring! We are especially interested in hiring new faculty in NLP. Closing date is 8 Oct. For more info, see below (or contact me) www.abdn.ac.uk/jobs/vacanci...

New blog: Reflections on blogging I am often asked about my experience blogging, sometimes by people who are considering writing their own blog. In this “meta” blog, I summarise my thoughts and experiences about my blog. ehudreiter.com/2025/09/23/r...

Aberdeen CS will probably be looking for a new lecturer in NLP. Formal advert is not out yet, but feel free to contact me informally if interested.

The registration page for #INLG2025 is now live! Join us in Vietnam at the Oct 29 - Nov 2 for the best conference on #NaturalLanguageGeneration 2025.inlgmeeting.org/registration... Curious to see what will be presented? Check out this list of accepted papers! 2025.inlgmeeting.org/accepted-pap...

New blog: Defining hallucination is not straightforward Many researchers assume that hallucination is a binary feature; either something is a hallucination or it is not. This is too simplistic. I describe some of the issues I have seen below. ehudreiter.com/2025/09/10/d...

At ACL, I engaged with 50 papers (went to oral, talked to poster person). Decided (looked at paper sometimes), that 3 of these robust, interesting, relevant to me; 2 of these 3 won awards. Hum, maybe in future I should focus on 40 award papers, ignore the other 3000?

Excited by recent positive evaluations of NLG apps developed by my students to encourage safer driving in UK and Nigeria. We see stat sig reductions in unsafe driving incidents in both countries. ehudreiter.com/2025/09/03/e...

Last week I had to deal with two cases of papers containing hallucinated references. This is not acceptable! Shows complete disdain for understand prev work, and suggests rest of paper may be fabricated. Ok to use LLM to suggest related work, but read (or at least skim) them!

Watched recording of ACL panel on generalisability (recommended to me). I share concerns about "LLM popcorn", but my biggest concern about NLP is lack of research diversity. Everyone does LLM, few people do impact or qual eval, little interest in genuine collab with other fields

New blog: I hate pay-to-publish The academic world has changed since I got my PhD in 1990. One of the worst changes is that researchers now often pay thousands of pounds to publish their work. Unfair to researchers with limited funding, and bad for science. ehudreiter.com/2025/08/19/i...

Very interesting meta-analysis of human-AI collab. Shows more effective in content creation (eg report writing) than in decision making, which does not surprise me When combinations of humans and AI are useful: A systematic review and meta-analysis www.nature.com/articles/s41...

Excited to announce the first-ever Workshop for Young Researchers in Natural Language Generation (YNLG), supported by @siggen.bsky.social, taking place on October 29, 2025 in Hanoi, Vietnam, co-located with INLG 2025. Call for Submissions is out now! ynlg-workshop.github.io

New blog: More on evaluating impact I got great feedback from recent paper and talk on eval impact, and summarise some of the suggested papers (including more examples of impact eval) and insightful comments (eg, about eval “ecosystem”) I received. ehudreiter.com/2025/08/05/m...

I'll be at ACL next week (Tue-Thur, not Sun/Mon). Look forward to meeting old friends and new people who want to connect! Ill also be giving an invited talk on impact evaluation at the GEM workshop on Thur 31 July

Really happy that this survey of NLP in cancer care, from my student Mengxuan Sun , has finally appeared (its been a saga). One key but depressing finding is that evaluation quality is uniformly dreadful by medical standards; NLP researchers just dont seem to care... doi.org/10.1016/j.ar...

Motivated by recent discussion with my group: Ignore subjective statements such as "I find LLMs to be incredibly useful for XX", especially when made by people (such as AI companies or gurus) who have strong biases/incentives/COI .

Nice example of using RCT to measure real-world impact of LLMs (and discovering that it is disappointing)

Ive written a "Last Word" opinion piece for CL about evaluating real-world impact. It * looks at how impact can be evaluated * shows via a structured survey that perhaps 0.1% of ACL Anth papers measure real-world impact * discusses why this is the case arxiv.org/abs/2507.05973

Looked at Google Scholar, nice to see that my h-index has reached 60

If a woman is considering IVF and uses an AI model to predict liklihood of success (having a baby), what explanations would help her trust the model? Info about what information the model considers is more important than info about how model works ehudreiter.com/2025/06/25/p...

Congratulations to my student Adarsa Sivaprasad for winning a best PhD student poster award at Healtac , for her work on "A conversational agent to address patient needs for out-of-distribution explanations"!

Fascinating keynote by Alison ONeil at Healtac. One key point is that in her space, omissions are bigger problem than hallucination since harder for readers to detect. Not surprisingly omissions more common for rare phenomenon

Reviews are variable. I see many complaints about poor reviews, but some reviews are good. I was very happy with recent set of reviews for a journal submission: positive, didnt wait long, constructive, made me think, will definitely improve my paper. My thanks to the reviewers!

New blog: The Aberdeen NLP Research Group Learn about our NLP research group! ehudreiter.com/2025/06/05/t...

Interested? Go to 2025.inlgmeeting.org #NLP #LLMs #AI #ML

📣 A Third Call for Papers: Come on in! NLPerspectives Workshop at EMNLP 2025! Submission deadline: June 27, 2025. Join us in Suzhou, China (Nov 5-9) or online to learn and chat about the latest on approaches in the context of perspectives in NLP research. 📝 #NLPerspectives #EMNLP2025

Its been 6 months since my NLG book was released. I summarise what I think are its key messages, for rule-based NLG, ML and neural NLG, requirements, evaluation, safety/testing/maintainability, and applications. ehudreiter.com/2025/05/14/k...

Someone just asked me how my blogs are licensed, they wnat to reuse them. If anyone else is interested, its a CC BY 4.0 license. (ie, you need to acknowledge source if you reuse them).

Marking UG/MSc projects in applied AI. Cant discuss individual cases, but some students spend most of their time understand domain, user requirements, cleaning data; and others spend almost all their time building and testing models. First group is closer to my perspective!

Really interesting discussion with local GPs about one of our "AI for patients" projects. Highlighted importance of specific use case, and how well this fit into NHS workflows and processes.

New blog: Even good leaderboards may not be useful, because they are gamed Most LLM benchmarks and leaderboards are garbage. Even the “good” benchmarks (eg SWEBench and Chatbot Arena) are compromised because they are being gamed by the big LLM vendors. ehudreiter.com/2025/05/05/g...

Opinions on real-world impact of LLMs range from "they're everywhere" to "they're useless". LLMs are most widely used for (A) software development and (B) writing assistance; in both cases they build on and (greatly) expand existing capabilities, so fit into existing workflows.

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

Looks like LLM vendors are aggressively gaming leaderboards, not surprising considering the large amounts of money at stake... But still seems to be little interest in moving to more real-world evaluation, I guess leaderboards are too convenient... arxiv.org/abs/2504.20879 x.com/Alex_Cuadron...

Playing around with GPT 4.5 for commmunicating medical info to patients, seems to work pretty well (other people have also said this). Shame it seems to be on the way out...

I dont usually tweet about politics, but @economist.com opinion piece "The thing about Europe: it’s the actual land of the free now" really made an impresson on me (its behind a paywall) www.economist.com/europe/2025/...

Kees van Deemter has written about his life in Artificial Intelligence - he's been working on AI even longer than me. arxiv.org/abs/2504.04142