New blog: My Eureka moments in research The most exciting moments of my career were discovering something new and exciting about NLG, etc. I describe a few of my “Eureka” moments. They are what I remember best, much more than acceptance of papers. ehudreiter.com/2026/01/30/m...

Realised that a lot of the URLs for older papers do not work in my publications page (ehudreiter.com/publications/) . For journal papers, I am replacing dead links with DOIs, and will use only DOIs in future!

New blog: Lets use AI to help people manage illness I am excited by the idea of using AI to help people manage ilness and health conditions. This isnt very sexy, but I think there is real potential to improve health outcomes and quality of life. ehudreiter.com/2026/01/19/l...

Other CS academics I know have done very different things in retirement: remained active in academia as emeritus, joined a startup, charitable work, moved to remote spot in Scot Highlands, write novels, etc. We did similar things as academics (research and teaching), but very diff in retirement!

AI hallucination is in the UK political news. Israeli fans were banned from a football match, and this ban was based on a report which included hallucinated material made up by MS Copilot www.theguardian.com/uk-news/2026...

Health experts: Your synthetic text "AI" overviews are misleading, for example see this about liver function tests. Google: Okay, we'll block "AI" overviews on that query. The product is fundamentally flawed and cannot be "fixed" by patching query by query. A short 🧵>>

Nice chat with some of my soon-to-submit PhD students. They all know how to conduct and write up research, have lots of ideas for future work, and have developed networks of collaborators. So they are ready to "leave the nest", which is good feeling for me as supervisor

New blog (personal): Retirement Plans: Travel and some academics I hope to retire soon, and many people are asking about my plans. Basically I want to do lots of travel, say involved in academia, and perhaps do some writing. ehudreiter.com/2026/01/06/r...

One nice thing about 2025 was that the two publications I was proudest of were single-author! Also many good papers with my students, but I get a special buzz from single-author papers

New blog: Do a sanity check on your experiments Researchers should do a “sanity” check on experiments. That is, manually inspect some (A) test/train data, (B) model/system output, and (C) evaluation output, looking for anything that seems strange. ehudreiter.com/2025/12/22/d...

One of main goals for 2025-26 is to get 6 PhD students to submit before I retire in summer 2026. So very happy that Nikolay Babakov has submitted and passed his viva, and Iniakpokeikiye Thompson has submitted. Getting there...

Colleague has discovered many bugs (eg incorrect annotations) in a respected 8-year old dataset he is using. Nobody warned him, and hard for him to warn others. Maybe most people just dont care if dataset is deeply flawed, as long as they can compute numbers and beat SOTA...

Making good LLM benchmark is hard. Avoid data contamination, reward hacking, saturation; ensure construct validity; rigorously test and validate, etc. Unfortunately, community places little value on above. Want to beat SOTA or competitors, dont care if BM used mean anything...

New blog: Do LLMs cheat on benchmarks LLMs often “cheat” on benchmarks via data contamination and reward hacking. This problem is getting worse, perhaps because of perverse incentives. Need to move beyond benchmarks and start measuring real-world impact. ehudreiter.com/2025/12/08/d...

Interesting chat about hallucination in patient information dialogues. When we ask domain experts to check statements such as "X increases liklihood of Y", response is often "depends on context" or "we dont know, need more experiments". Does this make statement a hallucination?

New blog: Hard to Change Poor Research Culture Research culture is very important but also very hard to change. I suspect this is one reason why it is so difficult to get people to do more rigorous and meaningful experiments. ehudreiter.com/2025/11/24/h...

Aberdeen CS is hiring a new lecturer for its "Joint Institute" with South China Normal University. Basically you would be based and do research in Aberdeen, but would be expected to go to China a few times a year and teach at SCNU. Closing 28 Nov www.abdn.ac.uk/jobs/vacanci...

I'm disturbing reports about chatbots encouraging children to kill themselves. such as www.bbc.co.uk/news/article... . Shame that the AI Safety community in general, and the @AISecurityInst in particular, seem to have little interest in this, very disappointing...

New blog: Understanding what users want from NLG When building an NLG system, it really helps to understand what users want; this came up several times at the recent INLG conference. I discuss some of our work in this space, and give a few suggestions. ehudreiter.com/2025/11/06/u...

I'm trying to understand OpenAI's healthbench. "HealthBench: Evaluating Large Language Models Towards Improved Human Health" doesnt say much about the BM(eg, very few examples). Are there other papers? I dont care how well model X performs, I want to judge if I can trust the BM

Just back from INLG. Nice event as always, but I am concerned that it is losing its uniqueness. Maybe for 2026 Ill suggest some special tracks which are interesting to INLG community but not ARR types (eg, user requirements/eval, non-LLM techniques).

New blog: Most common uses of AI in Healthcare Data on usage of AI in healthcare suggests that most common uses in 2025 are probably (A) giving personalised health information to patients and (B) helping clinicians write documents. ehudreiter.com/2025/10/21/m...

One of my main goals for 2025-26 is to help my 6 senior PhD students submit their PhDs before I retire. Glad to say that Nicolay Babakov has now done so, with viva scheduled for Dec. Other five students seem to be on track, which is encouraging.

Somewhat frustrated yesterday to once again read ACL paper which did all sorts of complex things (including the usual results tables showing best approach) on garbage data. With minimal ack of this in limitations. Most fundamental rule of CS is Garbage In, Garbage Out

New blog: Good diagrams for research papers Ive seen a number of diagrams recently which are too complicated and difficult to understand. I explain some of the problems I see and give advice. ehudreiter.com/2025/10/08/g...

Really interesting paper on real-world evaluation in IR. I should learn more about eval in IR, its not something Ive ever properly looked at dl.acm.org/doi/10.1145/...

Several people have asked me recently if I will still be able to contribute to research projects after I retire in summer 2026. Absolutely! I will have emeritus statius, and am very hapy to remain involved in research projects at Aberdeen amd elsewhere.

Aberdeen CS is hiring! We are especially interested in hiring new faculty in NLP. Closing date is 8 Oct. For more info, see below (or contact me) www.abdn.ac.uk/jobs/vacanci...

New blog: Reflections on blogging I am often asked about my experience blogging, sometimes by people who are considering writing their own blog. In this “meta” blog, I summarise my thoughts and experiences about my blog. ehudreiter.com/2025/09/23/r...

Aberdeen CS will probably be looking for a new lecturer in NLP. Formal advert is not out yet, but feel free to contact me informally if interested.

The registration page for #INLG2025 is now live! Join us in Vietnam at the Oct 29 - Nov 2 for the best conference on #NaturalLanguageGeneration 2025.inlgmeeting.org/registration... Curious to see what will be presented? Check out this list of accepted papers! 2025.inlgmeeting.org/accepted-pap...

New blog: Defining hallucination is not straightforward Many researchers assume that hallucination is a binary feature; either something is a hallucination or it is not. This is too simplistic. I describe some of the issues I have seen below. ehudreiter.com/2025/09/10/d...

At ACL, I engaged with 50 papers (went to oral, talked to poster person). Decided (looked at paper sometimes), that 3 of these robust, interesting, relevant to me; 2 of these 3 won awards. Hum, maybe in future I should focus on 40 award papers, ignore the other 3000?

Excited by recent positive evaluations of NLG apps developed by my students to encourage safer driving in UK and Nigeria. We see stat sig reductions in unsafe driving incidents in both countries. ehudreiter.com/2025/09/03/e...

Last week I had to deal with two cases of papers containing hallucinated references. This is not acceptable! Shows complete disdain for understand prev work, and suggests rest of paper may be fabricated. Ok to use LLM to suggest related work, but read (or at least skim) them!

Watched recording of ACL panel on generalisability (recommended to me). I share concerns about "LLM popcorn", but my biggest concern about NLP is lack of research diversity. Everyone does LLM, few people do impact or qual eval, little interest in genuine collab with other fields

New blog: I hate pay-to-publish The academic world has changed since I got my PhD in 1990. One of the worst changes is that researchers now often pay thousands of pounds to publish their work. Unfair to researchers with limited funding, and bad for science. ehudreiter.com/2025/08/19/i...

Very interesting meta-analysis of human-AI collab. Shows more effective in content creation (eg report writing) than in decision making, which does not surprise me When combinations of humans and AI are useful: A systematic review and meta-analysis www.nature.com/articles/s41...

Excited to announce the first-ever Workshop for Young Researchers in Natural Language Generation (YNLG), supported by @siggen.bsky.social, taking place on October 29, 2025 in Hanoi, Vietnam, co-located with INLG 2025. Call for Submissions is out now! ynlg-workshop.github.io

New blog: More on evaluating impact I got great feedback from recent paper and talk on eval impact, and summarise some of the suggested papers (including more examples of impact eval) and insightful comments (eg, about eval “ecosystem”) I received. ehudreiter.com/2025/08/05/m...

I'll be at ACL next week (Tue-Thur, not Sun/Mon). Look forward to meeting old friends and new people who want to connect! Ill also be giving an invited talk on impact evaluation at the GEM workshop on Thur 31 July

Really happy that this survey of NLP in cancer care, from my student Mengxuan Sun , has finally appeared (its been a saga). One key but depressing finding is that evaluation quality is uniformly dreadful by medical standards; NLP researchers just dont seem to care... doi.org/10.1016/j.ar...

Motivated by recent discussion with my group: Ignore subjective statements such as "I find LLMs to be incredibly useful for XX", especially when made by people (such as AI companies or gurus) who have strong biases/incentives/COI .

Nice example of using RCT to measure real-world impact of LLMs (and discovering that it is disappointing)

Ive written a "Last Word" opinion piece for CL about evaluating real-world impact. It * looks at how impact can be evaluated * shows via a structured survey that perhaps 0.1% of ACL Anth papers measure real-world impact * discusses why this is the case arxiv.org/abs/2507.05973

Looked at Google Scholar, nice to see that my h-index has reached 60

If a woman is considering IVF and uses an AI model to predict liklihood of success (having a baby), what explanations would help her trust the model? Info about what information the model considers is more important than info about how model works ehudreiter.com/2025/06/25/p...

Congratulations to my student Adarsa Sivaprasad for winning a best PhD student poster award at Healtac , for her work on "A conversational agent to address patient needs for out-of-distribution explanations"!

Fascinating keynote by Alison ONeil at Healtac. One key point is that in her space, omissions are bigger problem than hallucination since harder for readers to detect. Not surprisingly omissions more common for rare phenomenon

Reviews are variable. I see many complaints about poor reviews, but some reviews are good. I was very happy with recent set of reviews for a journal submission: positive, didnt wait long, constructive, made me think, will definitely improve my paper. My thanks to the reviewers!