Several people have asked me recently if I will still be able to contribute to research projects after I retire in summer 2026. Absolutely! I will have emeritus statius, and am very hapy to remain involved in research projects at Aberdeen amd elsewhere.

Aberdeen CS is hiring! We are especially interested in hiring new faculty in NLP. Closing date is 8 Oct. For more info, see below (or contact me) www.abdn.ac.uk/jobs/vacanci...

New blog: Reflections on blogging I am often asked about my experience blogging, sometimes by people who are considering writing their own blog. In this “meta” blog, I summarise my thoughts and experiences about my blog. ehudreiter.com/2025/09/23/r...

Aberdeen CS will probably be looking for a new lecturer in NLP. Formal advert is not out yet, but feel free to contact me informally if interested.

The registration page for #INLG2025 is now live! Join us in Vietnam at the Oct 29 - Nov 2 for the best conference on #NaturalLanguageGeneration 2025.inlgmeeting.org/registration... Curious to see what will be presented? Check out this list of accepted papers! 2025.inlgmeeting.org/accepted-pap...

New blog: Defining hallucination is not straightforward Many researchers assume that hallucination is a binary feature; either something is a hallucination or it is not. This is too simplistic. I describe some of the issues I have seen below. ehudreiter.com/2025/09/10/d...

At ACL, I engaged with 50 papers (went to oral, talked to poster person). Decided (looked at paper sometimes), that 3 of these robust, interesting, relevant to me; 2 of these 3 won awards. Hum, maybe in future I should focus on 40 award papers, ignore the other 3000?

Excited by recent positive evaluations of NLG apps developed by my students to encourage safer driving in UK and Nigeria. We see stat sig reductions in unsafe driving incidents in both countries. ehudreiter.com/2025/09/03/e...

Last week I had to deal with two cases of papers containing hallucinated references. This is not acceptable! Shows complete disdain for understand prev work, and suggests rest of paper may be fabricated. Ok to use LLM to suggest related work, but read (or at least skim) them!

Watched recording of ACL panel on generalisability (recommended to me). I share concerns about "LLM popcorn", but my biggest concern about NLP is lack of research diversity. Everyone does LLM, few people do impact or qual eval, little interest in genuine collab with other fields

New blog: I hate pay-to-publish The academic world has changed since I got my PhD in 1990. One of the worst changes is that researchers now often pay thousands of pounds to publish their work. Unfair to researchers with limited funding, and bad for science. ehudreiter.com/2025/08/19/i...

Very interesting meta-analysis of human-AI collab. Shows more effective in content creation (eg report writing) than in decision making, which does not surprise me When combinations of humans and AI are useful: A systematic review and meta-analysis www.nature.com/articles/s41...

Excited to announce the first-ever Workshop for Young Researchers in Natural Language Generation (YNLG), supported by @siggen.bsky.social, taking place on October 29, 2025 in Hanoi, Vietnam, co-located with INLG 2025. Call for Submissions is out now! ynlg-workshop.github.io

New blog: More on evaluating impact I got great feedback from recent paper and talk on eval impact, and summarise some of the suggested papers (including more examples of impact eval) and insightful comments (eg, about eval “ecosystem”) I received. ehudreiter.com/2025/08/05/m...

I'll be at ACL next week (Tue-Thur, not Sun/Mon). Look forward to meeting old friends and new people who want to connect! Ill also be giving an invited talk on impact evaluation at the GEM workshop on Thur 31 July

Really happy that this survey of NLP in cancer care, from my student Mengxuan Sun , has finally appeared (its been a saga). One key but depressing finding is that evaluation quality is uniformly dreadful by medical standards; NLP researchers just dont seem to care... doi.org/10.1016/j.ar...

Motivated by recent discussion with my group: Ignore subjective statements such as "I find LLMs to be incredibly useful for XX", especially when made by people (such as AI companies or gurus) who have strong biases/incentives/COI .

Nice example of using RCT to measure real-world impact of LLMs (and discovering that it is disappointing)

Ive written a "Last Word" opinion piece for CL about evaluating real-world impact. It * looks at how impact can be evaluated * shows via a structured survey that perhaps 0.1% of ACL Anth papers measure real-world impact * discusses why this is the case arxiv.org/abs/2507.05973

Looked at Google Scholar, nice to see that my h-index has reached 60

If a woman is considering IVF and uses an AI model to predict liklihood of success (having a baby), what explanations would help her trust the model? Info about what information the model considers is more important than info about how model works ehudreiter.com/2025/06/25/p...

Congratulations to my student Adarsa Sivaprasad for winning a best PhD student poster award at Healtac , for her work on "A conversational agent to address patient needs for out-of-distribution explanations"!

Fascinating keynote by Alison ONeil at Healtac. One key point is that in her space, omissions are bigger problem than hallucination since harder for readers to detect. Not surprisingly omissions more common for rare phenomenon

Reviews are variable. I see many complaints about poor reviews, but some reviews are good. I was very happy with recent set of reviews for a journal submission: positive, didnt wait long, constructive, made me think, will definitely improve my paper. My thanks to the reviewers!

New blog: The Aberdeen NLP Research Group Learn about our NLP research group! ehudreiter.com/2025/06/05/t...

Interested? Go to 2025.inlgmeeting.org #NLP #LLMs #AI #ML

📣 A Third Call for Papers: Come on in! NLPerspectives Workshop at EMNLP 2025! Submission deadline: June 27, 2025. Join us in Suzhou, China (Nov 5-9) or online to learn and chat about the latest on approaches in the context of perspectives in NLP research. 📝 #NLPerspectives #EMNLP2025

Its been 6 months since my NLG book was released. I summarise what I think are its key messages, for rule-based NLG, ML and neural NLG, requirements, evaluation, safety/testing/maintainability, and applications. ehudreiter.com/2025/05/14/k...

Someone just asked me how my blogs are licensed, they wnat to reuse them. If anyone else is interested, its a CC BY 4.0 license. (ie, you need to acknowledge source if you reuse them).

Marking UG/MSc projects in applied AI. Cant discuss individual cases, but some students spend most of their time understand domain, user requirements, cleaning data; and others spend almost all their time building and testing models. First group is closer to my perspective!

Really interesting discussion with local GPs about one of our "AI for patients" projects. Highlighted importance of specific use case, and how well this fit into NHS workflows and processes.

New blog: Even good leaderboards may not be useful, because they are gamed Most LLM benchmarks and leaderboards are garbage. Even the “good” benchmarks (eg SWEBench and Chatbot Arena) are compromised because they are being gamed by the big LLM vendors. ehudreiter.com/2025/05/05/g...

Opinions on real-world impact of LLMs range from "they're everywhere" to "they're useless". LLMs are most widely used for (A) software development and (B) writing assistance; in both cases they build on and (greatly) expand existing capabilities, so fit into existing workflows.

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

Looks like LLM vendors are aggressively gaming leaderboards, not surprising considering the large amounts of money at stake... But still seems to be little interest in moving to more real-world evaluation, I guess leaderboards are too convenient... arxiv.org/abs/2504.20879 x.com/Alex_Cuadron...

Playing around with GPT 4.5 for commmunicating medical info to patients, seems to work pretty well (other people have also said this). Shame it seems to be on the way out...

I dont usually tweet about politics, but @economist.com opinion piece "The thing about Europe: it’s the actual land of the free now" really made an impresson on me (its behind a paywall) www.economist.com/europe/2025/...

Kees van Deemter has written about his life in Artificial Intelligence - he's been working on AI even longer than me. arxiv.org/abs/2504.04142

Techniques for evaluating real-world impact include /B test, before/after eval, clinical trial, observational study. Examples of all of these in my blog ehudreiter.com/2025/04/08/e...

See jamanetwork.com/journals/jam... and www.pewresearch.org/science/2023...

New blog: Examples of evaluating real-world impact I describe several papers which measure real-world impact of NLP systems, using different methodologies. I hope these examples inspire and encourage more people to consider evaluating real-world impact! ehudreiter.com/2025/04/08/e...

Cursing fact that NLP abstracts are non-standardised and unstructured. Would be be much easier to scan literature if abstracts had standard struct (like medicine), eg problem, method, evaluation, discussion.

Colleague from humanities is super-excited about copilot as coding aid. Realised he's probably an ideal user - understands programming concepts, struggles with practicalities, is doing fairly simple things which many others in digital hum have done.

Come join us in Hanoi and discuss then latest advances in Natural Language Generation ✨ bsky.app/profile/sigg...

Many thanks to @beatricealex.bsky.social for inviting me to speak to the Edinburgh Clinical NLP group yesterday, about our work using AI and NLG to help patients. Really interesting questions and discussion.

Discussion today about how we manage patient expectations/concerns about use of AI in one of our health chatbots. Given all hype around AI, how stop people from assuming "AI magic" means bot can do anything, or "AI magic" means bot cant be trusted?

I gave a talk yesterday to some radiologists about AI. Not surprisingly, some preferred limited use of AI to make their life easier. Concern was also expressed about ability of NHS to deploy AI, I saw related article in Guardian today www.theguardian.com/technology/2...

New blog: Benchmarks distract us from what matters LLM benchmarks lead us to optimise LLMs for capabilities that are easy to BM even if of little interest to users; and to ignore capabilities) which are important to real users but hard to assess with BM ehudreiter.com/2025/03/26/b...

Journalist asked me how *I* would like LLM evaluated. Hum... ideally an indp org publish LLM evaluations for diff types of users (software devs, writers, etc). In each area, collection of (regularly updated) benchmarks and some human eval. Probably isnt going to happen...