New blog: Comparing performance of LLMs is not very interesting Comparing different LLMs in research papers may not mean much, because LLMs will be out of date when paper published. Finding behaviour which is shared by LLMs is interesting and worthwhile. ehudreiter.com/2026/03/24/c...

A student is writing up an experiment which includes a comparison of how well two LLMs do at a task. The community expects such comparisons, but why are they useful, since the LLMs being compared will be obsolete by the time people read this?

New blog: Please follow the rules for ARR/ACL papers ACL/ARR have rules and guidelines for how papers are written. Unfortunately many authors (and reviewers) ignore these, which makes their papers harder to read and less useful. Please follow the rules! ehudreiter.com/2026/03/16/p...

Looked at @METR_Evals blogs, some nice material on lessons, challenges, etc in using RCTs to evaluate LLMs. We have had RCTs in medicine for a long time and have built up a lot of knowledge about this, but RCTs in AI are very new, and we need to figure out how to do them well

Aberdeen CS is hiring a new lecturer, who will be expected to do some teaching in our new Mumbai campus. Closing date is 30 March www.abdn.ac.uk/jobs/vacanci...

In future, I will not read appendices when reviewing ACL papers (ARR guidelines say this is not expected). Appendices used to be a few pages of details for replication (eg hyperparam), but now its common to have 10-20 pages (Ive seen 50 pages). Dont expect me to read this!

I start my last-ever course today, MSc course on Natural Language Generation. My last lecture (on NLG evaluation) will be on 20 April. Hard to believe...

New blog: Questions from readers of my book A group who is reading my book sent me many questions, some of which we discussed in a call last week. I thought I would share the questions and my responses. ehudreiter.com/2026/03/03/q...

Great to see that my student Jawwad Baig has submitted his PhD! One of my main goals for 2025-26 is to help 6 PhD students submit before I retire. Halfway through the academic year, and three of the six have now submitted, so on track.

If you're not on the SIGGEN mailing list or in the NLG Discord server, you might not have seen that Barkavi Sundarajan has been leading a reading group about @ehudreiter.bsky.social's new book "Natural Language Generation". Join us Friday, 27 Feb, at 2pm UK time: discord.gg/hysgkK7Q?eve...

My PhD student Adarsa Sivaprasad is looking for people who have lived experience of IVF to help evaluate an AI chatbot which explains IVF outcome predictions. What is involved: 45 min online MS Teams call. Read details and sign up at: tinyurl.com/cc2aepf5

Id love to see old friends and meet new colleaugues at my retirement symposium!

New blog: Dont ignore omissions! Evaluation of LLMs focuses on accuracy and hallucination. Completeness and omission also important; does the text include all the key information? Omissions are a huge problem in medical NLG, and in other NLG tasks as well. ehudreiter.com/2026/02/11/d...

Fascinating paper on problems using LLMs to respond to health queries. LLMs do well on standard medical benchmarks but struggle (for example) to understand symptoms presented in a confused way www.nature.com/articles/s41...

Really interesting paper on AI coding assistants, which may be getting worse becuase of lower quality training data spectrum.ieee.org/ai-coding-de...

This is the first conference which my daughter Naomi (who is doing a PhD in medieval history) has helped to organise. Good luck to her and fellow organisers!

Nice talk by my colleague Jakub Zbrzeżny (Aberdeen Divinity Dept) on using LLM to translate biblical texts into and out of local Arabic dialect (n Hebron). Basically LLMs understand dialect, but cannot produce it. Will this encourage more young people to abandon their dialect?

Very excited to have a retirement symposium on NLG evaluation! Looking forward to seeing old friends and meeting new people!

Friend asked what I am focusing on in last half-year before retirement. Largely getting 5 PhD students to submit PhDs and have vivas. Fortunately all seem to be on track!

Wrote blog recently that authors should do sanity checks on papers. Readers should as well! Recently read interesting paper, but sanity check showed claims in paper did not match data. Either paper hallucinated or authors sloppy. ehudreiter.com/2025/12/22/d...

Interesting paper about different techniques to eval performance of medical decision support. Concludes that F1 is the worst technique, shame that is so heavily used in NLP and AOI.. doi.org/10.1016/j.la...

I liked recent Economist article "How to avoid common AI pitfalls in the workplace" (paywall). Little mention of benchmarks or AGI (but does mention contrast between rising benchmark scores and limited real-world impact). Instead focuses on pragmatic issues such as workflow.

The 5th Generation, Evaluation, and Metrics (GEM) Workshop will be at #ACL2026! Call for papers is out. Topics include: 🐟 LMs as evaluators 🐠 Living benchmarks 🍣 Eval with humans and more New for 2026: Opinion & Statement Papers! Full CFP: gem-workshop.com/call-for-pap...

New blog: My Eureka moments in research The most exciting moments of my career were discovering something new and exciting about NLG, etc. I describe a few of my “Eureka” moments. They are what I remember best, much more than acceptance of papers. ehudreiter.com/2026/01/30/m...

Realised that a lot of the URLs for older papers do not work in my publications page (ehudreiter.com/publications/) . For journal papers, I am replacing dead links with DOIs, and will use only DOIs in future!

New blog: Lets use AI to help people manage illness I am excited by the idea of using AI to help people manage ilness and health conditions. This isnt very sexy, but I think there is real potential to improve health outcomes and quality of life. ehudreiter.com/2026/01/19/l...

Other CS academics I know have done very different things in retirement: remained active in academia as emeritus, joined a startup, charitable work, moved to remote spot in Scot Highlands, write novels, etc. We did similar things as academics (research and teaching), but very diff in retirement!

AI hallucination is in the UK political news. Israeli fans were banned from a football match, and this ban was based on a report which included hallucinated material made up by MS Copilot www.theguardian.com/uk-news/2026...

Health experts: Your synthetic text "AI" overviews are misleading, for example see this about liver function tests. Google: Okay, we'll block "AI" overviews on that query. The product is fundamentally flawed and cannot be "fixed" by patching query by query. A short 🧵>>

Nice chat with some of my soon-to-submit PhD students. They all know how to conduct and write up research, have lots of ideas for future work, and have developed networks of collaborators. So they are ready to "leave the nest", which is good feeling for me as supervisor

New blog (personal): Retirement Plans: Travel and some academics I hope to retire soon, and many people are asking about my plans. Basically I want to do lots of travel, say involved in academia, and perhaps do some writing. ehudreiter.com/2026/01/06/r...

One nice thing about 2025 was that the two publications I was proudest of were single-author! Also many good papers with my students, but I get a special buzz from single-author papers

New blog: Do a sanity check on your experiments Researchers should do a “sanity” check on experiments. That is, manually inspect some (A) test/train data, (B) model/system output, and (C) evaluation output, looking for anything that seems strange. ehudreiter.com/2025/12/22/d...

One of main goals for 2025-26 is to get 6 PhD students to submit before I retire in summer 2026. So very happy that Nikolay Babakov has submitted and passed his viva, and Iniakpokeikiye Thompson has submitted. Getting there...

Colleague has discovered many bugs (eg incorrect annotations) in a respected 8-year old dataset he is using. Nobody warned him, and hard for him to warn others. Maybe most people just dont care if dataset is deeply flawed, as long as they can compute numbers and beat SOTA...

Making good LLM benchmark is hard. Avoid data contamination, reward hacking, saturation; ensure construct validity; rigorously test and validate, etc. Unfortunately, community places little value on above. Want to beat SOTA or competitors, dont care if BM used mean anything...

New blog: Do LLMs cheat on benchmarks LLMs often “cheat” on benchmarks via data contamination and reward hacking. This problem is getting worse, perhaps because of perverse incentives. Need to move beyond benchmarks and start measuring real-world impact. ehudreiter.com/2025/12/08/d...

Interesting chat about hallucination in patient information dialogues. When we ask domain experts to check statements such as "X increases liklihood of Y", response is often "depends on context" or "we dont know, need more experiments". Does this make statement a hallucination?

New blog: Hard to Change Poor Research Culture Research culture is very important but also very hard to change. I suspect this is one reason why it is so difficult to get people to do more rigorous and meaningful experiments. ehudreiter.com/2025/11/24/h...

Aberdeen CS is hiring a new lecturer for its "Joint Institute" with South China Normal University. Basically you would be based and do research in Aberdeen, but would be expected to go to China a few times a year and teach at SCNU. Closing 28 Nov www.abdn.ac.uk/jobs/vacanci...

I'm disturbing reports about chatbots encouraging children to kill themselves. such as www.bbc.co.uk/news/article... . Shame that the AI Safety community in general, and the @AISecurityInst in particular, seem to have little interest in this, very disappointing...

New blog: Understanding what users want from NLG When building an NLG system, it really helps to understand what users want; this came up several times at the recent INLG conference. I discuss some of our work in this space, and give a few suggestions. ehudreiter.com/2025/11/06/u...

I'm trying to understand OpenAI's healthbench. "HealthBench: Evaluating Large Language Models Towards Improved Human Health" doesnt say much about the BM(eg, very few examples). Are there other papers? I dont care how well model X performs, I want to judge if I can trust the BM

Just back from INLG. Nice event as always, but I am concerned that it is losing its uniqueness. Maybe for 2026 Ill suggest some special tracks which are interesting to INLG community but not ARR types (eg, user requirements/eval, non-LLM techniques).

New blog: Most common uses of AI in Healthcare Data on usage of AI in healthcare suggests that most common uses in 2025 are probably (A) giving personalised health information to patients and (B) helping clinicians write documents. ehudreiter.com/2025/10/21/m...

One of my main goals for 2025-26 is to help my 6 senior PhD students submit their PhDs before I retire. Glad to say that Nicolay Babakov has now done so, with viva scheduled for Dec. Other five students seem to be on track, which is encouraging.

Somewhat frustrated yesterday to once again read ACL paper which did all sorts of complex things (including the usual results tables showing best approach) on garbage data. With minimal ack of this in limitations. Most fundamental rule of CS is Garbage In, Garbage Out

New blog: Good diagrams for research papers Ive seen a number of diagrams recently which are too complicated and difficult to understand. I explain some of the problems I see and give advice. ehudreiter.com/2025/10/08/g...

Really interesting paper on real-world evaluation in IR. I should learn more about eval in IR, its not something Ive ever properly looked at dl.acm.org/doi/10.1145/...

Several people have asked me recently if I will still be able to contribute to research projects after I retire in summer 2026. Absolutely! I will have emeritus statius, and am very hapy to remain involved in research projects at Aberdeen amd elsewhere.