Quality estimation (automated metrics) are amazing. Truly. We would like to use them everywhere. That gets compute-expensive very quickly. We also don't know when they don't know. In "Early-Exit and Instant Confidence Translation Quality Estimation" (at EACL26) we fix that.

How often is human evaluation skipped in papers/workflows just because it's too difficult to set up? Yet even small humeval can give so much more signal than automatic metrics. Introducing Pearmut, Human Evaluation of Translation Made Trivial🍐 arxiv.org/pdf/2601.02933

Have you ever wondered how speech translation gets evaluated? Sadly, most speech evaluation downgrades to text-based metrics. Let’s do better! At IWSLT 2026, we’re launching the first-ever ✨Speech Translation Metrics Shared Task ✨!

Dissatisfied with EACL paper decisions? Fret not and submit your paper with ARR reviews to Multilingual Multicultural Evaluation workshop at EACL (both archival or nonarchival) until January 5th. 🔍🙂 multilingual-multicultural-evaluation.github.io

Now onwards to making language models transparent and trustworthy for everyone! 🚀 For those curious to know more about my thesis: - Web-optimized version: gsarti.com/phd-thesis/ - PDF: research.rug.nl/en/publicati... - Steal my Quarto template: github.com/gsarti/phd-t...

Do you have work on resources, metrics & methodologies for evaluating multilingual systems? Share it at the MME workshop 🕵️ co-located at EACL. Direct submission deadline in 10 days (December 19th)! multilingual-multicultural-evaluation.github.io

Let's talk about eval (automatic or human) and multilinguality at #EMNLP in Suzhou! 🇨🇳 - Efficient evaluation (Nov 5, 16:30, poster session 3) - MT difficulty (Nov 7, 12:30, findings 3) - COMET-poly (Nov 8, 11:00, WMT) (DM to meet 🌿 )

Grateful to receive the Google PhD Fellowship in NLP! 🙂 I am not secretive about having applied to 4 similar fellowships during my PhD before and not succeeding. Still, refining my research statement (part of the application) helped me tremendously in finding out the... inf.ethz.ch/news-and-eve...

📢 Announcing the First Workshop on Multilingual and Multicultural Evaluation (MME) at #EACL2026 🇲🇦 MME focuses on resources, metrics & methodologies for evaluating multilingual systems! multilingual-multicultural-evaluation.github.io 📅 Workshop Mar 24–29, 2026 🗓️ Submit by Dec 19, 2025

My two biggest take-aways are: - Standard testsets are too easy (Figure 1). - We can make testsets that are not easy (Figure 2). 😎

We saw increased momentum in participation growth this year: 36 unique teams competing to improve the performance of MT. Furthermore, we added collected outputs of 24 popular LLMs and online systems. Reaching 50 evaluated systems in our annual benchmark.

The 2025 MT Evaluation shared task brings together the strengths of the previous Metrics and Quality Estimation tasks under a single, unified evaluation framework. The following tasks are now open for participants (deadline July 31st but participation has never been easier 🙂 ):

You have a budget to human-evaluate 100 inputs to your models, but your dataset is 10,000 inputs. Do not just pick 100 randomly!🙅 We can do better. "How to Select Datapoints for Efficient Human Evaluation of NLG Models?" shows how.🕵️ (random is still a devilishly good baseline)

TIL that since python3.4 there's default `statistics` module with things like mean, mode, quantiles, variance, covariance, correlations, zscore, and more!. No more needless numpy imports!

Past iterations of the Terminology Shared Task don't come anywhere near the data quality and evaluation scrutiny of this one. In the era of LLM-as-MTs, participation has never been easier!

Thank you for your response. I will keep my score.

For the longest time I've been using Google Translate as a gateway to explain machine translation concepts to people as it's a tool that everyone knows. Now I get to contribute over the summer. 🌞 If you're near Mountain View, let's talk evaluation. 📏

arxiv submission process got an update! (still requires a manual bbl)

XCOMETs underperform because they do not match translators' subjective error annotation propensity. Using the granular p(error) value from XCOMET significantly boost their performance when calibration is possible → desirable for a fair evaluation 6/

Key takeaways for WQE evals: 1️⃣ Unsup. WQE shows promise (esp. uncertainty-based ones), interp approaches under-explored for MT 2️⃣ Calibration sets can help to ensure fair evaluations. 3️⃣ Use multiple annotators for robust rakings. More info ➡️ arxiv.org/abs/2505.23183 8/8

📢 New paper: Can unsupervised metrics extracted from MT models detect their translation errors reliably? Do annotators even *agree* on what constitutes an error? 🧐 We compare uncertainty- and interp-based WQE metrics across 12 directions, with some surprising findings! 🧵 1/

incredible monetization opportunity (this is a joke)

Ever looked down from a hot air balloon and despaired at how expensive it is to run thorough human evaluation of machine translation? Fret no more and come tomorrow at 11:00 to Hall 3 #NAACL2025.

Being in a hot air balloon in Albuquerque really makes one ponder *how to efficiently pick the best translation candidate without running expensive evaluation metrics on all of them.* See you tomorrow at 9:00 in Hall 3 #NAACL2025.

Multilinguality is happening at #NAACL2025 @crystinaz.bsky.social @oxxoskeets.bsky.social @dayeonki.bsky.social @onadegibert.bsky.social

Let's chat at #NAACL2025 about evaluation et al! ⚗️

(Automatic) span annotations are the future of evaluation and diagnosis in NLP! 🖊️

How do LLMs compare to human crowdworkers in annotating text spans? 🧑🤖 And how can span annotation help us with evaluating texts? Find out in our new paper: llm-span-annotators.github.io Arxiv: arxiv.org/abs/2504.08697

In the true sense of the word I am humbled to have been rejected by a few fellowships this year.

Overall 3.0 (borderline reject) but with 43% tariff adjustments it's 4.5 (borderline award).

Panopticon, but instead of prison cells, it's a stack of Overleaf tabs. You can’t watch them all at once, but at any moment, you *could* be watching any of them. And they know it.

Extending the quality and scale of WMT evaluation beyond machine translation! The data and insights out of this will be a goldmine. 🙂

Bad times to be doing Direct Assessment in Pakistan these days.

Following the MT Marathon, we're hosting a hackathon in Prague. Researchers and students from five institutions (+1 online) are working together to assess how robust #LLMs are to grammar errors in machine translation and related tasks. Thanks to EAMT for their support.

As my parents named me: v-i-l-backslash-singleapostrophe-e-m.

I'd like to follow more (NLP/ML) people on HuggingFace that I follow here. Please send me your handles! 🤗

The roots of our publishing craze can be traced to 1970s IBM posters

So many new changes coming to the WMT general machine translation shared task. Broader, sleeker, better!

Error Span Annotation for machine translation. Very high-level, very quick. See you at NAACL! youtu.be/995GDufPBw8 arxiv.org/pdf/2406.12419

Error Span Annotation for machine translation. Very high-level, very quick. See you at NAACL! youtu.be/995GDufPBw8 arxiv.org/pdf/2406.12419

It was today that I found out that the "bandit" in "multi-armed bandit" is not an outlaw *person* but a nickname for a casino machine. All reinforcement leraning courses somehow assumed this knowledge.

February is here, the honest month. www.youtube.com/watch?v=sjco...

Couldn't resist the urge to make the latest paper into a little guessing game. zouharvi.github.io/subset2evalu...

Vil\'em Zouhar, Peng Cui, Mrinmaya Sachan How to Select Datapoints for Efficient Human Evaluation of NLG Models? https://arxiv.org/abs/2501.18251

Updated anthology.min.bib (42M) so that it still fits within Overleaf upload limit. curl raw.githubusercontent.com/zouharvi/ant... > anthology.min.bib

I think NLP/ML conferences fake their acceptance rates. Allegedly it's 20-30% but according to my independent research from what I see posted here, almost everyone got paper accepted (close to 100%). ✈️🔫

The paper comes with mysterious fig 1. 👁️

The irony of this sentence is that it should be "recall" and "precision" instead of "precision" and "clarity".

What is the "alphabet"?

Quality estimation-enhanced user-facing MT has been my goal already since my bachelor's thesis. The relationship is complicated, but all the more fun to analyze. I've never worked with anyone as organized and prolific as Gabriele. 🙂