Let's talk about eval (automatic or human) and multilinguality at #EMNLP in Suzhou! 🇨🇳 - Efficient evaluation (Nov 5, 16:30, poster session 3) - MT difficulty (Nov 7, 12:30, findings 3) - COMET-poly (Nov 8, 11:00, WMT) (DM to meet 🌿 )

Grateful to receive the Google PhD Fellowship in NLP! 🙂 I am not secretive about having applied to 4 similar fellowships during my PhD before and not succeeding. Still, refining my research statement (part of the application) helped me tremendously in finding out the... inf.ethz.ch/news-and-eve...

📢 Announcing the First Workshop on Multilingual and Multicultural Evaluation (MME) at #EACL2026 🇲🇦 MME focuses on resources, metrics & methodologies for evaluating multilingual systems! multilingual-multicultural-evaluation.github.io 📅 Workshop Mar 24–29, 2026 🗓️ Submit by Dec 19, 2025

My two biggest take-aways are: - Standard testsets are too easy (Figure 1). - We can make testsets that are not easy (Figure 2). 😎

We saw increased momentum in participation growth this year: 36 unique teams competing to improve the performance of MT. Furthermore, we added collected outputs of 24 popular LLMs and online systems. Reaching 50 evaluated systems in our annual benchmark.

The 2025 MT Evaluation shared task brings together the strengths of the previous Metrics and Quality Estimation tasks under a single, unified evaluation framework. The following tasks are now open for participants (deadline July 31st but participation has never been easier 🙂 ):

You have a budget to human-evaluate 100 inputs to your models, but your dataset is 10,000 inputs. Do not just pick 100 randomly!🙅 We can do better. "How to Select Datapoints for Efficient Human Evaluation of NLG Models?" shows how.🕵️ (random is still a devilishly good baseline)

TIL that since python3.4 there's default `statistics` module with things like mean, mode, quantiles, variance, covariance, correlations, zscore, and more!. No more needless numpy imports!

Past iterations of the Terminology Shared Task don't come anywhere near the data quality and evaluation scrutiny of this one. In the era of LLM-as-MTs, participation has never been easier!

Thank you for your response. I will keep my score.

For the longest time I've been using Google Translate as a gateway to explain machine translation concepts to people as it's a tool that everyone knows. Now I get to contribute over the summer. 🌞 If you're near Mountain View, let's talk evaluation. 📏

arxiv submission process got an update! (still requires a manual bbl)

XCOMETs underperform because they do not match translators' subjective error annotation propensity. Using the granular p(error) value from XCOMET significantly boost their performance when calibration is possible → desirable for a fair evaluation 6/

Key takeaways for WQE evals: 1️⃣ Unsup. WQE shows promise (esp. uncertainty-based ones), interp approaches under-explored for MT 2️⃣ Calibration sets can help to ensure fair evaluations. 3️⃣ Use multiple annotators for robust rakings. More info ➡️ arxiv.org/abs/2505.23183 8/8

📢 New paper: Can unsupervised metrics extracted from MT models detect their translation errors reliably? Do annotators even *agree* on what constitutes an error? 🧐 We compare uncertainty- and interp-based WQE metrics across 12 directions, with some surprising findings! 🧵 1/

incredible monetization opportunity (this is a joke)

Ever looked down from a hot air balloon and despaired at how expensive it is to run thorough human evaluation of machine translation? Fret no more and come tomorrow at 11:00 to Hall 3 #NAACL2025.

Being in a hot air balloon in Albuquerque really makes one ponder *how to efficiently pick the best translation candidate without running expensive evaluation metrics on all of them.* See you tomorrow at 9:00 in Hall 3 #NAACL2025.

Multilinguality is happening at #NAACL2025 @crystinaz.bsky.social @oxxoskeets.bsky.social @dayeonki.bsky.social @onadegibert.bsky.social

Let's chat at #NAACL2025 about evaluation et al! ⚗️

(Automatic) span annotations are the future of evaluation and diagnosis in NLP! 🖊️

How do LLMs compare to human crowdworkers in annotating text spans? 🧑🤖 And how can span annotation help us with evaluating texts? Find out in our new paper: llm-span-annotators.github.io Arxiv: arxiv.org/abs/2504.08697

In the true sense of the word I am humbled to have been rejected by a few fellowships this year.

Overall 3.0 (borderline reject) but with 43% tariff adjustments it's 4.5 (borderline award).

Panopticon, but instead of prison cells, it's a stack of Overleaf tabs. You can’t watch them all at once, but at any moment, you *could* be watching any of them. And they know it.

Extending the quality and scale of WMT evaluation beyond machine translation! The data and insights out of this will be a goldmine. 🙂

Bad times to be doing Direct Assessment in Pakistan these days.

Following the MT Marathon, we're hosting a hackathon in Prague. Researchers and students from five institutions (+1 online) are working together to assess how robust #LLMs are to grammar errors in machine translation and related tasks. Thanks to EAMT for their support.

As my parents named me: v-i-l-backslash-singleapostrophe-e-m.

I'd like to follow more (NLP/ML) people on HuggingFace that I follow here. Please send me your handles! 🤗

The roots of our publishing craze can be traced to 1970s IBM posters

So many new changes coming to the WMT general machine translation shared task. Broader, sleeker, better!

Error Span Annotation for machine translation. Very high-level, very quick. See you at NAACL! youtu.be/995GDufPBw8 arxiv.org/pdf/2406.12419

Error Span Annotation for machine translation. Very high-level, very quick. See you at NAACL! youtu.be/995GDufPBw8 arxiv.org/pdf/2406.12419

It was today that I found out that the "bandit" in "multi-armed bandit" is not an outlaw *person* but a nickname for a casino machine. All reinforcement leraning courses somehow assumed this knowledge.

February is here, the honest month. www.youtube.com/watch?v=sjco...

Couldn't resist the urge to make the latest paper into a little guessing game. zouharvi.github.io/subset2evalu...

Vil\'em Zouhar, Peng Cui, Mrinmaya Sachan How to Select Datapoints for Efficient Human Evaluation of NLG Models? https://arxiv.org/abs/2501.18251

Updated anthology.min.bib (42M) so that it still fits within Overleaf upload limit. curl raw.githubusercontent.com/zouharvi/ant... > anthology.min.bib

I think NLP/ML conferences fake their acceptance rates. Allegedly it's 20-30% but according to my independent research from what I see posted here, almost everyone got paper accepted (close to 100%). ✈️🔫

The paper comes with mysterious fig 1. 👁️

The irony of this sentence is that it should be "recall" and "precision" instead of "precision" and "clarity".

What is the "alphabet"?

Quality estimation-enhanced user-facing MT has been my goal already since my bachelor's thesis. The relationship is complicated, but all the more fun to analyze. I've never worked with anyone as organized and prolific as Gabriele. 🙂

Trying to collect all the MT people here. I probably missed many. Ping me! bsky.app/starter-pack...

Trying to bring ML/NLP/etal people from ETH Zürich together. Ping me to add you. 🙂 bsky.app/starter-pack...

"ce n'est pas une poubelle" (EMNLP 2024)

Fun was had with this project. Marco is a great collaborator!

For the latest WMT we develop a new human annotation protocol for machine translation that has the annotators first highlight error spans with severity (minor/major) and only then assign a final numeric score.

Our presentation is starting soon! Come by for some tokenization and some Japanese snacks!