Not many people know this but this button was not invented to complain that reviewers didn't give you a higher score.

There are many things machine translation still can't do. Help us steer the next direction by contributing hard-to-translate inputs (and be on a cool paper).

The best rebuttal I read is just 4 sentences in 3 bullet points clarifying my listed weaknesses.

All my homies left for ACL/ICML and left me home managing a project *for which we're looking for coauthors* 🥸. Join us (also talk to @sethjsa.bsky.social @onadegibert.bsky.social @niyatibafna.bsky.social @patuchen.bsky.social @michellewastl.bsky.social @ayukh.bsky.social)

The ironic twist is that ARR *did* save us by solving our obsession with publishing papers. I know at least one person who deferred from publishing her paper because of all the annoyance&hurdles in reviewing/ACing.

Be the reviewer you want (or your AC wants) to have. (The 1 goes to the AI written paper. No thank you for wasting 2 hours of my life. Pleasure reviewing the rest.)

me: spending 6 hours checking proofs in a paper I'm reviewing someone reviewing my paper: the paper does not evaluate whether its existence increases human evaluation adoption in practice, overall 2.5

The issue with Typst is that it's so much better than LaTeX, same as why Java was winning in corporate world. It's annoying to do things in LaTeX that go beyond the basics (text, basic styling, images, figures, tables). As a result, all papers and their tex code look the same.

Beginning to think that the reviewing disaster in comptuer science is caused by us not even liking to read papers. Some parents pay kids 1$ for each finished book. I propose we give each researcher +0.1 citations for successfully reading a paper.

We are at @eamt2026.bsky.social with a large-scale stealth project called "[redacted] Translation [redacted]". Looking for contributors across all languages to challenge MTs. Talk to @patuchen.bsky.social, @sethjsa.bsky.social and me to be a coauthor!

We'll be hosting a tutorial at EAMT (already next week!), KONVENS and MT Marathon on human evaluation. Come learn with us! With support of @maikezufle.bsky.social and @patuchen.bsky.social

I reviewed for ICML and all I got was this lousy registration.

environmental storytelling

Deadline extension! - The task is simple: get audio + its translation and estimate how good it is. - Mark your name as the winner of the first Speech Translation Metrics Shared Task at IWSLT 2026 🏆 Predictions submission: May 7, 2026 Description paper: May 10, 2026

I love halucinated citations in papers. They serve as an obvious canary to AI-written papers. Without them, it takes a while to notice the discourse in writing doesn't make sense or that the science is shallow or unsound.

Come to La Palmaraie (EACL) for the first Multilingual Multicultural Evaluation workshop! 🧐 now. Organized by @pinzhen.bsky.social @hanxuhu.bsky.social @simi97k.bsky.social Wenhao Zhu @bazril.bsky.social Alexandra Birch @afaji.bsky.social Rico Sennrich @sarahooker.bsky.social

saddest conversation at eacl: - what do you work on? - mathematical modelling of evaluation - oh what kind of LLM is that?

all conference attendees under the age of 12 agree that playing subway surfer vastly improves the poster presentation experience

Machine translation is tough to evaluate, partly because most of what you throw at is too easy. That doesn't at all mean that translation is solved; we're just not doing a good job finding interesting inputs.

Quality estimation (automated metrics) are amazing. Truly. We would like to use them everywhere. That gets compute-expensive very quickly. We also don't know when they don't know. In "Early-Exit and Instant Confidence Translation Quality Estimation" (at EACL26) we fix that.

How often is human evaluation skipped in papers/workflows just because it's too difficult to set up? Yet even small humeval can give so much more signal than automatic metrics. Introducing Pearmut, Human Evaluation of Translation Made Trivial🍐 arxiv.org/pdf/2601.02933

Have you ever wondered how speech translation gets evaluated? Sadly, most speech evaluation downgrades to text-based metrics. Let’s do better! At IWSLT 2026, we’re launching the first-ever ✨Speech Translation Metrics Shared Task ✨!

Dissatisfied with EACL paper decisions? Fret not and submit your paper with ARR reviews to Multilingual Multicultural Evaluation workshop at EACL (both archival or nonarchival) until January 5th. 🔍🙂 multilingual-multicultural-evaluation.github.io

Now onwards to making language models transparent and trustworthy for everyone! 🚀 For those curious to know more about my thesis: - Web-optimized version: gsarti.com/phd-thesis/ - PDF: research.rug.nl/en/publicati... - Steal my Quarto template: github.com/gsarti/phd-t...

Do you have work on resources, metrics & methodologies for evaluating multilingual systems? Share it at the MME workshop 🕵️ co-located at EACL. Direct submission deadline in 10 days (December 19th)! multilingual-multicultural-evaluation.github.io

Let's talk about eval (automatic or human) and multilinguality at #EMNLP in Suzhou! 🇨🇳 - Efficient evaluation (Nov 5, 16:30, poster session 3) - MT difficulty (Nov 7, 12:30, findings 3) - COMET-poly (Nov 8, 11:00, WMT) (DM to meet 🌿 )

Grateful to receive the Google PhD Fellowship in NLP! 🙂 I am not secretive about having applied to 4 similar fellowships during my PhD before and not succeeding. Still, refining my research statement (part of the application) helped me tremendously in finding out the... inf.ethz.ch/news-and-eve...

📢 Announcing the First Workshop on Multilingual and Multicultural Evaluation (MME) at #EACL2026 🇲🇦 MME focuses on resources, metrics & methodologies for evaluating multilingual systems! multilingual-multicultural-evaluation.github.io 📅 Workshop Mar 24–29, 2026 🗓️ Submit by Dec 19, 2025

My two biggest take-aways are: - Standard testsets are too easy (Figure 1). - We can make testsets that are not easy (Figure 2). 😎

We saw increased momentum in participation growth this year: 36 unique teams competing to improve the performance of MT. Furthermore, we added collected outputs of 24 popular LLMs and online systems. Reaching 50 evaluated systems in our annual benchmark.

The 2025 MT Evaluation shared task brings together the strengths of the previous Metrics and Quality Estimation tasks under a single, unified evaluation framework. The following tasks are now open for participants (deadline July 31st but participation has never been easier 🙂 ):

You have a budget to human-evaluate 100 inputs to your models, but your dataset is 10,000 inputs. Do not just pick 100 randomly!🙅 We can do better. "How to Select Datapoints for Efficient Human Evaluation of NLG Models?" shows how.🕵️ (random is still a devilishly good baseline)

TIL that since python3.4 there's default `statistics` module with things like mean, mode, quantiles, variance, covariance, correlations, zscore, and more!. No more needless numpy imports!

Past iterations of the Terminology Shared Task don't come anywhere near the data quality and evaluation scrutiny of this one. In the era of LLM-as-MTs, participation has never been easier!

Thank you for your response. I will keep my score.

For the longest time I've been using Google Translate as a gateway to explain machine translation concepts to people as it's a tool that everyone knows. Now I get to contribute over the summer. 🌞 If you're near Mountain View, let's talk evaluation. 📏

arxiv submission process got an update! (still requires a manual bbl)

XCOMETs underperform because they do not match translators' subjective error annotation propensity. Using the granular p(error) value from XCOMET significantly boost their performance when calibration is possible → desirable for a fair evaluation 6/

Key takeaways for WQE evals: 1️⃣ Unsup. WQE shows promise (esp. uncertainty-based ones), interp approaches under-explored for MT 2️⃣ Calibration sets can help to ensure fair evaluations. 3️⃣ Use multiple annotators for robust rakings. More info ➡️ arxiv.org/abs/2505.23183 8/8

📢 New paper: Can unsupervised metrics extracted from MT models detect their translation errors reliably? Do annotators even *agree* on what constitutes an error? 🧐 We compare uncertainty- and interp-based WQE metrics across 12 directions, with some surprising findings! 🧵 1/

incredible monetization opportunity (this is a joke)

Ever looked down from a hot air balloon and despaired at how expensive it is to run thorough human evaluation of machine translation? Fret no more and come tomorrow at 11:00 to Hall 3 #NAACL2025.

Being in a hot air balloon in Albuquerque really makes one ponder *how to efficiently pick the best translation candidate without running expensive evaluation metrics on all of them.* See you tomorrow at 9:00 in Hall 3 #NAACL2025.

Multilinguality is happening at #NAACL2025 @crystinaz.bsky.social @oxxoskeets.bsky.social @dayeonki.bsky.social @onadegibert.bsky.social

Let's chat at #NAACL2025 about evaluation et al! ⚗️

(Automatic) span annotations are the future of evaluation and diagnosis in NLP! 🖊️

How do LLMs compare to human crowdworkers in annotating text spans? 🧑🤖 And how can span annotation help us with evaluating texts? Find out in our new paper: llm-span-annotators.github.io Arxiv: arxiv.org/abs/2504.08697

In the true sense of the word I am humbled to have been rejected by a few fellowships this year.

Overall 3.0 (borderline reject) but with 43% tariff adjustments it's 4.5 (borderline award).

Panopticon, but instead of prison cells, it's a stack of Overleaf tabs. You can’t watch them all at once, but at any moment, you *could* be watching any of them. And they know it.