Nice work by my student @gianlucavico.bsky.social on a topic close to home: crowdsourcing Piedmontese to test LLMs on non-standard orthography. New dataset covering tokenization, classification & translation.

On the Credibility of Evaluating LLMs using Survey Questions by @jlibovicky.bsky.social aclanthology.org/2026.mme-mai... Survey-based LLM value evals are unreliable — prompting & decoding choices skew results drastically.

So proud of my new PhD student Adnan Al Ali for presenting their master's thesis work at EACL! 🎓 A great contribution to understanding bias in AI text detectors across languages.

I'm at #EACL2026 in Rabat 🇲🇦. Find me and talk to me about tokenization or multilingual model eval. Also, check out our work on how to eval morphological plausibility of your tokenizer if you don't have gold segmentation data, but you happen to have morphosyntactic features 👇

Thanks for organizing this! I am definitely coming, and everyone interested in tokenization at #EACL2025 should too. 🫵

3rd run teaching ML to 250+ bachelor students (with great materials originaly by @straka-milan.bsky.social). Core philosophy: explain the math, implement algorithms from scratch, Kaggle-style competitions, all auto-graded. ufal.mff.cuni.cz/courses/npfl... But look what LLMs did to the course 👇

Spent time making AI-generated images of Bayes' Rule, Laplace Smoothing, Markov Chains & Shannon Entropy for class today 🎨🤖 Even though the images are objectively hilarious, none of the 50 students in the room laughed. Or even smiled. 💀

I reviewed papers evaluating LLM values using sociology questionnaires. Different methods, different results. Didn't trust them, so I tested it myself. Methodology matters. Short answers vs CoT, squared err vs KL div.: each changes which populations an LLM "aligns" with. www.arxiv.org/pdf/2602.04033

We have updated the pre-print on CUS-QA, benchmark for regional knowledge about Czechia, Slovakia and Ukraine arxiv.org/abs/2507.22752 Now, there are results of retrieval-augmented generation and more detailed analysis of model performance depending on the topic of the question or visual context.

We (= mostly @abyste.bsky.social) developed a way to evaluate how morphological a #tokenization is w/o gold segmentation labels. arxiv.org/abs/2601.18536 The key: align subword tokens with morphological features from UniMorph using IBM Model 1. To appear in EACL 2026 Findings.

Happy holidays! 🎄🎅🤩🎁

Attenzione! 🇮🇹 Know Piedmontese or Neapolitan speakers? @gianlucavico.bsky.social is collecting crowd-sourced translations to evaluate LLM performance on these regional languages. Partecipate!

With @andrei-a-manea.bsky.social, we posted a survey on multilingual vision-language models 👉 arxiv.org/pdf/2509.22123 We reviewed 31 models+21 benchmarks. There's a tension between language neutrality (same results across languages) & cultural awareness (context matters differently across cultures)

So proud of my PhD student @andrei-a-manea.bsky.social for his first first-author publication! 🎉 He presented this work last week at TSD. Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders arxiv.org/pdf/2504.21681

🧵 We're releasing CUS-QA - a new benchmark for testing LLMs on regional knowledge! Find out what your model knows about Czechia 🇨🇿, Slovakia 🇸🇰, and Ukraine 🇺🇦! 👉 Textual and visual questions, answers, and human judgment on model outputs! huggingface.co/datasets/ufa... www.arxiv.org/abs/2507.22752

Stay tuned, we will release the dataset soon...

We need to have poster fights at the end of every conference.

Just presented MAGBIG, a new dataset and evaluation methodology for gender bias in multilingual text-to-image generation. Grammatical gender matters when studying these biases across languages! Thanks to Felix Friedrich, @kathaem.bsky.social and all co-authors - it was fun to work on this together!

This week I am at #ACL2025NLP in Vienna 🎡🇦🇹. Find me 🕵️ or message 💌 me if you want to chat about multilinguality or tokenization. Stop 🛑 by our poster on gender bias in text-to-image generation on Monday aclanthology.org/2025.acl-lon...

TokShop @ #ICML2025 got way more submissions than expected! 📈 We could really use a few more reviewers to help out. If you have the capacity to review a #tokenization paper by Saturday, please fill out this form: forms.gle/32A6sQHQrMSb... 🙏

📣 Call for Paper Alert: TokShop @ ICML 2025 TokShop explores tokenization across all data modalities. Topics include: subword NLP techniques, multimodal approaches, multilingual challenges, post-training modification, alternative representations, and statistical perspectives.

Got a tokenization paper that just didn't make the cut for ICML? Submit it to the Tokenization Workshop TokShop at #ICML2025 -- we'd love to see it there! tokenization-workshop.github.io

Attending #NAACL2025 virtually. Since 2022, I've been training a classifier on papers I read to tackle the arXiv madness. Ran it on the NAACL proceedings for my personalized watch list. 🤓📺 However, it's far from perfect: Multilingual cultural awareness is great, but where is tokenization? 🤷

We're organizing ✨Tokenization Workhop✨ TokShop❗ Join us at @icmlconf.bsky.social in July in Vancouver 🇨🇦. Follow @tokshop.bsky.social for updates! Submit your paper by May 30.

Random take on the #TuringTest: Rather than testing machine intelligence, it can be a measure of societal awareness about #AI capabilities. The real objective isn't creating a machine that passes but educating people to think critically and avoid being deceived, so the machines do not pass the test.

Summaries of pre-prints that I noticed and liked on arXiv in March are now on my blog jlibovicky.github.io//2025/04/02/...

Our paper 'Beyond Literal Token Overlap: Token Alignability for Multilinguality' will be at #NAACL2025! We show that token alignability is a stronger predictor of cross-lingual transfer than literal token overlap. Read it here: arxiv.org/abs/2502.06468

Short notes about what pre-prints I noticed in December and January are now on my blog: jlibovicky.github.io/2025/02/07/M...

Join Mu-SHROOM 🍄, a SemEval 2025 shared task on detecting hallucination spans in multilingual LLM outputs! 🌍 Includes Czech with regional Czech questions 🇨🇿. Do you think you can spot when something isn’t true? 🤔 Try it out! 👉 helsinki-nlp.github.io/shroom #SemEval2025 #NLP

Happy holidays! 🎄🎅🤩🎁

Highlights from multilingual #NLP and machine translation papers I found on arXiv in November are now on my blog: jlibovicky.github.io/2024/12/05/M...

This is going to be fun! 🤓 We have three years to spend 6.5M CZK on improving multilingual tokenization. The goal is to make subwords more alignable across languages and help languages that suffer from over-segmentation with current models.

Just shared my takeaways from #EMNLP2024 on my blog: jlibovicky.github.io//2024/11/21/...

Hello Blue Sky! 👋 This is the official account of the Institute of Formal and Applied Linguistics (ÚFAL for short) at the Faculty of Mathematics and Physics, Charles University in Prague 🇨🇿. Here, we will share news from the life of the institute and our members.

Hello, Blue Sky 👋 🦋 Looking forward to reading and sometimes also posting about #NLP and related stuff! 🤓