📄 [ACL 2025 main] LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (doi.org/10.48550/arX...)

Three invited speakers will share their insights at TokShop! Hear from Yuval Pinter @uvp.bsky.social, Desmond Elliott @delliott.bsky.social, and Adrian Łańcuck on cutting-edge tokenization research. Don't miss these keynote presentations! #ICML2025 tokenization-workshop.github.io/speakers

It is with great pleasure that I share MAXMINDS 2.0, a new Max Planck program to support scholars in danger of displacement by war or natural disasters, and who have limited access to resources and institutional support. If you know affected scholars, please share. www.maxminds.mpg.de

📢I am hiring a Postdoc to work on post-training methods for low-resource languages. Apply by August 15 employment.ku.dk/faculty/?sho.... Let's talk at #ACL2025NLP in Vienna if you want to know more about the position and life in Denmark.

💡Beyond math/code, instruction following with verifiable constraints is suitable to be learned with RLVR. But the set of constraints and verifier functions is limited and most models overfit on IFEval. We introduce IFBench to measure model generalization to unseen constraints.

📣 I am happy to support Ph.D applications to the Danish Advanced Research Academy. My main areas of research include multimodal learning and tokenization-free language processing. Feel free to reach out if you have similar interests! Applications due August 29 www.daracademy.dk/fellowship/f...

Following #CVPR2025, #ICCV2025 implemented a new policy targeting accountability and integrity. PCs identified 25 highly irresponsible reviewers, resulting in the desk rejection of 29 associated papers, including 12 submissions that otherwise would have been accepted.

Huge thanks to everyone that attended the Copenhagen NLP Symposium last week. Thanks for our wonderful speakers @kylelo.bsky.social, @najoung.bsky.social, Yohei Oseki, @mziizm.bsky.social, and @loubnabnl.hf.co! @mariaa.bsky.social did a great job of summarizing the talks in these liveposts (quoted).

Thought-provoking interview with Meg about “AGI”

📯 Best Paper Award at CVPR workshop on Visual concepts for our (@doneata.bsky.social + @delliott.bsky.social) paper on probing vision/lang/ vision+lang models for semantic norms! TLDR: SSL vision models (swinV2, dinoV2) are surprisingly similar to LLM & VLMs even w/o lang 👀 arxiv.org/abs/2506.03994

I am looking forward to meeting people working on multimodality at #CVPR2025. You can find me hopping between the @vlms4all.bsky.social and Visual Concepts Workshops on Thursday. Feel free to reach out if you want to grab a coffee ☕ or a beer 🍻 during the week!

Announcing our recent work “Multilingual Pretraining for Pixel Language Models”! We introduce PIXEL-M4, a pixel language model pretrained on four visually & linguistically diverse scripts: English, Hindi, Ukrainian & Simplified Chinese. #NLProc

I am excited to announce our latest work 🎉 "Cultural Evaluations of Vision-Language Models Have a Lot to Learn from Cultural Theory". We review recent works on culture in VLMs and argue for deeper grounding in cultural theory to enable more inclusive evaluations. Paper 🔗: arxiv.org/pdf/2505.22793

📢 The Copenhagen NLP Symposium on June 20th! - Invited talks by @loubnabnl.hf.co (HF) @mziizm.bsky.social (Cohere) @najoung.bsky.social (BU) @kylelo.bsky.social (AI2) Yohei Oseki (UTokyo) - Exciting posters by other participants Register to attend and/or present your poster at cphnlp.github.io /1

Interested in multilingual tokenization in #NLP? Lisa Beinborn and I are hiring! PhD candidate position in Göttingen, Germany: www.uni-goettingen.de/de/644546.ht... PostDoc position in Leuven, Belgium: www.kuleuven.be/personeel/jo... Deadline 6th of June

Has anyone written anything about *scraping and text processing* for internet pretraining data? Practical details, which tools are used, which webpage elements are considered, how HTML to text conversion is done? (I know about work on quality filters, relevant but not quite what I'm looking for)

Had fun talking at the Spurious Correlations & Shortcut Learning at ICLR! One example I brought up, which I think provides an uncommon perspective: a case where spurious shortcuts can improve generalization... even to out-of-distribution sets where the spurious feature doesn't generalize! Thread:

What would you do if someone has rolled your dataset into their benchmark (cool!) but marked it as being available under a much more permissive license (not so cool)?

I'm recruiting a postdoc on an 18-month contract candidate.hr-manager.net/ApplicationI.... The position is about deploying LLMs in the Danish public sector. This is an interdisciplinary project that touches on technical, ethical, and legal aspects of LLM usage. Apply by 1 May 2025.

Very excited to release Kaleidoscope—a multilingual, multimodal evaluation set for VLMs, built as part of our open-science initiative! 🌍 18 languages (high-, mid-, low-) 📚 21k questions (55% require image understanding) 🧪 STEM, social science, reasoning, and practical skills

🚀 We are excited to introduce Kaleidoscope, the largest culturally-authentic exam benchmark. 📌 Most VLM benchmarks are English-centric or rely on translations—missing linguistic & cultural nuance. Kaleidoscope expands in-language multilingual 🌎 & multimodal 👀 VLMs evaluation

I'm really excited about this new benchmark for VLMs! It was great to do this as part of an open-science initiative and to meet so many great people.

📢Excited to announce our upcoming workshop - Vision Language Models For All: Building Geo-Diverse and Culturally Aware Vision-Language Models (VLMs-4-All) @CVPR 2025! 🌐 sites.google.com/view/vlms4all

Announcing a new event initiative: Feminist AI LAN Party! Katharine and I did a pilot event last year and we're now taking it to @pyconde.bsky.social. We've also open-sourced event kits to make it easy to host your own, including: 💣 hacking LLMs 📑 data development ✂️ zine making feministai.party

Would you present your next NeurIPS paper in Europe instead of traveling to San Diego (US) if this was an option? Søren Hauberg (DTU) and I would love to hear the answer through this poll: (1/6)

I'm also desperately looking for emergency reviewers for safety, bias, and alignment papers. Please let me know if you could write a review! 😭

Can anyone point me at concrete examples of the value being created by Large Language Models right now in 2025?

Inspired by languagelog.ldc.upenn.edu/nll/?p=68451 , I asked Claude for elephant-free rooms and honestly got much more than I expected

🌍 Join us in Copenhagen - fantastic city for work-life balance and awesome science. I am hiring PhD students & Postdocs in my group at SODAS Univ of Copenhagen to explore AI, network science & the science of science. 📍 Start: flex summer 2025 📌 Info&Apply: www.robertasinatra.com/2025/03/02/p...

I was reading a preprint from this week and I noticed, upon closer inspection, that 41/49 of the citations are from 2023--. It seems like the observation we made in 2020 (aclanthology.org/2020.acl-mai...) is becoming even more extreme in 2025.

Finland is making a significant investment in the future of machine learning / artificial intelligence by hiring 20 professors (at various levels) www.ellisinstitute.fi/ellis-instit...

🚨 Preprint alert 🚨 𝐂𝐚𝐧 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐭𝐲 𝐍𝐨𝐭𝐞𝐬 𝐑𝐞𝐩𝐥𝐚𝐜𝐞 𝐏𝐫𝐨𝐟𝐞𝐬𝐬𝐢𝐨𝐧𝐚𝐥 𝐅𝐚𝐜𝐭-𝐂𝐡𝐞𝐜𝐤𝐞𝐫𝐬? (arxiv.org/abs/2502.14132) Fact-checking agencies have come under intense scrutiny in recent months regarding their role in combating misinformation on social media.

We now have a form for postdoc applications: forms.gle/tiydAChgV1wL... I am looking at candidates on a rolling basis, so while there's no deadline, there's an advantage of throwing your name in the ring earlier than later

I've noticed a trend that Ph.D students are not finishing industrial internship projects before they return to university. Instead of the internship project being abandoned, it is completed during university time, sometimes paid for by tax payer-funded projects. Does anyone else see this happening?

Can Large Language Models THINK and UNDERSTAND? The answer from cognitive science is, of course, lolwut YES! The more interesting question is CAN TOASTERS LOVE? Intriguingly, the answer is ALSO YES! And they love YOU

Announcing our Multilingual Multimodal Exams initiative in collaboration with @cohere.com. We aim to collect MCQ-format exams for a challenging new multimodal benchmark. Submit 1000Qs for high/mid-resource, or 500 low-resource languages to be eligible for co-authorship.

During my time in the SIGGEN board, we received a request from the @aclmeeting.bsky.social executive board to create an overview of dual use issues in Natural Language Generation. In response, I carried out a survey. The results are here: arxiv.org/abs/2501.06636 Feedback is very welcome.

I am looking for emergency Area Chairs and Reviewers for the Language Grounding Track at the February 2025 ARR cycle. If you volunteer, then you will have no duties during the regular reviewing period. Emergency review: 18th to 26th of March, 2025. Emergency area chair: 7th to 11th of April, 2025.

Does anyone know where you can find the findings of BabyLM 2024?

Residual connections are 🔥, right? Wait, so why do we only use them to skip 1 layer? Not only Lucas Georges Gabriel Charpentier & David Samuel checked it. They found that this provided huge gains - winning the babyLM challenge🍼 arxiv.org/abs/2311.02265

Interesting talk by Tal Linzen at LSD this morning that included a surprising result about how much syntax can be learned from smaller corpora, such as CHILDES or Simple Wikipedia aclanthology.org/2023.acl-lon...