Bonus post advertising this other thread through the medium of "memes" which I've been told is what you have to do on social media.

It's great to see the idea of using games / interactions to evaluate LLMs gain traction, with textarena.ai and now ARC-AGI-3 being latest entrants. This is something we've been exploring since early 2023 with clembench ( clembench.github.io ), which we've been continuously maintaining & extending. »

📄 [ACL 2025 main] LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (doi.org/10.48550/arX...)

🚨 New pre-print! (Well, new & much improved version in any case.) 🚨 If you're interested in LLM post-training techniques and in how to make LLMs better "language users", read this thread, introducing the "LM Playpen".

The University of Potsdam invites applications for 5 postdoc positions, incl. Cognitive Sciences, incl. NLP (esp. cognitive). These are fairly independent research positions that will allow the candidate to build their own profile. Dln June 2nd. Details: tinyurl.com/pd-potsdam-2... #NLProc #AI 🤖🧠

There's indeed suddenly a bit of flexibility in a system that's not exactly known for that.. If there's anyone (post-doc, tenure-track, or more senior) in the #NLP space currently in the US who'd like to explore possiblities in Potsdam, contact me. 🤖🧠 www.nytimes.com/2025/05/14/b...

"We ablated both algorithm and hyperparameter choices [...]" When did "to ablate" take on the meaning "to systematically vary"? I've noticed this only recently, but it's seems to be super common now.

Update 2: New pre-print! Outcome of an ELLIS workshop last year, & more than a year of discussions and work, across labs and countries: Meet the Playpen, an environment for exploring learning in dialogic interaction. arxiv.org/abs/2504.08590 1/2

Update 2: New pre-print! Outcome of an ELLIS workshop last year, & more than a year of discussions and work, across labs and countries: Meet the Playpen, an environment for exploring learning in dialogic interaction. arxiv.org/abs/2504.08590 1/2

Update 1: New models added to our dialogue game-based agentic LLM leaderboard. TL;DR: GPT-4.1 as good as 4o, but much cheaper. Llama4 indeed not very good (decisively worse than 3.2 70B!). OLMo decent, but there's still a secret sauce that only closed labs have. clembench.github.io

Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Moment\`e, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, ... Playpen: An Environment for Exploring Learning Through Conversational Interaction https://arxiv.org/abs/2504.08590

Wenn die Grünen verhandeln könnten, würden am Tag vor einer Ankündigung über eine Einigung zur Schuldenbremse Söder und Dobrindt ankündigen, dass sie sich für immer aus der Bundespolitik heraushalten werden (und dass die CSU nie wieder einen Verkehrsminister stellen wird).

Press release by my Uni about our benchmark for LLMs as agents, which is now out in v2.0. Check it out here: clembench.github.io

Happy to see increasing interest in exploring social interaction as a learning environment! Along similar lines: We’re preparing a (complementary) challenge that will focus on exploring interaction for post-training, coming with a rich interaction environment to get things started. Stay tuned!

I'm not on X, so I'll use the opportunity of @karpathy.bsky.social 's post over there to plug our "clembench" project here. We've been doing exactly this--evaluating LLMs w/ conversational games--since early 2023, with several papers out by now (e.g. EMNLP 23). clembench.github.io

So, are we banning social network apps now whose owners potentially try to influence the political discourse in other countries? Asking for a supranational political and economic union.

I just randomly found this book on my bookshelf. It must have been transported there from an alternate timeline. “20 years of research on agents”? Preposterous! We all know that the very idea of software agents has only been invented last year by the LLM folks!

So my car needed to be towed this morning. It took the guy quite some time to get everything ready. Then the truck broke down. In the end, the tow truck was towed, and I got a new appointment. I think is probably an allegory for something, maybe the ending year 2024, or the coming year 2025.

me: I would really like to end this year with inbox zero. also me: I would really like to end this year with cookie jar zero / “pages remaining in the books I’ve started” zero. me again, expert problem solver: *creates IMAP folder “unprocessed emails from 2024”, selects all, moves 625 items*

These new models (using “inference time scaling”) bring out what many of us have been saying for a long time, namely that reasoning fundamentally is a discursive process. (What they are missing is that it is an intersubjective, interactive, repairable, and ultimately normative one.)

Looking forward to the first lecture of next year, where I can again use this meme I made a couple of years ago and multiply confuse the students in my "intro to NLP" class. (What is an "LP cover"? Who is that person?)

Can we discuss how stupid this photo button thing on the new iPhones is? Who thought that minimising the space where you can hold this damn thing without something unwanted happening is a good idea?

For a recent talk to a lay audience, I’ve used a metaphor which I think resonated: rely on an LLMs not more than you would rely on a dream. Use it to inspire you to work something out, but don’t be the one who has to say “this was once revealed to me in a dream”.

Some good news: The world now has one more doctor! Brielen Madureira passed her viva with flying colours (or, as we say in German, summa cum laude). She gave us quite some material to discuss in the viva, ending with the attached theses. (Remote: Luciana Benotti as fantastic examiner.)

Ok, this is starting to feel weird. The internet at my Uni has been down for almost 24 hours now, meaning: no new emails for almost 24 hours now. That’s like the dog finally catching the bus. What now??

Yesterday afternoon, as I was engaging in the German ritual of "Stoßlüften", a bird flew in through one window, shat on Pirmin Stekeler-Weithofer's dialogic commentary on Hegel's Science of Logic (specifically, vol. 2 on the Objective Logic, The Doctrine of Essence), and flew out through another.

I've just discovered dynamic backgrounds in Keynote! From now on my lectures will be, well, probably not at all more interesting, but at least 123.5% more psychedelic!

The disclaimer “$model can make mistakes” that you sometimes see on deployed LLMs is actually an interesting claim. I would argue that in the sense in which it will be understood, models *cannot* make mistakes. That sense requires an understanding of the possibility of error, and mastery of repair.

I'm thinking about organising an invited lecture series on "critical AI research" next summer semester, bringing together "outside" reflection on impact and "inside" reflection on practices. Who would be good speakers to invite? (Bonus points if w/in train distance of Berlin/Potsdam.)

This came out when birdchan was already in demise & bsky didn't exist yet — I'm super proud we pulled it off: a manifesto for moving beyond single-mindedness doi.org/10.1111/cogs... Part of the 'progress and puzzles in cognitive science' series; PDF & fellow travellers at markdingemanse.net/beyond/

My writing peaked early. (Also, remarkable constancy in research interests, although one didn’t talk about consciousness too much for most of the intermediate 25 years.)

Random find on my hard drive. Dragomir Radev’s 1995 FAQ on what NLP is, for the comp.ai.nat-lang newsgroup. (Note the sorting under “AI”…)

“I’m too sick to lecture tomorrow, so I can finally do this other thing!” … is not a healthy thing to think.

Irgendwie wird der Pollesch-Quatsch schon fehlen. (Das kichersüchtige Pollesch-Volksbühnen-Publikum aber nicht.) (Diese Google-Auskunft hätte ihm vielleicht gefallen.)

An important detail is that you have to ask the people if they want to do the COOL THING, say, in February. If you ask if they want to do it now, they will regretfully decline, because they are too busy. In a couple of months they will also be too busy, but that they cannot, will not believe.

The lard fly?

Prediction: 2025 is going to be the year of "meta cognition" in AI. Expect many more related terms from cognitive psychology to be taken, first as names for something vaguely inspired, then as fact. "Self-Regulation in LLMs through Self-Action Tokens", etc etc.

I myself, on the other hand, am not only tired of posts explaining why someone has left Xitter (because I’ve heard all of this a year ago already), I’m also tired of posts complaining how one is tired of posts explaining why someone has left Xitter (because they’ve heard all of this a year ago alr..

Not even a controversial opinion at this point, I think: The tech industry builds the least regulated, most dangerous products of our time.

When I first checked out this site a year ago or so, I only found a couple of people to follow, including the opossum picture bot. When I then occasionally checked in, my timeline would mostly be pictures of opossums. It’s … different now. 😅 @possumeveryhour.io

By way of introduction, here’s my paper with the worst “# citations / felt importance” ratio: aclanthology.org/2022.clasp-1.7 A sketch of a broadly pragmatist attempt at thinking about limitations of LLMs. Still haven’t found the time to spell this out more, but seems the most promising angle to me.

I feel reminded of the times of the fail whale. What would that be here? The hungry caterpillar? The lazy larva? The crazy chrysalis?

Somehow it still hasn't really taken off, my idea of enforcing more restrained discourse on social media by having people recite paragraphs from Hegel's "lord and bondsman" chapter before they can post. "So you crave recognition? Beware of the dialetic, friend! Interpret paragraph 183 of PoS."

After mass following a large bunch of people, my timeline is now a weird vibe mix of mid-period twitter, early tumblr, and “excited to share” LinkedIn. Not sure I can handle that. (Missing so far: The mastodon experience of not being able to see half of t replies, & 30% of posts being about ALT.)

Another disappointment. I wanted to create a starter pack “Starter Packs That don’t Contain Themselves”, and then the server crashed.

So that’s a disappointment. I wanted to create a starter pack “people who are me”, but it turns out the minimum number of people for a starter pack is 7.

Do we do introductions here? Anyway, here is mine: I’m a Professor of Computational Linguistics at the University of Potsdam. I am interested in understanding “understanding”, the process or activity by which an agent makes sense of its environment and, in interaction, of and with other agents.

Ok, I would like to give this thing here a chance. But I can’t until I find out whether there’s a setting that makes this app give me a chronological feed *that keeps my last position*. I can’t handle scrolling back in time.

Testing testing 🎤👏