Noce paper on how well benchmarks cover what people do at work arxiv.org/abs/2603.01203 Quote "these observations suggest that agent benchmarking effort is driven less by alignment with real-world employment structure or economic value, and more by methodological convenience."

@ai2.bsky.social I'm confused by the OLMoTrace paper. The paper says that the trace feature is useful for "Fact checking" (see image), but the Limitations section notes: "The retrieved documents should not be interpreted as [...] supporting evidence or citations for the LM output." A contradiction?

Article is in Dutch, but maybe understandable via Google Translate. TL;DR: AI poetry is used in 3 different ways. 1. To study progress in creative computing; 2. To explore new literary forms and to see how people interpret those; 3. To (deceptively) market AI with appeals to its 'humanity.'

New article on the evaluation of task-oriented chatbots: nejlt.ep.liu.se/article/view... #NLProc #AI

Nieuw artikel over Poëzie en kunstmatige intelligentie: www.aup-online.com/content/jour... (Open-access versie hier: research.tilburguniversity.edu/en/publicati...)

I should have known this: not all innovations are improvements.

PSA: Today I learned that quizzes created in the Canvas learning management system using the New Quiz Builder don't allow you to export all student answers to Excel/CSV. This has been a problem since 2021. Don't use the new quiz builder. Use the old one. community.instructure.com/en/discussio...

Surprising omission: the @aclrollingreview.bsky.social website does not have an easily findable button to sign up as a reviewer. Seems like a missed opportunity.

Some years ago, I bought Joseph Weizenbaum's book on Computer Power and Human Reason for a couple of dollars. Nice to have a classic work on computers and society. Following the AI boom, the price of secondhand copies has skyrocketed: more than 100 dollars for a used copy!

Nice essay on AI ethics by @frank.computer: www.frank.computer/blog/2025/05...

We just published a position paper on the evaluation of dialog systems, roughly asking: 1. Why don't we talk more about usage requirements for evaluation metrics? 2. Should we only adapt metrics to dialog, or can we also adapt dialog to suit metrics? URL: aclanthology.org/2025.gem-1.18/ #NLProc

I'm looking for studies on the mood/atmosphere/affective dimensions of photographs, but it's hard to find relevant work. Any recommendations?

The first papers are already submitted, but for anyone hoping for an extra week to work on their paper, I have very good news! The deadline for submitting to #INLG2025 has been extended until July 18th (end of the day anywhere on earth, AoE). Submit your work on generation! 2025.inlgmeeting.org

Does anyone know whether there are automatic solutions to detect either "ghost references" (academic sources that do not actually exist) or improper citations (misrepresenting claims in earlier work)? Would be a very interesting #nlp #nlproc challenge, and useful to flag potential fraud.

Just over a month left to submit to INLG2025, hosted in Hanoi, Vietnam! INLG is the annual @siggen.bsky.social conference for research on any aspect of Natural Language Generation: (large) language models, rule-based generation, applications, human/automatic evaluation of output quality, etc.

New open access textbook (in Dutch): tiu.trialanderror.org/projects/wer... It's all about quantitative content analysis, and how to analyse existing data in a responsible manner.

Curious new #NLProc/#NLP paper aiming to quantify the extent to which a language model can potentially generate all valid strings from a language without generating strings that do not belong to the language: arxiv.org/abs/2504.14370 Very similar to generative linguistics, but Chomsky is not cited.

Today we strike! My first day off since 2017 - Thanks @eppobruins.bsky.social

Mijn opiniestuk over domme AI-hype-stukjes op (o.a.) televisie, perfect verbeeld door Klöppings verhaal over ChatGPT als psycholoog. Wat zulk commentaar dom maakt is niet eens zozeer een verkeerd begrip van de technologie zélf, maar totale onkunde over de echte wereld. www.trouw.nl/opinie/opini...

INLG2025 will be in Hanoi, Vietnam. If you work on Natural Language Generation #NLG, please consider submitting! It's a really fun conference with attention for both #LLMs and more traditional natural language generation techniques. More information via: 2025.inlgmeeting.org #NLProc

If you are organizing a workshop in the United States, please consider adding a statement on remote attendance. Given the stories in the media about academia being under threat and grants being canceled based on their topic, it doesn't feel safe to go there if you work on those topics.

Results: Two weeks ago I was looking for Vision & Language models, and @dcatteeu.bsky.social helpfully suggested to look at the Open VLM Leaderboard. At least two models are not on the list, even though they look really nice: 1. Pixtral, by @mistralai.bsky.social 2. Molmo, by @ai2.bsky.social

i know people are coming here for catharsis and that's fine but when you are done screaming please pick something you can do that will help people and do it. you don't accomplish anything by rolling around in the fear all day

I'm looking for the current best Vision & Language models that can be run on a modern laptop, ideally through LMStudio. I currently have Qwen2.5-VL 7B and LLaVA 1.5 7B. Any others I should consider? #NLP #NLProc #ML #AI

I recently answered a question for the Dutch "AI helpdesk" on the future of LLMs: ikhebeenvraagoverai.nl/answers/wat-... Interesting exercise to write a short answer in plain language. I really enjoyed this! At the same time it's hard to provide a nuanced answer in ~1000 words. I hope it's useful.

What is a good pointer for a lecture/tutorial about the empirical process of LM/NLP/ML solution development? Rather than the technical details of models, optimization, etc. Basically, how to work with data, evaluation, measure generalization, overfitting, etc.

Not sure how documents like these (i.e., reports on community matters) should be published. I just went for ArXiv, at least for the first version of the report, but there should be a place to make these kinds of documents more findable.

During my time in the SIGGEN board, we received a request from the @aclmeeting.bsky.social executive board to create an overview of dual use issues in Natural Language Generation. In response, I carried out a survey. The results are here: arxiv.org/abs/2501.06636 Feedback is very welcome.

Good news everybody! You have an extra 7 days to prepare you submissions for #INLG2024! Please let everyone you know know! The new deadline for Friday, 7th of June 11:59 PM AOE! inlg2024.github.io/calls.html

Do you have experience writing ALT-text or any other image descriptions for accessibility purposes? (On BlueSky or elsewhere.) Please fill in our survey! URL: tilburghumanities.eu.qualtrics.com/jfe/form/SV_... Please share widely! #a11y #accessibility

New #NLProc paper on the evaluation of task-oriented dialogue systems: arxiv.org/abs/2312.13871 This took a while to complete (#slowscience), but it was really nice to be able to contribute to the paper.

I’m interested to learn more about science and technology studies (#sts). What is the best place to start? I’ve found this book by Sergio Sismondo: www.wiley.com/en-in/An+Int... Is that the recommended starting point or should I look elsewhere?