Good news (for me!) my gender bias paper from 2023 still replicates with GPT-5. Bad news (for everyone!) my gender bias paper from 2023 still replicates with GPT-5. arxiv.org/pdf/2308.14921 hkotek.com/blog/gender-...

🇦🇹I'll be at #ACL2025! Recently I've been thinking about: ✨linguistically + cognitively-motivated evals (as always!) ✨understanding multilingualism + representation learning (new!) I'll also be presenting a poster for BehaviorBox on Wed @ Poster Session 4 (Hall 4/5, 10-11:30)!

I did an interview w/ Pittsburgh's NPR station to share some of my views on the topic of the McCormick/Trump AI & Energy summit at CMU tomorrow. Despite being hosted at the university, there will not be opportunities for our university experts to contribute viewpoints at the event.

We have to talk about rigor in AI work and what it should entail. The reality is that impoverished notions of rigor do not only lead to some one-off undesirable outcomes but can have a deeply formative impact on the scientific integrity and quality of both AI research and practice 1/

Where does one language model outperform the other? We examine this from first principles, performing unsupervised discovery of "abilities" that one model has and the other does not. Results show interesting differences between model classes, sizes and pre-/post-training.

When it comes to text prediction, where does one LM outperform another? If you've ever worked on LM evals, you know this question is a lot more complex than it seems. In our new #acl2025 paper, we developed a method to find fine-grained differences between LMs: 🧵1/9

Hanging around NAACL and presenting this Thurs, 4:15 @ ling theories oral session (ballroom 🅱️). Come say hi, will also be eating many a sopapilla

I wasn’t super excited by o1, but as reasoning models go open-weights I’m starting to see how they make this interesting again. The 2022-24 “just scale up” period was both very effective and very boring.

Accept to NAACL main! See yall in NM ☀️

cat

I am once again asking for {cafe, food, work spots, things to see and do} for a place I will be visiting: the baaaay 🌁 (My first time visiting NorCal *ever* so the regular tourist spots are welcome!)

I don't remember who created this, where I got it from, or how long I've had it, but I have it on my slides as students walk in the first time we talk about Labov's NYC study. And it makes me chuckle every time I see it for some reason. "Very rhotic. Very stratified." 😆

When I started on ARL project that funds my PhD, the thing we were supposed to build was a "MaterialsGPT". What is a MaterialsGPT? Where does that idea come from? I got to spend a lot of time thinking about that second question with @davidthewid.bsky.social and Lucy Suchman (!) working on this:

got a new suitcase but the main beneficiary of this purchase was my cat

I always think classical music is great to work to, and then I put on something a little too emotional, and then I’m just left sitting motionless and staring at my screen

anyone have recs for good coffee shops to work from in nyc / working spaces in general? around nyu is a plus!

this is incredible

I love it when my reviewers read my paper.

@kanishka.bsky.social and I have made a starter pack for researchers working broadly on linguistic interpretability and LLMs! go.bsky.app/F9qzAUn Please message me or comment on this post if you've noticed someone who we forgot or would like to be added yourself!

Just got an email from the zombie journal soliciting my submissions so I thought I'd remind everyone that in 2015 the entire editorial board of "Lingua" resigned and created a new independent OA journal (Glossa). Don't submit to the new Lingua. slate.com/human-intere...

✨New pre-print!✨ Successful language technologies should work for a wide variety of languages. But some languages have systematically worse performance than others. In this paper we ask whether performance differences are due to morphological typology. Spoiler: I don’t think so! #NLP #linguistics

📢 NAACL reviews have been released! 🆕 feature alert: ARR now has review issue flagging! Thanks @jkkummerfeld.bsky.social & OR team for help with implementation, and other EiCs for supporting the idea! It'll be live after author response. More details here: aclrollingreview.org/authors#step... /1

💬 Have you or a loved one compared LM probabilities to human linguistic acceptability judgments? You may be overcompensating for the effect of frequency and length! 🌟 In our new paper, we rethink how we should be controlling for these factors 🧵:

had a great time meeting old n new friends in Miami @ EMNLP! if {you stopped at my poster, I stopped at your poster, we had a hallway chat, we met over beverages} feel free to reach out to chat online as well :D

It really was!!!

Don’t forget to come check out the hottest new thing in interlinear glossing @ EMNLP, Tuesday 2-3:30

(Hehe first bsky post!) I'll be at #EMNLP2024 💃🌴! Happy to chat about (among other things): ✨linguistically+cognitively motivated evaluation ✨NLP for low-resource+endangered languages ✨figuring out what features of language data LMs are *actually* learning I'll be presenting two posters 🧵: