Super excited that the Computer Use survey I've been working on w/ @anamarasovic.bsky.social for a while now is ready! Originally we were planning on a more traditional survey paper but as more surveys came out we decided on an interactive website survey.

Arriving to #ACL2025 #ACL2025NLP in a few hours! See you at the welcome reception & catch me at the poster session on 𝐓𝐮𝐞𝐬𝐝𝐚𝐲 (𝐉𝐮𝐥𝐲 𝟐𝟗) 𝐚𝐭 𝟏𝟎:𝟑𝟎𝐚𝐦, where Jesse will present our work introducing new tasks for supporting legal brief writing: arxiv.org/abs/2506.06619

Really excited about this! As backstory, Jesse Woo started this project when I taught a ML Datasets class at Columbia. Then we joined up with @anamarasovic.bsky.social and @fatemehc.bsky.social and really kicked it into high gear. Would not have happened without the full team!

Join us on June 11, 9am to discuss all things fine-grained! We are looking forward to a series of talks on semantic granularity, covering topics such as machine teaching, interpretability and much more! Room 104 E Schedule & details: sites.google.com/view/fgvc12 @cvprconference.bsky.social #CVPR25

We are so excited to have this amazing line-up of speakers!! Randall Balestriero, Kai Han, Mia Chiquier, Kenneth Marino (@kennethmarino.bsky.social‬), Elisa Ricci, Thomas Fel (@thomasfel.bsky.social‬)

We just dropped a new paper on studying LLMs on the “Blicket Test” to ask the question: do language models explore like adults or like children? We also show how to get them to act more like children (i.e. more like scientists). All credit to Anthony and team, this came together super well!

Are you tired of your static fixed benchmarks? Feel like your data is in a rut. You want to change something but you just feel stuck? Try ReCogLab! Really proud of this work and of my fantastic colleagues at Google DeepMind who put in so much hard work. See you all in Singapore!

People who actually believe in the promise of AI should be the most upset about the over-claiming, over-hyping and overt secrecy and unwillingness to expose your work to scrutiny that has come to characterize much of the “feel the AGI” crowd.

This is why open source and publishing is important. Maybe OpenAI didn’t do anything sus with held out splits. But if code and models are never released and the experiments and methods are not published or described in sufficient detail, we can’t reproduce it or scrutinize any of these decisions.

Just read a fantastic web agent paper. Game changer! * Treats it as an RL problem * Trains rather than just prompting * Beats closed models * Releases code and model so other people can build off of their work Many great ideas in this paper too, definitely read arxiv.org/pdf/2411.02337

Fun fact: Faculty do check the papers you put in your CV and notice when you try to make a workshop paper look like a full conference paper with deceptive wording.

Felix Hill was such an incredible mentor — and occasional cold water swimming partner — to me. He's a huge part of why I joined DeepMind and how I've come to approach research. Even a month later, it's still hard to believe he's gone.

Researcher: "We let the data speak for itself." Earlier that day:

the reviewer’s insightful suggestions have made our paper more accessible

Computer Vision: Fact & Fiction is now available on YouTube 🙌🏼 I made a playlist for it with the seven chapters. Enjoy this time capsule from two decades ago!

There’s still time to apply to work with me at Utah. If you’re interested in the intersection of language with perception and action and/or web agents, definitely apply! www.cs.utah.edu/graduate/pro... Deadline Dec 15

We’ll be at our poster soon if you’re interested in multimodal agents and continual learning. Can’t take a lot of credit, was only an advisor on the project. Gabriel Sarch (PhD candidate at CMU) did a great job on this project; you should come talk to him! Link to paper: arxiv.org/pdf/2406.14596

I’m at Neurips this week if you want to chat about multimodal agents (including web agents and robotics), datasets, knowledge-VQA, etc. Also, I’m hiring PhD students this fall at Utah; there’s still time to apply: www.cs.utah.edu/graduate/pro... Happy to chat if you’re at the conference!

For those of you attending #NeurIPS2024 in person: I'm from Vancouver and I made an extensive list of restaurants, bars, bookstores, etc., that I used to frequent when I still lived there. Enjoy! dippedrusk.com/posts/2024-0...

FYI, there’s a fake Google DeepMind account on here with about 3k followers right now. Please don’t follow or engage with it.

ahhh i feel right at home on bsky now that the seasonal "review system is broken!!11!1!" laments are going full swing. thanks everyone for making the transition go so smoothly <3

Okay, hot take time: I don’t like the ICLR continuous back and forth format. I find it exhausting as both an author and a reviewer. And the end result is often that reviewers don’t engage very much anyway so we might as well design it as a single response and then discussion.

I think my favorite part of Bluesky is that I can choose followers and feeds to *just* be about research. Twitter floods your feed with politics. Threads fills it with random celebrities. I can open Bluesky and just see research.

Daily paper: GPT-4V(ision) is a Generalist Web Agent, if Grounded arxiv.org/abs/2401.01614

Daily paper #4: Arboretum: A Large Multimodal Dataset Enabling AI for Biodiversity arxiv.org/pdf/2406.17720 Truly enormous dataset of animals/plants/fungi. Over 130M images, 300k species. Truly staggering scale and number of classes. A whole new scale of challenge for fine grain recognition.

Daily paper: Day 3 Open X-Embodiment: Robotic Learning Datasets and RT-X Models arxiv.org/pdf/2310.08864 Important work to move robotics towards larger datasets. 1M episodes from 22 types of robot in 311 scenes. The most impressive part of this paper is coordinating between this many people.

We now have a starter pack for ROBOTICS. This should be a fun group. Are you building robots or sharing thoughts/research on the state of robotics? 🤖🦿 Request to join if you would like to be added, before it fills up. go.bsky.app/61LAgEj

Paper #2: Vision Language Models are Blind arxiv.org/pdf/2407.06581 Definitely a provocative title. Shows some important places where VLMs underperform. Validates a sense many of us have that the vision side of VLMs has some room for improvement (low level vision especially).

Going to do my part to try to build content on Bluesky by trying to post a paper I like every day. First is one of the best papers I’ve read in a while doing essential work in examining how we evaluate agents: arxiv.org/html/2407.01... From @sayash.bsky.social @randomwalker.bsky.social at Princeton

I created a starter pack for human-AI collaboration in case that this would be helpful for some newcomers, which overlaps with several existing starter packs. Let me know if you would like to be added. go.bsky.app/MRV1Wa1

I made a starter pack with the people doing something related to Neurosymbolic AI that I could find. Let me know if I missed you! go.bsky.app/RMJ8q3i

Located 2 awesome HCI and Accessibility researcher starter packs thanks to @evanmpeck.bsky.social!! HCI list, by @comic-sans-soleil.bsky.social: bsky.app/starter-pack... HCI Accessibility list, by @jonfroehlich.bsky.social: bsky.app/starter-pack... Evan's post just now: bsky.app/profile/evan...

Love the starter packs here! Making it much easier to rebuild my feed rather than starting from 0.

Love the influx of academics to Bluesky. Will this be our new 🏠? To help grow the community, I just created an HCI+Accessibility "starter pack". Help me grow the list! Many are missing! go.bsky.app/PZxHNqP

Interesting. I reposted my Bluesky handle again on the bird app without the links and keywords and, huh, it already has triple the engagement in less than an hour. Wonder what could be going on 🤔🤔🤔

We think maybe we can move ML Twitter here? Suggestions on who to follow?