How good are LLMs at 🔭 scientific computing and visualization 🔭? AstroVisBench tests how well LLMs implement scientific workflows in astronomy and visualize results. SOTA models like Gemini 2.5 Pro & Claude 4 Opus only match ground truth scientific utility 16% of the time. 🧵

We’ve started a podcast! @awsto.bsky.social and @samps.phd host “Current Continuation,” a little interview series with PL researchers. The first two episodes are with @ranjitjhala.bsky.social and @satnam6502.bsky.social. sigplan.org/cc/

News🗞️ I will return to UT Austin as an Assistant Professor of Linguistics this fall, and join its vibrant community of Computational Linguists, NLPers, and Cognitive Scientists!🤘 Excited to develop ideas about linguistic and conceptual generalization (recruitment details soon!)

Evaluating language model responses on open-ended tasks is hard! 🤔 We introduce EvalAgent, a framework that identifies nuanced and diverse criteria 📋✍️. EvalAgent identifies 👩‍🏫🎓 expert advice on the web that implicitly address the user’s prompt 🧵👇

🚀Meet CRUST-Bench, a dataset for C-to-Rust transpilation for full codebases 🛠️ A dataset of 100 real-world C repositories across various domains, each paired with: 🦀 Handwritten safe Rust interfaces. 🧪 Rust test cases to validate correctness. 🧵[1/6]

A bit of a mess around the conflict of COLM with the ARR (and to lesser degree ICML) reviews release. We feel this is creating a lot of pressure and uncertainty. So, we are pushing our deadlines: Abstracts due March 22 AoE (+48hr) Full papers due March 28 AoE (+24hr) Plz RT 🙏

Come work with me! We are looking to bring on more top talent to our language modeling workstream at @ai2.bsky.social building the open ecosystem. We are hiring: * Research scientists * Senior research engineers * Post docs (Young investigators) * Pre docs job-boards.greenhouse.io/thealleninst...

🌟Job ad🌟 We (@gregdnlp.bsky.social, @mattlease.bsky.social and I) are hiring a postdoc fellow within the CosmicAI Institute, to do galactic work with LLMs and generative AI! If you would like to push the frontiers of foundation models to help solve myths of the universe, please apply!

encourage postdocs to apply 👇 @soldaini.net, myself and others from @ai2.bsky.social have been helping in project & also learning a ton---continued pretraining, creating domain-specific training data & evals---to build foundation models that scientists can use. promising area for open source LMs!

three things are certain in life: death, taxes, and Claude switching to concise mode during US business hours

Kudos to Usneek Singh. It was a pleasure to collaborate on this paper with the amazing folks at PROSE!

@ayushkhaitan.bluesky.social, Amitayush Thakur, and I are organizing an #AI4Math panel at the Joint Mathematics Meeting this month. Please spread the word among your math friends! We will post a summary of the discussion after the event.

Huge congrats to @prasannsinghal.bsky.social for being one of the 8 CRA Outstanding Undergraduate Researcher Award winners! It has been an absolute privilege to work with Prasann during his time at UT. (And he's applying for PhD programs this year...hint hint...) Prasann's work 🧵

@andersmoeller.bsky.social and I are co-chairing OOPSLA'26 and soliciting PC nominations. If you'd like to serve on the OOPSLA PC next year or know anyone (e.g., recent graduate) who you think would do a good job, please nominate them here: forms.gle/NVnzjcmbshoL...

The legendary Putnam math competition had its 85th edition yesterday. Coincidentally, George Tsoukalas will present our paper on PutnamBench, a next-generation #AI4Math benchmark, at #NeurIPS2024 this week: arxiv.org/abs/2407.11214. If you work on frontier AI for math/reasoning, talk to George!

I'll be at #NeurIPS2024 w/ - @fcyin.bsky.social's LoFiT: using interp to improve fine-tuning (Weds pm poster & MINT spotlight talk Sun) - @thomlake.bsky.social's analysis of Overton pluralism (Pluralistic alignment Sat) Please reach out to me to chat about interp, factuality, reasoning, &c!

Excited to visit Columbia next week!

I did a starter pack of ML/AI people at @utaustin.bsky.social Please distribute and feel free to self nominate! go.bsky.app/QLQznZg

Yay!!! I’m one of the cool people!

We got an 🥂 Outstanding Paper Award!! Cannot be more grateful 🥹 This is super validating for our long pursuit of computational work on QUD. Congrats to the amazing @yatingwu.bsky.social, Ritika Mangla, Alex Dimakis, @gregdnlp.bsky.social

I won't be at EMNLP, but come and see: 🔍 Detecting factual errors from LLMs (Liyan Tang) 🛠️ Detect, critique, & refine pipeline (Manya Wadhwa and Lucy Zhao) 🏭 Synthetic data generation (Abhishek Divekar) 📄 Fact-checking (Aniruddh Sriram) at FEVER t.co/fQbl0G7m23 (1st real post in the bluer skies!)

Wednesday at #EMNLP: @yatingwu.bsky.social will present our work connecting curiosity and questions in discourse. We built strong models to predict salience, outperforming large LLMs. 👉[Oral] Discourse+Phonology+Syntax2 10:30-12:00 @ Flagler also w/ Ritika Mangla @gregdnlp.bsky.social Alex Dimakis