ICL and SFT are the two most studied ways to adapt LMs. We understand each in isolation — but far less about how they might 𝗰𝗼𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗼𝗻𝗲 𝗮𝗻𝗼𝘁𝗵𝗲𝗿.

Imagine this: excited about the recent progress, you’ve built an agentic system that uses 🔧tools (API calls) to solve complex problems. What could go wrong? We studied agentic tool recovery—when your LLM selects a set of tools to execute, but one turns out to be unavailable or incorrect.

A core hurdles in AI safety eval is that benchmarks (e.g., those on jailbreak attacks) quickly become outdated shortly after they are released (e.g., saturate, contaminate, patched).

A core hurdles in AI safety eval is that benchmarks (e.g., those on jailbreak attacks) quickly become outdated shortly after they are released (e.g., saturate, contaminate, patched).

Excited to collaborate up with LMArena, NIH, and DataTecnica to launch BiomedArena! Our goal is to advance the use of LLMs in biomedical discovery and incorporate community-driven insights to help shape the future of biomedical AI. ⚔️ Check it out: biomedarena.ai

What’s really going on inside LLMs when they handle non-English queries? Niyati Bafna @niyatibafna.bsky.social 's recent work introduces the **translation barrier hypothesis**, a framework for understanding multilingual model behavior. Paper: huggingface.co/papers/2506...

🔈When LLMs solve tasks with a mid-to-low resource input or target language, their output quality is poor. We know that. But can we put our finger on what breaks inside the LLM? We introduce the 💥 translation barrier hypothesis 💥 for failed multilingual generation with LLMs. arxiv.org/abs/2506.22724

🚨New LLM benchmark🚨 We're releasing BiomedSQL🔬 for tabular reasoning over large-scale biomedical databases. This includes questions based on implicit scientific conventions—like statistical thresholds, effect direction, and drug approval status. 📄 Preprint: arxiv.org/pdf/2505.20321

Long-form inputs (e.g., needle-in-haystack setups) are the crucial aspect of high-impact LLM applications. While previous studies have flagged issues like positional bias and distracting documents, they've missed a crucial element: the size of the gold/relevant context.

There have been various efforts on disentangling "task learning" vs "task recall" in LLMs. We've recently explored a fresh angle by borrowing from cryptography: with substitution ciphers, we transform a given task into an equivalent, but cryptic (no pun intended!!) forms.

What is a university without "freedom of speech"? Apparently, ChatGPT has a better grasp than @nyuniversity. x.com/nebedaay/st...

**Certified Mitigation of Worst-Case LLM Copyright Infringement** TL;DR: We propose BloomScrub a framework to certifiably remove long verbatim quotes to reduce the risk of copyright violations.

Can LLMs can be co-pilots for peer review? Answering this requires evaluating *evaluate* whether LLMs can provide critiques that are *grounded* in the context of science papers. See @JiefuOu's dataset which has a collection of paper claims and their critiques: arxiv.org/pdf/2503.21717

📣📣📣 Tianjian @tli104 and I have refreshed our course material! self-supervised.cs.jhu.edu/sp2025/ These resources may be helpful if you're: (1) looking for slides to teach about LLMs, or (2) interested in diving deeper into the field.

I will be at #NAACL2025 to present our LLM creativity benchmark. Drop by if interested (Poster Session 8, Fri, May 2)! I'd love to chat about RL and its interpretability, data influence for post-training, CogSci for LLM. Feel free to reach out and let's have some coffee together ☕ !

Highlighting our #NAACL2025 papers 🧵🧵🧵

People rely on search engines/chatbots to access science. But what if you want a bird’s-eye view of science, or to identify over- and under-explored areas? We introduce 🔺Science Hierarchography🔺, the goal of organizing science papers into conceptual hierarchies. arxiv.org/abs/2504.13834

Highlighting our #ICLR2025 papers 🧵🧵🧵 (1) "GenEx: Generating an Explorable World" openreview.net/pdf?id=8NlU... TLDR— Physical exploration can be expensive, and even impossible. Our proposed policy mitigates this by enabling agents to form an imaginative model of the 3D world.

The flow of talent across institutions keeps research vibrant. Excited that students from my lab are off to top PhD programs! Muhan Gao @muhan_gao→ Texas A&M Zhouxiang Feng @FocusV857→ Rice Abe Hou @abe_hou→ Stanford Taiming Lu @TaiMingLu→ Princeton Dongwei Jiang @Dongwei__Jiang→ USC

Research is important, but so is recharging! Took the team out for some badminton fun today—amazing energy, lots of laughs, and a reminder of how lucky I am to work with this crew!

Several collaborators expressed frustration, feeling that reviewers are excessively harsh. Might this behavior be influenced by the anonymity of the review process—similar to the dynamic we often observe on platforms such as Reddit?

Can a simulated society of AI agents be used to assess the effectiveness of social policies? See Abe Hou @abe_hou 's study in the context of "vaccine hesitancy" where we can use historical data for comparison and validation. arxiv.org/abs/2503.09639

Hello world!