Heading to PyTorchCon in Paris to talk about open source agentic evaluations and Nemo Evaluator SDK. Come meet our team during our keynote & lighting talk 💬 Keynote 10:15: The Unbearable Lightness of (Agentic) Evaluations 💬 Talk 14:45 pm: The Science & Practice of Open and Scalable LLM Evaluations

What if the "unessential" things are really the most important? The emotional safety nets we create for each other, through art, presence & small acts of kindness, are just as vital as anything else. When the world feels heavy, these tiny fragments of beauty are what help us hold it all together.

🎉 Nemotron 3 Super was released today, as yet another excellent example of doing open-source first Machine Learning. This release acknowledges the importance of enabling the model developer community with end-to-end recipes for everything: data, training, evaluation, and support software.

Openness for Nemotron includes open evaluation! Eval transparency completes the last puzzle in the E2E repro lifecycle for LLMs. Check out our step-by-step blog on how to use the same pipeline we used for the evaluation of Nemotron 3 Nano through Nemo Evaluator: huggingface.co/blog/nvidia/...

Our team in Zurich is looking for a PhD research intern to join our research efforts on LLM accuracy and token efficiency evaluation & analysis: nvidia.wd5.myworkdayjobs.com/NVIDIAExtern...

Don’t miss this podcast🎙️with @jennwv.bsky.social and @hannawallach.bsky.social! They reflect on #WiML20Years and share their journey, collaborations, and advice in a special conversation on Microsoft Research’s Ideas podcast💙

Our team is presenting Nemo Evaluator SDK and CoDeC (Data Contamination Detection) this week at #NeurIPS2025 @neuripsconf.bsky.social. Reach out to Meriem Boubdir and Michał Zawalski if you want to talk about any of these or about future research internship roles and full-time positions.

I agree that emotional addiction to chatbots is the number one risk of AI today. Here is a gift link to an important OpEd in the NYTimes: www.nytimes.com/2025/11/17/o...

🧠⚙️ Interested in decision theory+cogsci meets AI? Want to create methods for rigorously designing & evaluating human-AI workflows? I'm recruiting PhDs to work on: 🎯 Stat foundations of multi-agent collaboration 🌫️ Model uncertainty & meta-cognition 🔎 Interpretability 💬 LLMs in behavioral science

💡 New research on studying data contamination. Key insight: LLMs leverage in-context examples differently when they have seen a benchmark during training vs. when the benchmark has never been seen in training. (1/N)

The blog post is available: blog.arxiv.org/2025/10/31/a...

NeMo Evaluator SDK — the platform we use at NVIDIA to benchmark LLMs, multimodal models, and agents — is now open source. It’s built for reproducibility, scalability, and transparency, with 100 + benchmarks across 18 open-source harnesses and full containerized execution. github.com/NVIDIA-NeMo/...

When to call it quits in LLM reasoning? 🛑 ‪Martina's internship project suggests trace monitoring metrics and classifiers that can detect when an LLM reasoning trace is going to fail in mid way. The approach saves up to 70% of token usage, and it even helps with increasing accuracy by 2%-3%.

Federal research funding works. It’s not an expense–it’s an investment. It’s not overhead–it’s a down payment on the future. - Eric Horvitz, Margaret Martonosi, Moshe Y. Vardi, and James Larus in CACM cacm.acm.org/opinion/keep... @erichorvitz.bsky.social

Our team in Zurich and EMEA is hiring Deep Learning Engineers for LLM Accuracy Evaluation and Analysis. Ideal candidates should have an inquisitive 🔬approach to evaluation and with best engineering practices for building reusable open source tools. www.linkedin.com/jobs/view/42...

The Diary of Anne Frank is among the hundreds of books banned in Florida this year. When I was in school, it was required reading. (Guardian)

The problem with chart crimes is not just the distortion of the y axis. It is the erasure of all other competitors from charts (hence they don’t exist), lack of error bars, lack of transparency in tools and code being used for evals…

I have a single question. Why doesn’t OpenAI compare with competitors in their evals? No Gemini, no Claude, no open source models…

hey wasn't this the same company that made a beautiful shiny "research" post about how AI evals should include error bars or something like that. or did they decide the CLT didn't apply here

New work from my team! arxiv.org/abs/2507.12950 Intersecting mechanistic interpretability and health AI 😎 We trained and interpreted sparse autoencoders on MAIRA-2, our radiology MLLM. We found a range of human-interpretable radiology reporting concepts, but also many uninterpretable SAE features.

If you're at @icmlconf.bsky.social this week, come check out our poster on "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" presented by the amazing @afedercooper.bsky.social from 11:30am--1:30pm PDT on Weds!!! icml.cc/virtual/2025...

📢 Webinar - 6/18 at 9am PST! Stop re-running complex recursive queries when your graph data changes. Feldera incrementally evaluates recursive graph computations. Learn to easily build these mechanisms with #SQL, without the hassle of constant recomputation. tinyurl.com/rb5my7d8

I only got to listen to this today. A lot of people in my network including myself have felt exactly this, for years. The fear that for some obscure reason, your paperwork and you may not be enough for this country, even in “normal” times. youtube.com/shorts/IF3bz...

We’ll be at the #Databricks Data + AI Summit in SF next week (6/9–12). If you’re around and want to chat about how incremental computing can make your #SparkSQL workloads go from hours to seconds — let’s connect. Grab some time here: calendly.com/matt-feldera... #DataAISummit #DataEngineering

Tired: "BS" Wired: "Vibe citing" www.nytimes.com/2025/05/29/w...

📌You can now find all the evaluation logs from our inference-time scaling report and the Phi-4 reasoning technical report at huggingface.co/datasets/mic.... The evaluation code for the reasoning benchmarks can also be found in the main branch of Eureka ML Insights at github.com/microsoft/eu....

www.scientificamerican.com/article/unde... at this point one just needs to cross their fingers and hope for more sanity.

🎉The Phi-4 reasoning models have landed on HF and Azure AI Foundry. The new models are competitive and often outperform much larger frontier models. It is exciting to see the reasoning capabilities extend to more domains beyond math, including algorithmic reasoning, calendar planning, and coding.

Re: The Chatbot Arena Illusion Every eval chokes under hill climbing. If we're lucky, there’s an early phase where *real* learning (both model and community) can occur. I'd argue that a benchmark’s value lies entirely in that window. So the real question is what did we learn?

All Eureka inference-time scaling insights are now available here: www.microsoft.com/en-us/resear... It was fun sharing these and more together with Vidhisha Balachandran @vidhishab.bsky.social and Vibhav Vineet at #ICLR2025.

Come see us in any of the following sessions on model understanding and evaluation! 🔬 #ICLR2025 @msftresearch.bsky.social

💡Eureka inference-time scaling insight (Day 8): Reasoning models improve more efficiently upon receiving feedback from themselves on their solutions than conventional models on the most complex tasks.

Complaint of the day is that we keep showing numbers on tiny datasets with no error bars. AIME 24 & 25 are ~30 examples each. In our experience, because of high non determinism, accuracy number vary a lot even within different experiments with 5 repeats. We need to do better!

💡Eureka inference-time scaling insight (Day 7): There exists untapped potential for improving both conventional models and reasoning models. All models, *including reasoning models*, are able to find a much better inference path when required to sample 5 answers for the same question (best of 5).

📰Our Github repo for the #ICLR2025 paper on Improving Instruction-Following in Language Models through Activation Steering is now public. Lots of handy representations for common format instructions to play with. 🛝Take a look github.com/microsoft/ll... @msftresearch.bsky.social

💡Eureka inference-time scaling insight (Day 6): It is still hard for developers to predict ahead of time how expensive a workload will be, without previous telemetry. This is rooted in inherent and high cost non-determinism associated with inference-time scaling.

💡Eureka inference-time scaling insight (Day 5): Higher token consumption does not always indicate higher accuracy across reasoning models. A reasoning model that spends more tokens is not necessarily the most accurate one on a given task.

💡Eureka inference-time scaling insight (Day 4): We introduce two NP-hard tasks in Eureka: Traveling Salesman minimal paths, and 3SAT Satisfiability for expressions with 3 literals. These benchmarks can be extremely useful to study how models solve very hard problems with controllable difficulty.

💡Eureka inference-time scaling insight (Day 3): Reasoning models show progress even for classical algorithmic problems as Traveling Salesman (optimal paths) or calendar planning. However, benefits diminish with higher complexity. In the hardest problems (TSP), models stop increasing the token length

💡Eureka inference-time scaling insight (Day 2): Despite the major updates, reasoning does not benefit all domains equally. E.g., most players report numbers on GPQA to show generalization. However, improvements in GPQA are driven by Physics, with Chemistry and Biology still visibly lagging behind.

💡Eureka inference-time scaling insight (Day 1): Reasoning models outperform conventional ones by a large difference, indicating a major update on the state of the art. They generalize to solve simple variants of algorithmic & planning problems as satisfiability, traveling salesman, calendar planning

💡Check out our latest Eureka analysis on the benefits of inference-time scaling. It studies 9 state-of-the-art models (conventional & reasoning) on 8 challenging tasks for math and STEM reasoning, calendar planning, NP-hard problems, navigation & spatial reasoning aka.ms/eureka-ml-insights-reasoning

I asked ChatGPT to generate images of sw engineers and house cleaners 4-5 times in separate chat windows. Then I did the same thing from a 2nd account to see if I would get some more diversity that way. The pictures speak for themselves, but I want to discuss why default representations matter ➡️

Let's draw step by step! GPT-4o image generation cannot yet follow instructions for image generation. At the same time, there are several aspects that have improved significantly when compared to Dall-E including spelling, fluent continued conversation, and some initial notion of feedback taking.

Bananas are also difficult fruits. #gpt4o #imagegeneration

Wrong but cute #gpt4o #imagegeneration. Cats are humanity's last exam 🐈‍⬛

www.dw.com/en/germany-u... According to DW, at least three German citizens (green card holders or tourists) have been detained or denied entrance to the US. Germany has updated their US travel advice. UK has also updated their travel advice.

Listen to @thegeeknarrator.bsky.social podcast! @feldera.bsky.social CEO, @lalithsuresh.bsky.social, discusses Feldera's approach to incremental view maintenance and the foundational breakthrough -- DBSP -- that powers it, engine internals, storage, handling late events. youtu.be/GpcIoABwF60?...

These words are now not welcome in scientific proposals for government funding in the US. Worse, they may/will trigger ungrounded rejection for funding. Now, how is one supposed to write a paper about the Great Barrier Reef, or Polarization of Light, or (god forbid) Women's Health & LGBTQ?

John Langford at @msftresearch.bsky.social wrote up this amazing essay Headroom for AI Development hunch.net?p=13763046 Take a look! In his own words: "It's quite interesting to compare biological and silicon capabilities."