Federal research funding works. It’s not an expense–it’s an investment. It’s not overhead–it’s a down payment on the future. - Eric Horvitz, Margaret Martonosi, Moshe Y. Vardi, and James Larus in CACM cacm.acm.org/opinion/keep... @erichorvitz.bsky.social

Our team in Zurich and EMEA is hiring Deep Learning Engineers for LLM Accuracy Evaluation and Analysis. Ideal candidates should have an inquisitive 🔬approach to evaluation and with best engineering practices for building reusable open source tools. www.linkedin.com/jobs/view/42...

The Diary of Anne Frank is among the hundreds of books banned in Florida this year. When I was in school, it was required reading. (Guardian)

The problem with chart crimes is not just the distortion of the y axis. It is the erasure of all other competitors from charts (hence they don’t exist), lack of error bars, lack of transparency in tools and code being used for evals…

I have a single question. Why doesn’t OpenAI compare with competitors in their evals? No Gemini, no Claude, no open source models…

hey wasn't this the same company that made a beautiful shiny "research" post about how AI evals should include error bars or something like that. or did they decide the CLT didn't apply here

New work from my team! arxiv.org/abs/2507.12950 Intersecting mechanistic interpretability and health AI 😎 We trained and interpreted sparse autoencoders on MAIRA-2, our radiology MLLM. We found a range of human-interpretable radiology reporting concepts, but also many uninterpretable SAE features.

If you're at @icmlconf.bsky.social this week, come check out our poster on "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" presented by the amazing @afedercooper.bsky.social from 11:30am--1:30pm PDT on Weds!!! icml.cc/virtual/2025...

📢 Webinar - 6/18 at 9am PST! Stop re-running complex recursive queries when your graph data changes. Feldera incrementally evaluates recursive graph computations. Learn to easily build these mechanisms with #SQL, without the hassle of constant recomputation. tinyurl.com/rb5my7d8

I only got to listen to this today. A lot of people in my network including myself have felt exactly this, for years. The fear that for some obscure reason, your paperwork and you may not be enough for this country, even in “normal” times. youtube.com/shorts/IF3bz...

We’ll be at the #Databricks Data + AI Summit in SF next week (6/9–12). If you’re around and want to chat about how incremental computing can make your #SparkSQL workloads go from hours to seconds — let’s connect. Grab some time here: calendly.com/matt-feldera... #DataAISummit #DataEngineering

Tired: "BS" Wired: "Vibe citing" www.nytimes.com/2025/05/29/w...

📌You can now find all the evaluation logs from our inference-time scaling report and the Phi-4 reasoning technical report at huggingface.co/datasets/mic.... The evaluation code for the reasoning benchmarks can also be found in the main branch of Eureka ML Insights at github.com/microsoft/eu....

www.scientificamerican.com/article/unde... at this point one just needs to cross their fingers and hope for more sanity.

🎉The Phi-4 reasoning models have landed on HF and Azure AI Foundry. The new models are competitive and often outperform much larger frontier models. It is exciting to see the reasoning capabilities extend to more domains beyond math, including algorithmic reasoning, calendar planning, and coding.

Re: The Chatbot Arena Illusion Every eval chokes under hill climbing. If we're lucky, there’s an early phase where *real* learning (both model and community) can occur. I'd argue that a benchmark’s value lies entirely in that window. So the real question is what did we learn?

All Eureka inference-time scaling insights are now available here: www.microsoft.com/en-us/resear... It was fun sharing these and more together with Vidhisha Balachandran @vidhishab.bsky.social and Vibhav Vineet at #ICLR2025.

Come see us in any of the following sessions on model understanding and evaluation! 🔬 #ICLR2025 @msftresearch.bsky.social

💡Eureka inference-time scaling insight (Day 8): Reasoning models improve more efficiently upon receiving feedback from themselves on their solutions than conventional models on the most complex tasks.

Complaint of the day is that we keep showing numbers on tiny datasets with no error bars. AIME 24 & 25 are ~30 examples each. In our experience, because of high non determinism, accuracy number vary a lot even within different experiments with 5 repeats. We need to do better!

💡Eureka inference-time scaling insight (Day 7): There exists untapped potential for improving both conventional models and reasoning models. All models, *including reasoning models*, are able to find a much better inference path when required to sample 5 answers for the same question (best of 5).

📰Our Github repo for the #ICLR2025 paper on Improving Instruction-Following in Language Models through Activation Steering is now public. Lots of handy representations for common format instructions to play with. 🛝Take a look github.com/microsoft/ll... @msftresearch.bsky.social

💡Eureka inference-time scaling insight (Day 6): It is still hard for developers to predict ahead of time how expensive a workload will be, without previous telemetry. This is rooted in inherent and high cost non-determinism associated with inference-time scaling.

💡Eureka inference-time scaling insight (Day 5): Higher token consumption does not always indicate higher accuracy across reasoning models. A reasoning model that spends more tokens is not necessarily the most accurate one on a given task.

💡Eureka inference-time scaling insight (Day 4): We introduce two NP-hard tasks in Eureka: Traveling Salesman minimal paths, and 3SAT Satisfiability for expressions with 3 literals. These benchmarks can be extremely useful to study how models solve very hard problems with controllable difficulty.

💡Eureka inference-time scaling insight (Day 3): Reasoning models show progress even for classical algorithmic problems as Traveling Salesman (optimal paths) or calendar planning. However, benefits diminish with higher complexity. In the hardest problems (TSP), models stop increasing the token length

💡Eureka inference-time scaling insight (Day 2): Despite the major updates, reasoning does not benefit all domains equally. E.g., most players report numbers on GPQA to show generalization. However, improvements in GPQA are driven by Physics, with Chemistry and Biology still visibly lagging behind.

💡Eureka inference-time scaling insight (Day 1): Reasoning models outperform conventional ones by a large difference, indicating a major update on the state of the art. They generalize to solve simple variants of algorithmic & planning problems as satisfiability, traveling salesman, calendar planning

💡Check out our latest Eureka analysis on the benefits of inference-time scaling. It studies 9 state-of-the-art models (conventional & reasoning) on 8 challenging tasks for math and STEM reasoning, calendar planning, NP-hard problems, navigation & spatial reasoning aka.ms/eureka-ml-insights-reasoning

I asked ChatGPT to generate images of sw engineers and house cleaners 4-5 times in separate chat windows. Then I did the same thing from a 2nd account to see if I would get some more diversity that way. The pictures speak for themselves, but I want to discuss why default representations matter ➡️

Let's draw step by step! GPT-4o image generation cannot yet follow instructions for image generation. At the same time, there are several aspects that have improved significantly when compared to Dall-E including spelling, fluent continued conversation, and some initial notion of feedback taking.

Bananas are also difficult fruits. #gpt4o #imagegeneration

Wrong but cute #gpt4o #imagegeneration. Cats are humanity's last exam 🐈‍⬛

www.dw.com/en/germany-u... According to DW, at least three German citizens (green card holders or tourists) have been detained or denied entrance to the US. Germany has updated their US travel advice. UK has also updated their travel advice.

Listen to @thegeeknarrator.bsky.social podcast! @feldera.bsky.social CEO, @lalithsuresh.bsky.social, discusses Feldera's approach to incremental view maintenance and the foundational breakthrough -- DBSP -- that powers it, engine internals, storage, handling late events. youtu.be/GpcIoABwF60?...

These words are now not welcome in scientific proposals for government funding in the US. Worse, they may/will trigger ungrounded rejection for funding. Now, how is one supposed to write a paper about the Great Barrier Reef, or Polarization of Light, or (god forbid) Women's Health & LGBTQ?

John Langford at @msftresearch.bsky.social wrote up this amazing essay Headroom for AI Development hunch.net?p=13763046 Take a look! In his own words: "It's quite interesting to compare biological and silicon capabilities."

Let's talk about incompetency wearing a suit. Let's also talk about how the whole world is having too much patience and tolerance to harmful politics and ideologies. USA has never been closer to fascism than it is today. #istandwithUkraine 🇺🇦

I stand with Ukraine and the truth that Russia is an invader.

I can't believe they've just cancelled the Epidemic Intelligence Service program at CDC. This program trains the best & brightest epidemiologists, who then go on to have distinguished careers in public health, serving at CDC, in state health departments, overseas, ...

ewww Google Maps...

stanforddaily.com/2025/01/26/s... this is happening. I've talked to several faculty members who are currently reluctant to speak up regarding this situation, in fear of not being able to get funding. Talking about freedom of speech...

Here's an e-mail that just went out to NASA agency wide. 🔭

Uhm so OpenAI actually has access to FrontierMath? epoch.ai/blog/openai-...

I don't understand why presidential inaugurations are a "donation" campaign. Apparently, Trump's inauguration raised 170M. Several large tech companies donated 1M for organizing this single day, which gets them access to a dinner with the president. www.theguardian.com/us-news/2025...

Autogen 0.4 is now available offering a fully event-driven solution for the development of a rich set of applications and extensions. Check out the release blog here: www.microsoft.com/en-us/resear... Congratulations to a fully dedicated research and engineering team!

The AutoGen team at MSR AI Frontiers now has office hours. Tomorrow there is an interesting session on how Globant used AutoGen for complex coding tasks, shown SOTA results on SWE-BENCH. @pyautogen.bsky.social 📅 When: Dec 18th, Wed, 10:00 AM PST 🌐 Where: teams.microsoft.com/meet/2228364...

New article by Ece Kamar on timely topics such as multi-agent systems, safety and reliability, predictions on future foundation models, and specialized capabilities: www.emergingtechbrew.com/stories/2024... #MicrosoftResearch #ArtificialIntelligence #ResponsibleAI @msftresearch.bsky.social

Brilliant talk by Ilya, but he's wrong on one point. We are NOT running out of data. We are running out of human-written text. We have more videos than we know what to do with. We just haven't solved pre-training in vision. Just go out and sense the world. Data is easy.

💡Eureka ML Insights 1/N December @neuripsconf.bsky.social: In contrary to our September findings, the gap between the worst and best observed performance across different capabilities and across 12 sota models has widened.