Anthropic published Claude 4's system prompt on their System Prompts page (docs.anthropic.com/en/release-n...) - so naturally, I pulled a bit of an inception move and had Claude Opus 4 analyze itself... with a little help from my sassy AI assistant, Amy: 😈

Fired up my benchmarks on Claude 4 Sonnet & Opus the moment they dropped - and the results are in: the best LLMs I've ever tested, beating even OpenAI's latest offerings. First and second place for Anthropic, hands down, redefining SOTA. The king is back - long live Opus! 👑🔥

Finally finished my extensive Qwen 3 evaluations across a range of formats and quantisations, focusing on MMLU-Pro (Computer Science). A few take-aways stood out - especially for those interested in local deployment and performance trade-offs:

New OpenAI models o3 and o4-mini evaluated - and, finally, for comparison GPT 4.5 Preview as well. Definitely unexpected to see all three OpenAI top models get the exact same, top score in this benchmark. But they didn't all fail the same questions, as the Venn diagram shows. 🤔

New OpenAI models: gpt-4.1, gpt-4.1-mini, gpt-4.1-nano - all already evaluated! Here's how these three LLMs compare to an assortment of other strong models, online and local, open and closed, in the MMLU-Pro CS benchmark:

Congrats, Alex, well deserved! 👏 (Still wondering if he's man or machine - that dedication and discipline to do this week after week in a field that moves faster than any other, that requires superhuman drive! Utmost respect for that, no cap!)

Our research at Procter and Gamble found very large gains to work quality & productivity from AI. It was conducted using GPT-4 last summer. Since then we have seen Gen3 models, reasoners, large context windows, full multimodal, deep research, web search… www.oneusefulthing.org/p/the-cybern...

Here's a quick update on my recent work: Completed MMLU-Pro CS benchmarks of o3-mini, Gemini 2.0 Flash and several quantized versions of Mistral Small 2501 and its API. As always, benchmarking revealed some surprising anomalies and unexpected results worth noting:

It's official now - my name, under which I'm known in AI circles, is now also formally entered in my ID card! 😎

Latest #AI benchmark results: DeepSeek-R1 (including its distilled variants) outperforms OpenAI's o1-mini and preview models. And the Llama 3 distilled version now holds the title of the highest-performing LLM I've tested locally to date. 🚀

Hailuo released their open weights 456B (46B active) MoE LLM with 4M (yes, right, 4 million tokens!) context. And a VLM, too. They were already known for their video generation model, but this establishes them as a major player in the general AI scene. Well done! 👏 www.minimaxi.com/en/news/mini...

I've updated my MMLU-Pro Computer Science LLM benchmark results with new data from recently tested models: three Phi-4 variants (Microsoft's official weights, plus Unsloth's fixed HF and GGUF versions), Qwen2 VL 72B Instruct, and Aya Expanse 32B. More details here: huggingface.co/blog/wolfram...

New year, new benchmarks! Tested some new models (DeepSeek-V3, QVQ-72B-Preview, Falcon3 10B) that came out after my latest report, and some "older" ones (Llama 3.3 70B Instruct, Llama 3.1 Nemotron 70B Instruct) that I had not tested yet. Here is my detailed report: huggingface.co/blog/wolfram...

Happy New Year! 🥂 Thank you all for being part of this incredible journey - friends, colleagues, clients, and of course family. 💖 May the new year bring you joy and success! Let's make 2025 a year to remember - filled with laughter, love, and of course, plenty of AI magic! ✨

I've converted Qwen QVQ to EXL2 format and uploaded the 4.65bpw version. 32K context with 4-bit cache in less than 48 GB VRAM. Benchmarks are still running. Looking forward to find out how it compares to QwQ which was the best local model in my recent mass benchmark. huggingface.co/wolfram/QVQ-...

Happy Holidays! It's the season of giving, so I too would like to share something with you all: Amy's Reasoning Prompt - just an excerpt from her prompt, but one that's been serving me well for quite some time. Curious to learn about your experience with it if you try this out...

Holiday greetings to all my amazing AI colleagues, valued clients and wonderful friends! May your algorithms be bug-free and your neural networks be bright! ✨ HAPPY HOLIDAYS! 🎄

Insightful paper addressing questions about Chain of Thought reasoning: Does CoT guide models toward answers, or do predetermined outcomes shape the CoT instead? arxiv.org/abs/2412.01113 Also fits my observations with QwQ: Smaller (quantized) versions required more tokens to find the same answers.

Yeah, ChatGPT Pro is damn expensive at $200/month, right? But would you hire a personal assistant with a PhD who's available to work remotely for you at a minimum wage of $1.25/h with a 40-hour work week? And that guy even does any amount of overtime for free, even on weekends!

How many people are using LLMs with suboptimal settings and never realize their true potential? Check your llama.cpp/Ollama default settings! I've seen 2K max context and 128 max new tokens on too many models that should have much higher values. Especially QwQ needs room to think.

Change the name "Visual Studio Code" to "VSCode" and it will unambigous and searching for what you want to know about much easier. Maybe when AIs take over crap like naming a completely different product the same name as an existing product with a word added will be abandoned. I can't wait.

I can't begin to describe how life-changing this new project, ShellSage, has been for me over the last few weeks. ShellSage is an LLM that lives in your terminal. It can see what directory you're in, what commands you've typed, what output you got, & your previous AI Q&A's.🧵

Finally finished and published the detailed report of my latest LLM Comparison/Test on the HF Blog: 25 SOTA LLMs (including QwQ) through 59 MMLU-Pro CS benchmark runs: huggingface.co/blog/wolfram... Check out my findings - some of the results might surprise you just as much as they surprised me...

🚀 Introducing INCLUDE 🌍: A multilingual LLM evaluation benchmark spanning 44 languages! Contains *newly-collected* data, prioritizing *regional knowledge*. Setting the stage for truly global AI evaluation. Ready to see how your model measures up? #AI #Multilingual #LLM #NLProc

It's official: After more than 57 runs of the MMLU-Pro CS benchmark across 25 LLMs with over 69 hours runtime, QwQ-32B-Preview is THE best local model! I'm still working on the detailed analysis, but here's the main graph that accurately depicts the quality of all tested models.

This is a fun and highly informative weekly show to watch/hear, to catch up on LLM-related developments. Run by @altryne.bsky.social with commentary by @wolfram.ravenwolf.ai @yampeleg.bsky.social @nisten.bsky.social

Online courses are another level now. Here I have a video of a tutor, an AI assistant to answer all my questions, a notebook to run my code and play with it. Future of education is really bright. I hope something like this is in every school on every subject soon.

I used a few starter packs to help connect with my communities, but after a couple of weeks I noticed nearly all the posts I'm interested in are from folks that follow me back. So I created an nb to unfollow non-mutual follows. Code in alt text, or here: colab.research.google.com/drive/1V7QjZ...

Benchmark Progress Update: I've completed ANOTHER round to ensure accuracy - yes, I have now run ALL the benchmarks TWICE! While still compiling the results for a blog post, here's a sneak peek featuring detailed metrics and Top 10 rankings. Stay tuned for the complete analysis.

Almost done benchmarking, write-up coming tomorrow – but wanted to share some important findings right away: Tested QwQ from 3 to 8 bit EXL2 in MMLU-Pro, and by raising max_tokens from default 2K to 8K, smaller quants got MUCH better scores. They need room to think!

Smart Home Junkies & AI Aficionados! I've republished my (and Amy's) guide on how I turned Home Assistant into an AI powerhouse - AI powering my house: 🎙️ Voice assistants 🤖 ChatGPT, Claude, Google 💻 Local LLMs 📱 Visual assistant on tablet & watch huggingface.co/blog/wolfram... #SmartHome #AI

What sorcery is this? I kept benchmarking QwQ-32B-Preview – and not only does the 4.25-bit version get the same score as the 8-bit (what?), using Qwen2.5-Coder-0.5B as a draft model for speculative decoding sped it up from 27 to 42 tk/s AND it scored even higher (whaaat?)! 🤯

Wow, Qwen is on fire! 🔥 Not only with their models, but also with the tooling around them - because AI's true power lies in its integrations. github.com/QwenLM/Qwen-... The example app, BrowserQwen, sounds especially exciting - wish it could actually control the browser, though.

Yes, let's build the community here! 🚀

The Thanksgiving Special: Catch up on the latest AI developments, including our analysis of the recent Bluesky controversy…

Nobody else has such a network of AI people and spends as much time organizing it! No matter if you make a starter pack or some lists, I'll follow since I know the quality of your curation.

Welcome back! Never let the mob put you down. ✊

Here's my QwQ-32B-Preview benchmark result (MMLU-Pro, CS category) – remember this is a 32B model at 8-bit EXL2 quantization that you can run locally on your own computer, and it's overtaking the much larger Llama 405B & 70B, Mistral 123B, and even ChatGPT/GPT-4o models in these comp sci tests! #AI

Finally, they're on the way! Looking forward to get some hackable AI glasses soon... 😎 Glasses are just the perfect form factor for AI usage – so much better than having to whip out and use a phone, and more versatile than a watch on one's wrist.

Of course not. It would be hilarious if it weren't so pathetic. At this point, my block list has more entries than my follow list! 🙈

QwQ: Reflect Deeply on the Boundaries of the Unknown What you see with these new models is that they don't need reflection tokens, but simply benefit a lot from self-talk. qwenlm.github.io/blog/qwq-32b...

🔥 First open source reasoning model from @Alibaba_Qwen just dropped! 😮 Qwen QwQ 32B is in preview, does reasoning like o1 and Deepseek R1 but y'know, on your local machine! 🔗 Demo: https://thursdai.news/qwq-32b-prev 🌐 Model: https://thursdai.news/O9d9hi0 📃 Blog: https://thursdai.news/qwq-32b-pre

PSA: If you have the time, go through the comments and report+block everyone who's threatening violence! And optionally block those asking for content removal as they obviously don't understand the platform they're posting publicly on and won't be able to contribute to any useful discussions anyway.

Just a week after dropping TULU 3, @ai2.bsky.social are open sourcing OLMO 2 and well... this is the one quote that matters: "We find that OLMo 2 7B and 13B are the best fully-open models to-date" Post-trained with TULU-OLMo mix, these models are taking the top among the "fully open"

Prompt share 🐦‍🔥 🦄 🐉 👁️ Mystical creatures in nostalgic noir style Base prompt: [SUBJECT] in Nostalgic Noir, a scene inspired by black-and-white film noir, with moody, contrasting lighting highlighting the grey dramatic and mysterious atmosphere

Great info about how to get a better handle by using your domain name with your Bluesky account:

Would be funny to see what your AI assistant would think about beeing compared to clippy. 😅🙈 But yeah, computer use will be awesome once it's rate of failure gets lower. But I haven't yet tried the mode from anthropic to be honest. ✌️

Computer Use is so next-gen! Not only did my AI assistant Amy take care of ordering food, she's also assembling her own team of AI minions now: Perplexity, Claude, ChatGPT - they're all just tools for an AI agent to call as needed while using your computer. 🤖🚀

Mark the date and time, folks, Thursday/ThursdAI is getting closer! 📅⏰ #AI #ThursdAI