happy year of the THIS GUY!! 🐍

I'll get straight to the point. We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff. It's much faster, more accurate, longer context, and more useful. 🧵

Random opinion on modern love, we need to normalize - breaking up & being friend - dating friend - friend groups being chill & supportive, with all of it (getting together, or breaking up) Starting romantic relations, without knowing your partner as a person is weird to me bias: I married a friend

My personal high belief / high conviction hot take: 32-72B is all we need for human level of AGI 🤖 Anything higher is just us being inefficient in architecture / code / etc #NeurIPS2024

Lots of people are only looking out for themselves which is why they fear everything.

PS: once we get 70B converted and working at ~128k context length We will be able to cover the vast majority of enterprise AI workloads without QKV attention Let that sink in 😎

QKV Attention is **not** all you need We release QRWKV6-32B-Instruct preview, a model converted from Qwen-32B instruct, trained for several hours on 2 MI300 nodes. Surpassing all previous known open linear models (StateSpace, Hybrid, etc) Unlocking 1000x+ lower inference cost

So yea, we just finished up the best subquadratic model for our QRWKV varient. Landing in hot during neurips Matching transformer level performance despite the lack of "Quadratic Attention", using RWKV Attention instead Proving Attention is **not** all you need

Did the RNG bless your NeurIPS lottery? Are you in SF? Wish to nerd out on AI & ML? Join our holiday get together + community potluck + discussion of the best new AI research with me, @swyx.io @eraqian @dylan522p @vibhuuuus 😎 At 4pm today lu.ma/25mwbwcm

Did you hear that joke about Ice Cream? Best I keep that frozen for awhile!

Repeat after me: I will build evals for my tasks. I will build evals for my tasks. I will build evals for my tasks.

This is such an interesting angle, because the number of large organizations that have this level of discipline to evaluating new models is likely to be tiny Running a bunch of test prompts specific to what your company does through a new model feels like it should be pretty low hanging fruit

A test of how seriously your firm is taking AI: when o-1 (& the new Gemini model) came out this week, were there assigned folks who immediately ran the model through your internal, validated, firm-specific benchmarks to see how useful it as? Did you update any plans or goals as a result?

I'm heading to #NeurIPS next week, I'll be in town Wednesday-Friday! I'll be at a couple things: - Wed 1-2pm: talking Transformer killers with @picocreator.bsky.social at @swyx.io @latentspacepod.bsky.social live! - Wed 11am: RedPajama poster (spotlight) with Maurice Weber 1/2

Just landed in Toronto 🇨🇦 Me as South East Asian: oooo… snow ☃️ My Canadian friends: that’s barely any snow ❄️ The true Canadian experience needs to have at least knee high snow I guess 🤣

For those in SF 🌉 before Neurips (8th Dec), doing a casual pre-neurips gathering; Food, Friends, Papers. To chill and talk, before flying over to Canada 🇨🇦 lu.ma/25mwbwcm

So what does one do, if they use more then 1TB of storage on huggingface? Seems like pro (~$9/month) is limited to 1TB?

So i heard ChatGPT is looking into adding ads now ... ( the circle of internet life )

Looking at what Qwen and Deepseek achieved at way less then 1/10th the budget (or even 1/100th) The biggest damage big AI did to western AI - was to convince everyone that, only they can build AI with more funding - and that no one else can Denying competition and progress locally

and guests @alpindale.bsky.social @capetorch.bsky.social and @picocreator.bsky.social for a great show today! Give them all a follow folks 👆 P.S - A shoutout to @presidentlin.bsky.social for helping as always

Over the past 18 month, the @latentspacepod.bsky.social paper club has had an unbroken streak of hosting EVERY SINGLE WEEK. We gained technical knowledge + insider know-how, built friendships, and grown a community of learners. Here's how to start your own paper club eugeneyan.com/writing/pape...

I have only one goal this #OpenAI #devday To help build up the Singapore AI community And somethings are looking promising (🇸🇬/acc)

@eugeneyan.bsky.social Turning this into a perpetual thread.... for every time i hear "LLM-as-a-judge" or something "similar" being discussed Today Oct 30 : Heavybit devguild AI Summit 3 event

Hello bluesky, my twitter is : x.com/picocreator And I guess I should start cross posting here as well