This is one of my all time favorite papers: openreview.net/forum?id=ByJ... It shows that, under fair experimental evaluation, lstms do just as well as a bunch of “improvements”

Really fun conversation with @natolambert.bsky.social!

Apparently there *is* another finbar(r) in Alberta.

New homeowner fear unlocked; someone hit and ran my neighbor’s garage

there’s a type of “not trying” which means not executing at the level of competence of a $XX billion corporation this is the complaint about eg Google products. They’re good! better than most startups! But not “trillion dollar corporation famed for engineering expertise” good.

I watched too many ski movies and now am trying to convince my wife we should move to Alaska

building my own mlp implementation from scratch in numpy, including backprop, remains one of the most educational exercises I’ve done

Love this. Very clean implementations of various inference optimizations.

Agreed! Folk knowledge is worth publishing!

my latest article for Artificial Fintelligence is up. i cover the evolution of Vision Language Models over the past few years, from complex architectures to surprisingly simple & effective ones. (link in next tweet)

the tech industry is like California: it gets a few of the most important things right, so it can make a lot of other mistakes and still be wildly successful

the remarkable success of the Google brain (and OpenAI) resident programs is an indication to me that smart, hardworking people can do more than you expect

My favorite thing about Bsky so far is not having the dumb algo requirements. Link in reply and not mentioning substack is stupid!

for all the work we researchers do, the best way to improve your model by far is to 1) use better data and 2) use higher quality data

If I was doing a phd, this would be one of my top choices for programs.

active learning is top of my list of "things that seem like they should work but don't" I haven't had much success when I actually implement it

Why does DeepSeek have so many GPUs? (Purportedly >10k H100s). Is this useful to their main hedge fund business?

Tulu is very exciting!

Deepseek r1 seems really good

If I was in a position to direct significant research resources/headcount I’d be putting a significant effort behind better exploration in RL.

honestly if houses in Whistler were less expensive I’d be way less motivated to earn money the prospect of retiring there motivates a non-trivial amount of my life

the amount of time I spend thinking about skiing rn is ridiculous

ok but seriously at some point we really need to solve exploration in RL

galaxy brain take: Google no moats memo is a psyop and a no-downside shot against openAI It’s a little too flattering to the conceits of open source etc

chain of thought reasoning keeps racking up the Ws

my thesis is that there’s gonna be one foundation model company for each major cloud provider that they either partner with or acquire, and these are the only companies that make money Azure: OpenAI Amazon: Stability (?) GCP: Anthropic/DeepMind

the most useful thing I’ve done in my career as a research engineer is to slowly build a bag of tricks and to try them on all the problems I come across

active learning seems like what everyone should be doing Particularly in RL

ok emergent properties of LLMs have nothing on emergent properties of babies my son just learned to say dada, very exciting

copilot makes it so easy to comment my code & write docstrings my coworkers should be paying github

the fact humans haven’t eliminated mosquitoes from the earth is the best argument against AI foomerism

not that “tweeted” is particularly dignified, but “as I skeeted yesterday” is egregiously bad

is anyone actually using cerebras in production

in the pnw they use greysky instead

skoot skoot mahout

there are so many aspects of productionizing ML models we know nothing about

it’s crazy how little LLM experience you need to be an “expert”

Ok American Jesus ads during the playoffs are a new experience

American fast food ads are so much more enticing than Canadian ones

how hard could it be to start a semiconductor company

I keep thinking about parameter quantization for large models I have periods where I’m convinced it’s the future of inference, and then other periods where m I’m convinced it doesn’t do anything Any anecdata out there? 👀