Morning shower thought: In the ICL era ('20-'23) the "statistically most likely text continuation" had some pull: if you see 10 examples of a pattern, the most likely continuation is one more example of that pattern. That's also how you get regurgitation or repetition collapse.

The LLM eras: 202x Pre-training (foundation) 2022 RLHF + PPO 2023 LoRA SFT 2024 Mid-Training 2025 RLVR + GRPO 2026 Inference-time scaling? 2027 Continual learning?

One thing I love is obsolete technology being encoded in our language. Words like “footage” and “rewind” and “dial (as a phone)” that literally describe a physicality that doesn’t exist anymore, yet lingers linguistically. Anybody have any other examples?

Will just leave this here

Saw someone confidently claim that LLMs are not neural networks. I think we have reached peak anti-AI derangement.

yeah basically you can't really use the site for anything remotely to do with ai unless you're an extremophile. there's a reason why most of the people who talk about ai the most are Combat Posters, anyone who does not thrive on conflict simply stops bothering to post at some point

On one hand, it's bad that Waymo went down all across the city. On the other hand, BART and Muni also shut down, and the Market St subway is still down this morning! As always, the problem is PG&E. We're just dealing with the consequences.

young people who want to learn have been given such a gift. I hope they know it

the funniest thing about bluesky's anti-AI bent is that it keeps running off machine learning researchers and academics rather than, you know, the people they claim to hate ('AI bros')

short bearded man at a podium: There are known gnomes, known un-gnomes, and unknown un-gnomes

Ike’s Love and Sandwiches, Mike’s Hatred and Calzones

And yet still better than Comcast

Like, could I have spent many hours learning how to paint a beard onto Mark Kelly? Sure! Instead, I asked AI to do it. Would the effect have been the same if I used mspaint to color his jawline gray? No. Am I an artist? No. Was I trying to be one? Also no. bsky.app/profile/rive...

Pantone just updated their Color Of The Year

According to all known laws of philosophy, there is no way an LLM can know something. Its model is too statistical to encode real knowledge. The LLM, of course, outputs the correct answer anyway. Because LLMs don't care what humans think is impossible.

CATO INSTITUTE: Despite DOGE slashing jobs, government spending in every month of 2025 “was greater than in every other year .. An observer who did not know when DOGE started could not identify it” on a chart. 🤡 www.cato.org/blog/doge-pr...

I’m a little rusty on this but churches have tickets now?

Alright, landed in SF. It was a pleasure to spend the last 10 or so hours with you all (despite United's best efforts). Now time for some Christmas decorating!

Yay, time for the best part of any long distance United flight, the pizza twist right before landing

This used to be conventional wisdom - and all commercial chat systems have tool calling with some sort of code interpreter. But LLMs can, in fact, do basic math. GSM8k is so saturated I cannot even find results from this year's models. Last year's are above >97%. It's not 2023 anymore!

tbh I think that the backbone of populism is that the fact that society has become so complicated it's very difficult for any single person to understand it. The modern world is a funhouse mirror nightmare for everyone who doesn't' have a boundless curiosity for everything

Being active on social media either gives you superhuman patience, or makes you go completely insane. No in between.

Slowly creating a list of Econ/PolicySky people that I will add to: (if you are in either and a mutual, let me know) bsky.app/profile/did:...

I'm pretty sure that at this point in time, the remaining opposition to renewables are nothing but the tantrums of those with interest in fossil fuels and/or could not get the logic of "there's no cost to obtaining the fuel" to sink into the brain.

I think I'm guilty of this myself. I've noticed groceries going up a lot in the last few months (esp beef), but meanwhile I'm still paying the same for internet that I was a decade ago, for more data, and there's probably a lot more examples of that.

One reason AI is throwing people off is that we built something "intelligent" but not "alive" in any meaningful sense of the word. Nothing else works like that - not animals, not humans. We expect intelligence to mean life, and it clearly doesn't. (This thought brought from watching Blade Runner)

I think this strengthens the case for Stancilism. Immediate post pandemic economy was *weird* in a way that made it hard to compare it to similar years. But we've got low unemployment and low inflation now. Very hard to make the case that now is worse than the Volcker shock.

So much political commentary/analysis comes down to whether you see 2024 as: 1) A sweeping and lasting change, signaling deeply and widely held issue beliefs, practically a binding referendum on American culture Or 2) The fourth close presidential election in a row, and impacted by global trends

Is Mitt a maker or a taker now? www.nytimes.com/2025/12/19/o...

One thing you learn working in a statistical field like ML is to "smell" when the data is wrong. Measurement error, confounders, baseline effects, sampling issues... Data doesn't match the intuition = most of the time, intuition is wrong. But sometimes...

I saw some commentary on a publisher adopting AI for translation, and I wonder if late medieval clergy reacted the same to the printing press. After all, a lot of modern English spelling mess came from sloppy Renaissance-era publishers! Yet you wouldn't argue against the printing press, would you?

No signs of an end to rapid gains in AI ability at ever-decreasing costs (which is a log scale) yet. I have to update this monthly or more frequently at this point. All AI benchmarks are flawed, but GPQA Diamond has been a pretty good one, though likely close to being maxed out.

Case in point: Gallego vs Kelly

the transformer model is not going to spontaneously uninvent itself. much consternation from this fact

Students learn early that correlation does not imply causation. Correct, but incomplete... Certain patterns of independence and conditional dependence constrain #causal structure very strongly. A simple example:

this labeler tells you a bunch of useful things but the killer app is the flag for if somebody’s caught 20+ blocks in a 24-hour period and I never want to be without it again

this is just a hypothesis on my part, but I also think this would change the nature of the interaction in a way that would dissuade (not make impossible) AI psychosis a little. the guy who compliments you for free is different than the guy you have to pay a dime to compliment you.

but my ask of you is to look at what protections the rest of the world have adopted in place of the blanket s230, and if you're talking to your legislators about this, point out that in order to repeal s230 they'd need to adopt those other measures too, and does the proposal do that, etc.

It was an actual joke that people made in 2016! www.vox.com/culture/2016...

Nothing Merrick Garland could’ve done would’ve stopped the Supreme Court majority from putting Trump above the law. Doesn’t mean Garland is above criticism. Hardly. But it does mean there was no move—no matter how fast—no strategy, no One Weird Trick. SOCTUS would’ve done whatever they needed to.

Is nobody even trying things before claiming AI can't do something? One of my main uses for LLMs is analyzing a codebase and telling me about, answering my questions about it, and finding things deep inside big ones that would take forever to find. And they've been good at this for a while now

Given their reputation for inaccuracy it feels a little absurd to ask an LLM to fact check something, but I've been getting similar high quality results from the best search-tool-using models recently for the prompt "fact check this"

wow, this paper really did make me think differently about the president’s pardon power, well worth reading

Compromise position: get rid of gender, introduce pronouns with de Bruijn indices

What happened is that 1) SFT and RL actually worked much better at that size (FLAN, InstructGPT) 2) There was still a lot of data around, and actually the data/param mix could be improved (Chinchilla) 3) The most unintuitive thing. Language pretrain was enough to build a good world model (GPT4)

vogt asks schake why so many americans have visibly soured on the liberal international order. she responds that a succession of presidents have failed to sell it to the american people to disagree, respectfully: you can't understand the crisis without iraq, afghanistan, and china

They invented Rome and suddenly a whole lot of roads appeared

It wouldn't be bad per se, but it would be a sign that we're still in normal thermostatic politics, rather than the generational realignment that a lot of people have predicted.

Yeah it's weird how the US insists that college should be a continuation of high school (including treating 20 year olds as "kids") You shouldn't have Gen Ed in college, that's what high school is for!