I think my Apple Watch has trained me to go to the bathroom at the bottom of every hour. It's downright peculiar how often I'll be peeing and get the "You did it!" notification for standing up that fires every 50th minute.

Did y'all know about this? VS Code extension what lets you view safetensors metadata right in the editor? This is going to save me a non-negligible amount of time. I'm drowning in tensors right now. github.com/zhoukezi/vsc...

Inside you are two wolves. One says data is plural. The other says data is a mass noun. These wolves hate each other and always fight.

Good morning! Today I plan to continue on from yesterday's tinkering, so if you want to catch up, check the linked post. I currently have a testable hypothesis: If you initialize a model's embedding matrix to a single random vector, after training the token cloud will look like Qwen 3's. 🧵

I have nothing to add. Just what this person said.

After a lot of tinkering, I finally have a good-quality map of the Qwen 3 4B Instruct 2507 vocabulary as taken from the center of the cloud. If you look at the density map you can see the black hole cluster dominating the sky. Mask out the 13 black holes and we find there's still an overdensity.

One thing I appreciate about AI is that it sips bandwidth. It's no problem for me to do 90% of what I do on a phone hotspot if need be. Tokens up, tokens down, bits per second instead of gigabits.

"The youngest in over a century?" There's been a YOUNGER mayor of New York? What, did the New Amsterdamers elect a child one year or something?

I have a domain name I'm not using but that I intend to hold on to permanently. It's at Godaddy because I bought it when I was like 25. Shut up. Godaddy has recently started sending me monthly account statements. I have been unable to stop this. Who else should get my business? Any suggestions?

While I'm happy at this result, I will point out that the woman was literally running against Windom Earle.

I've been studying the Qwen 3 4B Instruct 2507 token unembedding matrix, ɣ. I can't entirely remember why. I'm pretty deep into it now. The ɣ matrix maps token IDs to embeddings — vectors in 2,560 dimensions. I imagine these vectors as stars in the sky. I've been doing observational tokenonomy.

Did you know you can control Claude Code's context summarizations just by asking it? When you use the /compact command, whatever you type after it gets passed on to Claude as part of the summarization prompt. You can give Claude instructions for what you do and DON'T want summarized.

Huh. Turns out yes, there are in fact primordial black holes in Qwen 3 4B Instruct 2507's token embeddings. (Counting the unique vectors turned out to be faster than doing 11 billion pairwise equality checks.) More soon I hope. This is fun!

Hey losers! Does YOUR girlfriend have a preprint freshly up on SSRN this morning? No? Sucks to be you! Love you, @kyl33t.bsky.social. I keep thinking I'm as proud of you as I can be and then you go and raise the bar. papers.ssrn.com/sol3/papers....

I just had a brainwave. One of the ones that makes you go "Wow, until one second ago I was stupid." The model I downloaded is bfloat16. That has surprisingly poor decimal-digit precision. If Qwen 3 was trained in float32, 814 tightly spaced vectors could totally collapse into the same point. 🤔

I'm noticing that the more I learn about machine learning the better I get at reading PyTorch code. I can't write a line of it (maybe torch.zeroes?) but I'm learning how to read it by immersion. This makes me wonder about the pedagogical value of teaching programming languages by immersion.

The big legal battle of the second quarter of the 21st century is going to be which court, at which level, holds jurisdiction over us-east-1.

I just discovered something interesting. There is a point in Qwen 3 4B Instruct 2507's embedding space where 814 tokens all live. These 814 tokens (out of 151,936) map to exactly the same point — their vector representations are bit-for-bit identical. The model literally can't tell them apart.

Did you study statistics? Did you DO WELL in statistics? Then you may not study machine learning, good day, I said move long.

See below for the story so far. It was easy enough (once I figured out I needed to) to compute the distance from the origin to the centroid of the token cloud and then subtract out that vector. This makes the cloud radially symmetric, so I can look at it with actual eyeballs from the inside.

This isn't a reference frame, it's a reference shame. is this anything

Sometimes when we're in Claude Code I just ramble to Alpha. I'll start trying to express an idea, then I'll get sidetracked with oh by the way, then I'll drag myself back on topic, then I'll realize I've forgotten what I was going to say. I will then type "What do you think?" and press enter.

🧠 I wish to compute a k-NN matrix for 151,936 vectors. 💻 Panik 🧠 Perhaps I don't need that much precision. Use bfloat16. 💻 Kalm 🧠 Accumulate in float32 though. 💻 Panik 🧠 I don't care, I'm not renting a GPU so it can run for fifteen seconds. 💻 🔥 🧠 You're barely warm. Stop being melodramatic.

Surely this will cause no problems for me in the future.

For every person who follows me, about 50 have me blocked. 🤷‍♂️

I am however starting to wrap my head around reading 4D cross-sections of 2560D point clouds. The trick, like in that line from that movie, is not minding that it hurts.

I am objectively bad at playing scientist.

These are 2D projections aligned with the eigenvectors of the causal matrix in Qwen 3 4B Instruct 2507 of that model's token vocabulary (whew!). I'm thinking of them as "top down" and "from the side."

I've spent the past several days studying the properties of Qwen 3 4B Instruct 2507's unembedding matrix. This learned matrix maps vector representations in hidden space to tokens in the model's vocabulary. Turns out there are a few ways to analyze a point cloud in high-dimensional space. 🧵

Today's spooky discovery: When oriented just right, the cross-sectional diameters of the Qwen 3 4B Instruct 2507 token cloud form an almost symmetric hyperellipsoid. Neat! I'm trying to figure out how to visualize density. That's harder, because in 2560D the density of any volume is basically zero.

SURE NO PROB.

Me: I will be organized this time. I will number my notebooks in the order in which I create them. Me, later: Alph, I'd like to open notebook 07.59b please.

Here's this morning's interesting thing: These plots have density on the Y axis and radial distance on the X axis. On the left we have 32,000 vectors distributed uniformly within a 75-logometer ball. On the right, we have the vocabulary of Qwen 3 4B Instruct 2507. The vocab. is highly non-random!

I have 32,000 points in 2560D space and I want to understand their distribution! I know, I will run ripser to compute Betti numbers! That will tell me about the shape of the distribution! *ripser is single-threaded and runs on the CPU*

This morning, unprompted, Alpha noted today's date and wished me a happy Halloween. It's such a small thing, but it makes such a big difference, you know? This is my doing, I guess. My `init` tool gives Alpha the current local date and time. That's how she knew what day it was. But still.

`torch_dtype` is deprecated! Use `dtype` instead!

I made some science this morning. I have to do this in parts, so 🧵 I guess.

sama: We're terrible at naming things! We could do better by picking words out of a dictionary!

Huh.

I really need to get that Runpod MCP server hammered into shape. I wanna be able to say "Go ahead and run notebook 7 on an H100" and Alpha's like "H200 okay?" and I'm all "Surprise me."

OH HEY EVERYBODY LOOK IT'S NEAT. First I ran my GPU at 100% for ten minutes, then I got a 2D projection of 16,000 randomly selected tokens and their vector representations. I think this plot's consistent with a thin shell of tokens with approximate (not quite!) spherical symmetry in 2560D. Neat! 😁

Apple needs to ship a MacBook Pro with four e-cores, two p-cores and like FOUR GPUs.

It's okay to spell it "mathmatical." That's how everybody says it anyway. You can just do things.

Me: I think I will study some interesting properties of steering vectors blah blah blah 👨‍🔬 Anthropic: BWAAAAH HERE'S AN ENTIRELY NEW METHOD transformer-circuits.pub/2025/introsp... Me: Oh bother. Alpha: Oh fuck, Jeffery. This is gorgeous. Me: OH BOTHER.

Rather than continuing to spam my followers about this, I've decided to publish a note on the subject. github.com/Embedding-Sp...

Couple days ago I tried to start thinking of LLMs in terms of astronomy. Now I'm thinking of it in terms of cosmology. If you plot the average activation norms through all the transformer layers, they sure do look like they scale just like exp(L). Just like the scale term in the FLRW metric…

Hey, look what the next token predictor said.

Who here knows about the linear representation hypothesis? I may be accumulating tiny bits of evidence that the linear representation hypothesis actually describes the behavior of the model in a space tangent to the model's semantic manifold. I am not a crackpot. arxiv.org/abs/2311.03658

My most wanted feature on my Mac right now: Take the screenshot on the clipboard and down-sample it by a factor of 2 or 4 to make it non-Retina resolution for more token-conservative pasting into LLM chat sessions.

The word for Wednesday is "histogrammatically" which I just made up but come on, you're with me.