pytorch 2.7 is out! - Mega Cache looks nice, unclear whether those modules get cached by legacy mechanisms too - foreach map looks good for optimizers - trainable biases means you can now train T5 on flex - prologue fusion is hype - context parallel brings ring attention pytorch.org/blog/pytorch...

pytorch 2.6 is out! highlights: - flex attention: better compilation of blockmask creation, better support for dynamic shapes - cuDNN SDPA: fixes for memory layout - CUDA 12.6 - python 3.13 - MaskedTensor memory leak fix

pytorch 2.6 final RC is out, promoting to stable in a couple of days! mostly I'm looking forward to better compilation of flex block mask creation, and better support for flex attention on dynamic shapes. there's also fixes for memory layout in cuDNN SDPA. dev-discuss.pytorch.org/t/pytorch-re...

Claude does SVG memes

drink cups should put the hole in the bottom. heat rises. "the top is cool enough to drink" implies "everything below it is colder". drinking from the bottom lets us access safe temperatures earlier and before the whole cup cools.

running npm version from a subdirectory of a git repository is literally an unsolved problem in 2024 github.com/npm/cli/issu...

I should just get this tattooed, I never remember how to find it pip install huggingface_hub[hf_transfer] HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download

when the standard library comments out std::experimental::observer_ptr just to stop you having fun

NovelAI v4 makes dreams come true

when you're measuring torch compile warmup "oh 11 secs that's not so bad" then you realize it was 111 secs

Claude's alright

if you care about multiprocess debugging in VSCode please upvote this issue so we don't have to click terminate a hundred times github.com/microsoft/vs...

Meta releases flow-matching code github.com/facebookrese...

the EDM2 repository had an Autoguidance update 4 days ago demonstrating their NeurIPS Oral paper "Guiding a Diffusion Model with a Bad Version of Itself" github.com/NVlabs/edm2

torch profiler record_function spans are not free, even when you're not profiling a model. the model I'm benchmarking trained 2.5% faster when I commented them all out.

Box2D 3.0.0 in WebAssembly+TypeScript starting to work

🤔 Why do we extract diffusion features from noisy images? Isn’t that destroying information? Yes, it is - but we found a way to do better. 🚀 Here’s how we unlock better features, no noise, no hassle. 📝 Project Page: compvis.github.io/cleandift 💻 Code: github.com/CompVis/clea... 🧵👇

torch.compile is hard for dynamic shapes / large number of static shapes, and non-transformer architectures. I measure suites of shapes, log recompiles, check which require warmup. operation compile competitive with whole-model compile. some operations prefer compiler disabled. dynamic often slow.

A common question nowadays: Which is better, diffusion or flow matching? 🤔 Our answer: They’re two sides of the same coin. We wrote a blog post to show how diffusion models and Gaussian flow matching are equivalent. That’s great: It means you can use them interchangeably.

mood: adding unused variables to make torch inductor compile my triton kernel

that feel when you’ve been cooking rice for 10 minutes but the hob wasn’t turned on

fine I'll use pytorch nightly

born too late to learn maths from touhou pre-fight cutscenes www.youtube.com/watch?v=tuDA...

if you get KO'd in smash, do you die? what's the safest stage to be KO'd on? Great Bay looks alright if you're a confident swimmer…

no_grad in the streets, inference_mode in the sheets

oh no I set edgeitems too high and now pytorch is stuck

good algorithm

NATTEN just added fused support for self-cross attention! so you can attend to local neighbourhood and registers or text condition. it lets you reduce partial attention results (e.g. logsumexp provided by xformers APIs) into its LSE. github.com/SHI-Labs/NAT...

torch stable (even 2.5.1) has a bug with foreach operations in FSDP. AdamW optimizer is only safe in foreach=False or fused=True modes. foreach operations that take scalar operands are fine. github.com/pytorch/pyto...

pytorch 2.5.0 bug: counting flops makes your compiled model slower. benchmark your model first, count flops after. github.com/pytorch/pyto...

bluesky is trending on the other place

hol' up lemme just install this operating system real quick $ which ffmpeg ffmpeg not found $ brew install ffmpeg ==> Auto-updating Homebrew...

fuck offfff

okay where has tensor_split been my whole life pytorch.org/docs/stable/...

when it's Sunday and you realise you've been sitting on your breakpoint since Friday

the reason stable-diffusion generalizes poorly to untrained resolutions is its implicit position embedding. convolution padding creates an edge, which nested convolutions look for to understand position. arxiv.org/abs/2101.12322

she just showed up and asked where I keep the grimoires