@chhillee.bsky.social's talk at Jane Street is now up! youtu.be/139UPjoq7Kw?...

Getting different attention masks working for AstroPT (a proto-foundation model for astronomy github.com/Smith42/astr...), so much nicer to do it with Flex Attention vs custom CUDA kernels -- thank you for releasing it to the world 🫡

I judge social networks by how many FlexAttention users I can find on each one, and by that metric, Bluesky is doing pretty good!

If you'd like to influence what features the PyTorch distributed team work on in torchtitan (e.g. MoE, multimodal, context parallelism, etc.), go made your voices heard here!

First thought: Seems kinda "FlexAttention-y": https://bsky.app/profile/sungkim.bsky.social/post/3lbjbfmyqts27 Second thought: oh cool, they're already using FlexAttention! it's a nice usage of the `or_masks` and `and_masks` API - I think they do (causal & sliding_window) | (register_mask)