One day remaining before this closes! bsky.app/profile/benj...

We tried to set up a simple demo/tutorial model for the protein design ecosystem we've been developing, and it turned out a bit more interesting than we expected. 🧵 This was a team effort from a few people in my lab, including @antonoresten.bsky.social and others (not sure who is on this app)

My lab, at Karolinska, in Stockholm, is looking for a PhD student with a computational/quantitative background to work on probabilistic/generative models of proteins (structure and sequence). The research will involve methods development, and applications in vaccine design.

Transformer/attention folks: I saw massive activations in Qwen's keys, always towards the end of each head, especially in layer 1. Turns out this is directly driven by the key projection bias (which Qwen has but eg. Llama3 does not). These large values are where RoPE has the slowest(?) effect. Why?