Up next on stage, Dr. @edoardo-ponti.bsky.social ( @edinburgh-uni.bsky.social / NVIDIA) 🎤 “Adaptive Units of Computation: Towards Sublinear-Memory and Tokenizer-Free Foundation Models” Fascinating glimpse into the next gen of foundation models. #FoundationModels #NLP #TokenizerFree #ADSAI2025

🚀 By *learning* to compress the KV cache in Transformer LLMs, we can generate more tokens for the same compute budget. This unlocks *inference-time hyper-scaling* For the same runtime or memory load, we can boost LLM accuracy by pushing reasoning even further!

We propose Neurosymbolic Diffusion Models! We find diffusion is especially compelling for neurosymbolic approaches, combining powerful multimodal understanding with symbolic reasoning 🚀 Read more 👇

Sparse attention is one of the most promising strategies to unlock long-context processing and long-generation reasoning in LLMs. We performed the most comprehensive study on training-free sparse attention to date. Here is what we found:

🚀 Excited to welcome Dr. @edoardo-ponti.bsky.social to #ADSAI2025! Lecturer in NLP @edinburghuni.bsky.social , Affiliated Lecturer @cambridgeuni.bsky.social & Visiting Prof NVIDIA. 🎟️ Tickets for Advances in Data Science & AI Conference 2025 are live! 🔗Secure your spot: tinyurl.com/yurknk7y #AI

We created Approximate Likelihood Matching, a principled (and very effective) method for *cross-tokenizer distillation*! With ALM, you can create ensembles of models from different families, convert existing subword-level models to byte-level and a bunch more🧵

I have a scholarship for a PhD in efficient memory and tokenization in LLM architectures at @edinburgh-uni.bsky.social! Eligibility: UK home fee status Starting date: flexible, from July 2025 onwards. informatics.ed.ac.uk/study-with-u... Please contact me if you're interested!

Code and models for Dynamic Memory Compression are finally available! Stay tuned for architectures with even more efficient inference. developer.nvidia.com/blog/dynamic...

We're hiring a lecturer or reader in embodied NLP at the University of Edinburgh! Deadline: 31 Jan 2025 Call for applications: elxw.fa.em3.oraclecloud.com/hcmUI/Candid...

**Grounded typology**: a new paradigm. Traditionally, linguists posit functions to compare forms in different languages; however, these are aprioristic and partly arbitrary. Instead, we resort to perceptual modalities (like vision) as measurable proxies for function.

Two amazing papers from my students at #NeurIPS today: ⛓️💥 Switch the vocabulary and embeddings of your LLM tokenizer zero-shot on the fly (@bminixhofer.bsky.social) neurips.cc/virtual/2024... 🌊 Align your LLM gradient-free with spectral editing of activations (Yifu Qiu) neurips.cc/virtual/2024...

We had a blast at this year's @ellis.eu Dagstuhl seminar on "Modular and Agentive LLMs". Thanks everyone for participating!

Check out this piece on Strawberry 🍓/o1 we just authored on TheConversation! theconversation.com/ai-that-mimi... with @edoardo-ponti.bsky.social and Kolya 🚀

Several incredible NeurIPS tutorials this year. Worth navigating through the Swifties.

P.S. Make sure to follow @pnawrot.bsky.social!

Explore zero-shot routing of parameter-efficient experts with Phatgoose arxiv.org/abs/2402.05859 and Arrow arxiv.org/abs/2405.11157 w. github.com/microsoft/mttl 👉 github.com/sordonia/pg_mb… Part of "Dynamic Sparsity in ML" tut#neurips202424, feedback welcome and join for discussions! 😊

Last 5 days to apply for a PhD at #EdinburghNLP! Deadline: November 25 www.ed.ac.uk/studying/pos... If you are passionate about: - adaptive tokenization and memory in foundation models - modular deep learning - computational typology please message me or meet me at #NeurIPS2024!

Another nano gem from my amazing student Piotr Nawrot! A repo & notebook on sparse attention for efficient LLM inference: github.com/PiotrNawrot/... This will also feature in my #NeurIPS 2024 tutorial "Dynamic Sparsity in ML" with André Martins: dynamic-sparsity.github.io Stay tuned!