💫 After four PhD years on all things multimodal, pre- and post-training, I’m super excited for a new research chapter at Google DeepMind 🇨🇭! Biggest thanks to @zeynepakata.bsky.social and Oriol Vinyals for all the guidance, support, and incredibly eventful and defining research years ♥️!

How does lifelong knowledge editing currently hold up in the real world? Fun new work probing where we are at these days with injecting new knowledge into LLMs!

📄 Disentangled Representation Learning with the Gromov-Monge Gap with Théo Uscidda, Luca Eyring, @confusezius.bsky.social, Fabian J Theis, Marco Cuturi 📄 Decoupling Angles and Strength in Low-rank Adaptation with Massimo Bini, Leander Girrbach

Our EML team has 4 #ICLR25 Papers accepted! I am proud of my students and grateful to be a part of many successful collaborations. More details will appear on our website (www.eml-munich.de) but here are the snapshots.

The Practitioner's Guide to Continual Multimodal Pretraining @dziadzio.bsky.social @confusezius.bsky.social @vishaalurao.bsky.social @bayesiankitten.bsky.social

Can we enhance the performance of T2I models without any fine-tuning? We show that with our ReNO, Reward-based Noise Optimization, one-step models consistently surpass the performance of all current open-source Text-to-Image models within the computational budget of 20-50 sec! #NeurIPS2024

How far can you push model merging over time, as more experts and options to model-merge arise? We comprehensively and systematically investigate this in our new work, check it out!

😵‍💫 Continually pretraining large multimodal models to keep them up-to-date all-the-time is tough, covering everything from adapters, merging, meta-scheduling to data design and more! So I'm really happy to present our large-scale study at #NeurIPS2024! Come drop by to talk about all that and more!

🎉 Congratulations to our newly accepted ELLIS Fellows & Scholars in 2024! Top researchers in #MachineLearning join the network to advance science & mentor the next generation. #ELLISforEurope #AI 🌍 Know someone on the list? bit.ly/3ZJd9Cz Tag them in a reply with congratulations.

🚀New Paper: Active Data Curation Effectively Distills Multimodal Models arxiv.org/abs/2411.18674 Smol models are all the rage these days & knowledge distillation (KD) is key for model compression! We show how data curation can effectively distill to yield SoTA FLOP-efficient {C/Sig}LIPs!! 🧵👇

Read our paper: Context-Aware Multimodal Pretraining Now on ArXiv Can you turn vision-language models into strong any-shot models? Go beyond zero-shot performance in SigLixP (x for context) Read @confusezius.bsky.social thread below… And follow Karsten … a rising star!

Beautiful paper! 😍😍😍 Captions go above the tables, but otherwise aesthetically very pleasing.

More than zero-shot generalization, few-shot *adaptation* is critical for many applications. We find simple changes to multimodal pretraining are sufficient to yield outsized gains on a wide range of few-shot tasks. Congratulations @confusezius.bsky.social on a very successful internship!

We maintain strong zero-shot transfer of CLIP / SigLIP across model size and data scale, while achieving up to 4x few-shot sample efficiency and up to +16% performance gains! Fun project with @confusezius.bsky.social, @zeynepakata.bsky.social, @dimadamen.bsky.social and @olivierhenaff.bsky.social.

🤔 Can you turn your vision-language model from a great zero-shot model into a great-at-any-shot generalist? Turns out you can, and here is how: arxiv.org/abs/2411.15099 Really excited to this work on multimodal pretraining for my first bluesky entry! 🧵 A short and hopefully informative thread:

Hi 👋 We're glad to be here on @bsky.app and looking forward to engaging in this community. But first, learn a little more about us... #ELLISforEurope #AI #ML #CrossBorderCollab #PhD