We're very excited to introduce TAPNext: a model that sets a new state-of-art for Tracking Any Point in videos, by formulating the task as Next Token Prediction. For more, see: tap-next.github.io

Introducing MegaSaM! Accurate, fast, & robust structure + camera estimation from casual monocular videos of dynamic scenes! MegaSaM outputs camera parameters and consistent video depth, scaling to long videos with unconstrained camera paths and complex scene dynamics!

I love SfM, but it's way less useful than it should be because of a handful of characteristic failures. @zhengqi_li's new paper basically solves them all: -No parallax? ✅ -No calibration? ✅ -Dynamic scenes? ✅ -Dense geometry? ✅ Best of all, it's super fast.

🎥 Introducing MultiFoley, a video-aware audio generation method with multimodal controls! 🔊 We can ⌨️Make a typewriter sound like a piano 🎹 🐱Make a cat meow like a lion roars! 🦁 ⏱️Perfectly time existing SFX 💥 to a video. arXiv: arxiv.org/abs/2411.17698 website: ificl.github.io/MultiFoley/

Quark is out! Come check our work on generalized realtime 3D reconstruction. quark-3d.github.io PS: We're looking for interns!

Stop watching videos, start interacting with worlds. Stoked to share CAT4D, our new method for turning videos into dynamic 3D scenes that you can move through in real-time! cat-4d.github.io arxiv.org/abs/2411.18613

Check out CAT4D: our new paper that turns (text, sparse images, videos) => (dynamic 3D scenes)! I can't get over how cool the interactive demo is. Try it out for yourself on the project page: cat-4d.github.io

We just dropped CAT4D, text to dynamic 3D models that you can render in real time. Not posting a video because Bluesky is garbage in this respect; go straight to the real time viewer on a desktop browser and look around. The cat kneading dough is my favorite. cat-4d.github.io

Our group at Google DeepMind is now accepting intern applications for summer 2025. Attached is the official "call for interns" email; the links and email aliases that got lost in the screenshot are below.