How is AI helping robots to generalise their skills to unfamiliar environments? 🤖 🏠 In the latest episode, I chatted to Prof. Lerrel Pinto (@lerrelpinto.com) from New York University about #robot learning and decision making. Available wherever you get your podcasts: linktr.ee/robottalkpod

We just released RUKA, a $1300 humanoid hand that is 3D-printable, strong, precise, and fully open sourced! The key technical breakthrough here is that we can control joints and fingertips of the robot **without joint encoders**. All we need here is self-supervised data collection and learning.

When life gives you lemons, you pick them up. (trained with robotutilitymodels.com)

What would you love to know about #robot learning and decision making? Later this season, I'll be chatting to Prof. Lerrel Pinto (@lerrelpinto.com) from NYU about using machine learning to train robots to adapt to new environments. Send me your questions for Lerrel: robottalk.org/ask-a-question/

Is there a word for the feeling when you want to cheer for the other team?

Times Square

Slava Ukraini 🇺🇦

The robot behaviors shown below are trained without any teleop, sim2real, genai, or motion planning. Simply show the robot a few examples of doing the task yourself, and our new method, called Point Policy, spits out a robot-compatible policy!

It finally dawned on me. I couldn't figure out what was going in Washington, plus all the hype about humanoids, including the new genre of "humanoid theater" videos, unmoored from reality. Then I realized, we must indeed be living in a simulation. Scripted by an LLM. Our world is a confabulation!

This is important because the humble iPhone is one of the best accessories for embodied AI out there, if not actually the best. It's got a depth sensor, good camera, built-in internet, decent compute, and -- uniquely -- it has really good slam already built in.

We just released AnySense, an iPhone app for effortless data acquisition and streaming for robotics. We leverage Apple’s development frameworks to record and stream: 1. RGBD + Pose data 2. Audio from the mic or custom contact microphones 3. Seamless Bluetooth integration for external sensors

A useful “productivity” trick is to remind yourself that research should be fun and inspiring and if it’s not that something should change.

And I am an ally. If you are too, let the world know.

Just found a new winner for the most hype-baiting, unscientific plot I have seen. (From the recent Figure AI release)

One reason to be intolerant of misleading hype in tech and science is that tolerating the small lies and deception is how you get tolerance of big lies

Thank you to @sloanfoundation.bsky.social for this generous award to our lab. Hopefully this will bring us closer to building truly general-purpose robots!

yoinked this image from Twitter, then ordered my copy join me

A fun, clever idea from @upiter.bsky.social : treat code generation as a sequential editing problem -- this gives you loads of training data from synthetically editing existing code And it works! Higher performance on HumanEval, MBPP, and CodeContests across small LMs like Gemma-2, Phi-3, Llama 3.1

We have been working a bunch on offline world models. Pre-trained features from DINOv2 seem really powerful for modeling. I hope this opens up a whole set of applications for decision making and robotics! Check out the thread from @gaoyuezhou.bsky.social for more details.

To help people understand the importance of NIH, share what you’ve used their funding for (in easily understandable terms). I’ll start: my NIH postdoc funding helped me develop and test AI tools that could identify skin cancer across diverse skin tones.

If you’re in grad school, finding a therapist can be really helpful. The thing you’re doing is hard and it’s harder if you don’t have help managing imposter syndrome, stress, self esteem, and a whole bunch of other things.

omg a student somehow accidentally wrote an email addressed to a faculty-wide NYU listserv and my inbox is now a master class on who understands the difference between a listserv and an email chain

Humans vs Ants: Problem-solving Skills

At NYU Abu Dhabi today and in love how cat friendly the campus is!

This holiday season, take a moment to visit your local bookstore. It’s about more than finding a great book—it’s about supporting the small businesses that keep our communities thriving.

HOT 🔥 fastest, most precise, and most capable hand control setup ever... Less than $450 and fully open-source 🤯 by @huggingface, @therobotstudio, @NepYope This tendon-driven technology will disrupt robotics! Retweet to accelerate its democratization 🚀 A thread 🧵

Outstanding presentation, finally! DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control @jeffacce.bsky.social @lerrelpinto.com

Love this approach. Reminds me of a more detailed version of an idea I had. Will definitely look deeper into this ironj.github.io/eleuther/

New paper! We show that by using keypoint-based image representation, robot policies become robust to different object types and background changes. We call this method Prescriptive Point Priors for robot Policies or P3-PO in short. Full project is here: point-priors.github.io

Modern policy architectures are unnecessarily complex. In our #NeurIPS2024 project called BAKU, we focus on what really matters for good policy learning. BAKU is modular, language-conditioned, compatible with multiple sensor streams & action multi-modality, and importantly fully open-source!

Robot utility models are not just among the first learned models that work zero-shot on a mobile manipulator, but also provide a nuanced discussion on what works and what doesn't in data-driven robot learning.

Since we are nearing the end of the year, I'll revisit some of our work I'm most excited about from the last year and maybe a sneak peek of what we are up to next. To start of, Robot Utility Models, which enables zero-shot deployment. In the video below, the robot hasnt seen these doors before.

I'd like to introduce what I've been working at @hellorobot.bsky.social: Stretch AI, a set of open-source tools for language-guided autonomy, exploration, navigation, and learning from demonstration. Check it out: github.com/hello-robot/... Thread ->

How to drive your research forward? “I tested the idea we discussed last time. Here are some results. It does not work. (… awkward silence)” Such conversations happen so many times when meetings with students. How do we move forward? You need …

Then "ten simple rules" series from plos might count? It is a little bit more about how science is done, but insofar as it aims to write down practical tips that are often just spread by word of mouth it could be a good inspiration: collections.plos.org/collection/t...

I think we need an AMA series for Robotics / Embodied AI with an optional anonymous setting. Will be both fun and informative to new community members to absorb folk knowledge.

I collected some folk knowledge for RL and stuck them in my lecture slides a couple weeks back: web.mit.edu/6.7920/www/l... See Appendix B... sorry, I know, appendix of a lecture slide deck is not the best for discovery. Suggestions very welcome.

Interesting article but the author drank the Kool-Aid and never sought out other viewpoints: “Foundation models like GPT-4 have largely subsumed [previous] models that help robots with planning and vision, and locomotion and dexterity will probably soon be subsumed, too.”

Nice work Remi and team. We need more of this!

Want to learn / teach RL? 
 Check out new book draft: Reinforcement Learning - Foundations
sites.google.com/view/rlfound... W/ Shie Mannor & Yishay Mansour This is a rigorous first course in RL, based on our teaching at TAU CS and Technion ECE.

Wonderful blog post by Matt Mason: mtmason.com/the-beetle-r... Quote from it: “... When humans have to handle things not designed for their hands, it’s all tools! Look at their dentists and surgeons. Look at their watchmakers. ...”

We got stranded for a day in rural NY without electricity, running water, heat, internet, or cell service. It is crazy how difficult it is to live without these relatively modern inventions. I hope one day robots will join this list.

Super impressive. Can't wait to see how you're doing teleoperation. This is the best option I've found that uses VR (by @lerrel.bsky.social's team). open-teach.github.io

Our paper on convergent evolution of land crabs is out in its final final form. The journal didn't pick my image as the cover, so I'm sharing it here because it must be seen! academic.oup.com/sysbio/artic... 🦀🦑🧪

Postdoctoral fellowships are open at NYU. Reach out if you're interested in having me as a mentor: apply.interfolio.com/158723 Some non-exclusive topics of interest: - Scaling up RL - Multi-agent planning - Equilibrium computation

In Robot Utility Models, we collected a large number of demonstration videos — 1000 examples in 40 environments — using a tool called “the Stick." The lead author even demonstrated it live at CoRL on completely new furniture. This one really does 100% actually work - I’ve used it in my house.

Hi, I am new here and would love to interact with folks interested in robotics & AI. A bit about myself: I currently run a robotics lab at NYU, where we are building general-purpose robots. Some of our latest projects are on www.lerrelpinto.com. If you have questions ask away!

Very excited about this new project, DynaMem. It allows our robots to function in previously unseen environments, performing long-horizon manipulation tasks. Most importantly it *generalizes*, meaning you can try it out on a wide variety of homes and on different objects. (4x video)

In our latest project, we train robots to mimic a human video of the task by matching the object features using RL. We only need one video and under an hour of robot training. Project was led by Irmak Guzey w/ Yinlong Dai, Georgy Savva and Raunaq Bhirangi. More details: object-rewards.github.io