First blog post up on Robotics Simulation Infrastructure! I give a high-level overview, followed by an elementary example of better infrastructure for pose management. stoneztao.substack.com/p/robotics-s...

Wasn't at GTC but a sneak peak of some of my internship work at NVIDIA was! (sim2real cable routing presented by my manager Yashraj) www.youtube.com/watch?v=d5wm...

would people be interested in a blog series on "simulation infrastructure" where I look at popular robotics sim tools (e.g. mjlab, isaac...) and explain the pros and cons of their design decisions. The series would cover what I believe to be key pillars of good, usable, sim code

I will be in SF from November 10-17 🌉 If you work on something interesting and want to meet up let me know! Or if there’s a fun event i’d love some invites 😃 I’m also looking to visit robotics companies (especially startups!), if you have time lmk if I can visit!

1/1 neurips submission accepted! Will share more details later, covers an often overlooked aspect of on policy RL training when scaling to large scale parallelized environments

Rumors going around are that NeurIPS PCs had to reject ~400 papers because of venue constraints that were in the accept pile. Hard things like this happen but the PCs and conference should explicitly tell all the authors who had this happen to them. Or create something like accept w/o presentation.

Opensourcing a tool to calibrate camera extrinsics painlessly in a minute, no checkerboards! It's based on EasyHEC, using diff. rendering to optimize extrinsics given object meshes+poses. Crazy that even a piece of paper works too. Code: github.com/StoneT2000/s... (paper example in next post)

Big thanks to Jeremy Morgan (PhD @ USC) for helping massively accelerate the IK solvers in ManiSkill! Open source really helps grow this project (and i get to learn tons of new things like IK optimization) github.com/haosulab/Man...

might be the first time I made it personally on the trending developers list on github! One of the contributing factors being the lerobot sim2real code

Combining sim to real with cheap hardware like the so-100 really is making cutting edge robot learning so accessible and its wonderful

My latest post: The American DeepSeek Project Build fully open models in the US in the next two years to enable a flourishing, global scientific AI ecosystem to balance China's surge in open-source and an alternative to building products ontop of leading closed models. buff.ly/kvJQE3I

Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...

Excited to announce that I will be interning at @nvidia research this summer on robotics/embodied AI! I’ll be in seattle for the summer, let me know if you want to meet up and chat! 🦾

Rest of my lab has various presentations at RSS2025, please check out their awesome work! June 25 EEB 248 @ 10am - Hardware Optimization for In-Hand Rotation by K. Fay

The sim2real demo had some mixed success, hampered primarily by the lighting conditions of the outdoors. At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle

I’ll be at #RSS2025 from June 21 to June 23! I’ll be giving a presentation on ManiSkill3 on June 21, 5:30 PM We will also have two live demo sessions, on June 21, 12:30-2:00PM and 6:30-8:00PM. Swing by to see live demos of zero shot RGB sim2real, cool sim demos, and VR teleop!

Our computer vision textbook is now available for free online here: visionbook.mit.edu We are working on adding some interactive components like search and (beta) integration with LLMs. Hope this is useful and feel free to submit Github issues to help us improve the text!

it's happening.gif @stonet2000.bsky.social

Rendering nerds! Check out our latest work "Vector-Valued Monte Carlo Integration Using Ratio Control Variates" that has just gotten the best paper award at SIGGRAPH 2025. This paper presents a method that reduces variance of a wide range of rendering and diff. rendering tasks with negligible cost.

Code/tutorial released for🤗LeRobot zero-shot visual sim2real Reinforcement Learning! Train your SO100 robot in ManiSkill's fast simulator+renderer💨 and deploy a cube picking model zero-shot. Just in time for the LeRobot Hackathon, happy hacking! github.com/StoneT2000/l...

longer uncut video 17/18 successes increased the spawn area size and tuned it to be less occluded. Works well!

sneak peek of what Xander and I will show next week at RSS 2025 for the maniskill demo session: Zero shot visual sim2real (basic) manipulation, one camera and <1 hour of RL in sim with SO100 ~3 seconds to pick random color cubes, could be faster but goal is accessibility and low cost!

Is there any research on replay buffer sizes for off policy (sac) RL algorithms? How come we have just assumed that 1M replay buffer size is reasonable for the majority of tasks we test on (in robotics typically). Why not smaller? Can we make it smaller?

Very good thread on how we got into this whole stem education / research cutting situation and its impacts by @davidbau.bsky.social Grants that are otherwise perfectly fine if you remove the “dei part” are being cut without being put back into stem education/research?!?! x.com/davidbau/sta...

Nice to see the use of ManiSkill3 in this work! Simulation is not just useful for RL training. It provides some good cheap deterministic test beds, perfect for testing imitation learning scaling laws at scale. Years of data in hours

For almost a decade, the environments used for RL research were slow, a huge research bottleneck in cost and throughput. The shift to fast envs will change: - the speed of testing new algorithms and ideas - decrease shoddy results caused by high eval costs Everything is getting so exciting.

first thing I see on wechat channels when I get to China is a 1 hour uncut video of G1 speed walking on a track lmao

I’ll be in beijing for about 5 days, let me know if you want to meet up!

talk by @ncklashansen.bsky.social at the world models workshop starting now!

spotted at the robot learning workshop, 3 papers in a row all using maniskill! (and mujoco) seems like this lab likes to use our parallel rendering features and prebuilt tasks

I’ll be at the robot learning workshop today and giving an oral talk at 9:15 AM on ManiSkill3 in rooms Garnet 216/217 at ICLR. Come see the crazy things you can do with fast sim+rendering like fast visual RL and zero-shot RGB sim2real! ManiSkill was also accepted at RSS! Updated paper coming soon

no more experiments today apparently

strangely, in robotics there’s value to doing BC and RL. RL for generating super human data, BC (or vlas these days) to copy that super human data it’s why i still do research on learning from demos, rl, sim. It’s the most underexplored area at the moment with the promise imo

Will be in singapore for #ICLR2025 next week! I work at the intersection of simulation, embodied ai, and ml/rl, let me know if you want to meet! will share the posters/talks i’m presenting later, will definitely be at the workshop on robot learning

as someone who started in rl, I check if manipulation robots are modelled well and run fine in sim by checking if they can learn to pick up objects via RL. Slowly adding these test scripts for robot arms in ManiSkill, including SO100 here!

Honored to have NVIDIA feature my work! I'm grateful for the support and excited about all the recent progress in AI x robotics ✨

A sensible perspective on humanoids in manufacturing (TLDR: if you can make humanoids, you can probably make better, more manufacturing specific things) blog.spec.tech/p/humanoid-r...

worked on a nice new gallery/docs for robots in maniskill/sapien (inspired by mujoco menagerie). Given all the github issues on robot modelling (it's hard!) it seems like we should just model robots for most people, this is an effort to organize it more maniskill.readthedocs.io/en/latest/ro...

some exciting news, ManiSkill/SAPIEN now has experimental support for MacOS for the CPU simulation and rendering. You can now do your local debugging/development on Mac. Example shown here is a Push-T policy trained on my 4090 running on my mac! try now: maniskill.readthedocs.io/en/latest/us...

been learning warp recently primarily to add soft body sims back to ManiSkill. I’ve also been making some efforts to support mujoco/mjx as a physics backend earlier in the year. If these upgrades work then all of my efforts to learn warp / add mujoco will pay off a ton

people are beginning to post writeups for their submissions to the lux ai challenge season 3 and one of them rewrote the entire game engine in rust! and tackled the problem with RL github.com/IsaiahPressm..., the competition wrap-up report will be very interesting

Currently trying to do a kind of api setup where you can really write code like this to get easy sim2real real environment gym interfaces where the action/observation space is the same as the sim env, controllers are aligned etc. hoping to make sim2real more accessible

one of the worst years to apply for phd programs. Major drop in number of admits and funding might not even be guaranteed at UCSD (and likely other unis)

To learn more, great resources are: LeRobot Remi Cardene (open-source software), Alex Koch, Robot Studio (open-source hardware), @cpaxton.bsky.social, @stonet2000.bsky.social (news, papers). Also, 50 other grad students.

ManiSkill3 accepted as an oral presentation at the @iclr-conf.bsky.social robot learning workshop! See you all in Singapore🇸🇬, we have some cool demos demonstrating all the power of parallel rendering, heterogeneous sim and more! We will also update the arxiv with a v2 with some of these updates

diffusion policy learning to draw triangles on a canvas from vision, code for this was just released! Not too hard in fact, and I think drawing environments have some nice opportunities of analysis because there are many ways to draw a triangle. Same with cleaning envs (uploading those soon-ish)

Excited to share that I’ll be joining UC San Diego for my PhD, advised by Professor Hao Su! Many thanks to everyone who helped me along my research journey so far — I’m looking forward to continuing research in robot learning, manipulation, and simulation!

direct sim2real of a RGB dextrous manipulation policy trained via RL + distillation! Just from the videos you can see a big advantage of RL over pure imitation learning is learning fast behaviors+solve precise tasks more easily parallelrendering is doing a lot of wonders for people's research!

direct sim2real of a RGB dextrous manipulation policy trained via RL + distillation! Just from the videos you can see a big advantage of RL over pure imitation learning is learning fast behaviors+solve precise tasks more easily parallelrendering is doing a lot of wonders for people's research!

2 papers led by two UCSD undergrads I advised/co-advised were accepted to ICLR! Incredibly proud of both of them and will share/repost threads on them later