DataLoader Dispatching When constrained by a variety of reasons to where you can't include multiple copies (or mmaps) of datasets in memory, be it too many concurrent streams, low resource availability, or a slow CPU, dispatching is here to help.

Batch Sampler Sharding

Dataset Sharding When performing distributed data parallelism, we split the dataset every batch so every device sees a different chunk of the data. There are different methods for doing so. One example is sharding at the *dataset* level, shown here.

Hi all 👋 Back very briefly to mention I’m working on a new course, and there’s a star-studded set of guest speakers 🎉 From Scratch to Scale: Distributed Training (from the ground up). From now until I’m done writing the course material, it’s 25% off :) maven.com/walk-with-co...

AAAHHHHHHHHH BE NICE TO OPEN SOURCE MAINTAINERS OH MY GOD. SOME OF YOU ARE SO RUDE, WHO RAISED YOU

JK STRONG is still awesome. Solution (which they were already aware of this) was literally logout -> login! 11/10 customer support

Enshittification has come to my gym app. Lord help us all 😭

Recently did my first lifting competition, overall quite happy where things ended up (bar bench) and was a ton of fun. Might wait a year and give it another go, but I did get first in the weight class! 145kg squat 180kg deadlift 90kg bench

I guess work is done for the day, everyone can go home 🤷‍♂️

A federal judge in Maryland issued a temporary restraining order on Monday, blocking members of Elon Musk’s so-called Department of Government Efficiency team and anyone “working on the DOGE agenda at the Department of Education” from accessing sensitive data until March 10.

Day one of new program: ✅

Nothing humbles you more than a front squat. Going from 260lbs back squat to 115lbs front squat was *lovely*

Interesting discussions between John Ousterhout and "Uncle Bob" on software design: method length, comments, and test-driven development. It's a great example of "agree to disagree". There is no silver bullet in software design. Let's focus on writing easy-to-parse code 👀 github.com/johnousterho...

Estimated read time: 2-3 days Who’s going to be the first to start a study group?

After 6+ months in the making and over a year of GPU compute, we're excited to release the "Ultra-Scale Playbook": hf.co/spaces/nanot... A book to learn all about 5D parallelism, ZeRO, CUDA kernels, how/why overlap compute & coms with theory, motivation, interactive plots and 4000+ experiments!

Anyone in my bubble built a lack rack before? 👀 Eyeballing doing this for the new place

0/10 can’t recommend: * Selling your house * Buying a house (*after* putting your old house on the market) * Lifting competition All in the same week

Update: Ran the equivalent of this space locally via fastapi + hooked into the LLM commit message generator got the job done, and very fast (since it’s a T5 model under the hood) huggingface.co/spaces/mamik...

Me post seeing SmolLM2 quant is fast enough to run on my tiny laptop

Happy first intern day at @hf.co to all who celebrate 🤗 This years round of interns are 🔥

We are reproducing the full DeepSeek R1 data and training pipeline so everybody can use their recipe. Instead of doing it in secret we can do it together in the open! Follow along: github.com/huggingface/...

Wrote up my selfish personal argument for releasing code as Open Source: if you solve a problem and then release it under an Open Source license you will never have to solve that problem again for the rest of your career! simonwillison.net/2025/Jan/24/...

Me: huh where’s the cell signal go. Where’d the internet go Verizon: I suppose I’m forcibly done with work for the day, too bad I had to go to the next town over to say that 😅

Anyone in my mutuals looking to hire a really good Elixir dev for their team? My father is recently having some hard times and looking for work. Very knowledgeable both there and on ML

Hi all 👋 I work in OSS, typically in the ML framework scope (transformers Trainer, accelerate, etc). Big nerd on framework design. Also into 🏋️ & anime (so those may show up in the feed occasionally). muellerzr.github.io

Me: huh, why is the chainsaw man intro playing on repeat at the gym? Apparently my year of Spotify premium expired today. Quick trip to the car and we’re back in business for deadlifts. If you pay monthly for Spotify, don’t. $12/mo == $144. Or pay $99 for an egiftcard www.amazon.com/dp/B09MWM6TS1

Recently been digging into torchao’s float8 training paradigm. It interests me more than transformerengine or MS-AMP simply because it promises near exact training is possible between it and using raw bf16 github.com/pytorch/ao/t...

👋 for totally random reasons I’ll be exclusively over here from now on 🤗 Prepare for tech and lifting fun 😬

I’ll know I’m getting rusty when code I write with people’s API’s magically work the first time with things I try to do. For now though, remaining on the broken path we are

Time for an end-of-year detox, see you all in 2025 👋

Gentle reminder: support your local business and they’ll support you right back 🥰

Textual 1.0 has been released. 🥳 Three years in the making. A TUI framework that is bigger than the terminal. To celebrate, I want to give away some trade secrets. Because I am appalling at keeping secrets. Tell me what you think of the diagrams... textual.textualize.io/blog/2024/12...

Been a bit since I’ve hacked on GitHub actions. LLM’s does make this much more bearable and honestly fun to debug and work with.

Finally have outdoor lights on the house this year, and had many neighbors compliment them 🥹🥹🥹 (Didn’t put them up last year/didn’t own any, also not going to post a pic because it’s *the outside of my home*) Gimme a day or two though and I’ll post when I setup the Xmas village 🥰

ADHD is certainly fascinating. All year I’ve struggled to finish reading half a dozen books (I had a goal of reading 12 this year, 8 more than last.) Meanwhile, in the last ~3 weeks I’ve read 6, on pace for 2 more by next week.

Me: yay winter is here! Can’t wait to bike to the gym in 0C My bike tires: you sure about that boss? 😤🥲

If anything, I’m consistent

Want to clear up some misconceptions, pydantic is actually short for Pytholomew Daniel Ticonderoga, the inventor and patent-holder of the first disposable pencil that also forced you to write only on paper with dotted lines

This has been my secret sauce to all my writing from day 0. Write it to help me, but don’t discount that others might benefit too so make it public. At worst, you learn something from someone else’s viewpoint

Whelp large influx of folks here, hi 👋 I’m off and on here as long as things stay civil (for my own health) but I work at 🤗 on the Accelerate project. Also into lifting & anime. Hi!

new post: bluesky vs. huggingface for normal people jessbpeck.com/posts/bluesk... anytime i spend less than 5 months on a thing you know i have opinions. anyway, as always, feel free to argue with me/complain at me/ point out errors.

I'm looking for an intern! If you are: * Driven * Love OSS * Interested in distributed PyTorch training/FSDPv2/DeepSpeed Come work with me! Fully remote, more details to apply in the comments

How can you avoid the temptation to use a subprocess for sub-commands? This blog post from @muellerzr.bsky.social saved my day. muellerzr.github.io/til/argparse...

I wrote this in 15 minutes in Gradio

It’s nearly 2025, the fact I’m sitting here having to write my own custom interface and app just to sift through my notifications and extract all pull-request related GitHub notifications is astounding…

I'm a simple man. I see that Oura has an API, I then try to recreate the app so I don't have to pay $6/mo just to see my data and write custom visualizations

I’m very thankful to be surrounded by such wonderful friends on my birthday. 🥰