After 6+ months in the making and over a year of GPU compute, we're excited to release the "Ultra-Scale Playbook":
hf.co/spaces/nanot...
A book to learn all about 5D parallelism, ZeRO, CUDA kernels, how/why overlap compute & coms with theory, motivation, interactive plots and 4000+ experiments!