ICYMI: llm-d is officially a @CNCF Sandbox project! 🚀 We’re evolving #Kubernetes into SOTA AI infrastructure through a powerhouse coalition including Red Hat , Google Cloud , IBM Research, NVIDIA, Mistral AI, Hugging Face , and many more. www.cncf.io/blog/2026/03...

It’s official: llm-d has joined the cncf.io ! 🚀 Our mission to evolve Kubernetes into SOTA AI infrastructure just got a massive boost. This milestone belongs to our amazing community. Thank you for building this with us. 💜 We’re just getting started! 🔗 www.cncf.io/blog/2026/03...

It’s official: llm-d has joined the cncf.io ! 🚀 Our mission to evolve Kubernetes into SOTA AI infrastructure just got a massive boost. This milestone belongs to our amazing community. Thank you for building this with us. 💜 We’re just getting started! 🔗 www.cncf.io/blog/2026/03...

Deploying or scaling LLM inference? This is the room to be in. 📈 The vLLM Inference Meetup hits Boston on March 31! Join us for an evening of deep technical sessions, live demos, and real conversations with the community. 📅 Mar 31, 5PM 📍 314 Main St, Cambridge 🔗 luma.com/4rmkrrb7

LLMInferenceService is now fully production-ready and built on the high-performance @llm-d.ai framework. 𝗪𝗵𝗮𝘁’𝘀 𝗶𝗻𝗰𝗹𝘂𝗱𝗲𝗱? - KV-cache aware routing and disaggregated prefill-decode to maximize throughput.

Watch this preview of distributed tracing (llm-d 0.6) and Prefix Cache-Aware Routing. 🔹 State Tracking: llm-d tracks KV cache via ZMQ. 🔹 Smart Scoring: EPP pods tokenize prompts and query to find cached blocks. 🔹 Optimal Routing: Reqs go to the pod for the best cache hit.

The first llm-d NYC Meetup is live! Deep dive into the open-source stack for cloud-native inference with @IBMResearch, @AMD, and @RedHat State-aware scheduling & KV cache reuse P/D disaggregation Scaling MoE models AMD ROCm & llm-d Watch: www.youtube.com/watch?v=_ZBQ...

📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗠𝗮𝗿𝗰𝗵 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! We launched our newsletter publicly last year to share our contributions to upstream communities from our Red Hat AI teams. We’ve gained over 𝟭𝟯𝟬𝟬 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲𝗿𝘀!

Final Call: NYC 🗽 Registration for the llm-d Meetup closes, Tuesday March 10. Join the community this Wednesday at the IBM 1 Madison office for a deep dive into llm-d 0.5, MoE scaling, and KV-cache offloading. Don't miss a night of high-signal technical talks. 🎟️ Register now: luma.com/0crwqwg4

Planning to join us in NYC next week? 🏙️ Registration for the llm-d Distributed Inference Meetup closes this Tuesday, March 10th. Don't miss out on a night of technical talks and networking with the community at the IBM 1 Madison office. Grab your spot now! 🎟️ luma.com/0crwqwg4 #llmd #NYCMeetup

What’s on the agenda for next Wednesday's NYC meetup? 🛠️ Intro to llm-d 0.5 ⚡️ Distributed LLM serving on AMD 🧠 Lessons scaling Wide-EP and MoE 💾 KV-cache offloading & prefix scheduling Join the engineers building the future of open-source inference. Details: luma.com/0crwqwg4

Join us next week in NYC with the llm-d community for a deep dive into distributed inference. We’re talking llm-d 0.5, scaling MoE models, and KV-cache offloading. If you're building LLM infra, don't miss this. 📅 March 11th 📍1 Madison Ave Register: luma.com/0crwqwg4

In the latest llm-d release, we’re tackling high hardware costs with the new GPU Recommendation Tool! 📈 Evaluate throughput, latency, and cost-effectiveness before requesting expensive cluster resources. Check out the full demo: www.youtube.com/watch?v=Y26i...

Where to find the llm-d community over the next 2 months 🧵 We have a busy Spring ahead with sessions in NYC, Amsterdam, and Paris. If you're building open-source infrastructure for distributed inference, come join the conversation. ⬇️

NYC: Ready to go deep on Distributed Inference? 🗽 The llm-d community is hitting Manhattan on March 11th! Join us at the IBM Innovation Studio for a technical deep dive into the infra powering the next generation of LLM serving. 🧵

We'd like to announce that @kubernetes.io WG Serving has succeeded and will be disbanded! Thank you everyone who have participated and contributed to the discussions and initiatives! More details: groups.google.com/a/kubernetes...

In case you missed it, last week the llm-d community shipped the v0.5 release. Check out the post from the llm-d project owners to learn more about all the features we've included in this release. llm-d.ai/blog/llm-d-v...

👉 Check out the February newsletter here: https://inferenceops.substack.com/p/state-of-the-model-serving-communities-f81 👉 Subscribe to get future issues in your inbox: https://inferenceops.substack.com/ 🚀 Thanks to everyone who subscribed so far! Kudos to all contributors to this edition!

Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem, including updates from vLLM, KServe, @llm-d.ai, @kubernetes.io, and Llama Stack.

🏗️ llm-d v0.5: Sustaining Performance at Scale In our last release, we focused on breaking latency records. With v0.5, we’re shifting from peak performance to the operational rigor required to sustain those gains in production. 🧵👇 llm-d.ai/blog/llm-d-v...

Check out this recent talk from IBM Distinguished Engineer and llm-d project lead Carlos Costa at NVIDIA Dynamo Day. Breaking down our approach to open-source inference, moving beyond theory to provide verified blueprints for scaling LLMs in production. red.ht/4kjqGty

Scaling prod LLM inference shouldn't require a proprietary ecosystem. We’re demonstrating how a fully open-source stack handles high-performance, distributed workloads on Kubernetes. Huge shout-out to Sean Condon from @RedHat for this deep dive! 📺 youtu.be/OinY1Oooke0

Within the llm-d open-source project, our goal is to provide more than just documentation. We are providing blueprints for success that developers can trust for their production environments. 🚀 youtu.be/TNYXjZpLCN4

Want to learn more about open-source distributed inference? 🚀 Join contributors from vLLM and llm-d at NVIDIA Dynamo Day to see how the community is building the future of distributed inference. 📍 Virtual & Free 📅 Jan 22 | 8AM–1PM PT 🔗 nvevents.nvidia.com/dynamoday

If you see me around the hallway or at the sessions, I’d love to chat about: - Model inference (KServe, vLLM, @llm-d.ai) - @kubernetes.io AI Conformance Program - @kubefloworg.bsky.social & @argoproj.bsky.social - @cncf.io TAG Workloads Foundation - Open source, cloud-native, AI infra and systems

Excited to share that I'll be speaking at #KubeCon Europe in Amsterdam! You can find me in the following sessions: 1. Cloud Native AI + Kubeflow Day: Welcome + Opening Remarks: https://sched.co/2DZN3 2. Project Lightning Talk: Evolving KServe: https://sched.co/2EFyW

📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗝𝗮𝗻𝘂𝗮𝗿𝘆 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! We launched our newsletter publicly last year to share our contributions to upstream communities from our Red Hat AI teams. We’ve gained over 𝟭𝟮𝟬𝟬 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲𝗿𝘀!

Standardizing high-performance inference requires deep ecosystem collaboration. 🚀 Huge shoutout to @vllm_project and @IBMResearch on the new KV Offloading Connector. We’re seeing up to 9x throughput gains on H100s and massive TTFT reductions. 🧵 blog.vllm.ai/2026/01/08/k...

AI inference is like a busy airport: without a controller, you get gridlock. ✈️ Check out this breakdown by Cedric Clyburn from Red Hat on how llm-d intelligently routes distributed LLM requests. 🔹 Solves "round robin" congestion 🔹 Disaggregates P/D to save costs www.youtube.com/watch?v=CNKG...

Instead of naive load balancing hitting cold replicas in multi-turn chats, llm-d ensures context is reused. This demo shows a near 90% KV cache hit rate, a smoother time to first token, and a ~500ms drop in P95 tail latency. https://www.youtube.com/watch?v=H2N4c-E-iw8

Want to work on llm-d and vLLM as a full time job? @RedHat_AI is hiring a variety of roles around open source LLM inference. https://x.com/RedHat_AI/status/2001362060777586744

Want to understand the architecture under the hood of llm-d? ⚙️ We’ve curated our recent technical deep dives and talks from KubeCon, PyTorch Conf, and more into one central hub. Learn Kubernetes-native distributed inference from the source. 🧵👇 https://llm-d.ai/videos

🚀 Announcing llm-d v0.4! This release focuses on achieving SOTA inference performance across accelerators. From ultra-low latency for MoE models to new auto-scaling capabilities, we’re pushing the boundaries of open-source inference. Blog: https://t.co/qlQnzcT9O3 🧵👇

Learn more about the llm-d integration in KServe from the recent Cloud Native AI day at Kubecon. 👇 https://x.com/TerryTangYuan/status/1992995298105290794

Running massive models like DeepSeek-R1 requires serious distributed infrastructure. Enter llm-d ⚡️ Join @RedHat_AI's Rob Shaw for a deep dive into this open-source framework for optimizing distributed LLM inference using a "well-lit paths" approach 👉 https://www.youtube.com/watch?v=_xAXb70d4-0

🚀 llm-d v0.3.1 is LIVE! 🚀 This patch release is packed with key follow-ups from v0.3.0, including new hardware support, expanded cloud provider integration, and streamlined image builds. Dive into the full changelog: https://t.co/Wh6OGJ0KdO #llmd #OpenSource #vLLM #Release

llm-d just passed 2,000 stars on GitHub! ⭐️ A Kubernetes-native distributed LLM inference framework built for performance and scalability. Join the community today! https://llm-d.ai/docs/community

Going to #KubeCon Atlanta? Join the llm-d communites sessions exploring how to route and scale LLM inference on Kubernetes. From prefix-aware routing to multi-accelerator deployments - come learn what we've been building. Schedule: https://llm-d.ai/docs/community/events

Register now to meet with and learn directly from the llm-d and vllm core contributors! https://x.com/RedHat_AI/status/1983174562310451279

We're gathering feedback on our new "Wide-EP LWS" deployment pattern using Kustomize. Your input on how it compares to the previous Helmfiles approach is crucial for our v0.4 release cycle. Please share your thoughts in our short form! 👇 📝 https://t.co/HGDIusHtBu

🚀 Evolving for Impact! We're updating our llm-d SIG meeting schedule to a bi-weekly cadence. This gives our community more time for deep work between calls, making our sessions even more focused and productive. Here are the details 👇

We are thrilled to announce the release of llm-d v0.3! 🚀 This release is a huge milestone, powered by our incredible community, as we continue to build wider, well-lit paths for high-performance, hardware-agnostic, and scalable inference. 🧵Let's dive into what's new!

Running LLMs on Kubernetes? You've likely felt the pain of re-processing the same context tokens over and over (think RAG system prompts). This is a huge source of inefficiency in distributed inference. Let's break down how we're solving this with llm-d. 🧵

🗓️ Conference season is heating up, and the llm-d community is out in full force! Want to learn about efficient, scalable LLM inference directly from the experts? Here's where you can find us this month: 👇

In production LLM inference, this metric matters: KV-Cache hit rate. Why? A cached token is up to 10x cheaper to process than an uncached one. But when you scale out, naive load balancing creates a costly disaster: the "heartbreaking KV-cache miss." https://red.ht/46A4ynW

Your distributed vLLM setup is likely wasting GPU cycles. The culprit? Your load balancer. Standard round-robin is blind to KV-cache state, leading to cache misses that force costly re-computation of tokens Our latest post on the llm-d blog dives deep into this problem. 🧵

The llm-d community is crushing it! 🚀 We're constantly seeing amazing knowledge sharing, and our new Events page is the perfect spot to find it all. Case in point: an upcoming talk from Jeff Fan at @DigitalOcean diving into next-gen AI infrastructure with llm-d! 👇

The llm-d community is building incredible things! 🚀 Shout-out to Ernest Wong & Sachi Desai from Microsoft for their new blog post pairing llm-d with Retrieval-Augmented Generation (RAG) on Azure Kubernetes Service (AKS)! This is a must-read guide! 👇 https://t.co/DPfRUdTLJB

Pair llm-d Inference with KAITO RAG Advanced Search to Enhance your AI Workflows http://blog.aks.azure.com/2025/09/12/pair-llmd-and-rag-on-aks

With events & talks planned across the US, we're excited to connect with the community. Find us in Austin, Boston, San Francisco, and Atlanta over the next few months. First up, it's a packed week with three events. Here's where to find us. 👇