Most people think the Kubernetes scheduler makes scheduling decisions itself. Well, it doesn’t. The real power lies elsewhere. 🧵 Let’s talk about how the kube-scheduler is actually a plugin-first orchestration engine, not a decision-maker.

KubeCon goers: what are the Atlanta hotels you're booking that's not next to the boring downtown/convention center area?

Anyone have recommended readings on any scalable k8s multi-cluster work API implementations? Planning to roll out a similar model on our own and I want to study the state of the art and the rationale behind the existing model.

#KubeCon schedule is out! Team LinkedIn will give 3 talks this year (myself included): 1️⃣ Making Application Rollouts Observable (lnkd.in/gmnM-8Z4) 2️⃣ How We Upgrade the Whole ML Training Fleet Bi-weekly(lnkd.in/gG_a_Etc) 3️⃣ Evicted! All the Ways Kubernetes Kills Your Pods (lnkd.in/ggm9HKZF)

Our team at LinkedIn will give 3 talks at the #KubeCon 2025! Looks like we'll be meeting in Atlanta again. 👋🏼 Stay tuned.

Does anyone know why all cloud-native open source ecosystem uses pull model to expose their /metrics (instead of pushing the metrics)? Pull model is really poor when the metric cardinality exposed on a single endpoint increases.

We’ve been systematically evaluating the scalability of our fleet’s control planes, and the recent work on storage layer scalability in the Kubernetes community is going to prove pivotal as we scale further! Check out what we’ve been upto for the past couple of weeks ♥️

As we test Kubernetes clusters beyond the supported 5,000+ nodes limit at LinkedIn, we're finding out 'List API' calls are a bottleneck in cluster reliability. So I'm publishing a deep dive about why this is a problem and what you can do about them. ahmet.im/blog/kuberne...

Open sourcing a new kubectl plugin called `kubectl node-resource` that queries allocations/utilizations for all/subset of the Kubernetes nodes. It has optimizations for large clusters. We've needed this in our set up to quickly answer questions. github.com/ahmetb/kubec...

Glad to have @madhavjivrajani.bsky.social join us at LinkedIn compute team this summer. I’m constantly impressed by the depth of his kube-apiserver expertise.

DAYS SINCE ETCD BACKUPS SAVED OUR ASS IN PRODUCTION: 0 If you manage your Kubernetes clusters and think you won’t lose your etcd, you’re almost certainly wrong. It takes years for the disaster to happen and you’ll be glad you have invested in building a backup/restore system.

@kelseyhightower.com hi Kelsey, what was your toto model again? can’t find after the twitter nuke ☹️

@ahmet.dev and I shared our approach at #KubeConEU25 couple weeks ago with ~500 attendees (though the credit belongs to the entire team behind the platform). Great to see it resonate! Watch here: youtu.be/dDkXFuy45EA?...

Finally delivered our talk at #KubeCon to 500 attendees! Slides are up at kccnceu2025.sched.com/event/1txGQ/ and recording will be coming soon. If you wanna chat, I'm at the conference today.

Here we go again, this time in London. #kubecon maintainer summit

🇬🇧 Looking forward to seeing y'all at our #KubeCon talk about how we're building a bare-metal compute platform at LinkedIn to run thousands of services on high-6-digit number of machines! Schedule: sched.co/1txGQ

Our discussion for the @kubernetespodcast.com with @ahmet.dev and @ronaknathani.bsky.social of is finally live 🥳 I had a great time talking to these awesome people about how to leverage #k8s inside LinkedIn and what they learned along the way 🙌 Take a listen 👇 #KubernetesPodcast #k8s

This episode is a real treat- @ahmet.dev and @ronaknathani.bsky.social of @linkedin.com share lessons from running Kubernetes in production! Learn about the custom tools they've built for running Stateful workloads, how many clusters is the right number, & more! kubernetespodcast.com/episode/249-...

Looking forward to seeing y'all at #KubeCon London next week! (Who's going?)

New article on my blog that explains all the various ways Kubernetes can evict your workloads. This wasn't all documented in one place so I'm hoping this helps the teams operating with Kubernetes in production. You'll also find a cheatsheet in the article. ahmet.im/blog/kuberne...

Selam Türkiye 🇹🇷 Our Google /Dev/Cloud event is coming to Istanbul on March 13 🥳. We will have a full day of technical content and hands-on workshops about Google Cloud, GenAI, Cloud Runtimes, and more. Organized by the Cloud DevRel Team for developers 👨‍🏫 👩‍🏫 ⌨️ .

I might _attempt_ to write a blog about all the different things that causes Kubernetes evicts pods. There are so many, and every single one works differently.

What are some of the footguns you observe people keep adding to systems software? My pet peeves are: - environment variables for critical configs - default values for critical configs - assuming a value for a missing config in the code - soft-failing, trying to be smart around errors

Come work with me! We operate several hundred thousand machines that make up our compute fleet, and we orchestrate stateless/stateful/batch/GPU workloads on our Kubernetes infrastructure. (Seattle/Bay Area only, happy to refer if interested!) www.linkedin.com/jobs/view/sr...

Published a blog about common mistakes beginners make developing Kubernetes controllers (especially using controller-runtime) and how to write scalable and production-grade controllers. ahmet.im/blog/control...

📝 Drafting an article about Kubernetes controller development mistakes and anti-patterns we see beginners make based on what we learned over the past couple of years. I hope to publish on Tuesday. Anything you wanna add?

Xiaohongshu is an amazing app. I never used TikTok, but this app isn’t about dance videos to begin with. There’s actually a lot of useful content here, and you get to converse with Chinese users with auto-translated comments. Fantastic to peek behind the propaganda about PRC.

Should Kubernetes controllers+CRDs managing resources in the external world (non-k8s APIs) try to fix drifts (such as out of band modifications made in the external API) periodically –or assume they're the only actor, and wait for the next modification to reconfigure? This makes scaling harder.

Our KubeCon EU talk is accepted! It was a rejection for the North America earlier, but I'm glad we got in this time. Lots of exciting stuff to share soon.

Kubernetes community: We’re looking for a new teammate to work on low-level Kubernetes infrastructure primitives at LinkedIn. If you’re interested in high-scale bare-metal compute, please reach out to me. (The role is Bay Area & Seattle only.) www.linkedin.com/jobs/view/41...

Kubernetes community: We’re looking for a new teammate to work on low-level Kubernetes infrastructure primitives at LinkedIn. If you’re interested in high-scale bare-metal compute, please reach out to me. (The role is Bay Area & Seattle only.) www.linkedin.com/jobs/view/41...

Published my reading between the lines of OpenAI Kubernetes postmortem: ahmet.im/blog/openai-... Many lessons to learn here, IMO it comes down to: * unintentional coupling of data plane/control plane * not being able to shield k8s API from load * inherent risk of asynchronous service announcements

I wrote about the time when Kubernetes node-feature-discovery (NFD) caused an incident in our clusters, and why we're removing it from our stack. ahmet.im/blog/nfd-inc...

Something early career software engineers struggle with is the code quality of the tests. The tests ought to be readable and maintainable like your main code; not an unmaintainable copy pasta mess. Invest in tools to make your tests concise.

Is “disable retweets” a feature here, yet? I follow a lot of people for their opinions and to stay up to date with their work; not what they repost. 😞

TIL the Pacific salmon doesn’t have enough energy to swim back to the ocean after spawning. One of the local parks smells like rotten flesh. 🥴

Still surprised nobody has done a “find my Twitter follow list on Bluesky” tool. How are you converting Twitter followers to this platform, or are you building a new following from scratch?

Submitted a talk to #KubeCon EU today! 🤞 🤞

The most impressive technical feat I’ve seen at #KubeCon this year by far was the Gödel scheduler by ByteDance. A completely different scheduler design with sharding and optimistic concurrency achieving 2,000 pods/sec. Blog: www.cncf.io/blog/2024/04... Talk: youtu.be/PiUAdclow_4?...

Kubernetes isn't the only orchestrator that exists. Here's some features I wish it had youtu.be/9N9IOpyl3v8

Newest addition to my home office set up is the BenQ Halo light with both desk and back lighting. This will make the dark days of Seattle fall bearable.

Any app owner forcing their users to use SMS 2-factor authentication should be jailed. Trying to restore a phone without an eSIM (because Apple messed it up during a repair) and realized half of the apps need SMS to log in. 🤦‍♂️

It was great to catch up with many GCP & Azure friends at #KubeCon after so many years.

Big congrats to @thock.in and all the maintainers! #KubeCon

Go to @aurelievache.bsky.social and Gaëlle’s talk on building kubectl plugins and distributing on Krew today #KubeCon! sched.co/1i7lr

We've now published a blog about how LinkedIn runs large stateful apps with a generic operator on Kubernetes. www.linkedin.com/blog/enginee... Don’t forget to come to our #KubeCon session on Thursday.

Tonight was fun. #kubecon

Took a long break from #kubecon and I now see far fewer familiar faces.