So much goes into this. For any internal tool nerds, we’ve written a bunch about what we’ve had to build here: incident.io/building-wit...

Had a great time chatting about how we’ve been building AI SRE at @incident.io with @wparad.bsky.social If you’re working on AI systems or incident response, I’d hope this is a worthwhile listen!

The amount of Lime bike green I can see in the London naked cycle photos is making me reconsider ebiking as a social good

Was fun changing our Old Street billboard to focus on eng hiring last Friday. Went a few rounds with the team and felt this represented us well!

Wrote up a journey of optimising an LLM prompt for latency, reducing the time from 11s to just 2s by being careful with tokens. Loads of lessons here for people working with AI! incident.io/building-wit...

Yesterday we launched a microsite of stories from our AI team. To my knowledge, it’s the best repository of genuinely useful AI engineering content on the internet to date. I’m so proud of the team for this and can’t thank our designers enough for how amazing it looks. incident.io/building-wit...

It's been great fun sharing our approach to AI with Gergely, born out of the last year hitting our heads against AI. This is an awesome article that I'd encourage anyone building systems with generative AI to read! Thank you for the feature!

8 years ago Chris and I dealt with a horrible Postgres incident where RAID array battery failure was an extremely painful red herring. Good to see he’s back in the game after all that time. These disks must really be something!

Sometimes I wonder how many burritos a week is too many burritos

How is this a thing

I have a package that’s been stuck in customs for a week. Is there anything more irritating than a fedex tracker giving you zero info and saying “ETA Tuesday 11th 11am” when it’s now the next week 😂

Crazy to me that people can say this when ChatGPT/Claude/etc have hundreds of millions of daily active users. I have not once ever had a use for a blockchain payment. All my colleagues use AI daily. What is making people so resistant to seeing this?

Was playing with some CI build time improvements yesterday and booting big old machines is something I will always enjoy. 96 cores go brrrrrr

The average AI product experience is bad right now, and it’s because teams haven’t figured it out how to build with these tools yet. Building a prototype is 10% of the effort. The rest of the journey is uncharted waters and can be a real struggle. blog.lawrencejones.dev/ai-mvp/

Gotten into the habit of getting AI to check my code piece-by-piece as I build systems. Doesn't replace a human or a review, but it frequently catches issues I'd otherwise miss or only find once I'd deployed. Can't recommend this enough!

This was a fun chat about how we're building an AI incident responder, which: - Checks your Grafana dashboards - Looks at recent PRs - Finds past incidents like this one And drops a message just like this one I got this morning when I broke our evals 😂

It’s really important to remember when anyone speaks about DOGE that this is the person leading it. Just keep that in mind when the next tech bro says how ‘excited’ they are about the mission. www.theguardian.com/technology/2...

We've got a new game at work "Cold Case Mysteries" where people submit a picture of their fridge and then we all guess who it is. I thought this would be funny but did not anticipate a character assassination.

Kinda weird to see Zuck aping the behavior that lost Elon tens of millions of Twitter users. Makes me wonder what and how strong the incentive has to be for him to do something so actively against the top-line business interest.

Keep making eye contact with this weirdo across my desk and it’s kinda freaking me out

I just remembered a fun SRE interview we used to run where we’d break a VM in several ways and share an ssh session with the candidate to pair on fixing it. Lots of how we’d break the machine are no longer relevant today. Crazy how fast this stuff moves on.

Is the only issue I have with BlueSky: certain parts have extremely strong group-think. This is automating looking at the first few hits on Google nothing about it is ‘privacy breaching’ but because it’s AI and on X everyone wants to pile in.

Guys this whole 12 day thing is a terrible idea, this tree was emotionally load bearing 😭

Folks need to understand this. Pornhub is absolutely in the right here. They are blocking states who require this because they don't want to be responsible for your PI data! They do not NEED that info to provide their service, so collecting and maintaining it is inherently privacy violating!

I wrote some reflections on 2024 focusing on my experience at incident and everything that went by. Now the third year I’ve done this, hoping to make it a tradition. blog.lawrencejones.dev/2024/

I pushed all my git history into Gemini earlier, everything for the last year, then asked it to categorise PRs against themes. Eventually I got a graph of what I've been working on and when across the year. I've wanted to do this for years, cool how easy it is now.

This was a great listen. Loved the take on “one big text box” being the lazy access into LLMs pending us finding better UX and the implications of custom AI tutors for kids is kinda crazy.

I've maintained a vimrc for about 13 years. A better engineer would have removed things in that time, but I did not. Since then s/Vim/Neovim/, and s/vimscript/Lua/, which had put me off a full rebuild. Turns out for the price of a light hazing, Claude can upgrade the whole settings file for me!

Got a cool new meat thermometer that’s fully Bluetooth for Christmas so decided to try it out today. M&S only had a thin rib eye so it’s quite overkill but here we go!

At home unpacking after Christmas and trying to setup an AirTag I’ve been gifted for my keys. Finding myself getting weirdly emotional as I realise I should move my keys to the Apple keyring, abandoning the one I’ve used for the last ten years. How on earth have I got attached to a keyring.

Been looking forward to this for months, got a back catalogue of papers waiting for it to arrive! Really hoping it meets expectations🤞

My family dog stands here for extended periods of time slow cooking herself, refusing to move away. I expect having a fire each night is her favourite part of Christmas.

This is, from my experience, definitely a thing. It takes time to become proficient in how to use and instruct LLMs. And it’s very easy to non-actively practice where you use these tools but don’t actually get better at them.

This is savage and I appreciate the creativity that went into this, even if it makes me cry a bit

My Slack iOS app has become unusably slow recently, keyboard lag of about 5s at times. No idea if this was the iOS upgrade or the Slack app that’s done it, infuriating though.

Barbican is having a TV renaissance recently; Slow Horses, The Agency, The Day of the Jackel Seems brutalist architecture is like catnip for gritty TV crime directors. Kinda dig it.

Rhys turning up in this ridiculous inflatable Santa costume was one of the funniest things I’ve seen in a while. Trying to fit down a bank of desks and taking the whole stack of notebooks, pens, and keyboards with him was great.

Really excited about this. It’s probably the first time you can properly tell your CEO they don’t need to be in the call, just follow the updates. Would be great to remove that pressure from responders!

It’s wild that I never write database migrations or domain models by hand anymore, I load a tech spec into Claude and have it generate them for me. Totally different workflow from just a few months ago.

I’ve just been told I’m 5 days behind on my advent calendar and this is a good representation of my state of mind this month 😂

Just before the pandemic began I ran an incident response 'Game Day' to test how we'd respond remotely. After pushing some buttons I shouldn't, our drill turned quickly into a major outage. It's a fun one with some useful takeaways. blog.lawrencejones.dev/game-days-go...

London has its ups and downs but this group playing duck duck goose to while away a tube delay is the type of whimsy I value.

Matt did an amazing job of building a Go community in Twitter and I’d love to see that community live on here!

Biggest day of the year for the Spotify marketing team, would love to be a fly on the wall. What’s the strategy around notifying people? Do it geographically, get viral traction in small areas and domino up?

Until AI I'd never developed anything that felt so pay-to-play. We've put $ cost on all our interactions to get a sense of our usage and it can be quite wild.

"Who watches the watchers?" is funny as a phrase, less so as a problem you need to solve, which it became when building on-call. It's possible for us to receive a page saying "paging is broken" and while I hope we never do, thought it would be fun to share how it works. incident.io/blog/who-wat...

Wakeup pages. We don't like them, but they also cost the business real money in lost productivity. Take your salary, divide by 2,080, and 2x or 3x it. Then multiply that by the number of hours you spent staring into space today because you got woken up by an unactionable page.

🎉 We’re thrilled to announce that incident.io is opening a new office in San Francisco! 🎉 With London, NYC, and now SF, we’re officially working across three time zones to better support our customers, grow our team, and stay close to the action. See you in January 2025 🔥

Does anyone know how to clone a Claude 'project' so it copies across the existing project knowledge? We want to use projects more seriously at work (sharing them, curating them) but lack of cloning is going to be painful.

The team that writes the code needs to deploy it and be on-call for it. Full stop. Any other setup is going to lead to resentment, unfixed or slow to fix issues. But what about SRE? You aren't Google, so don't copy what they do. They had a specific problem to solve which you don't have.