At Etsy, we had an upgrade go out for MySQL with Chef. Meant to go to one server, instead upgraded all of production. During that time, it had to update very tall tables, which mean locking them from any reads or writes. Turned into a multi-hour downtime where we just had to wait for it to finish.
add a skeleton here at some point
about 1 month ago