Kevin McDonald (@sudorandom.infosec.exchange.ap.brid.gy)

IRC Log: rm -rf /var/opt/gitlab/postgresql/data The Incident Log: January 31, 2017 [23:00] *** Topic: DB Replication Lag | Status: 🔴 Critical [23:05] *** tired_sysadmin has joined [23:10] Replication is stuck again. The secondary node (db2) is refusing to sync. [23:11] I’m going to wipe the data directory on db2 and let it pull a fresh copy from master. [23:12] rm -rf /var/opt/gitlab/postgresql/data [23:12] Weird. It’s taking a while. Usually empty directories delete instantly. [23:13] Hey, why did the website just go 500? [23:13] … [23:13] I’m looking at my terminal prompt. [23:14] It says root@db1. [23:14] db1 is Prod. You are deleting Prod. [23:15] CTRL+C CTRL+C CTRL+C [23:15] Okay, I stopped it. How much is left? [23:16] Checking… The directory is 4.5KB. [23:16] We had 300GB of data. [23:17] Okay, don’t panic. We have 5 different backup mechanisms. Let’s check S3. [23:20] S3 bucket is empty. The backup script has been failing silently since version 8.1. [23:21] Check the Azure disk snapshots. [23:22] Not enabled. [23:23] …LVM snapshots? [23:24] We take them every 24 hours. We just lost 6 hours of data. [23:25] I am going to live stream the restoration on YouTube so people don’t kill us. Postmortem of database outage of January 31 https://kmcd.dev/posts/irc-log-rm-rf-/var/opt/gitlab/postgresql/data/