Deployment & Hosting blog

Case study

The staging snapshot that helped GitLab restore its database

GitLab's January 2017 postmortem documents what the team could recover and why transferring a usable copy took so long.

What the team had available

On January 31, 2017, an engineer working on a replication problem accidentally removed the primary database directory on GitLab.com. GitLab reported that its scheduled database dumps had been failing and that the failure emails had been rejected. Earlier that day, an engineer preparing load tests had taken an additional production snapshot and loaded it into staging.

The long path back

That snapshot gave the team a recovery point at approximately 17:20 UTC. Restoring from the staging environment involved a slow disk transfer that took around 18 hours. Staging also normally removed webhooks. Engineers used the original snapshot separately to recover the webhook table and add it to the restored production database. Database changes made after the snapshot could not all be recovered.

A lesson for a restore drill

Include one feature that your staging environment deliberately disables, such as a notification subscription. Check that its configuration survives recovery while keeping the delivery service disconnected during the drill. Time the full transfer over the storage path available during an incident. A successful restore of a tiny fixture can verify the instructions. A representative data volume tells the on-call team how much time the copy itself may consume.

Sources

All articles and case studies Describe your setup