Pretty simple: verify your backups!

GitLab.com is in crisis after experiencing a severe data loss caused by human errors and ineffectual backups.

What happened?

On Tuesday evening, one database experience a severe performance degradation, and the sysadmin tries to start an emergency database management.

[embed]https://twitter.com/gitlabstatus/status/826572933304827904[/embed]

 

But another (tired) sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a database replication process,wiping a folder containing 300GB of live production data.

[embed]https://twitter.com/gitlabstatus/status/826591961444384768[/embed]

In the Google Doc, the sysadmin notify:

“This incident affected the database (including issues and merge requests) but not the git repos (repositories and wikis).”

So not all is lost? Do not be too optimistic! The document concludes with the following:

“So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.”

A few hours later GitLab publishes a post on his blog where tries to reassure users on the situation:

Yesterday we had a serious incident with one of our databases. We lost 6 hours of database data (issues, merge requests, users, comments, snippets, etc.) for GitLab.com. Git/wiki repositories and self hosted installations were not affected. Losing production data is unacceptable and in a few days we’ll post the 5 why’s of why this happened and a list of measures we will implement.

As of time of writing, we’re restoring data from a 6-hours old backup of our database.

This means that any data between 17:20 UTC and 23:25 UTC from the database (projects, issues, merge requests, users, comments, snippets, etc.) is lost by the time GitLab.com is live again.

And finally:

Update 18:14 UTC: GitLab.com is back online

The reactions?

I like to quote this sentence from TheRegister:

The world doesn’t contain enough faces and palms to even begin to offer a reaction to that sentence.

[embed]https://twitter.com/nixcraft/status/826723474227892224[/embed]

 

 

[embed]https://twitter.com/nixcraft/status/826721366036533249[/embed]

 

 

[embed]https://twitter.com/TheRegister/status/826601732767567872[/embed]

 

[embed]https://twitter.com/drli0/status/826723911249301505[/embed]

 

[embed]https://twitter.com/mxrxdxn/status/826718020496130048[/embed]

 

[embed]https://twitter.com/eduardo_casas/status/826698378742472704[/embed]

 

Moral of the story?

  • Verify backups
  • Do not work late!

References

https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub