Google has had some trouble recently with Gmail, due to a software bug 150,000 or 0.08% 0.02% of the gmail accounts are effected.

The software bug was introduced when they were updating their software. It removed 0.02% of all the emails over all their servers. Because they back everything up on tapes too, they are able to restore all of those effected accounts.

I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers? Well, in some rare instances software bugs can affect several copies of the data. That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue.

To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they’re protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it’s taken us hours to get the email back instead of milliseconds.

So what caused this problem? We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version.

Edit: Google updated their estimate of accounts effected to 0.02%. I also updated and rewrote the whole post.