WOOF! Newsletter

June 11, 2024

Google's Big Boo-Boo—What We Can Learn from Their Deleting a $128B Pension Account

Got a backup in the cloud? Good! Did you put in a secondary cloud backup, too? You might want to after reading about Google’s latest accident.

“If you don’t have cloud backups, you don’t have backups.”

You’ve heard this sentiment before. It’s true, but when it comes to cloud backups, one is not enough.

Google proved that with one of their biggest slip-ups ever. One error wiped out an Australian pension fund’s ability to do business – and nearly destroyed all their backups, too!


The Backstory

UniSuper is a pension fund in Australia.  It manages over $128 billion in Australian dollars for is 600,000+ members. In 2023, UniSuper migrated its fund data to a Google Cloud account.

In early May 2024, UniSuper members began to see errors when accessing their accounts. It’s as if the accounts had been wiped away.

Turns out that’s what happened. Google Cloud deleted the entire account! No warning, no billing issue...everything was just gone.

Didn’t Google have a backup of UniSuper’s account? Yes, it did...but that had been deleted, too.

Fortunately, UniSuper had extra backups with another cloud service. Working with those backups and their own internal systems, Google eventually restored UniSuper’s operations.

UniSuper’s operations were down for 2 weeks from this. They documented everything here, if you want more details: https://www.unisuper.com.au/contact-us/outage-update

 

The Lesson – Always Have More Than One Backup

The UniSuper outage didn’t occur because of a cyberattack. It was an accident—a “misconfiguration” within Google Cloud, as they labeled it in statements.

UniSuper was smart to maintain a third-party cloud backup. This did help in the restoration process (though Google at first concentrated on restoring their account, sacrificing time). 

Fortunately for all of us, accidents like this one don’t happen often. What happens more often are things like:

  • Malware infiltrating your devices (and that includes cloud services)
  • Ransomware locking up your data
  • Hardware issues wiping out data/bricking phones

Without separate backup locations, these can take your backups down with them, like what happened to UniSuper.

Let’s not let that happen, okay? Here are industry best practices for your data backups.

 

Best Backup Practices - Microsoft 365 / Google Workspace Backups

For Microsoft 365 and Google Workspace tenants, make sure you have third-party cloud backups setup for all of your accounts.  The backups are reasonably priced and provide a failsafe in case your team (or even the provider) accidentally deletes data or you experience a malware attack.

 

Best Backup Practices - On-Premises Server and Workstation Backups

Aim for three backups total: 

  1. Local Backup
    • For on-premises servers, purchase a local backup solution that supports encryption.
    • Make sure to check your local backups regularly; no backup solution works continuously without issue.
    • Limit physical access to your local backup device. Lock your server room and provide access to authorized users only.
    • If you only have budget for two of the three recommended backups here, focus on your Primary and Secondary cloud backups.

  2. Primary Cloud Backup
    • Cloud backups protect against data loss in the event of IT hardware failures or a cyberattack.
    • The “cloud” means there’s an air gap between your server hardware and the server hardware where it’s backed up. 
    • Put cloud backups on your on-premises servers and critical workstations (C-Suite, Accounting, Developers, workers who travel, etc.).
    • Make sure your cloud backups use encryption and versioning. This prevents data corruption from a malware attack passing through to your cloud backup and corrupting it!
    • Set 30-day retention for your backups. Use 90 days if you are subject to SOX.
    • Your IT department or MSP should conduct monthly “spot test” restores of your cloud backups, to verify their restorability.
    • Once a year, perform a full backup restoration, to verify that you can restore the backups without issue (just in case!).

  3. Secondary Cloud Backup
    • The second cloud backup should be geographically separate from your primary cloud backup AND your office.
    • This provides an added layer of protection in case of a natural or manmade disaster (e.g., cyberattack, human error, etc.) at your primary cloud backup provider.
    • The second cloud backup can take the form of a mirror of your primary cloud backup, or a completely separate provider. (Mirror copies allow for fast restores.)
      • Like your primary cloud backup, test your mirror backups on a monthly basis to make sure your worst-case scenario backup is ready when needed.

Data Accidents Happen – Plan for Them

Behind every slick-looking tech company are a bunch of humans who make mistakes, just like everyone else. Google admitted this in a statement on the UniSuper incident:

"During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank.”

A blank parameter. That’s all it took to wipe out a major pension fund’s data and nearly destroy their ability to recover.

One of the sources we consulted ended with this line: “Don't put all your eggs in one cloud basket.” We couldn’t agree more.

 

Feeling like it’s time to revisit your backups? Contact PlanetMagpie's Support Team for a helping hand, at sales@planetmagpie.com.