"Where's the database server gone?"

And where was it gone?

@asgs just fell out of the cloud... When someone gave marketing access to the azure tenant and they deleted the whole container including backups.

@craig939393 omg 😯
what was their reasoning? 😆

@heyheni you give them too much credit

@craig939393 So what about that "dont give everyone root privileges thing. Especially then when they don't know what that is"? :D

@hypervtechnics you give my company too much credit.

POLP: a case study

@craig939393
When you can delete a backup without having to physically move or connect anything, it actually never was a backup in the first place!

@Oktokolo disagree with you there. With physical mediums there is just as much if not more danger. Archive should be read only and a backup that is not verified (on actual content) can be assumed to not exist.

@hjk101
That is why you have multiple backups.
If they are all accessible by some person online, they are accessible by users of the malware on that person's device too.

Protection against hardware failure and user error is not enough.
If a new 0-day leaks from NSA today, it can already be in your generic ransomware toolkit a week later.
But it may take weeks or even months until the infection is detected - usually by inability to use data or reading an extortion message...
Ransomware is a mature industry now (you can basically thank Bitcoin for that). Backup strategies (also) have to adapt.

@Oktokolo I've overcome a ransomeware attack by using the above.
To go in a bit more detail:
A backup is made to a bucket on a different cloud provider. The production cloud has write permissions to that bucket. Than on the backup cloud only a service account has write access to an archive bucket. A cloud function (using the service account) copies to the archive and afterwards tests the contents.
So nobody has access to tamper with the archive. Only with a key in the vault the backup cloud admin can be accessed.

@hjk101
You might get lucky, and ransomware never hits the device used to access that service account. It also isn't guaranteed, that online-wipeable "backups" actually get wiped.

But you can easily get 100% protection against all network-borne attacks to the point where the attacker even may have all credentials of all your cloud accounts and you still have a backup they just can't reach no matter what.
Just have an additional backup set consisting of the cheapest consumer HDDs which you store offline and use to weekly backup the most important data from the freshly filled cloud buckets.
Also protects against cloud provider dick moves.

@Oktokolo there is no way the service account can be accessed unless you suggest that the cloud provider itself gets hacked. Than we still have the production cloud but lose the archive.
There are other issues with physical mediums for us. You only feel safe using that and that's good to, just use that but it's not holy end all solution in my opinion.
I feel safe using the above setup. It survived a malicious admin and ransomeware attempt. I see that as enough proof, you can always say we were lucky

@rEaL-jAsEs
Backup as such is all about reducing the risk of data loss by having redundant storage which is likely not affected by the same desaster that hits the live data.

So you can just use the cheapest spinning rust drives. You only need to use a seperate one for each day of week, each week of month, each month of year and each year. That are 24 crappy disks for the first year and one extra added each year. Each of them isn't that reliable on its own. But they carry mostly the same data anyway and the redundancy is easy to scale up by just using more disks.
So in case of an instantly detected desaster, it is extremely likely, that you only lose one day, maybe two.
In case of detection of ransomware by extortion message on the screen, you might lose a month or even more (but most often it is only one to three weeks) if newer backups are also only containing mangled data.

The quality of the cheapest available consumer disks is definitely good enough for this.

@rEaL-jAsEs same! Our German team had a setup like Okto describes but that didn't end well, both hardware failure and human error (wrong disk inserted). Also quite a few assumptions are made on the nature of the data. Could never have worked with our team.

😳

@craig939393 after reading what happened if gave me second hand anxiety of prod being unrecoverably down