Not sure if it should be a joke or a rant, but something rather funny (at least to me) happened today.

TL;DR; Someone's outlook was crippled by 100k+ of warning notifications

So we have developed a server that has an internal database that wraps around an elastic search instance, that is managed by a POS vault/storage solution, that we have to use for legal reasons. The elastic search is "provided" by the software, but we keep this internal database just to be sure and totally not because it's unreliable POS.

Anyways, they take data integrity very seriously, so every warning our server produces is emailed to someone in charge to review it and if necessary forward it to us. This will be important later on.

Couple of days ago we got error forward when trying to write an entry into the POS software we get an error, because an object we tried to write already existed. After some investigation we concluded an entry was missing when the internal database was created, so we asked them to repopulate elastic search to solve this problem.

When start the server we always sync the internal database to the elastic search and emit a warning when an entry is missing in internal database or vice versa. And well... almost all of them were missing, which caused our server to emit ~40 warnings/ms. Poor outlook. Still investigating for the cause, but damn, I never expected I'd take down someone email account by accident