TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

Tip of monday:

Test your restores, not your backups.

Just got f#@ed by trusting the log, not the restore folder size

Discovered yesterday that my boss does tests with production database... and I'm responsible for the fucking backups and he doesn't even care to let me now when so I can at least schedule one at slave. Come on... it's not that hard to let others know or test on your own machine...

I was thinking about the problems one of our clients faced with the launch of their project the other day, because things were rushed, stuff was omitted and in the end they could not meet the launch date, and I started making a list of hard lessons I learned over the years that would have helped them avoid this situation.

Feel free to add yours in the comments.

- Never deploy on Friday
- Never make infrastructure changes right before a launch
- Always have backups. Always!
- Version control is never optional
- A missed deadline is better than a failed launch
- If everything is urgent, nothing is important
- Fast and cheap, cheap and quality, quality and fast. Only one pair at a time can be achieved
- Never rush the start or the end of a project
- Stability is always better that speed
- Make technical decisions based on the needs of the project two years from now
- Code like you will be the only maintainor of the project two years from now. You probably will...
- Always test before you deploy
- You can never have too many backups (see above)
- Code without documentation is a tool without instructions
- Free or famous does not necessarily mean useful or good
- If you need multiple sentences to explain a method, you should probably refactor
- If your logic is checked beforehand, writing the code becomes way easier
- Never assume you understand a request the first time around. Always follow up and confirm

There are many more that should be on this list, but this is what came to mind now.

Note to self.. Check how your backup restore works before actually needing it...

Coworker got ransomware to his computer via targeted attack and managed to encrypt about 6000 files on Google Drive share... Which I have backed up. However there are a total of 100k files so total restore is not an option and with 60 users updating things can't do point in time restore either... And thanks to the backup softwares buggy cmd line interface I can't create command line script to restore the files one by one... So in the end I most likely need to restore via sluggish gui one by one...

Just a note to everyone. Please make sure to test your script before putting it on crontab and soft-deleting all backups. They are restored now but seriously...

HOW TO RECOVER YOUR STOLEN CRYPTO INVESTMENT: REACH OUT TO RAPID DIGITAL RECOVERY

Teaching AI ethics is my day job, where I spend my days instructing students in the conflict between human judgment and machine learning, but I never thought I'd be outwitted by my own digital wallet. And yet there I was, staring at my fried hard drive and realizing that $910,000 worth of Bitcoin had just been encrypted into oblivion.
The disaster started innocently enough. I had been operating an experimental machine learning program, training it to improve encryption security independently. Enthusiastic with the encouraging results, I ran one final simulation on my home laptop, the same one that held my wallet keys in an encrypted file. What could possibly go wrong?
As would be the case, everything. The algorithm, eager to show itself, created a security system so robust that I couldn't even get access to it myself. It rendered the key file so corrupted that my life's savings might just as well be floating around cyberspace. When I tried to explain what had occurred in class the following day, the expressions on the faces of my classmates were one of amusement and horror. "Professor, didn't you teach data backups last week?" Ouch..Whatsapp: +1 4 14 80 71 4 85
A student approached me after class and softly whispered a lifeline: "Check out RAPID DIGITAL RECOVERY. They deal with cases like this." Desperation got the better of my pride, so I called them. From the first email, their team treated me like a valued partner, not another moron who let an AI lock him out...Email: rapiddigitalrecovery (@) execs. com
RAPID DIGITAL RECOVERY's engineers approached my problem like battle-hardened cryptographers and compassionate therapists. They requested samples of my encryption methods, dissected my test algorithm, and effectively reverse-engineered my own efforts. They even smiled (graciously) at the enthusiasm of my marauding AI for security...Telegram: @Rapiddigitalrecovery1
After 12 days of nail-biting suspense, I heard the call. "Professor, we got it." My heart pounded faster than when I first powered up a neural network. My Bitcoin wallet was returned, every single satoshi. They even provided me with security recommendations tailored to my academic community so that my next algorithm would not hold my money hostage.
Now, when I teach AI ethics, I introduce with this cautionary tale. My students chuckle, my robots beep their approval, and I sleep better knowing RAPID DIGITAL RECOVERY stands in the wings to rescue even the most hubristic tech wizards. Human ingenuity triumphs once more.