TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

Tip of monday:

Test your restores, not your backups.

Just got f#@ed by trusting the log, not the restore folder size

Discovered yesterday that my boss does tests with production database... and I'm responsible for the fucking backups and he doesn't even care to let me now when so I can at least schedule one at slave. Come on... it's not that hard to let others know or test on your own machine...

I was thinking about the problems one of our clients faced with the launch of their project the other day, because things were rushed, stuff was omitted and in the end they could not meet the launch date, and I started making a list of hard lessons I learned over the years that would have helped them avoid this situation.

Feel free to add yours in the comments.

- Never deploy on Friday
- Never make infrastructure changes right before a launch
- Always have backups. Always!
- Version control is never optional
- A missed deadline is better than a failed launch
- If everything is urgent, nothing is important
- Fast and cheap, cheap and quality, quality and fast. Only one pair at a time can be achieved
- Never rush the start or the end of a project
- Stability is always better that speed
- Make technical decisions based on the needs of the project two years from now
- Code like you will be the only maintainor of the project two years from now. You probably will...
- Always test before you deploy
- You can never have too many backups (see above)
- Code without documentation is a tool without instructions
- Free or famous does not necessarily mean useful or good
- If you need multiple sentences to explain a method, you should probably refactor
- If your logic is checked beforehand, writing the code becomes way easier
- Never assume you understand a request the first time around. Always follow up and confirm

There are many more that should be on this list, but this is what came to mind now.

Note to self.. Check how your backup restore works before actually needing it...

Coworker got ransomware to his computer via targeted attack and managed to encrypt about 6000 files on Google Drive share... Which I have backed up. However there are a total of 100k files so total restore is not an option and with 60 users updating things can't do point in time restore either... And thanks to the backup softwares buggy cmd line interface I can't create command line script to restore the files one by one... So in the end I most likely need to restore via sluggish gui one by one...

Just a note to everyone. Please make sure to test your script before putting it on crontab and soft-deleting all backups. They are restored now but seriously...