What is the worst production mistake you made?

Mine is restarting one of the main servers (wrong terminal window).

I deleted the wrong database table. Whoops!

Testing for SQL injection vulnerabilities... and succeeding...

Nothing that serious yet.
Try "rm -rf /" sometime 😉

One time a long time ago I accidentally deployed some infinite recursion. It brought down all the web servers and they wouldn't even take the code pushes with the fix so they all had to be restarted.

Accidentally batch updated a set of records in our CRM that triggered an email notification workflow.

Subscribed CRM users woke up to 2000 emails in their inbox.

@chrizzle lol I've done something similar

Not full on me, but the manager launched a Web site, posted about it on social media, and Then decided to look at it and make design changes. So, I'm making her changes on the fly and she's freaking out because it's live... Well, 20 minutes ago, the live server was my dev server. I wasn't the one who decided to make that IP the live server.

When I worked for a TV station the legacy application for scheduling shows didn't have a read only permissions so IT had read/write access in production for support purposes.

So one day I was testing changes in dev, scheduling new things only to realize that... Yeah, that wasn't Dev... And of course with my sense of humor I was scheduling crazy adult titles on the children's time slots.

Thank God my manager at the time had a good sense of humor and realized it was just a mistake.

@Yankeesrule LOL!

Btw, this rant topic is exactly the kind of thing I would want to do for the topic of the week feature we've been thinking about (I've posted about it in a few places). I think there are so many fun topics like this one and I would want to make sure they are easy to find and keep track of.

@dfox I am just glad the users caught it because I didn't realize what happened until my manager called me in. And angry parents are WAY worse than angry managers.

I work as a third party occasionally so I build sites for a company who deals directly with the client. I know him well, so when it comes to seeing the work on my Dev server I populate the Cms with dummy content. I spent ages re-writing Lorem Ipsum to contain hundreds of swear words but still resemble Latin (wish I still had it) as well as images of an unsavoury nature. It wasn't all like that just buried in places. Yep he sent the url and login to the client.

not really a production mistake but I wiped a live GPON config that had over 200 ONUs attached to it with no backup..... whoops, took a couple of days to rebuild as you could imagine the enpoint users where not impressed.

It wasn't me, but during highschool my friend was writing code to be used by the school. I'm still unaware of what the plan was, but to make a long story short the internet was taken down for the entire school system in my county.

@localhost yeah, i think i will pass on that one, haha ;)

Admittedly mine want too bad but man have i seen some things in my time.

The best by far have been someone deleting half a million customer records and another creating a search that deletes records it finds.

Still thinking of a good one for myself, but we had a kid working for a few weeks. One day he created himself an ftp login on our godaddy server we use for quick and dirty WordPress sites. He set the home directory for public_html. Not sure why, but he was unhappy with the setting so he deleted it to make a new one. Idk. But in godaddy, there's a box you need to make sure is unchecked that says 'delete users home directory'. He left it checked. Took down all our WordPress sites.

I once wrote an http interceptor for jobs which made a call to check something in cache for certain requests... Or so I thought. turns out it intercepted everything and queried the database. I effectively wrote a database denial of service tool into our own app. Let this be a lesson end to end test everything properly.

@dfox I did something similar at uni when we were learning about forking in c. Took down the machine we were remoted into, which apparently was also the head of ITs machine

@DaddyCool lol that's awesome.

I messed up a rsync command (with --delete of course) and it ended up deleting everything in /var/www on the production server.

Dropped my entire database from Compose.io when I thought I was doing a backup...

Processing 30k or so XML documents in parallel and found out about the automated load balancing took the sys admin used. He was watching the monitors and said all of the suddenly servers started disappearing (they were moving to other machines but not in any highly controlled way). He was one of the best sys admins ever...could always get extra compute and RAM...not like the shop now that is constantly running out of disk.

accidentally left off the where clause of an update... when I was updating the password for one of our users... ended up changing the passwords of something like 1000 users!! had a backup to restore to thankfully. That's the last time i try to slot in a quick update during a dota game (i work from home)

Leaving an environment=test in an API for booking flights.

@rossfrombritain same here but with the UIDs of the user accounts, which made it a bigger problem... oh and no backups :D

@dev-nope oh crap! what did you do?

@rossfrombritain well I ease myself, that I demonstrated the reason for doing backups

I moved a (non-rack) server over in the middle of the day ending up in the SCSI cable disconnecting from the external array. All processing halted for the next three hours as database transactions rolled back, etc.

Loving these guys and gals, keep them coming :)

We got featured today: http://stackify.com/co-founder-davi...

Devs have waaaay to much power