devRant - A fun community for developers to connect over code, tech & life as a programmer

The universe has taken a cactus.

It proceeded to gift the cactus with a toxin that greatly enhances the stimulus of pain.

After the universe watched it's miraculous creation it decided to shove it up so far my arse that my gag reflex turned on and I puked a lot of cactus.

Didn't sleep well, weekend hardware migration finish, today an old server got moved.

Some part, most likely the redundant PSU, had a short circuit - decided to take the switches out... Which are the only non redundant hardware...

There was only one critical system in the whole rack, that was one redundant firewall.

Guess what happened..... Naaaa?

*drum roll*

For whatever reason, the second firewall didn't kick in, so large part of internal network unreachable as VPN was on the firewall.

:thumbsup:

That's not cactus level yet.

Spontaneously a large part of the work at home crew decided to call, cause getting an email wasn't enough.

So while all the phones were ringing and we had the joyful fun to carefully take apart a whole rack to check for possible faulty wiring / electric burns / hardware damage and getting firewall up and running again...

Some dev decided to run a deployment (doable as one of the few working at the company at the moment -.-).

I work from home, but we had a conference phone call running the whole time so I could "deescalate" and keep others up-to-date. So me on headphone with conference call, regular phone for calls, while typing mails / sms for de-escalation.

Now we're reaching cactus level, cause being tortured by being annoyed out of hell by all telephone ringing, the beeping of UPS (uninterruptible power supplies), the screaming of admins from the server room and the roaring of air coolers…

Suddenly said dev must have stood in the midst of the chaos… and asked for help cause "the deployment broke, project XY is offline"...

I think it was the first time since years that I screamed at the top of my lungs.

Bad idea (health issues)… but oh boy was it a pleasure to hear my own voice echo through the conference speaker and creating an echoic sound effect.

It was definitely worth coughing out my loungs for the next hour and I think it was the best emotional outburst ever.

I feel a bit sorry for the dev, but only a tiny bit.

After the whole rack thing, the broken deployment fixing and the "my ears are bleeding and I think I will never be able to talk again" action...

We had to roll out several emergency deployments to fix CVEs (eg libexpat).

This day was a marvelous shit show.

I will now cry myself to sleep with some codein.