string excuses[]={
"it's not a bug it's a feature",
"it worked on my machine",
"i tested it and it worked",
"its production ready",
"your browser must be caching the old content",
"that error means it was successful",
"the client fucked it up",
"the systems crashed and the code got lost" ,
"this code wont go into the final version",
"It's a compiler issue",
"it's only a minor issue",
"this will take two weeks max",
"my code is flawless must be someone else's mistake",
"it worked a minute ago",
"that was not in the original specification",
"i will fix this",
"I was told to stop working on that when something important came up",
"You must have the wrong version",
"that's way beyond my pay grade",
"that's just an unlucky coincidence",
"i saw the new guy screw around with the systems",
"our servers must've been hacked",
"i wasn't given enough time",
"its the designers fault",
"it probably won't happen again",
"your expectations were unrealistic",
"everything's great on my end",
"that's not my code",
"it's a hardware problem",
"it's a firewall issue",
"it's a character encoding issue",
"a third party API isn't responding",
"that was only supposed to be a placeholder",
"The third party documentation is wrong",
"that was just a temporary fix.",
"We outsourced that months ago.","
"that value is only wrong half of the time.",
"the person responsible for that does not work here anymore",
"That was literally a one in a million error",
"our servers couldn't handle the traffic the app was receiving",
"your machines processors must be too slow",
"your pc is too outdated",
"that is a known issue with the programming language",
"it would take too much time and resources to rebuild from scratch",
"this is historically grown",
"users will hardly notice that",
"i will fix it" };

Worst thing you've seen another dev do? Long one, but has a happy ending.

Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.

The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.

Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.

At 5:00PM on a Friday, deploys, goes home.

Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)

After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)

We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.

Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.

Well, since I was responsible for logging, I was next in the hot seat.

For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."

Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.

A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.

Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).

!dev I'd just helped a client cut over to a new fiber connection and then left for Vegas, about 2 days into the trip my wife and I decided to hit a breakfast spot that had bottomless mimosa's, which was of course a claim we had to test.

As we are walking(stumbling) out of the restaurant I get a call that the connection has crashed and the entire car dealership is unable to sell cars, which they tell me is important functionality.

So I make it up to my room and break out the laptop, luckily the mgmt interfaces are still available externally so I'm able to log in and then have the fun challenge of 1) not falling off of my chair 2) not accidentally making a change that kills what connection I have in and 3) fixing their actual issue.

Took me almost an hour to find a simple OSPF issue but at least got them working and happy. However by that time I was beginning to sober up, which is the absolute worst thing that can happen while day-drinking and ended up basically causing me to be be hung-over for the rest of the night, including my wifes friends wedding, which she wasn't thrilled about...

The moral of this story is to make sure to NOT stop drinking while dealing with unexpected production impacting events.

So we hired an intern and his first task was to change a few things in email layout for our client, which is an investment bank.

I told to one of my developers to make his local database dump and setup the project for an intern. When intern completed the task, my developer thought that title "Dow Jones index crashed" was pretty funny title for a test.

What he didn't thought through enough, is that he forgot to configure fake SMTP server and he had production database dump with real email addresses.

I had really awkward 20 minutes conversation with our client. Fuck my life.

One of our senior dev enjoys berating the other devs because they don't check-in code according to his schedule (once a day, once an hour..he flip-flops a lot), then when they do, he 'reviews' their code, beating them up because of incomplete features, commented out code..petty..petty nonsense.
Ex. (this occurred couple of weeks ago).

Ralph: "The button click code in this event isn't complete"
Dev: "No, its not, the code in my development branch. You said it was best practice to check in code daily whether the code worked or not. I didn't finish the event last night and ..."
Ralph: "Exactly. Before you check any code into source control, it has to work and be 100% complete. What if someone moved that code into production? What happens if that code got deployed? I'm not even going talk about the lack of unit tests."
Dev: "Uh..well..the code is on the development channel, and I branched the project in my folder ...I didn't think it mattered.."
Ralph: "Ha ha...you see what happens when you don't think...listen..."
- blah blah blah for 10 minutes of hyperbole nonsense of source control check-in 'best practice'

This morning Ralph's computer's hard-drive crashed.
Ralph: "F-k! ..F-k! ... my f-king computer hard drive crashed!"
Me: "Ouch...did you loose anything important?"
Ralph: "A f-king week of code changes."
Me: "You checked everything into source control on Friday ...didn't you?"
Ralph: "F-k no!...I got busy...and...f-k!"
Me: "Look at the bright side, you'll have a good story to tell about the importance of daily check-ins"

Oh...if looks could kill. Karma...you're the best.

My app crashed in production and there's nothing in in the error logs..

Yay..

It is the year 2451 ad and mankind rules the galaxy with a lazy iron fist. There are roughly 14,000 civilizations, comprised of just over
17,000 intelligent species on a quarter of a million earth-like
worlds. And all of them call themselves 'the galactic empire'.
No one told them that twenty planets doesn't qualify them for the title "galactic."

Well, we could rule, if we wanted to. Most of its just backwaters that no one wants anyway. It turned out that the reason no one invaded earth before was because they were too busy fighting themselves. Stupidity it appears, is not a unique human quality.That and the sex robots. Theres more of them in the galaxy than actual meatbags. Many species had taken to artificial wombs and 'vatbabies', which is exactly what they are called. Those poor bastards will carry that label for life.

We never did break light speed, but most of the rich exist in hypersleep anyway. Most of them only wake up once a year or so. There are some that only creek out of bed to check their stock portfolio. I hear there is even one trillionaire thats up and about once a century to ask if we have broken light speed yet.

Despite all the progress over the last 400 years, historians all agree about the most significant event in modern history.
The lobster went extinct two hundred years ago on earth.
Theres been riots ever since.

* * *

In other news I'm still working on the game I guess. It's like totally the most okay indie game you'll ever play--if I ever finish it.

I put about a year of work into the NPC system, and then chatGPT came out.

After everything thats happened, at this point I may just make a game about an indie dev making a survival game, being stuck in the actual apocalypse or some weird political dysopia.

Put it on rewind, it was originally a zombie game. But at the time the market got flooded and steam sales for zombie games cratered. So I pivoted to something more along the lines of fallout. Then the flash market crashed, bunch of publishers folded, and adobe stopped support for flash (probably for the best). Then newgrounds, which I was gonna launch on for promotion (because actual marketing is expensive), ended support for flash.
Was going the route of kickstarter, and that year the KS market got flooded and the bar rose almost over night so you needed super high production quality out the gate, and a network of support you already built for months.

We had a brief nuclear war scare, and I watched the articles come out about market saturation for post-apocalypse games, so I pivoted back to zombies. Then covid happened and the entire topic was really fucked. So I went back to fallout meets rimworld. Then we had a flood of games doing that exact premise pretty much out of the fucking blue, so I went for a more single-survivor type game. Then ukraine happened and the threat of nuclear war has been slowly sapping the genre of its steam, on well, steam.
Then I was told to get a cancer screening which I can't afford. Then I broke a tooth and spent a month in agony.

Then a family member died. Then I made no money from the sale of a business I did everything to help get off the ground, then I helped renovate an entire house on short notice and sell it, then I lost two months living in a hotel
while looking for a new place to live. Then I spent two and a half years suffering low-level alcoholism, insomnia, and drifting between jobs.
Then I wrote amazing poetry. And then I rediscovered my love of math. And then I made out for the first time in over a year. And then I rediscovered my love of piano and guitar. And then I fell into severe depression for the last year. Then I made actual discoveries in math. And I learned to love my hobbies again, and jog, and not drink so much, and sing, and go on long drives, and occasional hikes, and talk to people again, and even start designing games and UIs again. And then I learned that doing amazing things without a lot of money is still possible, and then I discovered the sunk cost fallacy, and run on sentences, and how inside me there was a part of me that refused to quit because of circumstances I couldn't control, and then I learned that life goes on even when others lives have ended, even when everything and everyone never had an once of faith in you, and you've become the avatar of the bad luck brian meme..still, life goes on.

And we try to pick up the pieces, try, one more time, because the climb, and the fall, and the getting back up, is all there is.

What I would recommend, if you're thinking of making a game, or becoming an independent game developer, is, unless you have a *lot* of money upfront (think 50-100k saved, minimum, like one years income *bare* minimum), and unless you already have a full decade in the industry--don't make a game.

Just don't.

Three days ago my focus was shifted from a development role to a support role. I was shifted to replace another support guy who had used fraud to get the position. I have no experience with this role but there was decent KT and I'm catching on fine. During onboarding and KT I'm serving as the first contact for new tickets and whatnot...

Today I got a ticket with an error on our production instance that no one had ever seen before. It prevented the guy from using our service entirely. I tried to reproduce it and... I couldn't use the service either. No one could. Everything was down. I could see the sweat building on my manager's forehead.

Thankfully another member on my team has done a bit of support before, so we collaborated with each other and other teams throughout the day to figure out what's wrong and how to fix it. I'm listening to them chat remotely as we speak - so far I've been working on it 9 hours straight.

This service is used by everyone - it's a business critical service with due dates on actions and escalations to managers... Imagine if the support ticketing service for your company crashed. That means a lot of people are asking what's wrong, requiring extensions, etc. I've been answering to managers and seniors in the business throughout the day.

The best part? We figured out why the server went down, and the reason is fantastic: someone updated the server's code without telling anyone, and all they had done was remove critical parsing code. Just took it right out, pushed, redeployed. We don't know who did it or who even has access to do that. I guess I have some detective work cut out for me after we've fixed everything that was broken by that.

I miss coding already.

More and more, I am getting frustrated/depressed from the attitude of our customers who complain, moan and get angry about issues in their infrastructure, while at the same time, refusing to pay more so the issues could be mitigated.

Like, a client's angry with us today for having one of their non-production-critical databases inaccessible for... Hmm... About 8 hours now (So a whole workday).

Like... I get it, some of your employees couldn't work with it offline, but like... What the hell do we do? You keep data from as far back as several years ago in there, without partitioning, without exports, in a mix of innodb and myisam, so when the DB crashes, and its replication has to be reset from zero, reimporting all the data takes hours upon hours, and importing .sql files just takes time.

Or another client who got angry when their app fell out of the internet, cuz one of their myisam-based log tables crashed, and had to be repaired, with data spanning several years back, meaning it took hours to fix...

The more I work with these "basic" and "simple" infrastructure designs that is *not* redundant, or HA, the more I wonder -- How do the big names out there do it? How do you design systems with fault tolerance so a single DB table crash doesn't lead to the whole app getting inaccessible?

We have... One, exactly one, client, who uses MariaDB with Gallera, and that cluster is *amazing*, it just keeps chugging along, without a care in the world. But it cost them quite a lot, as they had to buy 3 DB servers, instead of 1...