This is why my embedded system keeps crashing

Worst thing you've seen another dev do? Long one, but has a happy ending.

Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.

The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.

Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.

At 5:00PM on a Friday, deploys, goes home.

Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)

After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)

We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.

Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.

Well, since I was responsible for logging, I was next in the hot seat.

For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."

Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.

A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.

Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

So, basically i am getting desperate, and i'm also angry... and want to cry, and i feel a failure.
My biggest error in this story is "believing"

First of all, I'm a starting entrepreneur as freelancer, started of 6 months ago, back then it all seemed bright, i had my first customer, they believed me, got a second assignment for that same customer, fulfilled the task in brilliance, and was ready to move on to a bigger customer base. Here's where shit goes wrong.

Working with another office that outsources people to various (goverment) contractors, Had a meeting with them, we would cooperate very soon. This was january.

PM: "We've got a gov Dept as a customer that wants to do project XYZ and starting in february, requirements are yadiyadiyadi, you up for it?
Me:"Sure, send me the specs, and timeframe and i'll apply with my offer)"
Project is about a bit backup system migration, lots of fancy shmancy tech used, 2 datacenters setup... a big project that can take up several months...
- sends offer- received an acceptance on the offer

PM " Great, we'll start end of february"
Me: "Ok, end of february is a go then, looking forward to cooperate"

FF to near end of february, receive msg from PM: "Project XYZ delayed until half if march"
Me: "Okay... what is the delay?"
PM:"Govt bureaucracy"
Me " Ok, let's keep in touch"

Near Half of March
PM " Project delayed again".
Me: " Okay... what's the reason?"
PM: "they have decided to take a different approach, and want 3 datacenters now"
Me: "That will change the offer i made for you, can you send me the specs?"
PM: "No, because they havent decided on the techs used, expect end of march"
Me: "Okay...but once you got the specs,s end them to me"

End of March
Pm: " Hy NeatNerdPrime, we still havent got any specs yet... they still forgot to make the shopping list"
Me: "(-_-) ... I thought they already had that covered"
PM:" No, new Govt budget cuts and lots of changes, basically, they still don't know what they really want. But we're just -delayed- , not -cancelled-"
Me: "Okay... i hope we get started soon"
PM: "expect launch beginning of april"
I was not really satisfied with this explanation.... but ok

Beginning of april, at april's folls day
PM "Project delayed again"
Me: " This is a joke right?"
PM: "Sadly, no, they delayed the project since they don't really know what they actually want, we are trying to give them the proposed solution... but they still need approval, and still need to set up shopping list"
Me:"when do we start then?"
PM: " I was told 17th of april'
Me:" I really hope it gets through, i had to decline some proposals for work for months due to this, this ain't funny"
PM: "I know, i'll make up for it"

15th of april...
PM " Project delayed again"
Me right now almost losing my shit
"why this time??? I thought you said 17th of april real launch!"
PM: "they fired their PM, need to hire a new one, delayed until May/June"
Me: "I've set aside some other assignments just because you said you were going to launch at those dates... This is really pushing my limits, can you give me SOME assurances?"
PM: "5th of june would be official go" Me: " Okay, i'm noting 5th of june in my agenda, let's get this through!"

And now i get a message...stating that the date of 5th of june , is for another project, at the same goct dept, but totally different project

WHAT THE FUCKING SHIT I'VE SPENT MONTHS WAITING FOR A PROJECT THAT WAS PRACTICALLY ALL SET AND GO AND EVERY SINGLE FUCKING TIME IT GETS DELAYED, EVERY FUCKING TIME I THINK "you know, i cannot accept these offers since that Govt project will start soon, i cannot do 2 assignments at once especially when they require me to work at govt office at the capital..." AND EVERY FUCKING TIME IT GETS DELAYED!!!

I feel at a loss now..... i've done i think the most horrible thing you can do as a independent and that's not accepting another assignment just because there was a concrete promise for a govt contract...

Almost dried up, nothing much left, had to do some spending because of a move to another apartment... I'm feeling really down, and angry...and down, but mostly angry, for not accepting those offers in the meantime...

System failure

Co-worker: hey, can you create an email?

Me: yeah, who needs one? There are no records indicating any new people starting for another two weeks.

Cw: if for Stan, he started today. Also he needs a computer set up.

Me: who the hell is Stan and why are there no records of this person?

Cw: he's new, he started today so we didn't need an email or computer before today.

Me: I get that they're new, but what happened to giving the IT department at least 3 days notice on new hires so I can make sure things get taken care of?

Cw: you know how it is around here, nobody gets notice for anything. So can you get that email and computer setup for me, he can't work without them.

Me: I get that we don't actually plan for anything around here and that 90% of my job is fixing that failure, but hiring someone isn't like a system failing, people don't just show up and say "I start today" they have to go through interviews and background checks and other stuff, someone besides this person knew they started today so I don't think it's too much to ask that I get an email when the offer is extended to the person so I can prepare a system.

Cw: well we interviewed him two weeks ago and he accepted the offer last week, he's here and waiting so just as soon as you can please.

Me: well here's an email, the computer is gonna have to wait, I have a lot going on today and I don't have any computers ready right now.

*Seriously tempted to make them wait till next week to cover the 3 days notice I've asked for 100 times*

(Written March 13th at 2am.)

This morning (yesterday), my computer decided not to boot again: it halts on "cannot find firmware rtl-whatever" every time. (it has booted just fine several times since removing the firmware.) I've had quite the ordeal today trying to fix it, and every freaking step along the way has thrown errors and/or required workarounds and a lot of research.

Let's make a list of everything that went wrong!

1) Live CD: 2yo had been playing with it, and lost it. Not easy to find, and super smudgy.

2) Unencrypt volume: Dolphin reports errors when decrypting the volume. Research reveals the Live CD doesn't incude the cryptsetup packages. First attempts at installing them mysteriously fail.

3) Break for Lunch: automatic powersaving features turned off the displays, and also killed my session.

4) Live CD redux: 25min phonecall from work! yay, more things added to my six-month backlog.

5) Mount encrypted volume: Dolphin doesn't know how, and neither do I. Research ensues. Missing LVM2 package; lvmetad connection failure ad nauseam; had to look up commands to unlock, clone, open, and mount encrypted Luks volume, and how to perform these actions on Debian instead of Ubuntu/Kali. This group of steps took four hours.

6) Chroot into mounted volume group: No DNS! Research reveals how to share the host's resolv with the chroot.

7) `# apt install firmware-realtek`: /boot/initrd.img does not exist. Cannot update.

8) Find and mount /boot, then reinstall firmware: Apt cannot write to its log (minor), listed three install warnings, and initially refused to write to /boot/initrd.img-[...]

9) Reboot!: Volume group not found. Cannot process volume group. Dropping to a shell! oh no..

(Not listed: much research, many repeated attempts with various changes.)

At this point it's been 9 hours. I'm exhausted and frustrated and running out of ideas, so I ask @perfectasshole for help.
He walks me through some debugging steps (most of which i've already done), and we both get frustrated because everything looks correct but isn't working.

10) Thirteenth coming of the Live CD: `update-initramfs -u` within chroot throws warnings about /etc/crypttab and fsck, but everything looks fine with both. Still won't boot. Editing grub config manually to use the new volume group name likewise produces no boots. Nothing is making sense.

11) Rename volume group: doubles -'s for whatever reason; Rebooting gives the same dreaded "dropping to a shell" result.

A huge thank-you to @perfectasshole for spending three hours fighting with this issue with me! I finally fixed it about half an hour after he went to bed.

After renaming the volume group to what it was originally, one of the three recovery modes managed to actually boot and load the volume. From there I was able to run `update-initramfs -u` from the system proper (which completed without issue) and was able to boot normally thereafter.

I've run updates and rebooted twice now.
After twelve+ hours... yay, I have my Debian back!

oof.

Story time:
Yesterday I wanted to go to the theater with my girlfriend. It was her idea because as a student you can get reduced tickets for the play, but only via the online store exactely two hours before the play starts. We had already tried two weeks before but with no success. So this time I said i want to be on my pc with a proper browser and not a mobile version like last time. So we are sitting at home me in front of their website on one screen and with a clock on the other screen. Two minutes realy i hit refresh and I get a selection for the reduced tickets, nice.

You would think.

After selecting the amount. ERROR: Can not get your tickets. I was like fuck they are already sold out because it's a popular play. But hey let's try again. I got one ticket but not the second one, okay strange lets try again, same ERROR again. WHAT the FUCK, no feedback what so ever. My girlfriend had then the idea that they maybe restricted the amount for reduced tickets to one (does not state this explicitly but hey lets give it a shot). Use second browser select one ticket. ERROR can not get you the amount of seats. Rage level near to a 1000 why did it work two minutes before but not anymore. Trying around for five more minutes finally got the second ticket.

Now the real fun begins.

Proceeding to checkout should not be that hard you would think, but you need to be registered for that. Okay so let's do that. The salutation is not required neither is the address for the tickets but you need to have a company name??!!!!! The fuck?? I am not self employed and neither are a most other people around here so why is this field mandatory? Beeing a little under stress I decided to found the "asdf" company with my girlfriend.

Now one would think checking out is easy. Not so fast.

After accepting the terms of service another ERROR, unable to accept your data. What data? I did not input anything new? Where does this come from? Ok never mind I am going to pay with credid card that must work!
ERROR: Internal paymentservice initialization failure! Sorry what? I thought maybe I was to long idle in this browser and they do not reserve the tickets for so long (which would be no surprise to me at this point). Let's try again. Nope same error.

Now my rage level was really over 9000 but we really wanted to go so I decided to call the customer SUPPORT. Or better to say I had a answering maching telling me for ten minutes how sorry they are that this takes so long, yeah you bet. Then and this is now really great: the support guy asks me: "What error do you see? Internal paymentservice initialization failure?" I was like, okay he knows this so they need to know how to handle it. FUCK NO. "Sorry I can't help you. This is our payment system maybe they (IT) are doing some maintenance I can't halp you. Call the theater directly good day." Sorry what just happened, you fuckers are the vendors for the tickets for nearly all big events around here and the theater explicitly states to call you for tickets but you can not help me? Like hell.

This process took 25 very frustrating minutes and I was really angry and wanted to quit, then I saw that there is also a paypal option which I had not tried. With very little hope i selected everything for the payment, registered with paypal and they told me I already had an account. So reactivated this five year old account payed with all the mobile passwords and tans to finally, after 30 fucking minutes, get a pdf file for a ticket. Repeated the last step for the second ticket and with some time left to get there we were off.

Damn, credit cards are so fucking secure these days that you hardly can BUY shit with them!
I need some special electronics that I only can get from a vendor in the US, which is overseas. Click click, buy, done. Well no, credit card refused. WTF? Click retry link. No, still refused. FUCK.
Called up the 24/7 hotline of my CC company. Oh yeah, that got blocked by the security system, somehow. We disable that for 20 minutes, just retry. Clicked retry link at the vendor. No failure mail. Hmmm, too good to be true?! Called up the electronics vendor. Yeah should work, stuff is in the warehouse stage. 40 minutes later: credit card declined. FUCK.
Called up the CC company again. Ok, disable blocker for one hour. Nice advice from them, tell the vendor it's only 45 minutes so that there's some buffer. Clicked retry link at the vendor and called them up to make sure that they retry before the time runs out.
LO AND BEHOLD, I could finally pay the shit!!

I worked on an amazing web app that tracked satellites and their failure rates. it was beautiful and worked fantastically. Me team worked like dogs to get it done three months. Our manager insisted it has to work perfectly for the demo and we delivered... We all got raises and everyone was happy, right? WRONG!

Demo day comes and management decides that they don't want to build the app because the customer pulled out. Sooo my manager then asked me why I wasted so much time building it when static images would have been fine. I lost my cool and yelled at my manager "YOU'RE THE ONE THAT TOLD ME TO GET A FULLY FUNCTIONAL SYSTEM WORKING!" Then I stormed out of the meeting.

It is still the coolest thing I ever built. Too bad it will never see the light of day.

Worst hack/attack I had to deal with?

Worst, or funniest. A partnership with a Canadian company got turned upside down and our company decided to 'part ways' by simply not returning his phone calls/emails, etc. A big 'jerk move' IMO, but all I was responsible for was a web portal into our system (submitting orders, inventory, etc).

After the separation, I removed the login permissions, but the ex-partner system was set up to 'ping' our site for various updates and we were logging the failed login attempts, maybe 5 a day or so. Our network admin got tired of seeing that error in his logs and reached out to the VP (responsible for the 'break up') and requested he tell the partner their system is still trying to login and stop it. Couple of days later, we were getting random 300, 500, 1000 failed login attempts (causing automated emails to notify that there was a problem). The partner knew that we were likely getting alerted, and kept up the barage. When alerts get high enough, they are sent to the IT-VP, which gets a whole bunch of people involved.
VP-Marketing: "Why are you allowing them into our system?! Cut them off, NOW!"
Me: "I'm not letting them in, I'm stopping them, hence the login error."
VP-Marketing: "That jackass said he will keep trying to get into our system unless we pay him $10,000. Just turn those machines off!"
VP-IT : "We can't. They serve our other international partners."
<slams hand on table>
VP-Marketing: "I don't fucking believe this! How the fuck did you let this happen!?"
VP-IT: "Yes, you shouldn't have allowed the partner into our system to begin with. What are you going to do to fix this situation?"
Me: "Um, we've been testing for months already went live some time ago. I didn't know you defaulted on the contract until last week. 'Jake' is likely running a script. He'll get bored of doing that and in a couple of weeks, he'll stop. I say lets ignore him. This really a network problem, not a coding problem."
IT-MGR: "Now..now...lets not make excuses and point fingers. It's time to fix your code."
IT-VP: "I agree. We're not going to let anyone blackmail us. Make it happen."

So I figure out the partner's IP address, and hard-code the value in my service so it doesn't log the login failure (if IP = '10.50.etc and so on' major hack job). That worked for a couple of days, then (I suspect) the ISP re-assigned a new IP and the errors started up again.

After a few angry emails from the 'powers-that-be', our network admin stops by my desk.
D: "Dude, I'm sorry, I've been so busy. I just heard and I wished they had told me what was going on. I'm going to block his entire domain and send a request to the ISP to shut him down. This was my problem to fix, you should have never been involved."

After 'D' worked his mojo, the errors stopped.

Month later, 'D' gave me an update. He was still logging the traffic from the partner's system (the ISP wanted extensive logs to prove the customer was abusing their service) and like magic one day, it all stopped. ~2 weeks after the 'break up'.

I've found and fixed any kind of "bad bug" I can think of over my career from allowing negative financial transfers to weird platform specific behaviour, here are a few of the more interesting ones that come to mind...

#1 - Most expensive lesson learned

Almost 10 years ago (while learning to code) I wrote a loyalty card system that ended up going national. Fast forward 2 years and by some miracle the system still worked and had services running on 500+ POS servers in large retail stores uploading thousands of transactions each second - due to this increased traffic to stay ahead of any trouble we decided to add a loadbalancer to our backend.

This was simply a matter of re-assigning the IP and would cause 10-15 minutes of downtime (for the first time ever), we made the switch and everything seemed perfect. Too perfect...

After 10 minutes every phone in the office started going beserk - calls where coming in about store servers irreparably crashing all over the country taking all the tills offline and forcing them to close doors midday. It was bad and we couldn't conceive how it could possibly be us or our software to blame.

Turns out we made the local service write any web service errors to a log file upon failure for debugging purposes before retrying - a perfectly sensible thing to do if I hadn't forgotten to check the size of or clear the log file. In about 15 minutes of downtime each stores error log proceeded to grow and consume every available byte of HD space before crashing windows.

#2 - Hardest to find

This was a true "Nessie" bug.. We had a single codebase powering a few hundred sites. Every now and then at some point the web server would spontaneously die and vommit a bunch of sql statements and sensitive data back to the user causing huge concern but I could never remotely replicate the behaviour - until 4 years later it happened to one of our support staff and I could pull out their network & session info.

Turns out years back when the server was first setup each domain was added as an individual "Site" on IIS but shared the same root directory and hence the same session path. It would have remained unnoticed if we had not grown but as our traffic increased ever so often 2 users of different sites would end up sharing a session id causing the server to promptly implode on itself.

#3 - Most elegant fix

Same bastard IIS server as #2. Codebase was the most unsecure unstable travesty I've ever worked with - sql injection vuns in EVERY URL, sql statements stored in COOKIES... this thing was irreparably fucked up but had to stay online until it could be replaced. Basically every other day it got hit by bots ended up sending bluepill spam or mining shitcoin and I would simply delete the instance and recreate it in a semi un-compromised state which was an acceptable solution for the business for uptime... until we we're DDOS'ed for 5 days straight.

My hands were tied and there was no way to mitigate it except for stopping individual sites as they came under attack and starting them after it subsided... (for some reason they seemed to be targeting by domain instead of ip). After 3 days of doing this manually I was given the go ahead to use any resources necessary to make it stop and especially since it was IIS6 I had no fucking clue where to start.

So I stuck to what I knew and deployed a $5 vm running an Nginx reverse proxy with heavy caching and rate limiting linked to a custom fail2ban plugin in in front of the insecure server. The attacks died instantly, the server sped up 10x and was never compromised by bots again (presumably since they got back a linux user agent). To this day I marvel at this miracle $5 fix.

Equality: no matter what company you are, facing Windows, you will fail!

Reminder (and probably some form of repost)

Food and sleep are key parts of human behavior.

I forgot and experienced system failure

Biggest challenge I overcame as dev? One of many.

Avoiding a life sentence when the 'powers that be' targeted one of my libraries for the root cause of system performance issues and I didn't correct that accusation with a flame thrower.

What the accusation? What I named the library. Yep. The *name* was causing every single problem in the system.

Panorama (very, very expensive APM system at the time) identified my library in it's analysis, the calls to/from SQLServer was the bottleneck

We had one of Panorama's engineers on-site and he asked what (not the actual name) MyLibrary was and (I'll preface I did not know or involved in any of the so-called 'research') a crack team of developers+managers researched the system thoroughly and found MyLibrary was used in just about every project. I wrote the .Net 1.1 MyLibrary as a mini-ORM to simplify the execution of database code (stored procs, etc) and gracefully handle+log database exceptions (auto-logged details such as the target db, stored procedure name, parameter values, etc, everything you'd need to troubleshoot database errors). This was before Dapper and the other fancy tools used by kids these days.

By the time the news got to me, there was a team cobbled together who's only focus was to remove any/every trace of MyLibrary from the code base. Using Waterfall, they calculated it would take at least a year to remove+replace MyLibrary with the equivalent ADO.Net plumbing.

In a department wide meeting:

DeptMgr: "This day forward, no one is to use MyLibrary to access the database! It's slow, unprofessionally named, and the root cause of all the database issues."
Me: "What about MyLibrary is slow? It's excecuting standard the ADO.Net code. Only extra bit of code is the exception handling to capture the details when the exception is logged."
DeptMgr: "We've spent the last 6 weeks with the Panorama engineer and he's identified MyLibrary as the cause. Company has spent over $100,000 on this software and we have to make fact based decisions. Look at this slide ... "
<DeptMgr shows a histogram of the stacktrace, showing MyLibrary as the slowest>
Me: "You do realize that the execution time is the database call itself, not the code. In that example, the invoice call, it's the stored procedure that taking 5 seconds, not MyLibrary."
<at this point, DeptMgr is getting red-face mad>
AreaMgr: "Yes...yes...but if we stopped using MyLibrary, removing the unnecessary layers, will make the code run faster."
<typical headknodd-ers knod their heads in agreement>
Dev01: "The loading of MyLibrary takes CPU cycles away from code that supports our customers. Every CPU cycle counts."
<headknod-ding continues>
Me: "I'm really confused. Maybe I'm looking at the data wrong. On the slide where you highlighted all the bottlenecks, the histogram shows the latency is the database, I mean...it's right there, in red. Am I looking at it wrong?"
<this was meeting with 20+ other devs, mgrs, a VP, the Panorama engineer>
DeptMgr: "Yes you are! I know MyLibrary is your baby. You need to check your ego at the door and face the facts. Your MyLibrary is a failed experiment and needs to be exterminated from this system!"

Fast forward 9 months, maybe 50% of the projects updated, come across the documentation left from the Panorama. Even after the removal of MyLibrary, there was zero increases in performance. The engineer recommended DBAs start optimizing their indexes and other N+1 problems discovered. I decide to ask the developer who lead the re-write.

Me: "I see that removing MyLibrary did nothing to improve performance."
Dev: "Yes, DeptMgr was pissed. He was ready to throw the Panorama engineer out a window when he said the problems were in the database all along. Didn't you say that?"
Me: "Um, so is this re-write project dead?"
Dev: "No. Removing MyLibrary introduced all kinds of bugs. All the boilerplate ADO.Net code caused a lot of unhandled exceptions, then we had to go back and write exception handling code."
Me: "What a failure. What dipshit would think writing more code leads to less bugs?"
Dev: "I know, I know. We're so far behind schedule. We had to come up with something. I ended up writing a library to make replacing MyLibrary easier. I called it KnightRider. Like the TV show. Everyone is excited to speed up their code with KnightRider. Same method names, same exception handling. All we have to do is replace MyLibrary with KnightRider and we're done."
Me: "Won't the bottlenecks then point to KnightRider?"
Dev: "Meh, not my problem. Panorama meets primarily with the DBAs and the networking team now. I doubt we ever use Panorama to look at our C# code."

Needless to say, I was (still) pissed that they had used MyLibrary as dirty word and a scapegoat for months when they *knew* where the problems were. Pissed enough for a flamethrower? Maybe.

We started a project in January for which I was the sole developer, to automate tedious interaction with a vendor's ticketing system. We have a storage environment with about 400,000 commodity disks attached(for this vendor-- there are other vendors too), in sites around the US and Canada. With a weekly failure rate of about 0.0005%, that means about 200 disks a week need to be replaced.

This work-- hardware investigation through storage appliance frontends, internal ticket creation, external ticket creation, watching the external ticket for updates to include in our internal ticket --was all manual, and for around 200 issues a week, it was done by one guy for two years. He was hopelessly behind. This is all automated now, and this morning, I pushed this automation from dev/test to production.

It feels great to see your work helping people around you.

Rant!

Been working on 'MVP' features of a new product for the past 14 months. Customer has no f**king clue on how to design for performance. An uncomfortable amount of faith was placed on the ORM (ORMs are not bad as long as you know what you are doing) and the magic that the current framework provides. (Again, magic is good so long as you understand what happens behind the smoke and mirrors - but f**k all that... coz hey, productivity, right?). Customer was so focussed on features that no one ever thought of giving any attention to subtler things like 'hey, my transaction is doing a gazillion joins across trizillion tables while making a million calls to the db - maybe I should put more f**king thought into my design.' We foresaw performance and concurrency issues and raised them way ahead of the release. How did the customer respond? By hiring a performance tester. Fair enough - but what did that translate into? Nothing. Nada. Zilch. Hiring a perf tester doesn't automagically fix issues. The perf tester did not have a stable environment, a stable build or anything that is required to do a test with meaningful results. As the release date approached, the customer launched a pilot and things started failing spectacularly with the system not able to support more than 15 concurrent users. WTF! (My 'I told you so' moment) Emails started flying in all directions and the hunt for the scapegoat was on (I'm a sucker for CYA so I was covered). People started pointing in all directions but no one bothered to take a step back and understand what was causing the issues. Numero uno reason for transaction failure was deadlocks. We were using a proprietary DB with kickass tooling. No one bothered to use the tooling to understand what was the resource in contention let alone how to fix the contention. Absolute panic - its like they just froze. Debugging shit and doing the same thing again and again just so that management knew they were upto something. Most of the indexes had a fragmentation of 99.8% - I shit you not. Anywho, we now have a 'war room' where the perf tester needs to script the entire project by tonight and come up with some numbers that will amount to nothing while we stay up and keep profiling the shit out of the application under load.

Lessons learnt - When you foresee a problem make a LOT of noise to get people to act upon it and not wait till it comes back and bites you in the ass. Better yet, try not to get into a team where people can't understand the implications of shitty design choices. War room my ass!

On my project the customer has re-signed into a contract several times when they have budget to continue work. The first time they got us to build the system was a huge success story because the team was assembled quickly and we did rapid development. Initialize repo to prod in 1.5 months. The customer asked for the same dev team. Strong dev team, a PM that doesn't take shit, and pure agile. Lets call her don't-take-shit PM.

When the customer re-signed the executive decided that she didn't like don't-take-shit PM. So the project manager gets replaced by play-by-the-rules PM who will comply with stupid requests and micromanagement. He isn't a bad PM but he tries to make everyone happy. The amount of management types executive installs on the project is massive, and development team is cut down in major ways. Customer and executive shit rolls down to the development team and we can't get anything done. The customer starts to lose faith because we can't get traction. They start demanding traditional waterfall/SDLC docs. Which causes more delay in the project.

So the executive decides that the PM can take a fall for it to save face for the company. She moves play-by-the-rules PM to another project. He starts handover to a new PM that has a history of being her pushover. The customer hadn't seen him yet so now we have push-over PM.

Play-by-the-rules PM is finally out of the project and instead of moving to a different account the company decides to "lay him off because there is no work". So basically they made him take the fall for the failure while promising reassignment, and instead let him go. This is so unfair..

Meeting with push-over PM yesterday and he shows us his plan. Identical to play-by-the-rules PM's plan that got him axed.We point that out and show him the docs that were made for it. His face clearly communicates "OH SHIT WHAT DID I SIGN UP FOR?"

Oh my God I'm a failure. Been working on this booking system backend for two weeks, refactored some code, and now it doesn't work at all.

I've gone back through the entire thing, and I can't find the problem.

Open up indeed, start browsing for low-skill jobs. Maybe the carnies will have me back!

*Re-reads error message, adds missing underscore to function call.

Why is it management always waits until point of failure for a major system to hear us about replacing systems with updated systems?

I used to work with a teacher in my last uni year.

The job consisted on doing a kinda-like management system for a business. It all began kinda "right", we agreed upon a price for 6 months of my work (a very lowball price, but it was just right because I was learning stuff that we were going to be using).

Fast-forward first six months, all I do is code frontend, mockup screens and whatsoever because this "business" hadn't give us proper requirements (Yeah, I told him to ask for them, but nothing came through).

So I was like well, I'll keep working in this project because I really want to finish it. Sidenote: I was doing all the "hard work", he didn't know how to code, and he calls himself a teacher... wtf).

Months go by, and a year goes round, in between these months, he spoke to me, that he wanted me that we kept working together, that we could renegotiate the payment (I asked him to give me my payment once the job was done). I agreed, but my uni residence period was coming along and I got an oportunity to go abroad to another country.

So there I was, in the need of money to buy my passport, plane tickets and other stuff, so I asked him for the payment.

Needs to be noted, that the last 6 months work was me doing tutorials on how to fucking use Linux, how to use PostgreSQL, how to fucking use CSS! He told me he would pay me extra for it.

The day came, and I received my payment... the exact amount we talked a year ago, I was like "Seriously dude?", but well, I needed the money and I didn't have time to argue, so we talked a little bit about me helping him and I told him "As long as I have time, I'll help, but remember that I'm going abroad to work for a small startup, so maybe I'll be up to my head with work" he agreed, we nod and then I left.

First week abroad came in and I was doing a shit-ton of stuff, then his first message comes around "Hey, I need more tutorials! ASAP! Before 6PM"

What.The.Fuck. I told you, son of a bitch, that I wouldn't be able to do them until weekend.. and it was monday!

So I ignored it, weeks went throught and my "angry mood" was fading away so I said to myself "Well, it's time to pick up that stuff again", I open Slack and I find a week old message with a document attached, it was a "letter", I just skimmed by it and read some keywords "deceptioned... failed me.."

Sure dude? Was I the failure? Becase, as far as I remember, you were the fucktard that didn't know how to fucking install a VM!

A week went by, and then randomly a friend of mine talks to me through Facebook:

E: Hey, how are you?
M: I'm fine, what's up?
E: What did you do to TEACHER?
M: Nothing, <explains all situation>
E: Well, It seems weird, that's why I wanted to talk with you, I believe in you, because I know you well, but TEACHER it's thrashing shit about you with all his students on all of his classes
M: Seriously?
E: Yeah, he's saying that you are a failure, irresponsible, that you scammed him

That moment, I for sure, lost all moral responsibility with him and thought to myself "He can go fuck himself with my master branch on his ass"

So when I got back to my country, I had to go around in school, avoiding him, not because I was ashamed nor anything by the way, just because I knew that If i ever had the disgrace to meet him face to face, my fists would be deep into his nose before he could say "Hey".

Moral of the story:
If you overheard that a teacher has a bad rep, not by one, nor two, but more than +100 people, maybe it's true.

Good thing my friends and others know me well and I didn't have repercutions on my social status, I'm just the guy that "fucked up TEACHER because I had the right and way to do it"

Issue in production. Multi billion dollar enterprise. Complex landscape. We sort of make things.

Turns out there is a single point of failure at a specific integration point. Kind of a lot stopped. When I reached out to the people knowing anything about it and I raised the issue that maybe we should make a slight change in how we do things they just brushed it off. Like it was nothing… 😬

No data was lost but everything was delayed for many hours. The _truth_ varied in different parts of the ecosystem causing potential wrong or suboptimal decisions to be taken.

When I asked why this LOS was not detected they told be they have no means of detecting it. 😬

I’m like, yeah, it’s 2023, we’re going to land on Mars and you can bet your ass we can detect it and you are just LAZY DEVELOPERS!

Anyway, I escalated (nicely) and they are now implementing a (more) resilient system and we’re helping the team detecting THEIR LOS in minutes instead of downstream services hours later (they are bad also but it’s not their fault!)

Stay safe!

If they followed my suggestion and went straight to debugging the server issues they would have been solved it from week 1 and everyone would have thought the migration had a minor performance hiccup. In fact, we have already done such at least twice before and nobody batted an eye.

Instead they self-labelled the migration a failure on first error, setting the stage for apologizing to the client, and put themselves on the spot for a whole staging / production signoff, replication / backup worfklow, almost a blue-green "seamless" deployment reminiscent of DigitalOcean.

Well they're not DigitalOcean, and anyone who has spent any time understanding users knows they will not participate in "new system" tests long enough to find or report issues.

So of course the migration stretched out to almost three months up until the whole reason for the migration - the rapidly escalating risk of the old provider disappearing - hit like a freight train and now they have to go through the problem of debugging the server like I told them to on week 1. Only this time they've set the client mindset against it, lost any chance of reverting, have had grave risk for data loss, and are under pressure to debug other people's code in real-time.

This is why I don't trust devs to do ops. A dev's first solution to any problem is to throw tech at it.

"Hello R., how are you?"
"Hi M.! I'm at the beach now, finally relaxing after months of work."
"I wanted to ask you this: did you remember, three years ago, when you helped me move the downloaded movies to my external hard disk?"
"Er... yes?"
"Well, today I tried to start my computer and it's showing me a black screen telling \"disk boot failure, insert system disk\", do you know how to solve it? Before you worked on it, it used to work as a charm..."

MTP is complete garbage. I want mass storage back.

The media transfer protocol (MTP) occasionally discovers new creative ways of failure. Frequently, directory listings take minutes to load or fail to load at all, and it freezes up infinitely (until disconnected) when renaming an item, and I can not even do two things simultaneously.

While files are being moved, I can not browse pictures or watch videos from the smartphone.

Sometimes, files are listed with the date 1970-01-01 (Unix epoch) instead of their correct date. Sometimes, files do not appear at all, which makes it unsafe to move directories from the device.

MTP lacks random access. If I want to play a two-gigabyte 4K 2160p video and seek in the video, guess what: I need to copy it to my computer's local mass storage first because MTP lacks random access.

When transferring high numbers of files, MTP has to slooooowly enumerate (or "prepare" or "calculate the time of") them all, which might even take longer than mass storage would need for the entire process. This means MTP might start copying or moving the actual files when mass storage is already finished.

Today, the "preparing to move" process was especially slow: five minutes for around 150 files! How am I supposed to find out what caused this random malfunction?

MTP sometimes drives me insane. I want mass storage back, at least for the MicroSD memory card, which uses a widely supported file system.

Imagine a 2010 $100 Android phone is better at file transfer than a 2022 $1000 Android phone (or iPhone, for that matter).

I remember a certain prank that amuses me till today....

Just add some devices to monitoring and the notification queue of the build chain / ... ...and wait patiently.

I still cry tears remembering an manager screaming what the hell "the poop train clogged the drain" means and why this is a critical system failure.

(Notice: next time check the mailing aliases of mailing aliases)

Although I can only recommend this if you know your team well. In my case we had a whole lot of fun after I got my head chewed off. XD (got an earful, but in the end he laughed his ass off)

So for those of you keeping track, I've become a bit of a data munger of late, something that is both interesting and somewhat frustrating.

I work with a variety of enterprise data sources. Those of you who have done enterprise work will know what I mean. Forget lovely Web APIs with proper authentication and JSON fed by well-known open source libraries. No, I've got the output from an AS/400 to deal with (For the youngsters amongst you, AS/400 is a 1980s IBM mainframe-ish operating system that oriiganlly ran on 48-bit computers). I've got EDIFACT to deal with (for the youngsters amongst you: EDIFACT is the 1980s precursor to XML. It's all cryptic codes, + delimited fields and ' delimited lines) and I've got legacy databases to massage into newer formats, all for what is laughably called my "data warehouse".

But of course, the one system that actually gives me serious problems is the most modern one. It's web-based, on internal servers. It's got all the late-naughties buzzowrds in web development, such as AJAX and JQuery. And it now has a "Web Service" interface at the request of the bosses, that I have to use.

The programmers of this system have based it on that very well-known database: Intersystems Caché. This is an Object Database, and doesn't have an SQL driver by default, so I'm basically required to use this "Web Service".

Let's put aside the poor security. I basically pass a hard-coded human readable string as password in a password field in the GET parameters. This is a step up from no security, to be fair, though not much.

It's the fact that the thing lies. All the files it spits out start with that fateful string: '<?xml version="1.0" encoding="ISO-8859-1"?>' and it lies.

It's all UTF-8, which has made some of my parsers choke, when they're expecting latin-1.

But no, the real lie is the fact that IT IS NOT WELL-FORMED XML. Let alone Valid.

THERE IS NO ROOT ELEMENT!

So now, I have to waste my time writing a proxy for this "web service" that rewrites the XML encoding string on these files, and adds a root element, just so I can spit it at an XML parser. This means added infrastructure for my data munging, and more potential bugs introduced or points of failure.

Let's just say that the developers of this system don't really cope with people wanting to integrate with them. It's amazing that they manage to integrate with third parties at all...

Inspiring moment: when the control system I wrote for a robot stopped the thing's EDF mere inches from my nose when the bot went out of control (for other reasons) during testing. Had it not stopped I would probably be without a nose, that EDF (Electric Ducted Fan) had fairly sharp blades. Very scary, but very thrilling too.

Each time my code affects something in the real world, it feels so damn awesome. Thankfully I've not come close to losing my nose (or other body parts) after that incident, but that incident inspired me to continue work on failure-proof control systems that enforce safety.

I really really hope that no one post this,a friend texted it to me and I wanted to share it because made my day.

Idk where it comes, so feel free if know where this came from to post it:

//FUN PART HERE
# Do not refactor, it is a bad practice. YOLO

# Not understanding why or how something works is always good. YOLO

# Do not ever test your code yourself, just ask. YOLO

# No one is going to read your code, at any point don’t comment. YOLO

# Why do it the easy way when you can reinvent the wheel? Future-proofing is for pussies. YOLO

# Do not read the documentation. YOLO

# Do not waste time with gists. YOLO

# Do not write specs. YOLO also matches to YDD (YOLO DRIVEN DEVELOPMENT)

# Do not use naming conventions. YOLO

# Paying for online tutorials is always better than just searching and reading. YOLO

# You always use production as an environment. YOLO

# Don’t describe what you’re trying to do, just ask random questions on how to do it. YOLO

# Don’t indent. YOLO

# Version control systems are for wussies. YOLO

# Developing on a system similar to the deployment system is for wussies! YOLO

# I don’t always test my code, but when I do, I do it in production. YOLO

# Real men deploy with ftp. YOLO

So YOLO Driven Development isn’t your style? Okay, here are a few more hilarious IT methodologies to get on board with.

*The Pigeon Methodology*
Boss flies in, shits all over everything, then flies away.

*ADD (Asshole Driven Development)*
An old favourite, which outlines any team where the biggest jerk makes all the big decisions. Wisdom, process and logic are not the factory default.

*NDAD (No Developers Allowed in Decisions)*
Methodology Developers of all kinds are strictly forbidden when it comes to decisions regarding entire projects, from back end design to deadlines, because middle and top management know exactly what they want, how it should be done, and how long it will take.

*FDD (Fear Driven Development)*
The analysis paralysis that can slow an entire project down, with developments afraid to make mistakes, break the build, or cause bugs. The source of a developer’s anxiety could be attributed to a failure in sharing information, or by implicating that team members are replaceable.

*CYAE (Cover Your Ass Engineering)*
As Scott Berkun so eloquently put it, the driving force behind most individual efforts is making sure that when the shit hits the fan, you are not to blame.

*laughing maniacally*

Okidoky you lil fucker where you've been hiding...

*streaming tcpdump via SSH to other box, feeding tshark with input filters*

Finally finding a request with an ominous dissector warning about headers...

Not finding anything with silversearcher / ag in the project...

*getting even more pissed causr I've been looking for lil fucker since 2 days*

*generating possible splits of the header name, piping to silversearcher*

*I/O looks like clusterfuck*

Common, it are just dozen gigabytes of text, don't choke just because you have to suck on all the sucking projects this company owns... Don't drown now, lil bukkake princess.

*half an hour later*

Oh... Interesting. Bukkake princess survived and even spilled the tea.

Someone was trying to be overly "eager" to avoid magic numbers...

They concatenated a header name out of several const vars which stem from a static class with like... 300? 400? vars of which I can make no fucking sense at all.

Class literally looks like the most braindamaged thing one could imagine.

And yes... Coming back to the network error I'm debugging since 2 days as it is occuring at erratic intervals and noone knew of course why...

One of the devs changed the const value of one of the variables to have UTF 8 characters. For "cleaner meaning".

Sometimes I just want to electrocute people ...

The reason this didn't pop up all the time was because the test system triggered one call with the header - whenever said dev pushed changes...

And yeah. Test failures can be ignored.
Why bother? Just continue meddling in shit.

I'm glad for the dev that I'm in home office... :@

TLDR: Dev changed const value without thinking, ignoring test failures and I had the fun of debunking for 2 days a mysterious HAProxy failure due to HTTP header validation...

I write web apps that show system health information, for support purposes. Whenever I talk to my boss about the general direction of what I'm writing he says, "I want one page that shows me everything."

This is an enormous company, with tens of millions of customers, and an infrastructure so big that there are literally millions of potential points of failure.

I hear this from management softs all the time: one page that shows me EVERYTHING. To me, that means he wants a red or green indicator that he can quickly check on his iphone while he's skiing.

I'm afraid that managing this kind of infrastructure is a bit more complicated than that. If it was that simple, you wouldn't have anyone to manage.

I love 2:30 am support calls, especially for a system I know nearly nothing about and when I'm supposed to be 4th in line for a call.

Hopefully I made the right decision. Now let's see if I can fall back asleep.

The classic failure a developer of absolute secured system do is subestimating the ingenuity of stupid morons

Did you ever had to integrate a fucking "API" that is done via mail bodies?

Fuck this shit! Who need responses about success or failure?! Guess this will take a long time to test this fucking piece of garbage... We don't get a test system, we need to test this with the production system of the other company. I hope their retarded application crashes when receiving malicious mails.

Not speaking about security, I bet everyone can send a mail to their stupid mail address and modify their data 🙈

And inside of this crap mail you also have to send the name, street and email of their company. Why do you fucking need this information?!

When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.

Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.

When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.

Speaking of which, there's more than six microservices per developer!

Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?

When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:

1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.

Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).

Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.

When confronted/advised about these issues the response from the CTO/CEO
varies:

(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".

Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.

There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".

It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?

Developed an app to display examination seats allocated by an algorithm developed by someone else.

The algorithm goes berserk and allocates randomly. Spectacular failure of the system follows.

Everyone blames me for all the trouble.

The algorithm developer smiles in the background. WTH!

Maybe it's just me. Maybe I'm the Charlie Brown of development and Lucy with the football is the XAMPP/MAMP/WAMP software in this world. EVERY. TIME. I. TRY. TO. SPIN. THIS. UP. IT. FAILS. It doesn't matter which tutorial I follow for which technology stack or CMS, the result is always the same. Something about the database or htaccess or some other stupid setting makes it impossible for me to create a simple dev environment on my system.

I have been doing this dance for 24 YEARS NOW!!!! The original programmer of Apache is a 2nd-degree acquaintance who used to be available to help me with this, but no more. I feel like a complete and utter failure as a web developer every time I try to set up XAMPP, and, the rare times I've succeeded and gotten a basic CMS up and running, I fail again and again with all these build/run/task tools I'm now supposed to be using. After a week of fiddling with my local dev environment, I give up and delete it all. I go right back to on-server development "the old fashioned way". WHY!? WHY IS THIS SO HARD?

I'm stepping on rakes here and about to quit. I'm probably just too OLD and STUPID for all these stacks and frameworks and tools and maybe even for this career now. I should probably quit and become a "facilities manager" at a tech firm somewhere, cleaning up the bathrooms and sweeping floors and watching all these young geniuses tut-tut about "Poor StackODev. I hear he had 24 years as a web developer, but then he snapped and he's never been the same."

I hate Windows! BSOD and many other *famous* errors waste my time resolving them.

This happens to me every 6 months and it is so fucking annoying !

The system repairs are so fucking futile! They take so much time *attempting repairs* and in the end give me a failure message.

What the hell is the point of this small projects team spending 2-3 months on developing extensive logging system for an internal application for inside and outside customers to use if your application isn’t going to log any of the fucking errors. Sure you write the failure status to the database, but it just says failure with an even more vague explanation than microsoft’s errors. “An error occurred”. No shit, that’s why I’m looking in the logs and database to debug the application to get these files on their merry way so our company can stay in compliance with the state, feds, and not pay out the wazzoo in fines. All our other applications state where the error occured such as “failed to connect to the email server”, why can’t this one.

Deadline was 2-3 days for product launch and doing distributed transactions was not an opinion as it requires heavy modifications.

I was doing money transfer app between one transactional system and one not transactional system so the way I did it was :
1. transfer money from one system to my app that was using Akka STM ( software transactional memory)
2. try to transfer money to second system
3. transfer money back on failure

There was no database, no state only transactional log as installing database would require to much time and paper work.

Sometimes transfer back failed so we need to look back at logs and search for money, it was quite easy cause there was error and there were not so many failed transactions like this.
About one or two in a month and everyone accepted that.

I started to write some sort of reconciliation thread but then was assigned to other work and it worked like this for couple of years transferring couple millions worth of transactions.

I hate the company (agency) I moved to...I've negotiated good pay and the project for cutting edge medical product which will change the world (cancer diagnose and it actually works).

Now the dark side I've got shit tier laptop which I don't want, overtime is payed 30% less, all the people in the agency from development team don't know shit and are mostly I would call them juniors (of course who would with enough seniority work with shit hardware and almost not payed overtime), only tap water and since this is the old part of town you instantly get sick, they treat people like shit.

The product dark side. We are actually working on crm for doctors to input patient data, we cannot have any real data because we are the agency people, product is being led by the guy who has 0 production experience (they choose the database basically with coin toss and emulated the mongodb in postgress with jsnob, they don't know how to build their own auth system hence my previous rant about b2c, they are using cognito and now moving to auth0 which probably won't fit their need because a lot of stuff needs to be custom), they are choosing every hipe tech out there without any prior experience. It's chaos...

I'm trying to guide them but i think this will be a huge expensive failure and that i need to leave asap.

There I feel better now, moral of the story, choose startups wisely.

Poorly built software is the other side of the coin of over-engineered software. They both exist because users carelessly use software products. By not exercising the code enough, or system failure not costing the business more penalties than they can bear, incompetent developers will continue to get away with building things haphazardly –not as relates to tech stack, but the nitty-gritty implementation details they gloss over without adequately thinking through

Because of this, there doesn't seem to be sufficient incentive for thorough planning –what could be referred to as over-engineering. Those fancy pedantry in code mostly goes unnoticed by the end user. Of course, this doesn't apply to big corporations in most cases. It's usually unexpected to see elementary bugs in them

We’re only random people living in random places, speaking random languages, eating random food, sleeping, studying and working random hours. Traveling to random points on a sphere.

Just random range is different.
Just random stuff happens on crossroads of two random dots and the entropy speed ups or slows down.

Nothing special at all.
Just a finite state machine iteration.

I mean the amount of effort we put into explanation of infinity is outstanding.

What if there is no infinity at all ?
What if infinity is just misunderstanding of our interpretation of the world around us. It’s just pixels, resolution, gaussian splatting, quantum state, you name it.

Hey man the world is flat. Just put it to the 2d space. How many space you need from a simulation perspective where your patient eyes can only see up to certain amount of light particles per second on a shitty lens.

Propose a world optimization techniques by slowing down subject perception, tiredness introduced. Compress memory, sleep introduced. Limit neurons, cpu power assigned. Deploy on cloud - put it to life. Exit 0 body failure. Exit 1 suicide. Kill -9 killed by tty from ip EARTH.X.Y

What you can do to make the world around this planet alive? Make it blink.

We developers are lazy and I believe that nature is even more lazy than us.

You think you’re going to elevator right now ? You’re going to the preloader. Looking at the window equals playing video from playback. Never goes live, just precomputed fsm. Cars, trains, airplains ? Preloaders everywhere. Highways to split traffic to cities and communication. The road and cities planning department is a matrix maintenance department. And don’t get me started about space.

Space is empty because it’s not even finished. So they put it all behind glass called milky way. You know how glass looked 500 years ago ? It was milky so it’s milky way so we don’t see shit.

If the space would be finished I’ll be starting writing this text from mars, finished it and sent from earth but no it’s light years guys, light years is not a second for a matter. Light year is a second of the the injected thoughts exchange only. Thoughts of the global computer called generative AI that they introduced on local computing devices called cloud.

Even the preloader system is not present, they left us with the one map and overpopulated demo. What a shit hole.I bet they’re increasing temperature right now to erase this alpha build and cash out. Obviously so many bugs here that his one can’t be fixed anymore. To many viruses.

Hope for 0days to start happening so we can escape using time travel or something.

I bet they cut a budget or something, moved the team to other projects. Or even worse solar system team got layoff off because we are just neurons that ordered to do it. And now we’re stuck in some maintenance mode, no new physics no new thoughts to pursue, just slow degeneration. I would pay more for the next run and switch to other galaxy far far away where they at lest have more modern light speed technology.

What do you think about it Trinity ? Not even worth wasting your time for that. No white rabbit this time.

I do not recommend this game at this stage of early access.
- only one available map despite promises for expansions over the years no single dlc arrived,
- missing space adventures
- no galaxy travel mode only a teaser trailers of what you can do in other “universes”
- developers don’t respond to complains
- despite diversity of species and buildings at first sight world looks to generic
- instead of new features bots with mind manipulation, AB testing and data harvesting was introduced
- death anti cheat mode installed

Fuck Arch Linux
Love the AUR, but the fact that every -Syu I've done led to system wide failure is unforgivable
I'm coming home Fedora, also screw you Debian and your broken packages