I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.

The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.

We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.

Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.

Needless to say, he was never allowed to touch code on a prod server ever again.

My mentor/guider at my last internship.

He was great at guiding, only 1-2 years older than me, brought criticism in a constructive way (only had a very tiny thing once in half a year though) and although they were forced to use windows in a few production environments, when it came to handling very sensitive data and they asked me for an opinion before him and I answered that closed source software wasn't a good idea and they'd all go against me, this guy quit his nice-guy mode and went straight to dead-serious backing me up.
I remember a specific occurrence:
Programmers in room (under him technically): so linuxxx, why not just use windows servers for this data storage?
Me: because it's closed source, you know why I'd say that that's bad for handling sensitive data
Programmers: oh come on not that again...
Me: no but really look at it from my si.....
Programmers: no stop it. You're only an intern, don't act like you know a lot about thi....
Mentor: no you shut the fuck up. We. Are. Not. Using. Proprietary. Bullshit. For. Storing. Sensitive. Data.
Linuxxx seems to know a lot more about security and privacy than you guys so you fucking listen to what he has to say.
Windows is out of the fucking question here, am I clear?

Yeah that felt awesome.

Also that time when a mysql db in prod went bad and they didn't really know what to do. Didn't have much experience but knew how to run a repair.
He called me in and asked me to have a look.
Me: *fixed it in a few minutes* so how many visitors does this thing get, few hundred a day?
Him: few million.
Me: 😵 I'm only an intern! Why did you let me access this?!
Him: because you're the one with the most Linux knowledge here and I trust you to fix it or give a shout when you simply can't.

Lastly he asked me to help out with iptables rules. I wasn't of much help but it was fun to sit there debugging iptables shit with two seniors 😊

He always gave good feedback, knew my qualities and put them to good use and kept my motivation high.

Awesome guy!

toxic workplace; leaving

I haven't wanted to write this rant. I haven't even wanted to talk to anyone (save my gf, ofc). I've just been silently fuming.

I wrote a much longer rant going into far too much detail, but none of that is relevant, so I deleted it and wrote this shorter (believe it or not) version instead. And then added in more details because details.

------

On Tuesday, as every Tuesday, I had a conference call with the rest of the company. For various, mostly stupid reasons, the boss yelled at and insulted me for twenty minutes straight in front of everyone, telling me how i'm disorganized, forgetful, how can't manage my time, can't manage myself let alone others, how I don't have my priorities straight, etc. He told the sales team to get off the call, and then proceeded to yell and chew at me for another twenty minutes in front of the frontend contractor about basically the same things. The call was 53 minutes, and he spent 40 minutes of it telling me how terrible I've been. No exaggeration, no spin. The issues? I didn't respond to an email (it got lost in my ever-filling inbox), and I didn't push a very minor update last week (untested and straight to prod, ofc). (Side note: he's yelled at me for ~15 minutes before for being horribly disorganized and unable to keep up on Trello -- because I had a single card in the wrong column. One card, out of 60+ over two boards. Never mind that most have time estimates, project tags, details, linked to cards on his boards, columns for project/qa/released, labels for deferred, released to / rejected from qa, finished, in production, are ordered by priority, .... Yep. I'm totes disorganized.)

Anyway, I spent most of conference call writing "Go fuck yourself," "Choke on a cat and die asshole," "Shit code, low pay, and broken promises. what a prize position," etc. or flipping him off under the camera on our conference-turn-video-call (switched due to connection issues, because ofc video is more stable than audio-only in his mind).

I'm just.
so, so done.

I did nothing the rest of the day on Tuesday, and basically just played games on Wednesday. I did one small ticket -- a cert replacement since that was to expire the next day -- but the rest was just playing CrossCode. (fun game, fyi; totally recommend.)

Today? It's 3:30pm and I can't be bothered to do anything. I have an "urgent" project to finish by Monday, literally "to give [random third party sales guy] a small win". Total actual wording. I was to drop all other tasks (even the expiring cert lol) and give this guy his small win. fucking whatever. But the project deals with decent code -- it's a minor extension to the first project I did for the company (see my much earlier rants), back when I was actually applying myself and learning something (everything) new, enjoying myself, and architecting+writing my own code. So I might actually do the project, but It's been two days and I haven't even opened single file yet.

But yeah. This place is total and complete shit. Dealing with the asshole reminds me of dealing with my parents while growing up, and that's a subject I don't want to broach -- far too many toxic memories.

So, I'm quitting as soon as I find something new.
and with luck, this will be before assface hires my replacement-to-be, and who will hopefully quit as soon as s/he sees the abysmal codebase. With even more luck, the asshole king himself will get to watch his company die due to horrible mismanagement. (though ofc he'll never attribute it to himself. whatever.)

I just never want to see or think about him again.
(nor this fetid landfill of a codebase. bleh.)

With luck, this will be one of my last rants about this toxic waste dump and its king of the pile.
Fourty fucking minutes, what the fuck.

“Yeah but you’re not a *real* developer”

Fuck. you.

I wrote 80% of this code base. I do 80% of the tickets/storyboard points. I do all of the QA. My nose is to the grindstone every fucking day honing this craft and sweating my balls off like a blacksmith staring into the red hot kiln while the sores of previous mistakes scream bloody murder from the unrelenting exposure to heat. I saw this amazing industry of opportunity, freedom and self examination and wanted in no matter what it took. I glued myself to every pithy resource I could possibly get my hands on and crawled through the muck and filth of it all until I could keep myself warm with the smallest spark of my own making. I stoked that spark until it became a fire and stoked that fire until I could set entire forests ablaze. I listened to the ungrateful people keeping warm by my combustion saying it “wasn’t hot enough” or “would have been a nicer colour if they did it” or “could have warmed up just fine jogging on the spot”. I made painstaking alterations to my ignition and watched my undeserving benefactors gradually be silenced and begin to sit quietly by the heat. I jumped into that inferno daily, was reduced to ash daily and emerged reborn daily. But you are right! I didn’t get scammed out of $40k+ studying technology in an archaic institution from instructors who don’t give a shit and answering “D all of the above” for 4+ years straight therefor my opinion doesn’t mean shit. Push your bullshit to prod and watch the server come burning out of the cloud as the apocalyptic swarm of angry tickets come flooding in why don’t you? Bet they didn’t teach you that in school. You’ve never poked around inside an open source codebase in your life. They are just a mystery boxes of magic that unless someone holds your hands with finely crafted instructions containing a 50/50 picture to word ratio you throw a hissy fit. Every problem that comes up instead of working to solve it you reflexively point to the first person in the room while thinking with your pea brain how you can possibly scapegoat them into taking the fall for whatever it is that’s come up today you couldn’t possibly understand.

Not a real developer?

Fuck. You.

One week, and it turned out to be worse than that.

I was put on a project for a COVID-19 program in America (The CARES Act). The financial team came to us on Monday morning and said they need to give away a couple thousand dollars.

No big deal. All they wanted was a single form that people could submit with some critical info. Didn't need a login/ registration flow or anything. You could have basically used Google Forms for this project.

The project landed in my lap just before lunch on Monday morning. I was a junior in a team with a senior and another junior on standby. It was going to go live the next Monday.

The scope of the project made it seem like the one week deadline wasn't too awful. We just had to send some high priority emails to get some prod servers and app keys and we were fine.

Now is the time where I pause the rant to express to you just how fine we were decidedly **not**: we were not fine.

Tuesday rolls around and what a bad Tuesday it was. It was the first of many requirement changes. There was going to need to be a review process. Instead of the team just reading submissions from the site, they needed accept and reject buttons. They needed a way to deny people for specific reasons. Meaning the employee dashboard just got a little more complicated.

Wednesday came around and yeah, we need a registration and login flow. Yikes.

Thursday came and the couple-thousand dollars turned into a tens of millions. The amount of users we expected just blew up.

Friday, and they needed a way for users to edit their submissions and re-submit if they were rejected. And we needed to send out emails for the status of their applications.

Every day, a new meeting. Every meeting, new requirements that were devastating given our timeframe.

We put in overtime. Came in on the weekend. And by Monday, we had a form that users could submit and a registration/ login flow. No reviewer dashboard. We figured we could take in user input on time and then finish the dashboard later.

Well, financial team has some qualms. They wanted a more complicated review process. They wanted roles; managers assign to assistants. Assistants review assigned items.

The deadline that we worked so hard on whizzed by without so much as a thought, much less the funeral it deserved.

Then, they wanted multiple people to review an application before it was final. Then, they needed different landing pages for a few more departments to be able to review different steps of the applications.

Ended up going live on Friday, close to a month after that faithful Monday which disrupted everything else I was working on, effective immediately.

I don't know why, but we always go live on a Friday for some reason. It must be some sort of conspiracy to force overtime out of our managers. I'm baffled.

But I worked support after the launch.

And there's a funny story about support too: we were asked to create a "submit an issue" form. Me and the other junior worked on it on a wednesday three weeks into the project. Finished it. And the next day it was scrapped and moved to another service we already had running. Poor management like that plagued the project and worked in tandem with the dynamic and ridiculous requirements to make this project hell.

Back to support.

Phone calls give me bad anxiety. But Friday, just before lunch, I was put on the support team. Sure, we have a department that makes calls and deal with users. But they can't be trained on this program: it didn't exist just a month ago, and three days ago it worked differently (the slippery requirements never stopped).

So all of Friday and then all of Saturday and all of Monday (...) I had extended panic attacks calling hundreds of people. And the team that was calling people was only two people. We had over 400 tickets in the first two days.

And fuck me, stupid me, for doing a good job. Because I was put on the call team for **another** COVID project afterwards. I knew nothing about this project. I have hated my job recently. But I'm a junior. What am I gonna say, no?

Me: So i've cloned the iOS project, i've run carthage, but it won't build.. Have I done something wrong?

Devs: Oh read this doc on github, we do loads of custom stuff. The depenedncy manager can't do it all by itself. You need to run `./scripts/boostrap.sh`

Me (another day): I've switched branches and i'm getting all these errors. Any ideas?

Devs: Ah this happens when someone modifies xyz. Read this pinned slack message. Run `./scripts/bootstrap.sh` again.

Me (another day): I've switched branches again, getting different errors, re-running boostrap didn't fix it.

Devs: Ah yeah, this happens when someone modifies abc. You need to run `./scripts/nuke.sh` and then boostrap when this happens.

Me (another day): Guys When I try to run the prod app its not building any ideas?

Devs: Ah yes have a look at this confluence link. You need to run `./scripts/setup_debug_release.sh`, then nuke, then boostrap and you'll be good.

Me: .... ok

Devs: Oh btw very important! do not commit any changes from `./scripts/setup_debug_release.sh`. It will break everything!

Me: ... no i'm sorry we have a much bigger problem than that. We need to talk ... like right now

Doot doot.

My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.

My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.

My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.

what a joke.

Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.

I told him it was going to happen again.

I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.

So stupid.

Just reduced 900 lines of horrible code into 106 lines of less horrible code..

QA: do those 106loc really do what the 900loc did ?
Me: yes
QA: really?
Me: yes
QA: are you sure?
Me: NO. I was lying. Maybe it does. Maybe it doesn't. Maybe you should run it and find out.
QA: *immediately releases for prod*

Welcome back to practiseSafeHex's new life as a manager.

Episode 2: Why automate when you can spend all day doing it by hand

This is a particularly special episode for me, as these problems are taking up so much of my time with non-sensical bullshit, that i'm delayed with everything else. Some badly require tooling or new products. Some are just unnecessary processes or annoyances that should not need to be handled by another human. So lets jump right in, in no particular order:

- Jira ... nuff said? not quite because somehow some blue moon, planets aligning, act of god style set of circumstances lined up to allow this team to somehow make Jira worse. On one hand we have a gigantic Jira project containing 7 separate sub teams, a million different labels / epics and 4.2 million possible assignees, all making sure the loading page takes as long as possible to open. But the new country we've added support for in the app gets a separate project. So we have product, backend, mobile, design, management etc on one, and mobile-country2 on another. This delightfully means a lot of duplication and copy pasting from one to the other, for literally no reason what so ever.

- Everything on Jira is found through a label. Every time something happens, a new one is created. So I need to check for "iOS", "Android", "iOS-country2", "Android-country2", "mobile-<feature>", "mobile-<feature>-issues", "mobile-<feature>-prod-issues", "mobile-<feature>-existing-issues" and "<project>-July31" ... why July31? Because some fucking moron decided to do a round of testing, and tag all the issues with the current date (despite the fact Jira does that anyway), which somehow still gets used from time to time because nobody pays attention to what they are doing. This means creating and modifying filters on a daily basis ... after spending time trying to figure out what its not in the first one.

- One of my favourite morning rituals I like to call "Jira dumpster diving". This involves me removing all the filters and reading all the tickets. Why would I do such a thing? oh remember the 9000 labels I mentioned earlier? right well its very likely that they actually won't use any of them ... or the wrong ones ... or assign to the wrong person, so I have to go find them and fix them. If I don't, i'll get yelled at, because clearly it's my fault.

- Moving on from Jira. As some of you might have seen in your companies, if you use things like TestFlight, HockeyApp, AppCenter, BuddyBuild etc. that when you release a new app version for testing, each version comes with an automated change-log, listing ticket numbers addressed ...... yeah we don't do that. No we use this shitty service, which is effectively an FTP server and a webpage, that only allows you to host the new versions. Sending out those emails is all manual ... distribution groups?? ... whats that?

- Moving back to Jira. Can't even automate the changelog with a script, because I can't even make sense of the tickets, in order to translate that to a script.

- Moving on from Jira. Me and one of the remote testers play this great game I like to call "tag team ticketing". It's so much fun. Right heres how to play, you'll need a QA and a PM.

*QA creates a ticket, and puts nothing of any use inside it, and assigns to the PM.
*PM fires it back asking for clarification.
*QA adds in what he feels is clarification (hes wrong) and assigns it back to the PM.
*PM sends detailed instructions, with examples as to what is needed and assigns it back.
*QA adds 1 of the 3 things required and assigns it back.
*PM assigns it back saying the one thing added is from the wrong day, and reminds him about the other 2 items.
*QA adds some random piece of unrelated info to the ticket instead, forgetting about the 3 things and assigns it back.

and you just continue doing this for the whole dev / release cycle hahaha. Oh you guys have no idea how much fun it is, seriously give it a go, you'll thank me later ... or kill yourselves, each to their own.

- Moving back to Jira. I decided to take an action of creating a new project for my team (the mobile team) and set it up the way we want and just ignore everything going on around us. Use proper automation, and a kanban board. Maybe only give product a slack bot interface that won't allow them to create a ticket without what we need etc. Spent 25 minutes looking for the "create new project" button before finding the link which says I need to open a ticket with support and wait ... 5 ... fucking ... long ... painful ... unnecessary ... business days.

... Heres hoping my head continues to not have a bullet hole in it by then.

Id love to talk more, but those filters ain't gonna fix themselves. So we'll have to leave it here for today. Tune in again for another episode soon.

And remember to always practiseSafeHex

One of our clients deploy their own server app. So this happened after a prod deployment. (4am)

*Cellphone rings while sleeping*

Client : we need you on the conference call now. URGENT!

*Gets on conference call*
*Client explain the problem*
*Explaining to the client that the problem is in their side (https connection not working, either network or certificate problem)*
*Client doesn't believe it and pushes me for a fix that I have no control on*
*4 hours later in a heated conversation*

Client : ok problem is on our side. We used our SSL certificate from staging with production and thought it would work.

Me :

Once upon a time there was a dev.
The dev had a resume that said he could dev.
We called the dev, he sounded intelligent.
We hired the dev, who was a bit green, on a three month probationary period.
The dev did very little.
When asked, we said he contributed to discussions, but seemed unclear about what to do, and maybe they could keep him as an intern if they wanted to have him at all.
They hired him. As a full time dev.
6 months later, that dev was shocked to find we could log into the servers with a privileged account.
We (his team mates) were sad.
We asked him to fix a few prod errors.
A little while later he said "Done!"
We then had to walk him through how to actually fix them, not just add a couple pieces of info to the table.
We were sad, again.
We asked him to fix some prod errors again.
We had to walk him through the process again
We expressed concerns to our superiors about his abilities because he was all theory, no hands on ability
They promoted him
We were sad
A few of us said "Fuck you guys, I'm going home"
They said OK
Now that guy is the only one that "knows" that code base
I get calls sometimes asking me questions.
I told them to pay me a consultant fee.
They said no
I said no
They called again
I laughed at them

Listen to the people who know when you ask them questions.
Listen to the people who know when they tell you there is a problem
Don't be like that company

"Hey, Root, someone screwed up and now all of our prod servers are running this useless query constantly. I know I already changed your priorities six times in the past three weeks, but: Go fix it! This is higher priority! We already took some guesses at how and supplied the necessary code changes in the ticket, so this shouldn't take you long. Remember, HIGH PRIORITY!"

1. I have no idea how to reproduce it.
2. They have no idea how to reproduce it.
3. The server log doesn't include queries.
4. The application log doesn't include queries.
5. The tooling intercepts and strips out some log entries the legendary devs considered useless. (Tangent: It also now requires a tool to read the logs because log entries are now long json blobs instead of plain text.)
6. The codebase uses different loggers like everywhere, uses a custom logger by default, and often overwrites that custom logger with the default logger some levels in. gg
7. The fixes shown in the ticket are pretty lame. (I've fixed these already, and added one they missed.)
8. I'm sick and tired and burned out and just can't bring myself to care. I'm only doing this so i don't get fired.
9. Why not have the person who screwed this up fix it? Did they quit? I mean, I wouldn't blame them.

Why must everything this company does be so infuriatingly complicated?

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

Devs: We need access to PROD DB in order to provide support you're asking us for.

Mgmt: No, we cannot trust you with PROD DB accesses. That DB contains live data and is too sensitive for you to fuck things up

Mgmt: We'll only grant PROD DB access to DBAs and app support guys

Mgmt: <hire newbies to app support>

App_supp: `update USER set invoice_directory = 54376; commit;`

----------------
I have nothing left to say....

First rant here. Long, but please bear with me:

So after slogging my ass off in various early stage startups for over 4 years and keeping up with the almost non-existent development process, I joined an organisation which has some of the brightest and smartest minds I have had the pleasure to work with.

Mind you, this company is the market leader in it's field and has a 50+ people in it's tech team and the quality of work is pretty impressive.

Now for this week's sprint, I was asked to develop a feature which already exists on the Android app and they want to introduce in the iOS app too. The backend APIs are all in place and all I need to do is build it with virtually no dependency. My PM asks me to start with the UI and ask the backend dev for the API list whenever I need them.This is where the story turns.

For my first API, I go to the backend dev and ask him to share the API documentation and he looks at me as if I have asked him to dance the fucking cha cha. With a straight face he tells me that, 'The organisation doesn't maintain any kind of documentation for it's APIs.' Now this really shocks me. Even in a 5 men tech teams I have worked on, we have always maintained a spec doc for the APIs and this is a company which is known for it's tech practices.

Being the new guy I compose myself and ask if they have anything for me here: Postman collection, a workflowy doc, a goddamn txt file; anything which might help me, and he laughs at my dilusion and says no.

Dejected, I ask for a way to get the APIs and I am told that there are only two ways: either I keep bothering the Android dev for the APIs(No, I don't have the access to the android repo and nor am I gonna get it) which he had worked on 4 months back or I install the prod app on my phone, and use Charles to get every fucking API which is really, really annoying.

I thought writing out this rant would make me feel better, turns out it just made me angrier. Why the fuck can't they document such an important thing!?

Root: Fleshes out missing data in some factories. Tests affected code and finds the change breaks some specs (but shouldn’t).

Root: Reaches out to spec author.

Root: Messages thundercunt (the ticket’s code reviewer) on slack about the specs and the reaching out. No response.

Root: Works on another ticket while blocked.

Root: Logs off.

Root: Talks with spec author chick in the morning. Decide to pair on specs later.

TC: Still no slack response.

Root: Gives update in standup. Mentions factories and broken specs. Mentions pairing with spec chick.

TC: Still no slack response.

Root: Pulled off tickets in favor of prod issue. Gets ignored by everyone else diagnosing prod issue. Investigates prod issue by herself. Discovers prod issue isn’t from bad code, but bad requirements — code works as requested. Communicates this with details. Gets ignored by people still diagnosing prod issue. Tries again. Gets ignored. Gives up. Works on non-blocked tickets instead.

TC: Still no slack response.

Hours later:

TC: Comments on PR telling me I broke specs (how did I not notice?), that I need to reach out to spec chick and work with her, and that I can’t resolve the ticket until it’s fixed and passes code review.

TC: Still no slack response. (21 hours later at this point)

TC: Logs off. Still no response (25 hours at this point)

———

Ignoring the prod issue for the moment…

I broke specs. No shit.
I need to talk with spec chick. No shit.
I can’t resolve the ticket. No shit!

Bitch, I told you all of this 21 fucking hours prior, and again 3 hours prior during standup. But no, I clearly “don’t communicate” and obviously have no bloody clue what I’m doing, either, so I need everything spelled out for me.

And no, I didn’t resolve the fucking ticket. Why the fuck would I if it still has pending changes? Do you even check? Ugh!

And what the fuck with that prod issue? I’m literally giving you the answer. fucking listen! Stupid cunts.

Why is it all of the women I work with are useless or freaking awful people? Don’t get me wrong, many of the men are, too, but I swear it’s every single one of the women. (Am I awful, too?)

Just. Ugh.
I can’t wait to leave this sewer of a company.

Oddly still a good day, though. Probably because I talked to recruiters and sent out my resume again.

Hello! I am Johnny Knoxville and tonight on Jackass...we test in prod with no back out plan in place...and then I will glue Bam's hand to his dick while he's sleeping.

Manager: How come the push to prod didn’t happen?

Dev: We told you at the scrum yesterday. To reiterate, our dev environment was crashing so it’s not safe to push to prod until that is fixed.

Manager: Ok well lets set a goal to fix that and push to prod happens today so that it guaranteed happens.

Dev: That was our goal yesterday and it definitely didn’t happen.

Manager: I AM AWARE OF THAT. The corrective action is that this time compliance with the goal is 100% ABSOLUTELY MANDATORY!!

Dev: We’ll do our best, can’t guarantee anything until we figure out what the nature of what is occurring on dev though.

Manager: NO. I AM THE BOSS. YOU WILL 100% ABSOLUTELY COMPLY WITH THIS. THAT IS AN ORDER. YOU WILL SUCCESSFULLY GET THIS UPDATE OUT TO PROD TODAY. ANYTHING LESS THAN THAT SHALL BE CONSIDERED INSUBORDINATION. I WANT STATUS UPDATES EVERY 15 MINUTES ON WHERE WE ARE AT WITH THIS.

Dev: …

Dev: Can I get you to send me that request in an email?

*Manager leaves the meeting*

// *****************************

Job search is ticking along. It’s tough going though because I currently make ~120k and the best offers I’ve received so far are all ~70k because “You only have 2 years experience so you couldn’t possibly have the skills to be worth 120k. You are are junior level developer and 70K is already overpaying for you. We can pay you more later™. No we will not give you that in writing”. Ah well, the hunt continues.

Colleagues sharing passwords.That was a big fat NO when I was a sysadmin - and for a good reason. But now, since I'm closer to development, it feels like no one really cares about the passwords. If I tell my colleague I'll take 10 minutes more because I can't log in, he OFFERS me his credentials. And sends them over saying "in case you need it". [the next day the same colleague was complaining his account is locked out. Oh, wonders! How on Earth...!]

But seriously, password sharing is a serious problem. I would fire the person on spot if I caught him sharing his credentials! This is the 8th deadly sin! IDC if they are for non-prod. Most people reuse their passwords in multiple systems, and even non-prod envs can bring the prod down! Or worse - install a trojan.

I login this morning

Everyone going crazy as prod is broken in a million ways. 2.5 million exceptions In 24 hours.

We talk for 1 hour and solve nothing.

I diagnose the 2 biggest sources of exceptions and explain every step I took with screen snippets to reach my conclusions

The one other competent person on my team agrees. Otherwise complete silence.

I'm told not to fix the issue because I am leaving in 6 weeks and other people have to start taking responsibility.

No one fixes the issues. Instead they leave early. Now it's the weekend and the product is fucked.

Fucking useless people. Can't wait to leave.

Me to a lead dev: hey, I noticed that junior guy pushed this bad code to prod that you approved.

Him: oh really that’s wrong? Ok we can fix it.

Me (cursing under my breath): no asshole, that’s not the fucking point. You should know enough to not approve such pull requests.

Week 278: Most rage-inducing work experience — I’ve got a list saved! At least from the current circle of hell. I might post a few more under this tag later…

TicketA: Do this in locations a-e.
TicketB: Do this in locations e-h.
TicketC: Do this in locations i-k.
Root: There’s actually a-x, but okay. They’re all done.
Product: You didn’t address location e in ticket B! We can’t trust you to do your tickets right. Did you even test this?
Root: Did you check TicketA? It’s in TicketA.
Product guy: It was called out in TicketB! How did you miss it?!
Product guy: (Refuses to respond or speak to me, quite literally ever again.)
Product guy to everyone in private: Don’t trust Root. Don’t give her any tickets.
Product manager to boss: Root doesn’t complete her tickets! We can’t trust her. Don’t give her our tickets.
Product manager to TC: We can’t trust Root. Don’t give her our tickets.
TC: Nobody can trust you! Not even the execs! You need to rebuild your reputation.

Root: Asks coworker a simple question.
Root: Asks again.
Root: nudges them.
Root: Asks again.
Coworker: I’ll respond before tomorrow. (And doesn’t.)
Root: Asks again.
Root: Fine. I’ll figure it out in my own.
TC: Stop making it sound like you don’t have any support from the team!

Root: Asks four people about <feature> they all built.
Everyone: idk
Root: Okay, I’ll figure it out on my own.
TC: Stop making it sound like you don’t have any support from the team!

Root: Mentions multiple meetings to discuss ticket with <Person>.
TC: You called <Person> stupid and useless in front of the whole team! Go apologize!

Root: Tells TC something. Asks a simple question.
Root: Tells TC the same thing. Asks again.
TC: (No response for days.)
TC: Tells me the exact same thing publicly like it’s a revelation and I’m stupid for not knowing.
TC: You don’t communicate well!

Root: Asks who the end user of my ticket is.
Root: Asks Boss.
Root: Asks TC.
Root: Fine, I’ll build it for both.
Root: Asks again in PR.
TC: Derides; doesn’t answer.
Root: Asks again, clearly, with explanation.
TC: Copypastes the derision, still doesn’t answer.
Root: Asks boss.
Boss: Doesn’t answer.
Boss: You need to work on your communication skills.

Root: Mentions asking question about blocker to <Person> and not hearing back. Mentions following up later.
<Person>: Gets offended. Refuses to respond for weeks thereafter.

Root: Hey boss, there’s a ticket for a minor prod issue. Is that higher priority than my current ticket?
Root: Hey, should I switch tickets?
Root: Hey?
Root: … Okay, I’ll just keep on my current one.
Boss: You need to work on your priorities.

Everyone: (Endless circlejerking and drama and tattling)

Recipe: "baked developer"

you will need:

- 1 day = 1 story point
- 10SP per sprint
- every team member must deliver all the SPs.

Now for every sprint slap on 20+ hours of mandatory meetings, mix with 2-5 days of ad-hoc tasks, which must be addressed, because they are blocking the release/other teams/prod, and make sure all the devs try not to spill no matter what, and you get a perfectly burned out team.

Brittle/crispy on the outside, mashed/soft on the inside

enjoy!

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---
<...>

< I log a JIRA ticket explaining what and how is wrong with the server with suggestions how to fix the problem so the app will not crash any longer (client own the server, has his own sysadmins -- I don't even had permissions to open syslog.. had to hack dmesg on their PROD server to pin-point the issue)
> no reaction from customer for weeks. I ping the ticket

× app crashes again

> no reaction from customer for weeks. I ping the ticket
> customer leaves a comment that their sysadmins are looking at it trying to figure out what might be wrong (ignoring what I wrote in ticket's description??? srsly?)

× app crashes again
< I post detail investigation details: snips from logs, screenshots, everything with crystal clear explanations.

> no reaction for weeks
......

well that's fun..

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Be me, new dev on a team. Taking a look through source code to get up to speed.

Dev: **thinking to self** why is there no package lock.. let me bring this up to boss man

Dev: hey boss man, you’ve got no package lock, did we forget to commit it?

Manager: no I don’t like package locks.

Dev: ...why?

Manager: they fuck up computer. The project never ran with a package lock.

Dev: ..how will you make sure that every dev has the same packages while developing?

Manager: don’t worry, I’ve done this before, we haven’t had any issues.

**couple weeks goes by**

Dev: pushes code

Manager: hey your feature is not working on my machine

Dev: it’s working on mine, and the dev servers. Let’s take a look and see

**finds out he deletes his package lock every time he does npm install, so therefore he literally has the latest of like a 50 packages with no testing**

Dev: well you see you have some packages here that updates, and have broken some of the features.

Manager: >=|, fix it.

Dev: commit a working package lock so we’re all on the same.

Manager: just set the package version to whatever works.

Dev: okay

**more weeks go by**

Manager: why are we having so many issues between devs, why are things working on some computers and not others??? We can’t be having this it’s wasting time.

Dev: **takes a look at everyone’s packages** we all have different packages.

Manager: that’s it, no one can use Mac computers. You must use these windows computers, and you must install npm v6.0 and node v15.11. Everyone must have the same system and software install to guarantee we’re all on the same page

Dev: so can we also commit package lock so we’re all having the same packages as well?

Manager: No, package locks don’t work.

**few days go by**

Manager: GUYS WHY IS THE CODE DEPLOYING TO PRODUCTION NOT WORKING. IT WAS WORKING IN DEV

DEV: **looks at packages**, when the project was built on dev on 9/1 package x was on version 1.1, when it was approved and moved to prod on 9/3 package x was now on version 1.2 which was a change that broke our code.

Manager: CHANGE THE DEPLOYMENT SCRIPTS THEN. MAKE PROD RSYNC NODE_MODULES WITH DEV

Dev: okay

Manager: just trust me, I’ve been doing this for years

Who the fuck put this man in charge.

Lead dev: Hey boss, you really do like Python right?

Me: No

Lead dev: Well it's cuz I was think....wait what? WTF do you mean no, you have automated a fuckload of BS with Python and we are still using it, why tf would you use Python if you don't like it?

Me: I like it enough for the automation scripts that we have and for parsing documents or generating glue scripts, its already installed in every server that we have, so testing bs in dev and then using them in prod is cake, it doesn't mean I LOVE python, I like it for what we use it.

Lead dev: Well ain't already bash and perl installed as well?

Me: Do you know bash and or perl?

Lead dev: No, don't you?....

Me: No......

L Dev: (using a Jim Carrey impersonation) WELLL ALLRIGTHY THEN! What is the other language that you used for X project?

Me: Clojure, do you remember that one?

* he said paren paren paren paren yes paren i space paren do close paren close paren etc etc
L Dev: (((((((yes (i (do)))))))) and nevermind, I'll get back to working more with Python

Me: das what I fucking thought esse

Every year my team runs an award ceremony during which people win “awards” for mistakes throughout the year. This years was quite good.

The integration partner award- one of our sysAdmins was talking with a partner from another company over Skype and was having some issues with azure. He intended to send me a small rant but instead sent “fucking azure can go fuck itself, won’t let me update to managed disks from a vhd built on unmanaged” to our jv partner.

Sysadmin wannabe award (mine)- ran “Sudo chmod -R 700 /“ on one of our dev systems then had to spend the next day trying to fix it 😓

The ain’t no sanity clause award - someone ran a massive update query on a prod database without a where clause

The dba wannabe award - one of our support guys was clearing out a prod dB server to make some disk space and accidentally deleted one of the databases devices bringing it down.

The open source community award - one of the devs had been messing about with an apache proxy on a prod web server and it ended up as part of a botnet

There were others but I can’t remember them all

That feeling when your client connection is more stable than the connection of a fucking game server... Incompetent pieces of shit!!! BEING ABLE TO PUT A COUPLE OF SPRITES DOESN'T MAKE YOU A FUCKING SYSADMIN!!!

Oh and I sent those very incompetent fucks a mail earlier, because my mailers are blocking their servers as per my mailers' security policy. A rant from the old box - their mail servers self-identify a fucking .local!!! Those incompetent shitheads didn't even properly change the values from test into those from prod!! So I sent them an email telling them exactly how they should fix it, as I am running the same MTA on my mailers (Postfix), at some point had to fix my mailers against the exact same issue as well, and clearly noticed in-game that they have deliverability problems (they explicitly mention to unblock their domain). Guess why?! Because their server's shitty configuration triggers fucking security mechanisms that are built against rogue mailers that attempt to spoof themselves as an internal mailer, with that fucking .local! And they STILL DIDN'T CHANGE IT!!!! Your fucking domain has no issues whatsoever, it's your goddamn fucking mail servers that YOU ASOBIMO FUCKERS SHOULD JUST FIX ALREADY!!! MOTHERFUCKERS!!!!!

One of the biggest reality checks you will run into when starting your first dev related job - and which they don't teach you about in school - is that a lot of the time will be spent working with other people's code, and rewriting it into "your own" is rarely an option.

You might be super into making things, but not everyone manages to maintain that same spark while taking over a 15 year old project with fundamental issues that have to be triaged "for now" because you need a hotfix on this other specific thing out in prod before lunch.

There are no gods now. They left the company years ago and nobody knows why they used the windows registry as a user repo.

TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

Joined a new company / team to work on an iOS app that has 2 different backend environments "Dev" and "Prod". Also being referred to in iOS speak as "Debug" and "Release".

Been trying to get accounts on these backends (no sign up in app, its controlled via another process). Eventually get access to "Dev" for one of the regions, so I load up "Debug" and its not working.

This is odd, so I open the Android app and load "Dev" and it works? I then Notice Android has "Dev", "QA", "Staging" and "Prod" for every region where as iOS only has 2 of these.

So I go back to iOS and find the file for the settings and it has iOS Debug assigned a variable for the backend Dev ... which is actually pointing to QA. Because they use QA to Debug and not Dev.

... confused? join the club

I just love it when our clients decide to make a clone of live production server..then put it immediately online..and don't tell anyone about this.. and then start bitchin how data gets doubled all of a sudden..

Yeah, no shit sherlock.. you have two prod servers for 'hot swapping' and some services may only be running on one at a time.. You even have a manual on how to switch primary to secondary (turn off services on primary first, then turn them on on secondary and all)..or in case primary actually dies, just turn on services on secondary and you're good to go, right?

So how do ya think cloning the one with running services and putting the clone immediately online will work out?! 🤔

God, I thought it was common sense to not do that..but here I am, bitchin about how people fail to RTFM.. :/ or use brain..fuck..

Somewhere in a lonely break room
There's a guy starting to realize that eternal hell has been unleashed unto him.
It's two a.m.

It's two a.m.
The boss has gone
I'm sitting here waitin'
This desktop's slow
I am getting tired of fixin' all my coworkers' problems
Yeah there's a bug on the loose
Errors in the code
This is unreadable
Rubber ducky can't help
I cannot debug, my whole life spins into a frenzy

Help I'm slippin' into the programming zone
Git push to the prod
Set up a repo
My hard drive just crashed
All my code is gone
Where am I to go
Now that I've broke my distro
Soon you will come to know
When you need Stack Overflow
Soon you will come to know
When you need Stack Overflow

I'm falling down a spiral
Solution unkown
Disgusting legacy, ugly code
Can't get no connection
Can't get through to commit

Well the night weights heavy
On my confused mind
Where's the error on this line
When the CEO comes
He knows damn well
To keep his distance
And he says

Help I'm slippin' into the programming zone
Git push to the prod
Set up a repo
My hard drive just crashed
All my code is gone
Where am I to go
Now that I've broke my distro
Soon you will come to know
When you need Stack Overflow
Soon you will come to know
When you need Stack Overflow
When you need Stack Overflow

When you need Stack Overflow, a ha
When you need Stack Overflow
When you need Stack Overflow, a ha
When you need Stack Overflow

When you need Stack Overflow, a ha
When you need Stack Overflow
When you need Stack Overflow, a ha
When you need Stack Overflow

Highlights from my week:

Prod access: Needed it for my last four tickets; just got it approved this week. No longer need it (urgently, anyway). During setup, sysops didn’t sync accounts, and didn’t know how. Left me to figure out the urls on my own. MFA not working.

Work phone: Discovered its MFA is tied to another coworker’s prod credentials. Security just made it work for both instead of fixing it.

My merchant communication ticket: I discovered sysops typo’d my cronjob so my feature hasn’t run since its release, and therefore never alerted merchants. They didn’t want to fix it outside of a standard release. Some yelling convinced them to do it anyway.

AWS ticket: wow I seriously don’t give a crap. Most boring ticket I have ever worked on. Also, the AWS guy said the project might not even be possible, so. Weee, great use of my time.

“Tiny, easy-peasy ticket”: Sounds easy (change a link based on record type). Impossible to test locally, or even view; requires environments I can’t access or deploy to. Specs don’t cover the record type, nor support creating them. Found and patched it anyway.

Completed work: Four of my tickets (two high-priority) have been sitting in code review for over a month now.

Prod release: Release team #2 didn’t release and didn’t bother telling anyone; Release team #1 tried releasing tickets that relied upon it. Good times were had.

QA: Begs for service status page; VP of engineering scoffs at it and says its practically impossible to build. I volunteered. QA cheered; VP ignored me.

Retro: Oops! Scrum master didn’t show up.

Coworker demo: dogshit code that works 1 out of 15 times; didn’t consider UX or user preferences. Today is code-freeze too, so it’s getting released like this. (Feature is using an AI service to rearrange menu options by usage and time of day…)

Micromanager response: “The UX doesn’t matter; our consumers want AI-driven models, and we can say we have delivered on that. It works, and that’s what matters. Good job on delivering!”

Yep.
So, how’s your week going?

Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: No.
Me: ...
Dev: The Testers are going to be pissed.

🎄🎊 MERRY CHRISTMAS 🎊 🎄

When you have no option but to push untested code to prod, also, you don't trust your shitty team.

New country, new company, new team, new projects.
I'm supposed to be the TL of a team working on a React project.

A guy in his late 40s celebrates himself as "the senior", he basically just finished watching a youtube thing, React 101 crash course or similar. The other two juniors who did only Wordpress so far venerate him like a god.

The code, of course, is one on the finest pieces of crap I ever had the pleasure to deal with in my life: naturally a bunch of JQuery plugins for everything, no tests, no state management, side effects everywhere, shared state and globals like hell, everything written in ES3/ES5 style, no types, no docs, build and deploy totally manual, deep props drilling at every level... and not to mention the console.log() shipped in prod.

First day, already headache.
Full rewrite start tomorrow.
Hiring real devs as well.

That time I thought I crashed a prod server which I had no access to physically turn it back on. Turned out it was doing its weekly reboot. Scary 3 minutes.

I've seen many devs doing crazy things in my entire career till now. But this one dude stands out.
He used to:
- Push binaries to git repo
- Use some old libraries which were used during Indus Valley civilization
- Had no sense of database and used to delete random data from it and call it as TESTING (Thank God! I never gave him prod access)
- And on top of this, he had an ass full of attitude!

When will I fuckin learn that
a) customers lie
b) customers are sloppy
c) customers are wrong
d) customers do not do their work (properly)
e) customers want us to do their (dirty) work
f) possibly all of the freakinly above?! + khm....

They will fuckin aaaalwaaaays say sth is not working after the update..
And I will alwaaaays assume I fucked up something..even if I didn't touch that part of the code/data..
And almost aaaaalways it turns out that the bug they complain about is how the system worked (or didn't work) before the update and/or some fuckup from their side..

Anyhow, I rushed over, grabbed the files went testing in dev..wtf, output is different, mine is ok, theirs is..wtf is that shit?!

Transfer newly built dll to test..same shit as on prod..wtf?! How?!

I assumed they have thing A correctly linked to thing B.. ofc thing A was linked to thing C in their case and in another case (our test) to correct thing B..
I got chillies when grabbing files, that
I should have tripple checked that they didn't fuck up something on the link part, but I just assumed they know what they were doing & that they checked they linked correct files with correct content already, before being pissy that the update fucked up things.. riiiight!! :/

I wanted to find solutions to this fuckup asap so I disregarded my gut feeling..yet again!! Fuuuck!
I've spent too much time trying to find ways to fix a bug that wasn't even a real bug to begin with.. :/

Fuuuuuck!!

So yeah, always treat the customers like they are 3yrs old & have no clue what they are doing & check exactly wtf they were indeed trying to do..it will save you time & nerves..

And note to self: reread this shit daily!! And imprint it in your brain that everything is not always your fault!!

So apparently there was a prod issue all day today because some server app couldn't start up.

The error was caused by missing Params...

And seems everyone that got asked to look at it are pretty clueless.

I looked at the text msgs now cuz my phone wouldn't stop vibrating from notifs (yes it got that bad) and I'm thinking, of all the ppl you guys could've asked, no one reached out to me... Who well knows exactly it works, and how to fix it.

But the issue with the team is everyone works on everything but most people have no idea how the apps actually work...

Jack of all trades, master of none...

Anyways, sitting on the sidelines watching the chaos ain't too bad either....

Me: eh probably shouldn’t push to prod on Friday, maybe not even before the first...

Co worker: no balls

Me:

I was asked to look into a site I haven't actively developed since about 3-4 years. It should be a simple side-gig.
I was told this site has been actively developed by the person who came after me, and this person had a few other people help out as well.
The most daunting task in my head was to go through their changes and see why stuff is broken (I was told functionality had been removed, things were changed for the worse, etc etc).
I ssh into the machine and it works. For SOME reason I still have access, which is a good thing since there's literally nobody to ask for access at the moment.
I cd into the project, do a git remote get-url origin to see if they've changed the repo location. Doesn't work. There is no origin. It's "upstream" now. Ok, no biggie. git remote get-url upstream. Repo is still there. Good.
Just to check, see if there's anything untracked with git status. Nothing. Good.
What was the last thing that was worked on? git log --all --decorate --oneline --graph. Wait... Something about the commit message seems familiar. git log. .... This is *my* last commit message. The hell?
I open the repo in the browser, login with some credentials my browser had saved (again, good because I have no clue about the password). Repo hasn't gotten a commit since mine. That can't be right.
Check branches. Oh....Like a dozen new branches. Lots of commits with text that is really not helpful at all. Looks like they were trying to set up a pipeline and testing it out over and over again.
A lot of other changes including the deletion of a database config and schema changes. 0 tests. Doesn't seem like these changes were ever in production.
...
At least I don't have to rack my head trying to understand someone else's code but.... I might just have to throw everything that was done into the garbage. I'm not gonna be the one to push all these changes I don't know about to prod and see what breaks and what doesn't break
.
I feel bad for whoever worked on the codebase after me, because all their changes are now just a waste of time and space that will never be used.

Took me 6+ full days.

The feature does not work. Repro is unknown, so only prod is experiencing the issue.. Which rules out the debugger option. Sometimes there's an entry seen in logs: "java.lang.StringIndexOutOfBoundsException". Nothing more - just that. No stack, no class, no nothing. Is it my code that's buggy? Is it some config? Integration? Unexpected response...? A bug in a lib? Is dimm faulty ir maybe server's shared libs are off?

Turns out I used a closing parentheses instead a closing curly bracket in an error message that's supposed to be interpolated...

String message = "{some-business-rule-related-error-message-key)";

took me 6+ full days... But I found it. Took the rest of that Friday off to walk in a park and enjoy my life :)

I feel JS has taken over...

The good.

Easy to learn, thus everyone knows it

The bad.

Everyone knows it, so npm has disgusting code that's used in prod.

Npm itself is still an unstable mess at times.

No one knows what it is to be a good programmer .

Dozens of frameworks every year since everyone thinks they can make it a bit better.

Classes still suck , no interfaces !

On chat today.
Dude: can you run a script for me? We don't have permission.
Me: what kind of script? Who wrote it?
Dude: posts screenshot of DML select/update statement he tried to run.
Me: I'm a DBA. We don't run DML for people.
Dude: Oh. Can you give me the password?
Me: examine script and notice he tried to run it on QA DB.
Me: No. We don't memorize passwords, and this is QA; you need to check the password out of the safe. You also need a change ticket to DevOps, and they will run it for you.

At that point I ended the discussion, because running anything in QA or Prod without a change ticket gets you fired. And I like my job. Really annoyed.

Have a couple I want to air today.

First was at my first gig as a dev, 4-5 months out of school. I was the only dev at a startup where the owner was a computer illiterate psycopath with serious temper tantrums. We're talking slamming doors, shouting at you while you are on the phone with customers, the works...

Anyways, what happened was that we needed to do an update in our database to correct some data on a few order lines regarding a specific product. Guess who forgot the fucking where-clause... Did I mention this boss was a cheap ass, dollar stupid, penny wise asshole that refused to have anything but the cheapest hosting? No backups, no test/dev/staging environment, no local copies... Yeah, live devving in prod, fucking all customers with a missing semi-colon (or where clause).

Amazingly, his sheer incompetance saved my ass, because even if I explained it, he didn't get it, and just wanted it fixed as best we could.

The second time was at a different company where we were delivering managed network services for a few municipalities. I was working netops at that time, mostly Cisco branded stuff, from Voice-over-IP and wifi to switches and some routing.

One day I was rolling out a new wireless network, and had to add the VLAN to the core switch on the correct port. VLAN's, for those who don't know, are virtual networks you can use to run several separated networks on the same cable.

To add a VLAN on a Cisco switch one uses the command:

switchport access vlan add XYZ

My mistake was omitting the 'add', which Cisco switches happily accept without warning. That command however can be quite disruptive as it replaces all of the excisting VLAN's with the new one.

Not a big deal on a distribution switch supplying an office floor or something, but on a fucking core switch in the datacenter this meant 20K user had no internet, no access to the applications in the DS, no access to Active Directory etc. Oh and my remote access to that switch also went down the drain...

Luckily a colleague of mine was on site with a console cable and access to config backups. Shit was over within 15 minutes. My boss at that time was thankfully a pragmatic guy who just responded "Well, at least you won't make that mistake again" when we debriefed him after the dust settled.

Me: “Hey boss, you assigned these things to me that I’m not qualified for and have no experience in. We should really hire someone with the specialized skills in this”

Boss “I agree. It’s a role I desperately think we should have hired for a long time ago”

Me “Ok so about these tickets the-“

Boss “I need you to write up a justification for this role, what kind of work the person would be doing and what budget implications we will incur”

Me “You’re asking me to write a job description for a class of work I’ve already admitted I have no experience or qualifications doing MYSELF?”

Boss “Correct”

Me “and I’m still responsible in the meantime for getting these other tickets done still aren’t I?”

Boss “Yes”

Me “Very well. I’ll email you a recap of this discussion then so we can come back to it later when we start hiring for the role”

(and so my ass is sufficiently covered when I inevitably bring down prod and people start asking why I broke prod)

There's this huuuge project I was a part of for half a year. I was kicked off along with a few dozens of other devs (>60% of total manpower) a while ago for particular company reasons.

Now the remaining devs are oh-so-enjoying their time there..

1. workload has not changed
2. deadlines have not changed
3. no one will have Christmas-NewYear vacation
4. a new k8s-based infra is scheduled to roll out to PROD on Dec 23 (k8s is still far from ready - might need a few more months)

The most fun part is that it's not client's mgmt who has decided for #4 -- it's our own....

Boys.. Girls.. Save yourselves.

So I joined this financial institution back in Nov. Selling themselves as looking for a developer to code micro-services for a Spring based project and deploying on Cloud. I packed my stuff, drove and moved to the big city 3500 km away. New start in life I thought!
Turns out that micro-services code is an old outdated 20 year old JBoss code, that was ported over to Spring 10 years ago, then let to rot and fester into a giant undocumented Spaghetti code. Microservices? Forget about that. And whats worse? This code is responsible for processing thousands of transactions every month and is currently deployed in PROD. Now its your responsibility and now you have to get new features complied on the damn thing. Whats even worse? They made 4 replicas of that project with different functionalities and now you're responsible for all. Ma'am, this project needs serious refactoring, if not a total redesign/build. Nope! Not doing this! Now go work at it.
It took me 2-3 months just to wrap my mind around this thing and implement some form of working unit tests. I have to work on all that code base by myself and deliver all by myself! naturally, I was delayed in my delivery but I finally managed to deliver.
Time for relief I thought! I wont be looking at this for a while. So they assign me the next project: Automate environment sync between PROD and QA server that is manually done so far. Easy beans right? And surely enough, the automation process is simple and straightforward...except it isnt! Why? Because I am not allowed access to the user Ids and 3rd party software used in the sync process. Database and Data WareHouse data manipulation part is same story too. I ask for access and I get denied over and over again. I try to think of workarounds and I managed to do two using jenkins pipeline and local scripts. But those processes that need 3rd party software access? I cannot do anything! How am I supposed to automate job schedule import on autosys when I DONT HAVE ACCESS!! But noo! I must think of plan B! There is no plan B! Rather than thinking of workarounds, how about getting your access privileges right and get it right the first time!!
They pay relatively well but damn, you will lose your sanity as a programmer.
God, oh god, please bless me with a better job soon so I can escape this programming hell hole.
I will never work in finance again. I don't recommend it, unless you're on the tail end of your career and you want something stable & don't give a damn about proper software engineering principles anymore.

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

School gave me 3 DigitalOcean droplets to try out Kubernetes in the cloud, awesome!

Wrote an Ansible script to not only simply install docker and add users but also add kubernetes, nice!

Oh wait, error?! Well I should've known this wasn't going to be easy... ah well no problem. Let's see... Ansible is cryptic as always, it can't connect to the API server? Is it even running?

Let's ssh to the master, ah nothing is running, great. Let's try out kubeadm init and see what happens, oh gosh, my Docker version has not been validated! No problem, let's just downgrade!

How do I do that? Oh I know, change the version in the role! Wait that version doesn't exit? Let's travel to Docker's website and see what versions exist of docker-ce, oh I see, it needs a subversion, no problem.

Oh that errors too? Wait then what... Oh I need a ~ and a ubuntu and a 0 somewhere, my mistake!

Let's run it again! Fails!

Same ssh process, oh wait...

Oh god no...

Kubernetes requires 2 cores and these things only have 1...

Welp, time to ask the teachers to resize my droplet by a small amount tomorrow, hopefully I'll get a new error!

----------------------------------------------

My adventure so far with Kubernetes. I'm not installing it for any serious/prod reason, just for educational purposes. K8s seems like 'endgame' to me, like one of the 'big guys' that big enterprises use so I'm eager to throw stuff at a droplet and see what happens.

Going further down the rabbit hole tomorrow!

Wish me luck :3

(And yes, I could've figured this all out beforehand with documentation, but this is more fun in my opinion)

So... Heard back from a recruiter today. Lovely lass.

I’d passed over a submission for her tech demo.

The brief was basically just to create a small simple module that calculates shit, nae effort.

But, when the recruiter had me on the phone she said “I know it’s a silly small module but try and run it up like you would a production ready app”.

The job spec and recruiter were keen on me demonstrating TDD, not specific on js version, final runtime, etc. The job was a senior spec at a higher salary range. So it warranted some effort, and demonstrating more than a simple module.

“Okay, cool, nae bother, let’s crack on.”

The feedback in the response from the dev today:
“He’s over-engineered tests, build...”

SUCK MY LEFT TESTICLE YOU FUCKWIT.

Talk to your recruiters, not me.

The feedback included a phrase I never hope to hear from a developer I work with:
“Tests are good but...” 😞

It was a standard 98% test suite from an RGR cycle, no more or less than I’d expect in prod.

The rest of the feedback was misguided or plain wrong. It was useful to see because I know now when they say they have “high standards” they mean: we listen to the dude who put the factory pattern in a JS brief.

Oh shit also: “someone’s done chmod 777” was in there as a sarcastic comment in the feedback. It was his fucking unarchive tool 😞

My response was brief and polite: “cheers for the consideration, all the best, James”

It’s honestly not worth warning them. Or, asking why they’d criticise something they’d asked me to do.

If you want a shitty js module, ask for a shitty js module and no more.

We've worked 5 months to decompose a complex and huge monolith into microservices, deployed in prod with zero defects. And finally moving to AWS, one by one.
How can i explain this work to bunch of 5 year olds? i.e. i've to present this to top level management with no tech knowledge.

I'm thinking of: Lets say a family of 6 people want to travel for 30 holidays to another country. A monolith can be equivalent to having everyone's luggage in huge bag, microservices can be packing luggage in sizable chunks acceptable by airlines.

I'm bad at explaining, can someone help with better example?

TL;DR;

Idiot hard coded database host on the app... Pushed to prod and suddenly shit wasn't working... Took me 10 minutes to figure out what was going on...

Wrote a passive aggressive git message and commited.
Before updating prod my boss turns around to me and the following took place:

Boss: is there any problem with the server?

Me: yes, someone (i know who was ) hard coded the test db IP and it broke the backend.

Boss: oh, but will it affect the mobile app?

Me: well, it won't work but I'm already pushing the fix.

Boss: no..err.. I mean... Will I have to make any change to the mobile app?

Me inside: wtf dude... For real?! Get your shit together...

Me: no. It good, I already fixed it.

Boss: OK. Thanks

TL;DR;

Moron hard coded dB's host and stupid boss can't get shit together nor ask who did it to take precautions...

I got cut from a contracting job yesterday I have 3 weeks left in the contract. They said I worked well with the team, had a great work ethic but didn't think I had strong enough tech skills. In the past this would have hurt my feelings and it does a little but I think my tech skills are fairly high. There were three devs working on 66 apps with no tests, some source control but most of the code in source control was older than code deployed in prod, no automatic builds, people would wait a week before checking in code, others would check in code that would not build. Today the boss asked if I had messed with app pools on the prod iIs server because something was wrong. I said no because never remote into the server. Anyway it is the end of the world and I feel fine.

I JUST HAD ONE OF THOSE DAYS THAT MAKES ONE WANT TO BANG TWO BRICKS ON HEAD SND END THE PAIN THE STORY STARTS YESTETDAY WITH ISSUES AFTER A MIGRSTION AND THEY ASK ME TO HELP TROUBLESHOOT EVEN THOUGH I'M A DEV DBA AND THE ISSUE IS IN QA/SAT AND I HELP ANYWAY AND THEY CAN'T FIND A VIEW AND SO I LOOK EVERYWHERE AND CAN'T DOING IT EITHER AND IT DIDN'T EXIST IN PROD OR DEV SO I TELL THEM IT'S NOT THERE, AND THEY ARE LIKE, CAN YOU RETRIEVE IT FOR US AND I'M LIKE FROM WHERE? I DON'T KEEP VIEWS IN MY BUTT AND YOU GUYS ARE SMOKING CRACK AND THE GIVE ME THEIR QUERY WHICH CONTAIN THE VIEE ANYWAY AND THEY SAY CAN YOU RUN IT AND IT RUNS AND WORKS AND THEY CAN'T MAKE IT WORK AND IT WORKS BECAUSE IT DOESN'T CALL THE VIEW THEY HAVE ME SO NO PROBLEM THERE SO I FINALLY ASK THEM ARE YOU POINTING TO THE CORRECT DATABASE AND THEY'RE LIKE OH MAN WE TOLD YOU THE WRONG DATABASE AND SO I LOOK AT THE RIGHT DATABASE AND FIND THAT THE GRANTS ARE MISSING AND YEAH THANK YOU FOR TAKING EIGHT HOURS OF MY LIFE BECAUSE WE WERE IN THE WRONG DB YOU GAVE ME AND I HOPE THE FLAG OF A THOUSAND CAMELS INVEST YOUR ARMPITS AND THE CHIGGERS OF A THOUSAND SOUTHERN LAWNS INGEST YOUR SOCKS AND UNDERWEAR. YAAAAAA!!!!

that moment when you were still new to git commands and just typed in random commands while thinking of a stupid dog meme saying I have no idea what im doing then accidentally pushing something in prod and had no idea what to do cause the lead dev was not around... aahhh those were the days

On my project the customer has re-signed into a contract several times when they have budget to continue work. The first time they got us to build the system was a huge success story because the team was assembled quickly and we did rapid development. Initialize repo to prod in 1.5 months. The customer asked for the same dev team. Strong dev team, a PM that doesn't take shit, and pure agile. Lets call her don't-take-shit PM.

When the customer re-signed the executive decided that she didn't like don't-take-shit PM. So the project manager gets replaced by play-by-the-rules PM who will comply with stupid requests and micromanagement. He isn't a bad PM but he tries to make everyone happy. The amount of management types executive installs on the project is massive, and development team is cut down in major ways. Customer and executive shit rolls down to the development team and we can't get anything done. The customer starts to lose faith because we can't get traction. They start demanding traditional waterfall/SDLC docs. Which causes more delay in the project.

So the executive decides that the PM can take a fall for it to save face for the company. She moves play-by-the-rules PM to another project. He starts handover to a new PM that has a history of being her pushover. The customer hadn't seen him yet so now we have push-over PM.

Play-by-the-rules PM is finally out of the project and instead of moving to a different account the company decides to "lay him off because there is no work". So basically they made him take the fall for the failure while promising reassignment, and instead let him go. This is so unfair..

Meeting with push-over PM yesterday and he shows us his plan. Identical to play-by-the-rules PM's plan that got him axed.We point that out and show him the docs that were made for it. His face clearly communicates "OH SHIT WHAT DID I SIGN UP FOR?"

This is what happened today in our dayli:

Lead: We need to profile our software

Me: You can use the chrome devtools as remote profiler, even on prod, or make HAR files for later inspection.

Lead: Yeah but no that’s just collecting data on every tick, we need something like “has been called x times”

Me: Yeah but you can filt -

Lead: Yeah no, so back when I wrote code in Delphi...

Me: *oh god no not this again*

Lead: ... We could have clicked a button in our IDE and it would wrap the function call with the API call to profile that function ...

Me, to the secret dev group in slack: doesn’t a simple method decorator and node performance api help with that?

The people in the group: We had this topic last Friday all day...

Me: oh well *get’s coffee and ignores lead*

Boss wanted me to make changes in company's website which was based on wordpres s.

I knew it could be done by tweaking some JS code, but I have very less experience with wordpress

But wordpress is easy man(Internet told me).

Give me 5 minutes, you will see the changes in production.
Being lazy af I directly logged in to ftp, checked out some files, updated some code, I was good to go.

Before pushing it, I opened the website and it was GONE ٩(๑´0`๑)۶

Now there was no public_html in the root.
I was fucked. I have accidentally deleted the website that had no backup.

And the best part I was on leave from
next day.

I was looking everywhere for backups, looked into google cache to get the contents. I have to recreate the complete site now.

Just when I was asking questions on choice of my profession and simultaneously looking here and there in FTP for backups,
I found the jewel "public_html".

It happens out that I have accidentally moved the folder to some other directory.

Phewww.

Moved it back to root. Site was up and running.

Reassured myself that I deserve to be a dev.

Backed up complete site, made the changes.

Uploaded it.

And the best part, amount of wordpress I learned in those three hours was way more than I could have learnt in many weeks.

Lessons Learnt :

A) ALWAYS keep backups.
B) You SHOULD NOT make changes on prod directly
C) You become superhuman when your brain know you are going to be fucked 😂

Developing a notification API, sends emails to subscribers, email API can take only 100 IDs at once, so partitioned the email list and send mails in blocks of 100.

Forgot to reset the list after every block, so each new partition got appended to the existing list and kept going on.

Ran it against a test DB, which was recently refreshed with near-prod data !!! Thousands of emails went out of the app server in one shot and everybody receiving numerous duplicate emails. Especially the ones in the very first partition.

Got an incident raised by the CEO himself reg the flurry of emails. But, things were out of our hands, quite literally. All emails are queued up in the exchange server.

Called up the exchange server team, purged​ the queued emails. No other emails were sent/received during this whole episode.

Thanks to Iterables.partition in the present day.

I'd never do anything "risky" in a prod environment if I considered it so at the time, but in retrospect there's *lots* of things considered risky now (both from a security and good practice viewpoint) that were standard practice not long ago:

- Not using any form of version control
- No tests (including no unit tests)
- Not considering XSS vulnerabilities
- Completely ignoring CSRF vulnerabilities
- Storing passwords as unsalted MD5 hashes (heck that was considered very *secure* in the days of plaintext password storage.)

...etc. I'm guilty of all of those previously. I daresay in the future there will be yet more things that may be standard practice now, but become taboos we look back on with similar disdain.

Central team: No, your team must be doing something wrong. Our pipeline is super-configurable and works for any situation! You just have to read the docs!

Me: Where are the docs?

Central team: Uhh, well, umm... we'll hook you up with a CI/CD coach!

Me: Okay, cool. In the mean time, can you point me at the repo where all the base scripts are?

Central team: Sure, it's here.

Me, some weeks later: Yeah, uhh, the coach can't seem to figure out how to make our Prod deployment work either.

Central team: That's impossible! It's so easy and completely configurable!

Me: Well, okay... but, here's the thing: your pipeline IS pretty "configurable", in the sense that you look for A LOT of variables...

Central team: See! We told you!

Me: ...none of which are actually documented, so they're just about useless to me...

Central team: But, but the coach...

Me: ...couldn't make heads or taisl of it either despite him literally being ON YOUR TEAM...

Central team: Then your project must just be architected wrong!

Me: Well, we're not perfect, so could be...

Central team: Right!

Me: ...but I think it's far more likely that the scripts... you know, the ACTUAL Python scripts the pipeline executes... while it took me DAYS to get through all your levels of abstraction and indirection and, well, BULLSHIT... it turns out they are incredibly NOT flexible. They do one thing, all the time, basically disregarding any flexibility in the pipeline. So, yeah, I'm thinking this is probably one of this "it's you, not me" deals.

Central team: Waaaaahhhhhhhh!!!!!!

FUCKING FUCK ANGULAR!!!!
LIKE FUCK IT IN THE ARSE AND BURN THE MOTHERFUCKER WHILE LAUNCHING A MISSILE ON IT TO BE SURE!

(ﾉ≧∇≦)ﾉ ﾐ ┸━┸

So I am making something on angular and I got everything running in ng serve(development environment) , after handling all issues and showing it to my boss man he approves and asked to put it up on prod for a demo , doesn’t sound like an issue , I make the prod build on cli and BAM! 16 errors ? No issues right?, I’ll just google the issue. Googles.... there aren’t no clear solutions to it as the angular version keeps changing and nobody knows what broke it, I mean people have the issue,but like 100 reasons that can cause it,
HOLY LORD RELEASE A NEWER VERSION AFTER MENDING THE OLD ONE
But nooooooo!

Angular Dev:We fucked this one, lol what should we do boss man?
Angular boss man: lol just leave it, we need to build the new version with newer bugs,

P.S. I like angular, but it’s like a underdeveloped framework, too many issues and too many changes

Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?

Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!

Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..

And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..

When will X be finished?!

Um..I don't know cuz thing Y is blocking me from doing X implementation and it makes no sense.. I have no idea what is wrong with it and can't tell how long it'll take me to fix, hopefully by the end of today I'll have thing Y fixed enough so I can finish thing X..

Ok, will you be able to update production today?! Just put it on production when you're done with X.

Uuuumm... Yeah... // still need to know how X will behave in cases a, b, c & b+c...

I'm writing this while waiting for abc feedback.. this is going great.. :/

I know prod is not used as prod yet, but still... fuuuuu!!

IMHO technical dept is kind of like smoking cigarettes for some decades.

You were told that shit will hit the fan but you do not take proper action. And one day you'll realize that you fucked up (or not, also seen that).

Worked for a company in IT, where we maintained an ERP which was "in progress" for over a decade. The basic implementation was done by people with zero technical understanding. To clarify: not self coded. Software was bought. We are talking about integrating the system.

Therefore, the foundation was like a wet noodle. When I joined that company, I told them that they need to address that. I told them that things will get slower and slower and that shit will hit the fan if no proper actions taken.
Even made a list with flaws I found. With potential risk and actions to take, that could then be measured.

At that time, five people worked in said department (including me).

People did not want to listen. "Would be too expensive to rewrite stuff".

Nothing has changed about the wet noodle, but I tried to fix as many things in a working system as I could. Felt like heart surgery, because changes got implemented and "tested" in prod. No version control, no documentation, everyone implemented things like they felt (no guidelines for consistency).
A lot of small fuckups that summed up over the years.

I left the company after two years because I had the chance to land a job as a dev.

Been around two years now since I left. Now 9 people work in that department with around the same efficiency as us 5 people back then.

The new employees struggle to be productive, because things are just implemented poorly and not maintainable anymore.

Had some dialogs with them some time ago. Everything I told them would happen, actually happened. What a suprise :-|
I will not go into too much detail about all the shit that's going on there, as it would be just too much (and my morning coffe is almost finished).

I think that we all know the difference between "not beautiful, but does the job" and "oh, that will backfire - badly". And I wish that my communication skills increase so that people start listening in future.

Returning from vacation tmr... But apparently we just had a "someone at github deleted all the repos and no backup"-like prod issue....

Well guess I say least can say "¯\_(ツ)_/¯ not my area, can't help u"

I didn’t turn down a dev freelance project when the client decided against going with best practices because the solution I offered was a well-established design pattern but created a need for a financial management change she didn’t like. I stupidly built what she asked for. It worked fine in the 3rd party vendor test environment but failed on production. After hours of analysis of code to ensure no changes happened to my source during test->prod deployment, and the vendor denying they had config differences between them, and the client refusing to pay, all I could do was abandon the project.

So last week I really fucked up

I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.

So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.

Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"

FUUUUUUUUUUUUCK!!!

Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.

Needless to say, I had a fairly good weekend.

Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert

Oh so day continued....

My boss just asked me before I left... You know that report we wanted automated, you said you'd get it done by today, is it done yet?

Me: well uh who dragged me into a PROD issue because no one else knows how to investigate... EVEN THOUGH I BUILT AND SHOWED U HOW TO USE THEM... SEVERAL TIMES. (no i didn't say this last but that's what went on in my head).

Oh and I figured out what the issue was...

Common misconception is that low efficiency technologies (react native, in this case) are completely useless. That is not true at all. Im building a smart home, and yes i want a mobile app, but 90% is just requests to endpoints, so its way simpler to just copy paste some buttons inside a js file than do it properly with native. Would i use it in prod? Fuck no. Would i hack together a bunch of buttons? Fuck yes.

I am doing some freelance work for a client who is thankfully mindful about security. I found out that they are so strict with their access because they had a huge data breach last year.

Today I was given access to their repo for connecting to their AS400. In the docker file the username and password were included and were the same for dev and prod. They also are performing no sql injection prevention. They are just joining strings together.

Rant from a previous gig I just remembered that reignited my fury lol

Suddenly, CSV exports became massively critical to our product's success. "They were always part of the plan, if we don't have them the product is a failure". Plot twist, they were NOT always part of the plan. And our backend is not at all designed for querying the combinations of data you're asking for.

Nevermind we've been entirely focused these last few months on making the new user experience as slick as possible because "our customers want cake, not meat and potatoes". Forget the fact that, in order to meet the deadlines, my team coupled the backend a little too much with the needs of the frontend because otherwise integrations took too long. We NEED fucking CSV exports of everything you can fucking imagine.

No. Fuck you. If you want it, it's gonna take at least 2 engineers and a month, and according to you we only have a few weeks of runway. No, I'm not compromising jack shit, this is the reality we live in. This is going to go nuclear in production if we don't do it right. Either give us the month and bankrupt the company, or fucking drop it.

Or...you could go cry to the frontend team for solutions. And convince them to page through ALL of the data and generate CSVs in the fucking browser. Sure, it sort of works in QA with the miniscule amount of data we have there, but how'd that work out for you in prod?

Jesus fucking christ why are you people such incompetent morons, and how the fuck did you become executives??

Designers use plugin in WP to modify UI. CSS and JS is saved (somewhere?) in database. No version control. Changes are made across different envs at the same time and all need to be migrates to prod later.

Pls

I Just realized that during my sort conversation with my boss tonight where I said I haven't learned anything new from this job in years. That usually I bring what I learned during my own time to work. He said "didn't you learn something new investigating this issue?"

NO I DO NOT WANT TO FUCKING HAVE TO HAVE A PROD ISSUE IN ORDER TO LEARN SOMETHING NEW....

Love my new job but fuck they are way behind in any kind of modernization.

Just saw a demo over zoom where someone was showing the team how to change the margin on an error page.

They literally changed the HTML directly in prod using the VIM. So first of all no web modernation because there was no build, no deployment, not git, no pipeline - NOTHING!

This project went from 40 people to around 200 in 6 mos. You can't have all these people in prod just making fucking changes.

If this job did not pay 110k a year I would bail.

Managers at my jobs for the most part leave me be. Though I often have no clue whether I'm doing OK. I guess no news is good news right?

My worst experience isn't that bad. At one place, I was the only tester working on things coming from 20-30 devs. After about a year+, the company finally hired more testers, but it was still only 3 of us.

We were in the final stages of releasing a build to prod. It was going smoothly, or so we thought. At the last minute, I found a buried bug that was a showstopper.

A lot of hatred on me that day, that once it was fixed, and the release was finally deployed, I just shut off my laptop and left. I took all the blame because I was the one who found it rather than blaming the team as a whole for not finding it earlier. Oh well. Stuff happens.

Let's knock on wood that I don't run into worse higher up stories.

1 Week ago I took down the development version of an internal demo, leaving only the prod one (temporary space issue). Colleagues were told n+15 times via email, face to face, comments on tickets etc. The prod one has also been live for weeks, and again, they were told when it went up.

This just happened:

Colleague 1: practiseSafeHex can you help me, the demo doesn't seem to be working.

Me: *logs in*, *click around* ... seems fine dude. Are you using the correct URL: <prod-url>.

Colleague 1: let me try again and see.
Colleague 2: practiseSafeHex i'm with Colleague 1, were trying to use the demo, and its not working. Can you have a look, we need it.

Me: I just told him, its fine, I think you have the wrong URL <prod-url>. Can you try again.

Colleague 2: No I have the right one, can you check it.

Me: *does nothing*, yep i've had a look at it, can you try again: <prod-url>.

Colleague 2: Ah its back now, cheers.

They are with a customer now, so I won't say anything, when they get back, one of them is being castrated.

We had 1 Android app to be developed for charity org for data collection for ground water level increase competition among villages.

Initial scope was very small & feasible. Around 10 forms with 3-4 fields in each to be developed in 2 months (1 for dev, 1 for testing). There was a prod version which had similar forms with no validations etc.

We had received prod source, which was total junk. No KT was given.

In existing source, spelling mistakes were there in the era of spell/grammar checking tools.

There were rural names of classes, variables in regional language in English letters & that regional language is somewhat known to some developers but even they don't know those rural names' meanings. This costed us at great length in visualizing data flow between entities. Even Google translate wasn't reliable for this language due to low Internet penetration in that language region.

OOP wasn't followed, so at 10 places exact same code exists. If error or bug needed to be fixed it had to be fixed at all those 10 places.

No foreign key relationships was there in database while actually there were logical relations among different entites.

No created, updated timestamps in records at app side to have audit trail.

Small part of that existing source was quite good with Fragments, MVP etc. while other part was ancient Activities with business logic.

We have to support Android 4.0 to 9.0 of many screen sizes & resolutions without any target devices issued to us by the client.

Then Corona lockdown happened & during that suddenly client side professionals became over efficient.

Client started adding requirements like very complex validation which has inter-entity dependencies. Then they started filing bugs from prod version on us.

Let's come to the developers' expertise,
2 developers with 8+ years of experience & they're not knowing how to resolve conflicts in git merge which were created by them only due to not following git best practice for coding like only appending new implementation in existing classes for easy auto merge etc.

They are thinking like handling click events is called development.

They don't want to think about OOP, well structured code. They don't want to re-use code mostly & when they copy paste, they think it's called re-use.

They wanted to follow old school Java development in memory scarce Android app life cycle in end user phone. They don't understand memory leaks, even though it's pin pointed by memory leak detection tools (Leak canary etc.).

Now 3.5 months are over, that competition was called off for this year due to Corona & development is still ongoing.

We are nowhere close to completion even for initial internal QA round.

On top of this, nothing is billable so it's like financial suicide.

Remember whatever said here is only 10% of what is faced.

- An Engineering lead in a half billion dollar company.

I hate having to deal with our IT service desk. Every time it takes enormous energy to get to the right people and make them understand that no, you are not an idiot, but you actually have a technical issue.

Sure thing they do have a few competent nice folks there too I've gotten to know over time and they indeed have to deal with a ton of dumb non-tech savvy idiots on a daily basis. However, if my job title mentions "software" and "engineer" they should at least assume I'm an idiot in tech. Or something. Every single time I need to open a ticket, even for the simplest "add x to env y", I need to quadruple check that the subject line is moron-friendly because otherwise they would take every chance to respond "nah we can't do that", "that's not us", or "sry that's not allowed". And then I would need to respond, "yes you do:) your slightly more competent colleague just did this for us 2 weeks ago".

Now you might imagine this is on even another level when the problem is complex.

One of our internal apps has been failing because one of the internal APIs managed by a service desk team responds a 500 status code randomly but only when called with a specific internal account managed by another service desk team.
(when I say "managed by", that doesn't mean they maintain it, it just mean they are the only ones who would have access to change something)

Yesterday I spent over a fucking hour writing a super precise essay detailing the issue, proving a million times it's not on our end and that they need to fix it. Now here is an insight to what beautiful "IT service" our service desk provides:

1) ticket gets assigned to a "Connectivity Engineer" lady

2) few hours later she responds and asks me to give her the app and environment IDs and grant her access to those
(naturally everything in my email was ignored including these two IDs)

3) since the app needs to be in prod for the issue, I make a copy isolating the failing part and grant her access to the original "for reference" and the copy to play with

4) few hours later I get an email from the env that some guy called P made changes to the actual app, no changes to the copy
(maybe they immediately fixed the app even though I asked them to only touch the copy)
I also check the env and the live app had been shared with another 2 people giving them editing rights:)

5) another few hours pass and the lady responds that she had been chatting with P (no mention of who tf that guy is) and that P has a suggestion that might work and I should test it, "please see screen shot" for details:

These motherfuckers sent me a fucking screenshot of the env config file where "P has edited a few parameters" that might help. The screenshot had a 16 line part of the config json with a bunch of IDs and Base64 params which HE EDITED LOCALLY.

Again, because I needed a few iterations to realise what I've just witnessed:
These idiots modified some things in the main app (not the copy) for hours. Then came to the conclusion that the config needs some IDs and params updated. They downloaded the config json. Edited it locally. Did not fucking upload it back to the main or test app. Did not test it live. Did not CC in or direct the guy with changes to me. Did not send me the modified config file. Did not even paste the new IDs into the email. But TOOK A FUCKING SCREENSHOT OF THE MODIFIED FILE AND SENT THAT SHIT TO ME. And then had the audacity to ask me to test it when they had access to it and that's literally their fucking job.

I had to compare the fucking screenshot to the live config file and manually type in the changes.

And no, it still doesn't work. And Now I have to get back to them showing it still fails the same way but I just can't deal with these people. Fuck. Was hoping by the time I write it all down it'd be better, and it does feel a bit better, but I still need to get this app fixed. And I can only do it through these... monkeys. I just can't. Talking to these people drains my life energy... I'm just sad.

An intern made a very bad impression on the first day.

This was before I become a developer. I was working in commercial art sales. One day, I had an appointment to onboard two new interns together.

Intern 1 shows up and I ask her for her signed confidentiality agreement. The boss had sent it out a week before and told me the interns were bringing the signed paperwork on their first day. I see the surprised look on her face and she says she forgot. She’s lucky I had access to another copy. If I didn’t, things could have gotten pretty awkward if I had to contact my boss, who was out of office. If there’s no signed agreement, I can’t onboard her and I’d have to send her home. The appointment was made with intern 1’s availability in mind, so intern 1 could have spent her time coming to the office for nothing and being turned away because of a stupid mistake she made.

While we wait for intern 2 to arrive, I try to engage in small talk with intern 1. I try to get to know her a little better and I ask “are you still in college/university?” She word vomits that she thought she had graduated, but six months later she hadn’t received her diploma and she called the school and they told her her pre-college credits had not transferred, so she’s finishing those credits now.

Oh, intern, you should have just simplified all this to “I’m finishing up my degree” or “yes, I’m still in college.” This is TMI. You don’t want to give out information about yourself that could put you in a bad light. You need to know to be discreet about yourself. You’re 22 years old. It’s really bad judgement to say this to your supervisor (me) and we’ve only known each other for ten minutes. I’m not your friend, I’m your supervisor. Honestly, I thought the explanation didn’t make sense because she would have found out about the credits when she tried to transfer them and when she applied for graduation. I didn’t prod for more details.

I did have to tell my boss about intern 1 forgetting the paperwork. It’s not something the intern would be reprimanded for, but it is something that’s not a good sign. The paperwork had been sent by the boss a week prior. It’s troublesome that an intern would forget to complete an important task that was sent by the boss. This was never a problem with prior interns.

Boss did freak out because boss thought I onboarded intern 1 without intern agreeing to the confidentiality agreement. Boss hadn’t considered an intern would forget the paperwork and didn’t tell me what to do if this did happen. I reassured boss that I had printed a new copy and had intern 1 sign the agreement.

I didn’t say anything about the word vomit. The content was troubling, but I was concerned this would be gossip and I wasn’t out to sabotage the intern.

Forgetting the paperwork and the word vomit were signs the intern wasn’t reliable. Intern had trouble taking direction even when it was written down. She’d do stupid things like invite her boyfriend to the office for hours and let BF sit at the boss’s desk—boss caught her and boss’s office is visible from our public viewing floor, so visitor did see this too. I suspected she might have an diagnosed learning disability.

In the end, intern didn’t ask for a reference letter. Boss said that if intern asked for one in the future, the answer would be no.

Intern 1 is the reason why I don’t want to be in change of interns ever again even though I’m not in art sales anymore.

ideal sprint fallacy.
total days 10 , total hours(excluding breaks ) 8 hrs per day= 80 hrs per dev

code freeze day = day 8, testing+ fixing days : 8,9,10. release day : day 10
so ideal dev time = 7days/56 hr
meetings= - 1hr per day => 49 hrs per dev

- 1 day for planning i.e d1 . so dev time left . 6 days 42 hrs.
-----------
all good planning. now here comes the messups

1. last release took some time. so planning could not happen on d1. all devs are waiting. . devtime = 5 days 35 hrs.

2. during planning:
mgr: hey devx what's the status on task 1?
d: i integrated mock apis. if server has made the apis, i will test them .
mgr : server says the apis are done. whats your guestimate for the task completion?
d : max 1-2 hrs?
m : cool. i assign you 4 hrs for this. now what about task 2?
d : task told to me is done and working . however sub mgr mentioned that a new screen will be added. so that will take time
m : no we probably won't be taking the screen. what's your giestimate?
d : a few more testing on existing features. maybe 1-2 hrs ?
m: cool
another 4 hrs for u. what about task 3?
d : <same story>
m : cool. another 4 hrs for u. so a total of 12 hrs out of 35 hrs? you must be relaxed this sprint.
d : yeah i guess.
m cool.
-------
timelines.
d1: wasted i previous sprint
d2 : sprint planning
d3 : 3+ hrs of meetings, apis for task 1 weren't available sub manager randomly decided that yes we can add another screen but didn't discussed. updates on all 3 tasks : no change in status
d4 : same story. dev apis starts failing so testing comes to halt.
d5 : apis for task1 available . task 3 got additional improvement points from mgr out of random. some prod issue happens which takes 4+ hrs. update on tasks : some more work done on task 3, task 1 and 2 remains same.
d6 : task1 apis are different from mocks. additionally 2 apis start breaking and its come to know thatgrs did not explain the task properly. finally after another 3+ hrs of discussion , we come to some conclusions and resolutions
d7 : prod issue again comes. 4+ hrs goes into it . task 2 and 3 are discussed for new screen additiona that can easily take 2+ days to be created . we agree tot ake 1 and drop 2nd task's changes i finish task 2 new screens in 6 hrs , hoping that finally everything will be fine.

d8 : prod issue again comes, and changes are requested in task 2 and 3

day 9 build finally goes to tester
day 10 first few bugs come with approval for some tasks
day 11(day 1 of new sprint) final build with fixes is shared. new bugs (unrelated to tasks. basically new features disguised as bugs) are raised . we reject and release the build.

day 2 sprint planning
mgr : hey dev x, u had only 12 hrs of work in your plate. why did the build got delayed?

🥲🫡

Found this line in code that's been in prod for 10 years

"int kak = object.getValue()"

There is no contextual reason for it to be called kak, which means shit in Afrikaans

Rolled out a new application I built almost entirely by myself 2 days ago... But my dev group is understaffed and has a project manager who is literally the most clueless person I have ever met, so as a result, we don't have a functional/useful dev/test/prod framework and no standards for how to deploy apps. So my past 2 days were comprised of fixing bugs in the live system that could probably have been caught if I had the time and resources to get everything thoroughly tested. It's stable now, but damn our management for being generally idiots. Our motto appears to be "Fuck it, we'll do it live"

Since I have seen a lot of people uploading this kind of stuff lately, here is Xiaomi's test in production, back in 2017 November...

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

New normal. New app to build.

- Still have to maintain older systems in parallel
- That leaves 1 week for developing the new app
- Slept 2 hours a day, coding coding coding
- Tested the shit out of the app because.. hey, its to help the customers' safety and health... I don't mind staying up late
- Finished the app in 5 days, code is now on prod
- Could barely look myself at the mirror because I look like shit
- Btw the app requires an external device as an input, the existing device works flawlessly based on my testing
- We need more devices
- Clueless manager bought new model instead. He assumed everything is fine, no testing is required
- Tested the app with new device model, doesn't work
- Deadline closing in
- Thanks, there goes my sleep
- THANK YOU

Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.

The non tl;dr:

For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......

Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments

I'm currently between jobs and have a few rants about my previous job (naturally). In retrospect, it's somewhat therapeutic to range about the sheer brainfuckery that has taken place. Enjoy!

First, let me set the scene: legacy B2B web app made with LEMP stack and sencha ext.js 3 + 4 (don't ask) and a lot of madness. Let's call that app "Alpha".

Alpha is a self made CMS build for typical ERP stuff. Yes, a self made CMS: entities are containers, containers have types and fields and values. Like so many legacy PHP apps, it does not have a dedicated FE: the HTML is rendered on the server and then spewed out to the browser.

Easy right? Coding like it's 1999! But there was a twist: Because everything is basically a container, the HTML-templates are saved in the DB. Along with the nessary JS and the CSS. And the translation variables. Why? Because fuck you! That's why. Who needs a git history anyways.

For some reason, Alpha was kinda slow.

There was also an editor, that allowed you to modify templates (web, mail, pdf) on the fly in prod. Because templates contain repeating data (header/footer), one template could contain additional templates. Much confusion. You could change templates via migration (slow, boring) or just ctrl-c/ctrl-v that sucker (fast, much excitement).

Did I mention Alpha was slow?

On with the rant: e-mails! How do they work? Noone knows. How to send mails asynchronous in PHP? Witchcraft is the only possible answer to that riddle. Here is your enterprise™ solution:
1. create mail
2. insert mail into DB
3. WAIT UP TO 59 SECONDS FOR A FUCKING CRON TO SEND MAIL

Why? "Because that way, we can resend mails in case the network is down :)"

Same procedure for the SOAP-API (db-queue + cron). You read that right: all requests to various other systems are processed once a minute.

Alpha slow.

Alpha was only one of several systems. Imagine a bunch of monolithic php apps, interconnected via SOAP, REST and GraphQL like a godamn intergalactic orgy. Image having to debug that cluster fuck.

Let's say there is a bad request. These things happen. No biggie. Remember the db-queue? Let's try to send the bad request a second time! And a third time! Still no luck? How odd. Let's create a specific file in a specific directory: a LOCK-file. Now, "the db-queue is on hold and no request gets processed :)"

Golly gee thanks Alpha.

Anyhow, did you know that MySQL has a join limit of 61 tables?

I deployed to prod on a friday right before finishing work.
No risk no fun.

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

Started a new job as junior developer. One of my first task was to sent a simple notification on an event in out product. Write the code, test that it works, push to devops.

Code compiles, tests pass, it’s deployed to internal test env. Check that my notification works in the test env. No problem.

It’s deployed to the customers test environment. It works and customer accepts it for prod.

We release to prod and of course it fails. Seems to be a simple string.Format that fails for god knows why. After 3h of debugging on prod without success we decide to roll it back.

Today we decided to try it on a backup of the prod db since one of the strings was taken from the db. Still working. No matter what data I input when trying it locally it still wont reproduce the issue we saw on prod.

Fuck this

I've now worked on both monolithic solutions and microapps/microservices. I gotta say I'm not sold on the new approach. There's so much overhead! You don't have to know your way around one solution -- no, now you need to know your way around 100 solutions. Debugging? Yeah, good luck with that. You don't have to provision one environment for dev, test, staging, and prod. No, now you need 100 environments per... environment. Now, you need a dedicated fulltime devops person. Now devs can check in breaking changes because their code compiles fine in that one tiny microapp. The extra costs go on and on and on. I get the theoretical benefits but holy crap you pay for it dearly. Going back to monolithic is so satisfying. You just address the bug or new feature head on without the ceremony and complexity. You know you're not crapping on other people's day (compilation-wise) because the entire solution compiles.

...and yeah, I'm getting old. So get off the lawn! ;)

Got a PROD issues caused by missing data from upstream. Told them about it last week. 3 days of no replies.

Finally got a reply that they will look into it. It's been almost a week since I first told them but no responses and no solution yet.

I couldn't fall asleep because I guess I drank too much coffee but also cuz the thought of this pissed me off.... So got up and sent a work email basically threatening to escalate this all the way up, put their asses in hot water.

Though not sure if I can but.... Maybe I can take a disability leave for needing to mentally recover from going mad at other people that can't do their jobs but aren't fired...

How come something works absolutely perfectly in dev but not in prod?
I was making a desktop app in election js and everything is working perfectly. No problem at all. But then I create the installer/distributable and nothing shows on the screen. And out of curiosity, I wanted to see the error log and it shows an unknown error, I didn't even know from what thing the error is being generated. And after I fixed that, another problem came with Asana Api. I mean, if it's a public API, why do you have to block it with cors? I hate cors!
And after all of it, there's more to it. I mean, why can't you just show the errors in dev?

Holy. Shit. Tests. I am testing. All week. Nothing but tests. I am one with the spec. You know what I realised today? Tests are a bit like life. Life is just one big spec suite that takes 75 years to run. Except there's no prod environment at the end of the DMT tunnel waiting for our green pass docket to say 'take me Lucifer, you absolute visionary: I'm ready'. We're all just a spec with no application. We're doomed. Nothing matters. I need to lie down

Microsoft :

"YOLO All Devops pipeline are now .net 8"

Me, lunching PROD release : "Whatever, I don't use any advanced shit"

Pipeline : "Can't connect to your nuget soure (private) : 401"

Me : What ? You just connected to it in prevous step in pipeline !

Azure : Yah, but prev st ep was using different syntax so it worked

Me : Ok, here you go, same syntax

Azure : Still nop.
Me : WHY ?!!!
Azure : I won't tell you. 401 !!

Me : Fuck you, I 'll rewrite this step from scrath !
30 mins later

Azure : Naaaan, stil 401.
Me : HOW ?!!! You literally use this nuget feed in 2 prev steps with no issues.
Azure : Fuck you
Me : fuck you too.
15 mins later
me to me... Hmm, I've already restored all packages. They "should" stay where they are..
Me : Take that azure : Ignore restoring for this step"
Azure : Oh yeah, that works ! All compiled !

WTF ?!!!! And why they release cha ges without even sending a mail. because fuck us apparently.

God I hate devops....

In a previous job, I was trying to organize a common repository with our shitty business partner so we could both be able to contribute our part so our work would not overlap. Not like they cared anyways.

One thing I quickly noticed is those fuckers would just straight up commit untested changes on master and cripples our whole testing and prod deployment at times because we were depending on a shitty IoT service they provided us onto which we had no control whatsoever.

I told my boss, who was often complaining about them being unreliable in the first place, I would simply restrict them from merging and commiting to develop or to master without my approval. We cannot keep working like this.

He told me that we could not impose on them our work practices and that I should not try to piss them off. To be diplomatic.

I politely and professionally refused to do it, but he did change his mind in the end. He and I left not too long after. I guess he felt obliged to respond that having his job at stake but you cannot condone voluntarily shitty work.

I just joined a new company.
Their CI pipeline is to give root access to staging and prod servers to every developer in the company and the manually git pull each repository (8-10 repos per server) and manually set nginx and port configurations. And if this wasn't enough, all of the 30 sites they have are basically the same site and they make the changes manually for each tenant (no env file). I'm amazed at how hard some people are willing to work.

So I made an update to my React Native app. I changed UI of a couple of screen, added a few animations here and there, refactored how my graphQL resolvers work in the backend(no breaking changes), changed how data gets loaded into the database etc.

It worked in dev so I figured hey let's deploy it. Today is(was because it's now 3am but more on that later) a national holiday so no one goes to work so no one will use my app so I have an entire day to deploy.

I started at 15:00(because i woke up at 13:00 lol). I tested the update once again in dev and proceeded to deploy it to prod. I merged backend to master, built docker images, did migrations on the db, restarted docker-compose with new images. And now for the app. I run ./gradlew assembleRelease and it starts complaining that react-native-gesture-handler is not installed. Ugh, rm -rf node_modules && yarn install. It worked. But now gradlew crashes and logs don't tell me anything. Google tells me to change a bunch of gradle settings but none of them work. Fast forward 5h, it's around 20:00 and I isolated the issue to, again, react-native-gesture-handler. They updated from 2.2.4 to 2.3.0 which didn't fucking compile. 2 more hours passed (now 22:00) and I got v2.3.1 working which fixed the problem in 2.3.0 but made my app crash on startup. YOUR FUCKING LIBRARY GETS 250K WEEKLY DOWNLOADS AND YOU DONT EVEN BOTHER CHECKING IF IT COMPILES IN PROD ON ANDROID?! WHAT THE FUCK software-mansion?

After I solved that, my app didn't crash. Now it threw an error "Type errors: Network Request Failed" every time I fetch my legacy REST API(older parts use rest and newer use graphql. I'll refactor that in the next update). I'll spare you the debugging hell i went through but another 5h passed. Its 3am. My config had misspelled url to prod but good for dev... I hate myself and even more so react-native-gesture-handler.

Just launched a successful deployment today. Took 4 months to build everything. It's probably not a big deal but I'm just really happy today that everything worked right away, and almost no deployment hiccups at all.

Only one issue popped up, but come to find out it's a particular thing about the Prod environment and nothing to do with my code.

Gonna go celebrate now, before more work comes in. (hey, can you refactor this for me?)

Until today, I had assumed deploying stuff to prod would NOT be one of my responsabilities in this company. Apparently that's not the case.

Had to deploy my code and pray it didn't break anything. Why is this a big deal at all?

Well you see, there is no repository. At all. No git, no svn, not even duplicate folders. No tests, no pipeline. Just a bunch of CPanels.

Had to manually copy files and folders from the development site to the production site and partially copy a database. "Just drag and drop" were the instructions I was given.

As if using CakePHP2, PHP5 and having to parse fucking Excel files wasn't bad enough, now I have to deal with one of the worst ways to deploy code.

Fuck it, I'm switching on the looking-for-job flag on linkedin.

Update: for those of you who know cougar woman/ my self proclaimed "work mom" in my previous posts (bitch who keeps stealing my lunch), yesterday she really tipped me over the edge. So I'm fucking hauling ass on my sprint work because I had to take over another team member's tasks (because he "doesn't have time" for it being prod support but all I see him doing is shitposting at his desk) and someone from another team asked the cougar a question about something. she comes up to me all demanding like "HEY you, you got the worksapce open?"
I was like um... no I don't. I'm working on the other application.
and then she's like "Look at my laptop and show him". Okay bitch let me just drop everything I'm doing to help a random person. The fact that she commanded me instead of just asking me pissed me off. Not even a single sign of "please". I'm tired of her truly. She is a snake. Even to her friend on our team. Every time he's out she's like "hey where is _____? huh why isn't he here??? hmmm" in an instigating way to remind everyone that he might've taken off for no reason. When I was too sick to volunteer at an event she organized in the morning on Saturday, she asked me the following Monday, "oh did you drink too much? lmao. a spiteful, grudge holding bitch for sure

I some time's feel stupid when my Ubuntu start acting up on prod, purge this and that ,fuck apache can't start system CTL that ps ax grep in the p#$#@* , damn apt install nginx full good, mysql can't start too ,OK apt install mariadb-client else percona && apt postgresql thanks ,god no client noticed ..

Story time;

Major project, multi million budget, huge business and IT coordination, board level status updates, meeting started back in March 2018 for a Go Live of Aug 2019.

Based on draft requirements (and experience) I request the test environment be built for half of the work. Turns out that no one told Server Eng and they are out of space in both dev and prod until Q2 of 2019. We went from Green to Red because a Service Request.

Two (2) senior developers and one (1) senior tester left our team and I am left with two (2) Java legacy applications that are hard to maintain. Here is a list of things I hate about these old webapps (let's call them app A and B):

1. App A depends on 80% web services. If one web service for a product or warehouse goes down, work flow is impeded while prod support team checks with the core services team for repair

2. App B is a maven project with multiple modules dependent on libraries that are dependent on company's internal libraries. So if we want to upgrade to OpenJdk 9 and up, the project will definitely produce a lot of errors due to deprecated/unsupported codes

3. App A is dependent on Tibco and I have no experience on that

4. App B's continuous integration build tool is Jenkins and the jobs that build it has a shell script that wasn't updated during the tech upgrade enhancement. The previous developer who did the knowledge transfer to me didn't tell me about this (it should be considered a defect on her part but she already resigned)

5. App A when loaded in eclipse IDE is a pain to work with since it is only allowed to build a war file using ant. I have to lookup in quick search instead of calling shortcuts (call hierarchy) because the project wasn't compiled via eclipse.

6. It's impossible to debug app A because of #5

7. Both applications have high priority and complex enhancements and I have no other teammates to help me

8. You never know what else can go wrong anytime

Lessions I learned so far from my first big node/npm project with tons of users:
1) If you didn't build something for a while, expect 3 hours of resolving version conflicts for every two weeks since the last build.
2) Even if the tests pass, run the containers on your own machine and make sure that the app doesn't randomly crash before deploying
3) Even if the app seemed to work on your own machine, run the tests again in an environment mimicking prod at most 15 minutes before replacing the running containers.
4) Even if all else indicates that the app will work, only ever deploy if you expect to be available within the 4 hours following a deployment.
5) Don't use shrinkwrap for anything other than locking every version down completely. A partial shrinkwrap will produce bugs that are dependent on the exact hour you built the app _and_ the shrinkwrap file, and therefore no one will ever have seen them other than you.
6) Avoid gyp, and generally try not to interface too much with anything that doesn't run on node. If parts of your solution use very different toolchains, your problems will be approximately proportional to the amount of code. And you'd be surprised just how much code you're running. (otherwise it's more logarithmic because the more code the less likely a new assumption is unique)
7) Do not update webpack or its plugins or anything they might call unless you absolutely need to
8) Containers are cool but the alpine ones are pretty much useless if you have even just one gyp module.
9) There's always another cache. To save yourself a lot of pain, include the build time in every file or its name that the browser can download, and compare these to a fresh build while debugging to assert that the bug is still present in the code you're reading
+1) Although it may look like it, SQLite is far from a simple solution because the code and the bindings aren't maintained. In fact, it'll probably be more time consuming than using a proper database.

Today I think it became apparent why we have so many data and prod issues...

No one understands SOLID design principals... Particularly Separation of Concern. No wonder all the Spring (DI) modules look like a mess of God functions that are impossible to reuse and difficult to extend...

Let me check Slack
Just before I go to bed
Just in case — OH NO

It’s not what you think
It isn’t like I broke prod
… request makes me cringe.

This condition over here which is never supposed to be true sometimes becomes true, no one know how or why, just change it manually in prod whenever that happens...

Want to hear another joke?

Blue Prism allows you to export stuff from version 6.7 to 6.3.

However they changed 𝘷𝘦𝘦𝘦𝘦𝘳𝘺 slightly the way they store the position of the nodes. No new features -or at least nothing that you would care about- but the structure of the node itself want went from
```
<positionx>1</positionx>
<positiony>2</positiony>
<width>3</width>
<height>4</height>
```
To
```
<position x=1 y=2 w=3 h=4></position>
```

The whole project collapsed to a single point, catastrophic consequences as far as exception handling. A generic "fuck you" for no real reason other than the sheer malice of those beasts of burden who developed Blue Prism in the first place.

And I have two different versions of Blue Prism on dev and prod :)

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

I am forced to work with a client's notoriously slow SOAP api. Slow in this case is 1.5-2s per request.
The api is structured rather... creatively... at the same time. So we have to bombard it with thousands of requests to build our data base with historical SOAP data. Also the data sometimes is a couple of hours late, giving a flat line (all values at 0) until retroactively fixing the output for the same requests.
So to fill one dev data base with a year's worth of historical data (nice to have when testing a dashboard application) we hammer the api with ~20k requests (~1 million if we want to be thorough).
Best thing about that: There is no staging/test api and the prod api seems not to handle lots of requests at the same time very well...
Latest thought: Maybe we could put a varnish cache in front of the SOAP for testing. Better have wrong data, than nothing at all and we don't kill the prod clients every time we ramp up a new instance.
Also that would dramatically decrease the 4.2 hours of data pumping to about 7 minutes after the first run.

Small chaotic startup that never grew up (15 years atm).
Hosts/maintains a number of apps/sites for various customers.

At some point, someone decides that a CMS would be usefull to maintain the content across all products. Forgoing all sense, reason and the very notion of "additional maintenance and dev" it is decided that one should be built in-house.

Fast forward a number of years.

Ops performs routine maintenance on prod-servers. A java-patch accidently knocks out one of the pillars a 3rd party lib the CMS uses for storing images. CMS basically burst in to flames causing a.... significant incident.

Enter yours truly to fix the mess.
Spend a few days replacing the affected 3rd party lib. Run tests on CMS in test and staging environments. Apply java-patch. All seems fine.

When speaking to frontenders and app-devs, a significant hurdle present itself:
All test/staging instances of all websites/apps/etc ALL USE PRODUCTION CMS. Hardcoded. No way around.

There is -no- way to properly test and verify the functionality of any changes made to the home-brewed CMS.
My patch did indeed work in the end.

But did the company learn anything? Did they listen to my reasoning, pleading or even anguished screams for sanity?

No.

Due to covid, mgrs decided to fire 10% but could not negotiate schedule increase with internal IT. With no promotions or hikes, few full stacks we have leave.
Now am working with 2 data engg doing cloud java microsvs work while learning. Their first delivery was applauded by their mgr who is under pressure to retain them.
I as arch review their code. No unit tests, print statements all around, shoddy exception handling, variable naming issues. We have Sonar by default in our build. They ignore the report. I ask them about it. Seems mgr told them he is getting a contract person from another team on part time basis to do/fix. I share my confusion.

Mgr calls me up and checks if we can put it as tech debt backlog and deploy to prod !!!

Not in prod today, but was part of a group project that we handed in and which got us an A.

The project was to write a PID controller for a robot that would drive along a track using a sensor to follow markings on the floor. During development we were drawing graphs of the PID parameters and sensor input every tick, which caused a bit of lag but no worries - we'll turn it off for the trial runs.

Imagine our pikachu shock meme when we turned off the graphs and our calibrations were suddenly *way* off since we had been oversteering all along to compensate for the lag.

There wasn't enough time to optimize it before the deadline and using sleeps didn't produce the same "type" of lag, so we just made the graph minimize itself when it opened. To this day I wonder if the professor ever saw it or if we got the A despite it.

Don't refractor for fun!

An anecdote from my previous company. A developer had written a shitty java console app for fetching stock prices. About 3000 LOC. just one java GOD class. So, when me and my friend looked at it, we were amazed how that code works with all that if conditions spanning 100LOC. so. My dear friend underestimated the complexity. Since it just fetches stock price and puts in database right. I can write it in few days and much better one. So, he started writing code in an OO way. Three days later I see he still working on it. Having a glimpse at code. The app is now Object oriented shitty and ugly.

Guess what new code never goes in prod too.

Learning
Don't underestimate complexity of app.
Be empathic about fellow developer. Don't think he has written a shitty code. Think why he had to do so.
Don't work on refractors if there is no one to guide you.

yay!!! Another PROD issue on something that we've had PROD issue on and well should be an easy fix... Cuz the root cause is prolly bc the idiot that implemented the logger had no idea what the fuck they were doing... And well the whole team didn't bother to ask me (and never listen to me anyway).... WHO WOULD PROBABLY SHOW THEM HOW THEY FCKED UP...

BUT ....

¯\_(ツ)_/¯

Yesterday whole 12 hours we were working on deployment about a feature X that has deadline yesterday itself.
Everything damn perfectly running on Test env but not on Prod.
We made Prod into Dev/Test/Fucking garabage env. Haha.
I was laughing to myself at same time crying hard in my deep heart.

Business guys chasing PM
PM chasing us
And from morning till night we were in same room. Had lunch, and dinner only went out for toilet and to refil water bottles.

And found that feature Y is not working at same time that is related to our feature X. Fucking we have been wasted hours on it.

One of my devs got so fucked up emotionally that he messed up the code (not his fault) he didnt had his lunch and dinner. Had to console him later that its not his fault. Poor guy not sure whether he slept or not; will find out in few hours.

Anyways reported a bug.
But that bug assigned to us for fixing.
Are you fucking kidding me.
Anyways no choice. Had to do it.
Hope today everything goes good or horribly bad. FYI no deployments on Friday damn we are in stalememt till Monday.

Fuck that bug
Or
May be fuck our stupiditiy while makiing mistakes.

Question to the frontend devs: I'm currently trying to optimize part of our JS. How do sourcemaps influence loading/rendering Speed in the Browser?

In the `webpack.config.js` we use uglify as minimizer for js. It has the option `sourceMap` Set to `true`. Now as I understand it, this caused it to Inline a sourcemap inside the JS file (which leads to a slightly bigger file size and thus longer load time for the generated file...).

My Question: should sorcemaps be enabled for prod (minified) files? Why yes, why no? Do they help the browser parse the JS faster?

Other dev in group chat: we need to stop syncing so much data from prod to uat because it's crashing the uat db...

Me thinking: no really... U just realized it's not a good idea to dump most of the prod db into uat every week?

¯\_(ツ)_/¯

This project is just a complete clusterfuck... But nvm. We had to integrate a third party service pushing data into our system. Btw the service wasnt even working correctly. But that is just the tip of the iceberg. Its friday around lunch time. Message appears "what is the status of the integration?" Yeah havent started working on it. Last info was service is not stable. I doubt that this will be done this week. Next message from PO: "We will all push hard to get this done today and deploy to prod." Why? Because this dumbasses said to the customer this will be deployed eod. And by we you mean the devs once again doing overtime. Has this shit stopped? No. Like for the last two weeks its like we promised the customer xyz to be deployed tomorrow. Not a single dev was asked how long it takes to add this

When your IT VP starts speaking blasphemy:

"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.

We need to do everything possible to minimize the load on our prod database server.

Here are some guidelines we’re implementing immediately:

· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.

Please keep the communication level high and stay vigilant in protecting our prod environment!"

RIP most of what I do at work.

Dumb mistake from when I was still working:

My work laptop’s SSD went haywire, and I/O would spike every 10 minutes or so for ~50 ms. The hardware guy said he could replace the SSD right away, or I could endure it for a few weeks and get a new laptop instead. Obviously, I agreed to wait. The stutter noticeably affected screen rendering, but I didn’t notice any other issues. Little did I know that every time it happened, all input was ignored (as in: not queued). Normally it wouldn’t matter, because hitting a random ~50 ms window is hard. How-the-f×ck-ever…

A few days later — without getting into “why” — I was forced to apply a patch in production. So I opened an SSH session to prod in one terminal, spun up a dev environment in another, copied the database schema from prod to dev, and made sure to test everything. No issues, so I jumped to prod, applied the patch, restarted services, jumped back to dev, and cleaned up the now-unnecessary database. Only to discover that my “jumped back to dev” keystroke didn’t register.

Losing faith in Netflix and their awesome open source projects.

Had a hard time trying to install Security Monkey : poor quality quickstart Ubuntu-only, almost no documentation, same instructions for latest (aka dev) and stable (aka prod) version, no depencies list ... oh and the UI display well only on Chrome ..

Then you surrender and just want to check the dockerized version they provide : it doesn't work neither (build fail or back end process just shut down) !!

I'm done ...

Weekly Out-Of-Context Thread

I'm looking for comments on rants. Not just any comments people, but comments that are non-sequiters, comments that are hilarious, or make no sense when taken from their original rant.

Anyone participating should link back to the original post to give credit. It would be great to get some regulars in here, but ideally we can prod some people into action that aren't on as much.

POST for our AMUSEMENT SLAVES! POST FASTER I SAY! FASTER!

Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?

# The detailed version #

We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.

But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are

So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.

I guess the saying goes: the only people that know how it works is whoever wrote it and God.

I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.

I think in Agile there's supposed to be artifacts during development but never seen em.

Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.

I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.

And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...

The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.

Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.

Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.

Had a client whom was using the staging system on my server as cdn, remote computing, etc... because his prod server was a cheap vhost while the vm was a beast compared to it. I shut it down without telling. I just got a call that his site is now slow a f and full of errors.

I kindly told him that there was a recent security breach called dirty cow. Then I told him that I shut the vm down because it would mean security risk for him since there are no patches available yet and only Power on again with there was work for me to do.

If you want resources pay for them

If a team uses multiple languages and stacks (Have, JS, Python) do you think it's better to have everyone use/constantly switch between them or have dedicated developers for each language (ie. 80% main, 20% others)?

--END QUESTION, ANSWER NOW BEFOREHAND CONTINUING---
---BEGIN RANT---

My boss likes keeping the team "will rounded" so everyone does everything. One month in working in Java, the next with Node web apps. When I switch to node, it takes like a week of "wtf doesn't it work.... what changed, is it a big?" And usually end it"oh right I remember I need to ..."

And also always... "How the fuck do I write tests in {some reading framework} again?"

So feels like everyone is just a generalist and no one is a master/has time to develop mastery. I don't know if it's just me (1/3 Senior developers on the team that has to do everything) or if I'm the only one that complains... Not that it makes a difference... (Only option to really be heard is to resign but I need to somewhere else to work and finding one is hard for personal reasons)

And well this is the biggest reason I would leave the team. No time for mastery, no standardization/shared knowledge (everyone does their own thing but probably not well and no time for testing or documentation; how the fuck does whatever you wrote work, how do we use it, what the fuck did you put in prod that does ... And where the fuck did you put it cuz it's not in ANY of our repos).

I always feel one day soon it will come crashing down and I can say "I told you so" but will then it's too late and I'll be there one cleaning it up... Again

Confession: a very important feature of the website I'm developping wasn't working for a certain time. The boss wasn't aware because he doesn't go on the site, and I only found out last week because I needed to implement a new feature that used the previous one. Problem: the bug was only on production, not on local (and of course we don't have test server).
I took advantage of the absence of my boss today to clear the situation by making all of my tests on prod. I hope no customer tried to pass a command today, but it's finally repaired. I am both proud and shameful.

Ops teams that are "assuring" stability of production by being the only ones who can click to deploy to Prod add no value.

This is a continuation of my previous rant about admob being not very informative when it comes to invalid traffic and the resulting restriction in ad delivery.

I then wanted to use admob mediation to hang in facebook ads. My app is written with Xamarin.Forms.

So first I needed to make some facebook configuration - create an account, let my app review, create some ad placements and other shit. I came to the point where I had to put in a link to my privacy policy and the link could not be accepted due to some SSL fuckup -.-'

I then found out that there is an issue with my SSL Chain. With the help of whatsmychaincert.com I solved that issue. Little side note here: I have limited knowledge of that stuff and my cousin helped me set up my homepage so I had no idea what I was doing. Did a snapshot and luckily I did not needed that as everything worked :)
This took me around half an hour just so I can paste the fucking link to activate my app in facebook developer portal.

After that I made the whole mediation configuration shit - not an issue as google documented this quite well but it took some time.

Now comes the shitty part. To use admob mediation you need adapters to the other ad network. I found a nuget package with exactly what I needed just to find out that it is outdated. So I pulled the repo and saw that this thing is an aar binding library. Never did that stuff so I read some docs again. Updated the package and consumed it in my app.

The google docs then said "Use this mediation test shit to check if you did everything correct before going prod" - aar binding nr. 2 (but I am now familiar with that :P). This thing then told me that facebook ads could not be loaded because the SDK version is outdated -.-' SDK version comes from another nuget package which is referenced by the first aar thingie. I tracked that thing back to a repo where I found out that they are indeed totally behind. So I downloaded the aar, made a binding lib and bound that to my first aar binding lib as that depends on this.

Put that all back in my app - tested mediation and fucking finally after 6 hours everything comes together! all lights are green and things work.

Sorry if this is not quite a rant but it was quite a journey and I just had to share it.

This place I’m consulting at just had a new Directory of Change, first policy she made for IT is mandatory 3 working days wait for any release in Test and 2 weeks for Prod. That includes application config value update in Test database. We think she might have misunderstood her job title Ms. Director of No Change..

I’m finishing up a thing for someone. It’ll be in prod soon. I’m nervous. I keep picking at it but it’s “done.” It’s just a silly script. But I haven’t written anything in a long time that someone else is gonna actually use. I write things no one else uses to make my life easier.

Should I be even be testing if no one cares.

I keep asking Devs regarding the functionalities of their system for testing and then, realize it's already on Prod even before I test in beta or enable to test in beta.

Do I exist!!!

Now, for the nth time, I have started to test when things are almost there or already there in Prod.

I should keep myself in adrenaline mode always on from now on.

Need to do something worthy.

The worst way to start a new job.

This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...

Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place

Remember how I was - against all that was promised - assigned to a time-sensitive front-end (so definitely not my forté) project about a month ago? Remember how I struggled with the choices of how to go about it - switching from F# (Fable) to Rust (Yew) to eventually settling in with Vue and TS?

Yeah, I’m glad I went that way, even though there could’ve probably been better choices out there: my part is done now, even though it’s not quite prod ready yet (close tho), the team who’ll maintain it takes it over now, after I finish dealing with my current minor issue. And damn their front-end guy is GOOD. Makes me feel very inferior in that department. Well, I am. Back to back-end, thank you very much...

But I have an issue here, that bothers me. I’ve produced a codebase that’s obviously written on a tight schedule: no tests, no documentation, a few embarrassing hacks/workarounds and so forth. I actually feel bad for leaving it out of my hands to them in such a state...

Had a definite week from hell... a bunch of prod issues that only I could fix (that's a whole other rant for another day!)... a piece of code totally kicking my ass for days... a hosting environment that was unstable seemingly every time I needed to do something in it (and that killer piece of code could ONLY be properly tested there, naturally!)... a service that my app depends on flaking out with no indication what the problem was and another team responsible for it that is based off-shore so aren't responsive when I need them to be... a metric shit-ton of procedural bullshit dropped on my head... an immense amount of stress due to the lead-up to a prod rollout next month that absolutely CANNOT fail without huge ramifications for the business but not enough help to ensure it gets done.

But, with all that said, I DID manage to get that killer piece of code working late on Friday after slamming my head against the wall for over a week on it (and ultimately re-writing it from the ground-up on Thursday and Friday)... so, the week of hell ended on a high note at least, which is always a Very Good Thing(tm)!

So I'm assigned once again to fix a new someone else created and that seems to be the case whenever there's an issue...

Boss just assigns it to whoever is most likely to be able to investigate it... which is basically me. Other than the little time I can use to develop stuff, I'm usually cleaning up other people's messes.

And these other people are to busy working on new crap to properly explain how their existing code/processes/changes works.

And well the fact that anything breaks in production (that's not due to upstream one off issues) whoever does not think he needs to take responsibility for it.

So everyone else and especially me has to spend time understanding the shit they wrote and fixing it for them.

How do I tell my boss this nicely that we need clearly definitely ownership and whenever a component blows up in prod, the guy that wrote the code fixes it no matter what? Thereby incentivizing him to not write shit code in the first place and be more proactive in making sure it doesn't in the first place since he knows otherwise he's doing overtime to fix it?

Is it just me or is there really no such thing as a dev job where something doesn't blow up due to poorly tested and designed code every other day?

Pushed some changes to PROD today. Go to login and check changes .. noooooope!

Still a bit new to Symfony 5... but I'm just not a fan right now. The login screen just jumps back to itself. No login failed message and prod log had a size of 0 so that was no help.

Traced this thing way down into the CSRF Authentication functions. \is_callable(...namespace) just returning null so no go on getting a token for isTokenValid() =/
ugh! This is truly the most torturous junk I've ever seen. Nothing in the logs so I decided to just use the good old ECHO'HERE' debugger.

What was the issue you might ask?... effin' yaml file
Fix for now is to set the session handler_id back to null

Sigh, what is it with these cowboy SQL Devs? Why the fuck is this a pattern for anything?

New contract, new idiots, sigh.

EDIT: Had to change picture because Prod is different to Dev (but no dev has been done since release....smh)

This got me fucked up. Listen yo.

So we have this issue on our modal right. The issue keeps poppin. It's a hotfix because its in prod. So my senior and I were on it. After a few hours, I showed him the part of the code that is buggy. It's 50 lines of code of nested if-else, else-if. And so we're still fighting it. He redid everything since we're using angular2 he did a subject, behavior-subject all that bs and I was still trying to understand what's the bug, because it's happening on the second click and so I did my own thing and found the cause bug and showed it to him, its this:

setTimeout( () => {}, 0)

the bootstrap-modal doesn't allow async inside it (I dont why, its in the package). So he explained to me why it's there. So I did my own thing again and find a workaround which I did, a one-line of angular property, showed it to him he didn't accept it because we'll still have to redo it with subjects and he was on it. I said ok. Went back to my previous issue. The director came in and ask for a fixed, my senior came up to me and told me to push my fix. Alright no problem. So we good now. Went back to our thing bla bla bla, then got an email that we will have a meeting, So we went, bla bla bla. The internal team wants a support for mobile, senior said no problem bla bla bla, after the meeting he approaches me and said (THIS IS WHERE IT GOT FUCKED UP) we wont be supporting bootstrap4 anymore because of the modal issue and since we're going to support mobile and BOOTSTRAP4 grid system is NONINTUITIVE we are moving to material design because the grid system is easier. I was blown away man. we have more than 100 components and just because of that modal and mobile support shit he decided to abandon bootstrap. Mater of fact its the modal its his code. I'm not expert in frontend but I looked at the material design implementation its the same thing other than the class names. OHHH LAWD!

I was hoping that I never have to build stupid websites again, but here I am...

The thing I hate the most, is the hassle to have an easy to update dev, staging and prod environment. Fuck wordpress, fuck drupal, fuck joomla.

git pull && composer install && npm install should be all that's needed to get the latest code in an environment.

composer require *** to install plugins. No stupid web interface where users install plugins in production env.

I don't want to create database dumps just because these fuckers think that you should store configurations in the database.

Is there any clean CMS primarily for professional programmers? Or are they all just made for retarded subhumans?

I've having issues trying to form a proper branching strategy for my mean stack app deployment.

Heroku creates staging and prod branches for my web app so I'm a bit confused if I need my own staging branch?

Currently I have this: feature -> dev -> staging -> heroku staging (the staging branch seems useless)

Also, Heroku allows you to promote heroku staging to heroku prod, so there's no point in making master push to heroku prod.

I'm thinking of making my strategy to the following, but wasn't sure of any pitfalls or anything I'm overlooking long term.

feature -> dev -> master -> heroku staging -> manual promote to heroku prod.

Any suggestions?

Wish me luck.

Deploying Blazor app for the first time in prod in about a month.

Did tests. Curent infrastructure can hold around 70 concurent sessions with no problems. (probably more, 70 was the limit in my browser).

I tested each sessiuon with 70.000 line table. (Yep whole 70.000 lines for each session with a virtual scroll).

Shit is fast. Too fast even. I'm waiting for other shoe to drop, but so far in simulated tests it's amazing.

Let's talk in 2 month AFTER prod deploy.