I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.

The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.

We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.

Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.

Needless to say, he was never allowed to touch code on a prod server ever again.

She: I like people who takes risks.
He : I develop in a prod server.
She: My place or you place? 😉

Before an interview prepare a list of questions for them, they expect it!
My list to give inspiration:

Describe your company culture? - if the response is buzzword heavy, avoid.

What’s the oldest technology still in use? - all companies have legacy systems but some are worse than others

Describe your agile process? - a few companies I’ve interviewed with said they are agile but it’s actually kanban

Are developers involved with customers?- if they trust you to talk to customers you can infer trust to do your job ( I’m sure others will disagree)

Describe your development environment?- do they have such a thing as dev, test and prod?

These are the only ones I can remember but should give others a bit of inspiration I hope 😄

Why is this CSS not updating???

Ctrl + F5

Still!!!

Let's check in incognito..

F*CK, really!!!!

.... Ooooooh right I'm checking prod not dev.

Welcome back to practiseSafeHex's new life as a manager.

Episode 2: Why automate when you can spend all day doing it by hand

This is a particularly special episode for me, as these problems are taking up so much of my time with non-sensical bullshit, that i'm delayed with everything else. Some badly require tooling or new products. Some are just unnecessary processes or annoyances that should not need to be handled by another human. So lets jump right in, in no particular order:

- Jira ... nuff said? not quite because somehow some blue moon, planets aligning, act of god style set of circumstances lined up to allow this team to somehow make Jira worse. On one hand we have a gigantic Jira project containing 7 separate sub teams, a million different labels / epics and 4.2 million possible assignees, all making sure the loading page takes as long as possible to open. But the new country we've added support for in the app gets a separate project. So we have product, backend, mobile, design, management etc on one, and mobile-country2 on another. This delightfully means a lot of duplication and copy pasting from one to the other, for literally no reason what so ever.

- Everything on Jira is found through a label. Every time something happens, a new one is created. So I need to check for "iOS", "Android", "iOS-country2", "Android-country2", "mobile-<feature>", "mobile-<feature>-issues", "mobile-<feature>-prod-issues", "mobile-<feature>-existing-issues" and "<project>-July31" ... why July31? Because some fucking moron decided to do a round of testing, and tag all the issues with the current date (despite the fact Jira does that anyway), which somehow still gets used from time to time because nobody pays attention to what they are doing. This means creating and modifying filters on a daily basis ... after spending time trying to figure out what its not in the first one.

- One of my favourite morning rituals I like to call "Jira dumpster diving". This involves me removing all the filters and reading all the tickets. Why would I do such a thing? oh remember the 9000 labels I mentioned earlier? right well its very likely that they actually won't use any of them ... or the wrong ones ... or assign to the wrong person, so I have to go find them and fix them. If I don't, i'll get yelled at, because clearly it's my fault.

- Moving on from Jira. As some of you might have seen in your companies, if you use things like TestFlight, HockeyApp, AppCenter, BuddyBuild etc. that when you release a new app version for testing, each version comes with an automated change-log, listing ticket numbers addressed ...... yeah we don't do that. No we use this shitty service, which is effectively an FTP server and a webpage, that only allows you to host the new versions. Sending out those emails is all manual ... distribution groups?? ... whats that?

- Moving back to Jira. Can't even automate the changelog with a script, because I can't even make sense of the tickets, in order to translate that to a script.

- Moving on from Jira. Me and one of the remote testers play this great game I like to call "tag team ticketing". It's so much fun. Right heres how to play, you'll need a QA and a PM.

*QA creates a ticket, and puts nothing of any use inside it, and assigns to the PM.
*PM fires it back asking for clarification.
*QA adds in what he feels is clarification (hes wrong) and assigns it back to the PM.
*PM sends detailed instructions, with examples as to what is needed and assigns it back.
*QA adds 1 of the 3 things required and assigns it back.
*PM assigns it back saying the one thing added is from the wrong day, and reminds him about the other 2 items.
*QA adds some random piece of unrelated info to the ticket instead, forgetting about the 3 things and assigns it back.

and you just continue doing this for the whole dev / release cycle hahaha. Oh you guys have no idea how much fun it is, seriously give it a go, you'll thank me later ... or kill yourselves, each to their own.

- Moving back to Jira. I decided to take an action of creating a new project for my team (the mobile team) and set it up the way we want and just ignore everything going on around us. Use proper automation, and a kanban board. Maybe only give product a slack bot interface that won't allow them to create a ticket without what we need etc. Spent 25 minutes looking for the "create new project" button before finding the link which says I need to open a ticket with support and wait ... 5 ... fucking ... long ... painful ... unnecessary ... business days.

... Heres hoping my head continues to not have a bullet hole in it by then.

Id love to talk more, but those filters ain't gonna fix themselves. So we'll have to leave it here for today. Tune in again for another episode soon.

And remember to always practiseSafeHex

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

Certified enganeers!
Spaghetti masters!!
Those who dev in prod!!!
Push to master!!!
The "it's a feature" assholes!
The Madman Muntz misers!

The computer science and engineering fields are filled to the brim with fakers, phony's and Python Lords. I try to fight them every time but they outnumber me. Apparently money is more attractive than expressing your own passion.

First rant here. Long, but please bear with me:

So after slogging my ass off in various early stage startups for over 4 years and keeping up with the almost non-existent development process, I joined an organisation which has some of the brightest and smartest minds I have had the pleasure to work with.

Mind you, this company is the market leader in it's field and has a 50+ people in it's tech team and the quality of work is pretty impressive.

Now for this week's sprint, I was asked to develop a feature which already exists on the Android app and they want to introduce in the iOS app too. The backend APIs are all in place and all I need to do is build it with virtually no dependency. My PM asks me to start with the UI and ask the backend dev for the API list whenever I need them.This is where the story turns.

For my first API, I go to the backend dev and ask him to share the API documentation and he looks at me as if I have asked him to dance the fucking cha cha. With a straight face he tells me that, 'The organisation doesn't maintain any kind of documentation for it's APIs.' Now this really shocks me. Even in a 5 men tech teams I have worked on, we have always maintained a spec doc for the APIs and this is a company which is known for it's tech practices.

Being the new guy I compose myself and ask if they have anything for me here: Postman collection, a workflowy doc, a goddamn txt file; anything which might help me, and he laughs at my dilusion and says no.

Dejected, I ask for a way to get the APIs and I am told that there are only two ways: either I keep bothering the Android dev for the APIs(No, I don't have the access to the android repo and nor am I gonna get it) which he had worked on 4 months back or I install the prod app on my phone, and use Charles to get every fucking API which is really, really annoying.

I thought writing out this rant would make me feel better, turns out it just made me angrier. Why the fuck can't they document such an important thing!?

Manager: How come the push to prod didn’t happen?

Dev: We told you at the scrum yesterday. To reiterate, our dev environment was crashing so it’s not safe to push to prod until that is fixed.

Manager: Ok well lets set a goal to fix that and push to prod happens today so that it guaranteed happens.

Dev: That was our goal yesterday and it definitely didn’t happen.

Manager: I AM AWARE OF THAT. The corrective action is that this time compliance with the goal is 100% ABSOLUTELY MANDATORY!!

Dev: We’ll do our best, can’t guarantee anything until we figure out what the nature of what is occurring on dev though.

Manager: NO. I AM THE BOSS. YOU WILL 100% ABSOLUTELY COMPLY WITH THIS. THAT IS AN ORDER. YOU WILL SUCCESSFULLY GET THIS UPDATE OUT TO PROD TODAY. ANYTHING LESS THAN THAT SHALL BE CONSIDERED INSUBORDINATION. I WANT STATUS UPDATES EVERY 15 MINUTES ON WHERE WE ARE AT WITH THIS.

Dev: …

Dev: Can I get you to send me that request in an email?

*Manager leaves the meeting*

// *****************************

Job search is ticking along. It’s tough going though because I currently make ~120k and the best offers I’ve received so far are all ~70k because “You only have 2 years experience so you couldn’t possibly have the skills to be worth 120k. You are are junior level developer and 70K is already overpaying for you. We can pay you more later™. No we will not give you that in writing”. Ah well, the hunt continues.

Dev: * In the middle of pushing to prod *

Manager: Hey btw I forgot to mention the client asked for these 5 features awhile back as a part of this update and they just reminded me about them, I haven’t created tickets yet or told anyone about this. Will these features be included in the update that’s going out today?

Dev: …

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Be me, new dev on a team. Taking a look through source code to get up to speed.

Dev: **thinking to self** why is there no package lock.. let me bring this up to boss man

Dev: hey boss man, you’ve got no package lock, did we forget to commit it?

Manager: no I don’t like package locks.

Dev: ...why?

Manager: they fuck up computer. The project never ran with a package lock.

Dev: ..how will you make sure that every dev has the same packages while developing?

Manager: don’t worry, I’ve done this before, we haven’t had any issues.

**couple weeks goes by**

Dev: pushes code

Manager: hey your feature is not working on my machine

Dev: it’s working on mine, and the dev servers. Let’s take a look and see

**finds out he deletes his package lock every time he does npm install, so therefore he literally has the latest of like a 50 packages with no testing**

Dev: well you see you have some packages here that updates, and have broken some of the features.

Manager: >=|, fix it.

Dev: commit a working package lock so we’re all on the same.

Manager: just set the package version to whatever works.

Dev: okay

**more weeks go by**

Manager: why are we having so many issues between devs, why are things working on some computers and not others??? We can’t be having this it’s wasting time.

Dev: **takes a look at everyone’s packages** we all have different packages.

Manager: that’s it, no one can use Mac computers. You must use these windows computers, and you must install npm v6.0 and node v15.11. Everyone must have the same system and software install to guarantee we’re all on the same page

Dev: so can we also commit package lock so we’re all having the same packages as well?

Manager: No, package locks don’t work.

**few days go by**

Manager: GUYS WHY IS THE CODE DEPLOYING TO PRODUCTION NOT WORKING. IT WAS WORKING IN DEV

DEV: **looks at packages**, when the project was built on dev on 9/1 package x was on version 1.1, when it was approved and moved to prod on 9/3 package x was now on version 1.2 which was a change that broke our code.

Manager: CHANGE THE DEPLOYMENT SCRIPTS THEN. MAKE PROD RSYNC NODE_MODULES WITH DEV

Dev: okay

Manager: just trust me, I’ve been doing this for years

Who the fuck put this man in charge.

Lead dev: Hey boss, you really do like Python right?

Me: No

Lead dev: Well it's cuz I was think....wait what? WTF do you mean no, you have automated a fuckload of BS with Python and we are still using it, why tf would you use Python if you don't like it?

Me: I like it enough for the automation scripts that we have and for parsing documents or generating glue scripts, its already installed in every server that we have, so testing bs in dev and then using them in prod is cake, it doesn't mean I LOVE python, I like it for what we use it.

Lead dev: Well ain't already bash and perl installed as well?

Me: Do you know bash and or perl?

Lead dev: No, don't you?....

Me: No......

L Dev: (using a Jim Carrey impersonation) WELLL ALLRIGTHY THEN! What is the other language that you used for X project?

Me: Clojure, do you remember that one?

* he said paren paren paren paren yes paren i space paren do close paren close paren etc etc
L Dev: (((((((yes (i (do)))))))) and nevermind, I'll get back to working more with Python

Me: das what I fucking thought esse

Me passing time on the weekend

Random call from unknown number

Turns out it's the manager

M: hey , how is your weekend going ...

Me: nothing much ... Whatsup ?

M : yeah well , we wanted to push some minor adhoc fixes as some clients wanted it urgently

The Devops folks need developer support . Can you pitch in and monitor

Me : I'm not aware of what changes are going , i don't think i can provide support

M : don't worry it's minor changes , it's already tested in pre prod , you just need to be on call for 30 mins

Me : ugh okay .. guess 1 hr won't hurt

M: thanks 👍🏽

Me: *logs in

*Notices the last merged PR
+ 400 lines , implemented by junior dev and merged by manager

*Wait , how is this a *minor* release...

*Release got triggered already and the CI CD pipeline is in progress

*5 mins later

*Pipeline fails , devops sends email - test coverage below 50%

Manager immediately pitches in ...

M: hey , i see test coverage is down , can you increase it ?

Me: and how do u suppose I do that ?

M : well it's simple just write UTC for the missing lines ... Will it take time ?

Me : * ah shit here we go again

Yeah it will take time , there are around 400 lines , I am not aware of this component all together

Can you ask junior dev to pitch in and write the UTC for this

*Actually junior dev is out on a vacation with his girlfriend

M : well he's out for the weekend , but
as a senior dev , i expect you to have holistic understanding of the codebase and not give excuses ,
this is a priority fix which client are demanding we need this released ASAP

Me : * wait wat ?

---

I ended up being online for next 3 hours figuring out the code change and bumping up the UTC 🤦🏾

Jr. Dev: We got an issue in prod.
Sr. dev: Okay, what is the issue?
Jr. dev: Don't know, just got an email.
Sr. dev: Did you check the service?
Jr. dev: Which service?
Sr. dev: What have you debugged so far?
Jr. dev: I explained you the issue and waiting for your inputs.

😑😑😑😑😑

One of the biggest reality checks you will run into when starting your first dev related job - and which they don't teach you about in school - is that a lot of the time will be spent working with other people's code, and rewriting it into "your own" is rarely an option.

You might be super into making things, but not everyone manages to maintain that same spark while taking over a 15 year old project with fundamental issues that have to be triaged "for now" because you need a hotfix on this other specific thing out in prod before lunch.

There are no gods now. They left the company years ago and nobody knows why they used the windows registry as a user repo.

Joined a new company / team to work on an iOS app that has 2 different backend environments "Dev" and "Prod". Also being referred to in iOS speak as "Debug" and "Release".

Been trying to get accounts on these backends (no sign up in app, its controlled via another process). Eventually get access to "Dev" for one of the regions, so I load up "Debug" and its not working.

This is odd, so I open the Android app and load "Dev" and it works? I then Notice Android has "Dev", "QA", "Staging" and "Prod" for every region where as iOS only has 2 of these.

So I go back to iOS and find the file for the settings and it has iOS Debug assigned a variable for the backend Dev ... which is actually pointing to QA. Because they use QA to Debug and not Dev.

... confused? join the club

I really, really, fucking god damn it REALLY need to move a legacy project from the grave yard server and get it in git, and then build a dev environment for it, so I can stop making incredibly volatile changes direct to PROD (backend, frontend and DB all at once and then test it while it’s live and being used, but fuck me if I can be bothered digging through a 10GB code base and attempting to make it work in a multi-environment setup when it’s going to be a long trip down the error logs until it works again 😱🔫

Dev: I think I just deleted data I shouldn't have. Can you do a rollback?
Me: No, I'm a DBA, I don't touch data. Did you commit?
Dev: Yes, I committed.
Me: Your data is gone.
Dev: But don't you guys do backups?
Me: We backup the Prod servers. We only back up Dev on request. Did you request a backup in Dev?
Dev: No.
Me: ...
Dev: The Testers are going to be pissed.

When will I fuckin learn that
a) customers lie
b) customers are sloppy
c) customers are wrong
d) customers do not do their work (properly)
e) customers want us to do their (dirty) work
f) possibly all of the freakinly above?! + khm....

They will fuckin aaaalwaaaays say sth is not working after the update..
And I will alwaaaays assume I fucked up something..even if I didn't touch that part of the code/data..
And almost aaaaalways it turns out that the bug they complain about is how the system worked (or didn't work) before the update and/or some fuckup from their side..

Anyhow, I rushed over, grabbed the files went testing in dev..wtf, output is different, mine is ok, theirs is..wtf is that shit?!

Transfer newly built dll to test..same shit as on prod..wtf?! How?!

I assumed they have thing A correctly linked to thing B.. ofc thing A was linked to thing C in their case and in another case (our test) to correct thing B..
I got chillies when grabbing files, that
I should have tripple checked that they didn't fuck up something on the link part, but I just assumed they know what they were doing & that they checked they linked correct files with correct content already, before being pissy that the update fucked up things.. riiiight!! :/

I wanted to find solutions to this fuckup asap so I disregarded my gut feeling..yet again!! Fuuuck!
I've spent too much time trying to find ways to fix a bug that wasn't even a real bug to begin with.. :/

Fuuuuuck!!

So yeah, always treat the customers like they are 3yrs old & have no clue what they are doing & check exactly wtf they were indeed trying to do..it will save you time & nerves..

And note to self: reread this shit daily!! And imprint it in your brain that everything is not always your fault!!

Have a couple I want to air today.

First was at my first gig as a dev, 4-5 months out of school. I was the only dev at a startup where the owner was a computer illiterate psycopath with serious temper tantrums. We're talking slamming doors, shouting at you while you are on the phone with customers, the works...

Anyways, what happened was that we needed to do an update in our database to correct some data on a few order lines regarding a specific product. Guess who forgot the fucking where-clause... Did I mention this boss was a cheap ass, dollar stupid, penny wise asshole that refused to have anything but the cheapest hosting? No backups, no test/dev/staging environment, no local copies... Yeah, live devving in prod, fucking all customers with a missing semi-colon (or where clause).

Amazingly, his sheer incompetance saved my ass, because even if I explained it, he didn't get it, and just wanted it fixed as best we could.

The second time was at a different company where we were delivering managed network services for a few municipalities. I was working netops at that time, mostly Cisco branded stuff, from Voice-over-IP and wifi to switches and some routing.

One day I was rolling out a new wireless network, and had to add the VLAN to the core switch on the correct port. VLAN's, for those who don't know, are virtual networks you can use to run several separated networks on the same cable.

To add a VLAN on a Cisco switch one uses the command:

switchport access vlan add XYZ

My mistake was omitting the 'add', which Cisco switches happily accept without warning. That command however can be quite disruptive as it replaces all of the excisting VLAN's with the new one.

Not a big deal on a distribution switch supplying an office floor or something, but on a fucking core switch in the datacenter this meant 20K user had no internet, no access to the applications in the DS, no access to Active Directory etc. Oh and my remote access to that switch also went down the drain...

Luckily a colleague of mine was on site with a console cable and access to config backups. Shit was over within 15 minutes. My boss at that time was thankfully a pragmatic guy who just responded "Well, at least you won't make that mistake again" when we debriefed him after the dust settled.

4:55: Everything looks good in prod.
4:56: Deploy new feature after all is well in Dev.
4:57: Prod goes to shit!
4:58: Call wife to tell her I'm not coming home at 5.
4:59: Prod looks fine w/o anyone doing anything.
5:00: Leave work.
5:25: Get yelled at by wife for leaving at 5 after telling her I couldn't.

I once ran a batch import job to stress test with much more data than usual in staging... so I thought.
I sshd into prod by accident. History search can be a bastard at times...

The moment I realized what I was doing was crazy. I was shivering and thought I get fired if anything went wrong.

fortunately I just duplicated the original data for the test, and the system was built to ignore unnecessary updates... so the data was correct and nothing went wrong.

Not an active stupid dev choice but still something I will remember for a while.

I used to work on a production management team, whose job was, among other things, safeguarding access to production. Dev teams would send us requests all the time to, "run a quick SQL script."

Invariably, the SQL would include, "SELECT * FROM db_config."

We would push the tickets back, and the devs would call us, enraged. I learned pretty quickly that they didn't have any real interest in dev, test, or staging environments, and just wanted to do everything in prod, and see if it works.

But they would give up their protests pretty fast when I offered to let them speak to a manager when they were upset I wouldn't run their SQL.

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

Today in horror stories: setting up the local dev environment for a project I'm taking over creates changes and notifications in prod.

Oh and my local dev env is somehow receiving data from actual patients. I don't know how.

So... Heard back from a recruiter today. Lovely lass.

I’d passed over a submission for her tech demo.

The brief was basically just to create a small simple module that calculates shit, nae effort.

But, when the recruiter had me on the phone she said “I know it’s a silly small module but try and run it up like you would a production ready app”.

The job spec and recruiter were keen on me demonstrating TDD, not specific on js version, final runtime, etc. The job was a senior spec at a higher salary range. So it warranted some effort, and demonstrating more than a simple module.

“Okay, cool, nae bother, let’s crack on.”

The feedback in the response from the dev today:
“He’s over-engineered tests, build...”

SUCK MY LEFT TESTICLE YOU FUCKWIT.

Talk to your recruiters, not me.

The feedback included a phrase I never hope to hear from a developer I work with:
“Tests are good but...” 😞

It was a standard 98% test suite from an RGR cycle, no more or less than I’d expect in prod.

The rest of the feedback was misguided or plain wrong. It was useful to see because I know now when they say they have “high standards” they mean: we listen to the dude who put the factory pattern in a JS brief.

Oh shit also: “someone’s done chmod 777” was in there as a sarcastic comment in the feedback. It was his fucking unarchive tool 😞

My response was brief and polite: “cheers for the consideration, all the best, James”

It’s honestly not worth warning them. Or, asking why they’d criticise something they’d asked me to do.

If you want a shitty js module, ask for a shitty js module and no more.

On my project the customer has re-signed into a contract several times when they have budget to continue work. The first time they got us to build the system was a huge success story because the team was assembled quickly and we did rapid development. Initialize repo to prod in 1.5 months. The customer asked for the same dev team. Strong dev team, a PM that doesn't take shit, and pure agile. Lets call her don't-take-shit PM.

When the customer re-signed the executive decided that she didn't like don't-take-shit PM. So the project manager gets replaced by play-by-the-rules PM who will comply with stupid requests and micromanagement. He isn't a bad PM but he tries to make everyone happy. The amount of management types executive installs on the project is massive, and development team is cut down in major ways. Customer and executive shit rolls down to the development team and we can't get anything done. The customer starts to lose faith because we can't get traction. They start demanding traditional waterfall/SDLC docs. Which causes more delay in the project.

So the executive decides that the PM can take a fall for it to save face for the company. She moves play-by-the-rules PM to another project. He starts handover to a new PM that has a history of being her pushover. The customer hadn't seen him yet so now we have push-over PM.

Play-by-the-rules PM is finally out of the project and instead of moving to a different account the company decides to "lay him off because there is no work". So basically they made him take the fall for the failure while promising reassignment, and instead let him go. This is so unfair..

Meeting with push-over PM yesterday and he shows us his plan. Identical to play-by-the-rules PM's plan that got him axed.We point that out and show him the docs that were made for it. His face clearly communicates "OH SHIT WHAT DID I SIGN UP FOR?"

!dev && rant

Can we talk about banks? Those fuckers! Suposed to keep our money save and be competent... They today gave me the biggest scare of my live and I've run one an update query on a prod db without a where clause! (Okay I knew we had a backup but still pretty scarry moment!)

As a few know, besides being a dev I help to organize a small openair music festival here in Switzerland. The openair was this weekend. Every thing wen't well, until I checked our ebanking account today. There was only 2/3 of the money that should be there. A quick call to the bank and they told me, nope they never received it. As we've thrown it in a secure locker during the night, we didn't receive any receipt or something like that. It took those fuckers 3.5 hours to actually go and check the looker, just to find the remaining money in the corner of it. What the fuck people, can't you open your fucking eyes and not give me a fucking heartatack? I thought you guys are professionals!

Note locker: we get a key to open it from the outside, place our payment during the night, as soon as we close it, it falls inside a vault, so there it's a pay in only system, for lack of a better word, I called it locker.

My heart is still beating like mad, because of them.

that moment when you were still new to git commands and just typed in random commands while thinking of a stupid dog meme saying I have no idea what im doing then accidentally pushing something in prod and had no idea what to do cause the lead dev was not around... aahhh those were the days

I JUST HAD ONE OF THOSE DAYS THAT MAKES ONE WANT TO BANG TWO BRICKS ON HEAD SND END THE PAIN THE STORY STARTS YESTETDAY WITH ISSUES AFTER A MIGRSTION AND THEY ASK ME TO HELP TROUBLESHOOT EVEN THOUGH I'M A DEV DBA AND THE ISSUE IS IN QA/SAT AND I HELP ANYWAY AND THEY CAN'T FIND A VIEW AND SO I LOOK EVERYWHERE AND CAN'T DOING IT EITHER AND IT DIDN'T EXIST IN PROD OR DEV SO I TELL THEM IT'S NOT THERE, AND THEY ARE LIKE, CAN YOU RETRIEVE IT FOR US AND I'M LIKE FROM WHERE? I DON'T KEEP VIEWS IN MY BUTT AND YOU GUYS ARE SMOKING CRACK AND THE GIVE ME THEIR QUERY WHICH CONTAIN THE VIEE ANYWAY AND THEY SAY CAN YOU RUN IT AND IT RUNS AND WORKS AND THEY CAN'T MAKE IT WORK AND IT WORKS BECAUSE IT DOESN'T CALL THE VIEW THEY HAVE ME SO NO PROBLEM THERE SO I FINALLY ASK THEM ARE YOU POINTING TO THE CORRECT DATABASE AND THEY'RE LIKE OH MAN WE TOLD YOU THE WRONG DATABASE AND SO I LOOK AT THE RIGHT DATABASE AND FIND THAT THE GRANTS ARE MISSING AND YEAH THANK YOU FOR TAKING EIGHT HOURS OF MY LIFE BECAUSE WE WERE IN THE WRONG DB YOU GAVE ME AND I HOPE THE FLAG OF A THOUSAND CAMELS INVEST YOUR ARMPITS AND THE CHIGGERS OF A THOUSAND SOUTHERN LAWNS INGEST YOUR SOCKS AND UNDERWEAR. YAAAAAA!!!!

Just learned that yesterday someone suggested putting the dev team on "workspace", when I was on leave.

My first question, "what the hell is workspace?"
"It's a remote environment..."
Okay I get it. Are you kidding me? Doing development on remote desktop?

My second question, "Why the hell did someone suggest that?"
"We have had issues with devs using MySQL but the target prod will be using PostgreSQL. That caused issues, inconsistencies... And we found some issues after deployment."

Okay so much for DB agnostic. I called it out that everyone now install PostgreSQL on local. Problem solved, hopefully.

Why we had MySQL in the first place? Yes DB agnostic is one of the reason. The other being I'm more familiar with MySQL so it's quicker to perform tasks (like "can you clone that environment for me" and "can you fix the data on XYZ"). But that's trivial.

Just some ridiculous suggestion that set me off.

This is what happened today in our dayli:

Lead: We need to profile our software

Me: You can use the chrome devtools as remote profiler, even on prod, or make HAR files for later inspection.

Lead: Yeah but no that’s just collecting data on every tick, we need something like “has been called x times”

Me: Yeah but you can filt -

Lead: Yeah no, so back when I wrote code in Delphi...

Me: *oh god no not this again*

Lead: ... We could have clicked a button in our IDE and it would wrap the function call with the API call to profile that function ...

Me, to the secret dev group in slack: doesn’t a simple method decorator and node performance api help with that?

The people in the group: We had this topic last Friday all day...

Me: oh well *get’s coffee and ignores lead*

Boss wanted me to make changes in company's website which was based on wordpres s.

I knew it could be done by tweaking some JS code, but I have very less experience with wordpress

But wordpress is easy man(Internet told me).

Give me 5 minutes, you will see the changes in production.
Being lazy af I directly logged in to ftp, checked out some files, updated some code, I was good to go.

Before pushing it, I opened the website and it was GONE ٩(๑´0`๑)۶

Now there was no public_html in the root.
I was fucked. I have accidentally deleted the website that had no backup.

And the best part I was on leave from
next day.

I was looking everywhere for backups, looked into google cache to get the contents. I have to recreate the complete site now.

Just when I was asking questions on choice of my profession and simultaneously looking here and there in FTP for backups,
I found the jewel "public_html".

It happens out that I have accidentally moved the folder to some other directory.

Phewww.

Moved it back to root. Site was up and running.

Reassured myself that I deserve to be a dev.

Backed up complete site, made the changes.

Uploaded it.

And the best part, amount of wordpress I learned in those three hours was way more than I could have learnt in many weeks.

Lessons Learnt :

A) ALWAYS keep backups.
B) You SHOULD NOT make changes on prod directly
C) You become superhuman when your brain know you are going to be fucked 😂

A guy who had the same nationality as the enterprise we were working for was promoted from JUNIOR js developer to UX/UI coordinator for the entire department just because he was 2 year older than me (26 vs 28). Literally he was a junior dev and went to that.

One day he was accusing me of writing a piece of code which led prod to downtime. I was in the office, he was in another country with our manager and technical director next to him and we were talking over internal conference system. I shown git history + his name + his code and he was saying ‘that’s not true!!!’.

I couldn’t resist and I began to yell something like ‘You fucking fuck piece of shit cocksucker...’ for 5 minutes. Since that day i was the god on my project for UI/UX side.

Even now he is in the same place on the same position...

PS: more stories to come with this guy

// My First Rant

We have a developer that almost everyone adjust to what he want to avoid talking or working with him.

I have office mates that doesn't want to give tasks to him just to avoid working with him.

Even our devOps guy just did what he want so he would stop talking.

One bad experience of our devOps guy with him is that his infrastructure or other AWS stuff was blame why his APIs is not working. It turns our that his url for the database has FUCKING SPACES.

Not sure if a good practice but he wants the base url of our Endpoint to be set in environment variables instead of having DEV/PROD/TESTING and base the endpoint from there.

He said that he was given permission to study a language but he doesn't even ask for permission.

So last week I really fucked up

I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.

So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.

Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"

FUUUUUUUUUUUUCK!!!

Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.

Needless to say, I had a fairly good weekend.

Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert

I am doing some freelance work for a client who is thankfully mindful about security. I found out that they are so strict with their access because they had a huge data breach last year.

Today I was given access to their repo for connecting to their AS400. In the docker file the username and password were included and were the same for dev and prod. They also are performing no sql injection prevention. They are just joining strings together.

I didn’t turn down a dev freelance project when the client decided against going with best practices because the solution I offered was a well-established design pattern but created a need for a financial management change she didn’t like. I stupidly built what she asked for. It worked fine in the 3rd party vendor test environment but failed on production. After hours of analysis of code to ensure no changes happened to my source during test->prod deployment, and the vendor denying they had config differences between them, and the client refusing to pay, all I could do was abandon the project.

Few years ago as a junior android dev with couple years of self taught experience of working in startups I submitted a simple android app assignment for a junior android dev role. Assignment had only like 8 requirements so I followed them to the letter. That didn't end well.

App was simple just 3 screens. Login screen with username and password input fields, login button.

Had to call a login endpoint after login button was clicked, redirecting to home screen, calling items endpoint, displaying a list of items and when an item was clicked passing item data and redirect to item details screen.

Needless to say big swinging dick senior was not impressed. UI was not perfect, I forgot to display a loading animation when fetching data, didnt handle back button properly.

I agreed with some points but other comments were clearly just nitpicking: his preferred variable naming conventions, his opinions on architecture that was not up to his standard (official google arch at the time was not up to his standard).

He also was mad that app wasn't prepared for release to googleplay (another out of the ass requirement). Like I would prepare a 3 screen app for prod release that he will forget ever existed after 20min of his review.

Lots more of nitpicking, encapsulation this encapsulation that, omg now hes shocked that there are a few warnings after the project is built.

Regardless my self confidence was destroyed at that point and after few more negative experiences I dropped android dev alltogether for a couple years and switched to game dev.

After game dev ran its course I went back to android dev and found a supportive place where I could grow.

Looking back, they were actually hiring atleast a mid level for a junior position but I was grilled as a senior. The guy literally didnt wrote any single positive thing in that review about my code even tho my senior peers said my project was decent back then, its just that I didnt handle a few edge cases and that's all.

I looked up the guy in linkedin, turns out hes a uni dropout who posts all books that he red about software dev in his education section of his linkedin profile. Found a bunch of other narcissistic stuff on his profile. Guy was a fucking idiot. Even if I worked under him it would have probably sucked.

Learned some important lessons I guess. Always get a second, 3rd and 4th opinion and dont take criticism too seriously. Always check what kind of person is providing feedback.

IMHO technical dept is kind of like smoking cigarettes for some decades.

You were told that shit will hit the fan but you do not take proper action. And one day you'll realize that you fucked up (or not, also seen that).

Worked for a company in IT, where we maintained an ERP which was "in progress" for over a decade. The basic implementation was done by people with zero technical understanding. To clarify: not self coded. Software was bought. We are talking about integrating the system.

Therefore, the foundation was like a wet noodle. When I joined that company, I told them that they need to address that. I told them that things will get slower and slower and that shit will hit the fan if no proper actions taken.
Even made a list with flaws I found. With potential risk and actions to take, that could then be measured.

At that time, five people worked in said department (including me).

People did not want to listen. "Would be too expensive to rewrite stuff".

Nothing has changed about the wet noodle, but I tried to fix as many things in a working system as I could. Felt like heart surgery, because changes got implemented and "tested" in prod. No version control, no documentation, everyone implemented things like they felt (no guidelines for consistency).
A lot of small fuckups that summed up over the years.

I left the company after two years because I had the chance to land a job as a dev.

Been around two years now since I left. Now 9 people work in that department with around the same efficiency as us 5 people back then.

The new employees struggle to be productive, because things are just implemented poorly and not maintainable anymore.

Had some dialogs with them some time ago. Everything I told them would happen, actually happened. What a suprise :-|
I will not go into too much detail about all the shit that's going on there, as it would be just too much (and my morning coffe is almost finished).

I think that we all know the difference between "not beautiful, but does the job" and "oh, that will backfire - badly". And I wish that my communication skills increase so that people start listening in future.

TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.

Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.

management logic.

dev : calling api on every product scroll is a stupid idea. we shouldn't do it. what if user has 100s of products bought?
mgmt : it isn't a practical scenario. in prod, we checked the data and we rarely have customers with more than 20 products
dev : 😮🤷‍♂️

dev : this is a rare issue that only happens for very old devices from this specific manufacturer. even manufacturers have acknowledged this.
mgmt : we don't care. fix it, as per data this error has been logged for more than 12 times (from 1 user only)
dev : 😮😢

We had 1 Android app to be developed for charity org for data collection for ground water level increase competition among villages.

Initial scope was very small & feasible. Around 10 forms with 3-4 fields in each to be developed in 2 months (1 for dev, 1 for testing). There was a prod version which had similar forms with no validations etc.

We had received prod source, which was total junk. No KT was given.

In existing source, spelling mistakes were there in the era of spell/grammar checking tools.

There were rural names of classes, variables in regional language in English letters & that regional language is somewhat known to some developers but even they don't know those rural names' meanings. This costed us at great length in visualizing data flow between entities. Even Google translate wasn't reliable for this language due to low Internet penetration in that language region.

OOP wasn't followed, so at 10 places exact same code exists. If error or bug needed to be fixed it had to be fixed at all those 10 places.

No foreign key relationships was there in database while actually there were logical relations among different entites.

No created, updated timestamps in records at app side to have audit trail.

Small part of that existing source was quite good with Fragments, MVP etc. while other part was ancient Activities with business logic.

We have to support Android 4.0 to 9.0 of many screen sizes & resolutions without any target devices issued to us by the client.

Then Corona lockdown happened & during that suddenly client side professionals became over efficient.

Client started adding requirements like very complex validation which has inter-entity dependencies. Then they started filing bugs from prod version on us.

Let's come to the developers' expertise,
2 developers with 8+ years of experience & they're not knowing how to resolve conflicts in git merge which were created by them only due to not following git best practice for coding like only appending new implementation in existing classes for easy auto merge etc.

They are thinking like handling click events is called development.

They don't want to think about OOP, well structured code. They don't want to re-use code mostly & when they copy paste, they think it's called re-use.

They wanted to follow old school Java development in memory scarce Android app life cycle in end user phone. They don't understand memory leaks, even though it's pin pointed by memory leak detection tools (Leak canary etc.).

Now 3.5 months are over, that competition was called off for this year due to Corona & development is still ongoing.

We are nowhere close to completion even for initial internal QA round.

On top of this, nothing is billable so it's like financial suicide.

Remember whatever said here is only 10% of what is faced.

- An Engineering lead in a half billion dollar company.

36 in previous job (sysadmin) - had to run 3 shifts in a row. Boy was I a vegetable in the end....
18 in current job (java dev), when I messed up PROD db

Pressing ctrl+s in sql dev when checking packages.. you accidentally press space or sth and later ctrl s it compiles.. then shit gets real when you accidentaly lock up everything on prod db..

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

Holy fuckin shit. Fuck java updates.

Today we deployed to production with a java app. For whatever reason it didn't work, just throwing SSL errors left and right.
Same app works on dev and qa

There's only one tiiiiny difference: Java 8 141 on dev/qa, Java 8 171 on prod

Guess what happened in 171 ... they updated the CA

rip 4h debuggin

TL;DR

Got Salesforce customised by an external company, took over the code and wanted to cry.

So my company decided to get a new crm for sales with contract generation and a whole lot of fluff. Specs included an easy to implement API to connect to our in-house software and an easy to adjust contract Wizard.

After month of checking various companies the CEO finally settled on...
Drumroll
The cheapest German based company.
Turns out the only part that is based in Germany is the sales department, development is based in Poland which made for interesting times during implementation because of the language barrier that comes with non native english speakers.

We as in our development department for told that we wouldn't need to worry about the solution because it would all be developed and maintained by the new company. As we are fairly small that was more then welcome.

Fast forward to integration day. No docs for the API available, contract Wizard is hardcoded, bunch of errors and inconsistencies. Get tickets for them.
Ok we can deal with this. Just tell the ticket writer he needs to address the problems with the external. Yep, doesn't work. External fixes bugs and introduces then elsewhere.
Fml. Ok I'll take a look into the code. Ugh, Java, I hate that shit but at least I don't need to worry about all the fluff, just the code, so it's ok.
No repository present, code is developed in the Dev environment and pushed to prod. Ugly but works. Code comes with a lot of functions but only one real class called "CommonUtilities" ... even the web API is defined in there. Meanwhile my colleague throws out the need for the API because we will just directly tie to SOQL. I'll let's check what's going on here, nice you reused Lead/Account/Contact for the branch offices of companies as well...
Is not like Salesforce has a bunch of logic tied to these objects...
Nice the required implementation of an automatic holiday import is not even there, just a custom object populated with this year's data...

Tell CEO how badly all of this was handled. Nice note this dumpster fire is our new in-house project because CEO cancelled the contract with the external.

Ok we can deal with this, let's set up a repo, define the CI/CD and get the extensions for vscode. Nice now this all makes sense. Fix all bugs and reimplement the contract Wizard using custom objects that sales can change so the contract is actually easily customizable by a non dev. Implement branches as their own object to avoid opportunities, triggers and the likes to be executed for them.
Took a whole day. Why did these 2000 lines of code that was shipped to us take 3 month to implement?

Lesson learned: never trust an external to just do a good job.
New rule implemented by company to always have a Dev check in regular intervals on projects handled by externals for standards and overall logic

10k down the drain for what amounted to 2 weeks tops (one dev) if we had handled the implementation of Salesforce and all the requirements in-house

Rolled out a new application I built almost entirely by myself 2 days ago... But my dev group is understaffed and has a project manager who is literally the most clueless person I have ever met, so as a result, we don't have a functional/useful dev/test/prod framework and no standards for how to deploy apps. So my past 2 days were comprised of fixing bugs in the live system that could probably have been caught if I had the time and resources to get everything thoroughly tested. It's stable now, but damn our management for being generally idiots. Our motto appears to be "Fuck it, we'll do it live"

Since I have seen a lot of people uploading this kind of stuff lately, here is Xiaomi's test in production, back in 2017 November...

I hate dev politics...

PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.

Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.

Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!

Me: That's dumb... the code works on test... it's not the code.. it's the application.

Senior dev: %$*^$>&÷^> $

Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!

Senior Dev: OK

Me: *runs Jenkins job the deploys the new code and restarts the application*

PM: YAY you fixed it!!

Senior Dev: Did you sump put those errors like I said.

Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.

ideal sprint fallacy.
total days 10 , total hours(excluding breaks ) 8 hrs per day= 80 hrs per dev

code freeze day = day 8, testing+ fixing days : 8,9,10. release day : day 10
so ideal dev time = 7days/56 hr
meetings= - 1hr per day => 49 hrs per dev

- 1 day for planning i.e d1 . so dev time left . 6 days 42 hrs.
-----------
all good planning. now here comes the messups

1. last release took some time. so planning could not happen on d1. all devs are waiting. . devtime = 5 days 35 hrs.

2. during planning:
mgr: hey devx what's the status on task 1?
d: i integrated mock apis. if server has made the apis, i will test them .
mgr : server says the apis are done. whats your guestimate for the task completion?
d : max 1-2 hrs?
m : cool. i assign you 4 hrs for this. now what about task 2?
d : task told to me is done and working . however sub mgr mentioned that a new screen will be added. so that will take time
m : no we probably won't be taking the screen. what's your giestimate?
d : a few more testing on existing features. maybe 1-2 hrs ?
m: cool
another 4 hrs for u. what about task 3?
d : <same story>
m : cool. another 4 hrs for u. so a total of 12 hrs out of 35 hrs? you must be relaxed this sprint.
d : yeah i guess.
m cool.
-------
timelines.
d1: wasted i previous sprint
d2 : sprint planning
d3 : 3+ hrs of meetings, apis for task 1 weren't available sub manager randomly decided that yes we can add another screen but didn't discussed. updates on all 3 tasks : no change in status
d4 : same story. dev apis starts failing so testing comes to halt.
d5 : apis for task1 available . task 3 got additional improvement points from mgr out of random. some prod issue happens which takes 4+ hrs. update on tasks : some more work done on task 3, task 1 and 2 remains same.
d6 : task1 apis are different from mocks. additionally 2 apis start breaking and its come to know thatgrs did not explain the task properly. finally after another 3+ hrs of discussion , we come to some conclusions and resolutions
d7 : prod issue again comes. 4+ hrs goes into it . task 2 and 3 are discussed for new screen additiona that can easily take 2+ days to be created . we agree tot ake 1 and drop 2nd task's changes i finish task 2 new screens in 6 hrs , hoping that finally everything will be fine.

d8 : prod issue again comes, and changes are requested in task 2 and 3

day 9 build finally goes to tester
day 10 first few bugs come with approval for some tasks
day 11(day 1 of new sprint) final build with fixes is shared. new bugs (unrelated to tasks. basically new features disguised as bugs) are raised . we reject and release the build.

day 2 sprint planning
mgr : hey dev x, u had only 12 hrs of work in your plate. why did the build got delayed?

🥲🫡

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

A couple of years ago I was working on a fairly large system with a complex (by necessity) access control architecture.

As is usually the case with those projects, it's awkward for developers to repro bugs that have to do with a user's accesses in production when we are not allowed to replicate production data in test, let alone locally.

We had a bug where I ended up making myself a new row in the production database for a thing I could have access to without affecting real data to repro it safely. I identified the bug so I could repro it in dev/test and removed the row and ensured everything worked normally, whew scary.

Have you ever walked into the office one day, and everyone is hunched over in a semicircle around one person's workstation, before one turns around to look at you and says - after a pause - "... ltlian?.."

Turns out I had basically "poisoned the well" with my dummy entity in a way where production now threw 500 for everyone BUT me who had transitive access to this post-non-entity. Due to the scope of the system, it had taken about a day for this to gradually propagate in terms of caching and eventual consistencies; new entities coming in was expected, but not that they disappear.

Luckily I had a decent track record for this to be a one-off. I sometimes think about how I would explain testing in prod and making it faceplant before going home for the day, other than "I assumed it would be fine". I would fire me.

I've now worked on both monolithic solutions and microapps/microservices. I gotta say I'm not sold on the new approach. There's so much overhead! You don't have to know your way around one solution -- no, now you need to know your way around 100 solutions. Debugging? Yeah, good luck with that. You don't have to provision one environment for dev, test, staging, and prod. No, now you need 100 environments per... environment. Now, you need a dedicated fulltime devops person. Now devs can check in breaking changes because their code compiles fine in that one tiny microapp. The extra costs go on and on and on. I get the theoretical benefits but holy crap you pay for it dearly. Going back to monolithic is so satisfying. You just address the bug or new feature head on without the ceremony and complexity. You know you're not crapping on other people's day (compilation-wise) because the entire solution compiles.

...and yeah, I'm getting old. So get off the lawn! ;)

So here we go again.
Same "web designer", same me, same website (based on wordpress, completely redone front-end UI, full of ACF to keep their fucking data).

WD: ok, I'll need you to add an information on the project page template in order to show a gallery with slider at the bottom and "film+visuals" at the top if there is a gallery.
Me: ok

1 week later, after like 40 e-mails between us deciding if it was ok or not on DEV server and pushing it to PROD

Me: ok, done
WD: hey, if it's just a film there should be written "FILM" while there's nothing showing, also the gallery must have same height as the film above

Me, internally: why the fuck didn't you fucking tell me before pushing to PROD? are you fucking dumb or something?
Me, via email: ok, i'll check it now...

Ah.... Just saw a message in team chat that we had a prod issue caused by the use of a hack/duct tape solution...

Dev just replied "as a fix I will do it the right way"

Me thinking: ¯\_(ツ)_/¯ I told you so... Now you learned it the hard way

How come something works absolutely perfectly in dev but not in prod?
I was making a desktop app in election js and everything is working perfectly. No problem at all. But then I create the installer/distributable and nothing shows on the screen. And out of curiosity, I wanted to see the error log and it shows an unknown error, I didn't even know from what thing the error is being generated. And after I fixed that, another problem came with Asana Api. I mean, if it's a public API, why do you have to block it with cors? I hate cors!
And after all of it, there's more to it. I mean, why can't you just show the errors in dev?

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

So I made an update to my React Native app. I changed UI of a couple of screen, added a few animations here and there, refactored how my graphQL resolvers work in the backend(no breaking changes), changed how data gets loaded into the database etc.

It worked in dev so I figured hey let's deploy it. Today is(was because it's now 3am but more on that later) a national holiday so no one goes to work so no one will use my app so I have an entire day to deploy.

I started at 15:00(because i woke up at 13:00 lol). I tested the update once again in dev and proceeded to deploy it to prod. I merged backend to master, built docker images, did migrations on the db, restarted docker-compose with new images. And now for the app. I run ./gradlew assembleRelease and it starts complaining that react-native-gesture-handler is not installed. Ugh, rm -rf node_modules && yarn install. It worked. But now gradlew crashes and logs don't tell me anything. Google tells me to change a bunch of gradle settings but none of them work. Fast forward 5h, it's around 20:00 and I isolated the issue to, again, react-native-gesture-handler. They updated from 2.2.4 to 2.3.0 which didn't fucking compile. 2 more hours passed (now 22:00) and I got v2.3.1 working which fixed the problem in 2.3.0 but made my app crash on startup. YOUR FUCKING LIBRARY GETS 250K WEEKLY DOWNLOADS AND YOU DONT EVEN BOTHER CHECKING IF IT COMPILES IN PROD ON ANDROID?! WHAT THE FUCK software-mansion?

After I solved that, my app didn't crash. Now it threw an error "Type errors: Network Request Failed" every time I fetch my legacy REST API(older parts use rest and newer use graphql. I'll refactor that in the next update). I'll spare you the debugging hell i went through but another 5h passed. Its 3am. My config had misspelled url to prod but good for dev... I hate myself and even more so react-native-gesture-handler.

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

Small chaotic startup that never grew up (15 years atm).
Hosts/maintains a number of apps/sites for various customers.

At some point, someone decides that a CMS would be usefull to maintain the content across all products. Forgoing all sense, reason and the very notion of "additional maintenance and dev" it is decided that one should be built in-house.

Fast forward a number of years.

Ops performs routine maintenance on prod-servers. A java-patch accidently knocks out one of the pillars a 3rd party lib the CMS uses for storing images. CMS basically burst in to flames causing a.... significant incident.

Enter yours truly to fix the mess.
Spend a few days replacing the affected 3rd party lib. Run tests on CMS in test and staging environments. Apply java-patch. All seems fine.

When speaking to frontenders and app-devs, a significant hurdle present itself:
All test/staging instances of all websites/apps/etc ALL USE PRODUCTION CMS. Hardcoded. No way around.

There is -no- way to properly test and verify the functionality of any changes made to the home-brewed CMS.
My patch did indeed work in the end.

But did the company learn anything? Did they listen to my reasoning, pleading or even anguished screams for sanity?

No.

I am forced to work with a client's notoriously slow SOAP api. Slow in this case is 1.5-2s per request.
The api is structured rather... creatively... at the same time. So we have to bombard it with thousands of requests to build our data base with historical SOAP data. Also the data sometimes is a couple of hours late, giving a flat line (all values at 0) until retroactively fixing the output for the same requests.
So to fill one dev data base with a year's worth of historical data (nice to have when testing a dashboard application) we hammer the api with ~20k requests (~1 million if we want to be thorough).
Best thing about that: There is no staging/test api and the prod api seems not to handle lots of requests at the same time very well...
Latest thought: Maybe we could put a varnish cache in front of the SOAP for testing. Better have wrong data, than nothing at all and we don't kill the prod clients every time we ramp up a new instance.
Also that would dramatically decrease the 4.2 hours of data pumping to about 7 minutes after the first run.

Story time;

Major project, multi million budget, huge business and IT coordination, board level status updates, meeting started back in March 2018 for a Go Live of Aug 2019.

Based on draft requirements (and experience) I request the test environment be built for half of the work. Turns out that no one told Server Eng and they are out of space in both dev and prod until Q2 of 2019. We went from Green to Red because a Service Request.

Want to hear another joke?

Blue Prism allows you to export stuff from version 6.7 to 6.3.

However they changed 𝘷𝘦𝘦𝘦𝘦𝘳𝘺 slightly the way they store the position of the nodes. No new features -or at least nothing that you would care about- but the structure of the node itself want went from
```
<positionx>1</positionx>
<positiony>2</positiony>
<width>3</width>
<height>4</height>
```
To
```
<position x=1 y=2 w=3 h=4></position>
```

The whole project collapsed to a single point, catastrophic consequences as far as exception handling. A generic "fuck you" for no real reason other than the sheer malice of those beasts of burden who developed Blue Prism in the first place.

And I have two different versions of Blue Prism on dev and prod :)

I upgraded a Linux server one time and data that was serialized in yaml stopped being parsed properly.

It turns out the libyaml people decided to change how hashes were handled, which made any previous hashes come back as blank.

A whole database of valid data in dev was coming back invalid in prod. It was maddening.

It took a day to figure out the problem and how to update the data to the new format in rails.

I now serialize in json.

Well, this happens time to time...

I'm freelancing as a backend guy. I like to take care of all infrastructure before really starting to build anything, this mostly includes dev/staging/prod environments with some linear promotion strategy. So.. I did this API. Still on staging, proceeding with the development as planned, everything goes according the timeline.

And then.. this happens... At some point PM told frontend guy that it's time for production (without notifying me), so the frontend guy does what "anyone" would do in this case - tells PM to create DNS record for production to point to staging app.

Time passes, I'm still unaware of this. But I'm starting to see some quality entries in the DB, not the usual QA crap. I write to them that they're doing good job and continue with my tasks.

One of the tasks required some major DB change. I could've written migrations script, but since we're not in "production" yet, I just wipe the DB and recreate schema as I need it.

In 10 minutes the furious PM starts shouting that "production" is down and I need to fix it ASAP.

I'm lost, I'm asking questions, I'm slowly understanding what's happening...

So I want to grab some coffee, sat back down, wrote politely that they suck, added a finger emoji and terminated the contract.

Felt like the right thing to do as I definitely don't want to continue within the same "team".

Yesterday whole 12 hours we were working on deployment about a feature X that has deadline yesterday itself.
Everything damn perfectly running on Test env but not on Prod.
We made Prod into Dev/Test/Fucking garabage env. Haha.
I was laughing to myself at same time crying hard in my deep heart.

Business guys chasing PM
PM chasing us
And from morning till night we were in same room. Had lunch, and dinner only went out for toilet and to refil water bottles.

And found that feature Y is not working at same time that is related to our feature X. Fucking we have been wasted hours on it.

One of my devs got so fucked up emotionally that he messed up the code (not his fault) he didnt had his lunch and dinner. Had to console him later that its not his fault. Poor guy not sure whether he slept or not; will find out in few hours.

Anyways reported a bug.
But that bug assigned to us for fixing.
Are you fucking kidding me.
Anyways no choice. Had to do it.
Hope today everything goes good or horribly bad. FYI no deployments on Friday damn we are in stalememt till Monday.

Fuck that bug
Or
May be fuck our stupiditiy while makiing mistakes.

Other dev in group chat: we need to stop syncing so much data from prod to uat because it's crashing the uat db...

Me thinking: no really... U just realized it's not a good idea to dump most of the prod db into uat every week?

¯\_(ツ)_/¯

1. For my employer to invest in QA. Honestly, even if I'm 101% confident about my code, if nobody tests it other than me, I would advise against prod-ing(Is that a word?) it.

2. For recruiters so stop expecting a Full stack dev to be perfect in both ends (especially with an entry level salary. Stop taking advantage of them!!). Just stop using the term full stack entirely, please.

3. For API docs of other companies to be deserving of the title 'Documentation'. I'm so tired of figuring out other API parameters via trial and error. Just make your docs as clear as you can please, so we don't have to bother each other with so much email.

That's all for now. Thanks dev Genie.

Losing faith in Netflix and their awesome open source projects.

Had a hard time trying to install Security Monkey : poor quality quickstart Ubuntu-only, almost no documentation, same instructions for latest (aka dev) and stable (aka prod) version, no depencies list ... oh and the UI display well only on Chrome ..

Then you surrender and just want to check the dockerized version they provide : it doesn't work neither (build fail or back end process just shut down) !!

I'm done ...

Today, I'm making a revolutionary change to our code base. I'm finally deprecating a script that lived in this goddamn repo for way too long. There's about 10 copies of the same script in 10 different directories. The script copies all the code into a "dev" folder and then runs "sed s/prod/dev/g" on all the files, and then overwites a bunch of it with some files suffixed with ".dev". Finally, after fighting the so-called architects, the devs and everyone else that seems to have gotten used to the pain this cursed dumpster fire script has caused us, my merge request is now open and ready to go to get rid of this insanity. Now we won't have to deal with as many "surprises" that happen every goddamn time we deploy to production, overwriting all our hard work by accident, and relieve some of my OCD of having the same script in 10 different places in the repository.

First dev in prod. pretty much ever for me

Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?

# The detailed version #

We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.

But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are

So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.

I guess the saying goes: the only people that know how it works is whoever wrote it and God.

I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.

I think in Agile there's supposed to be artifacts during development but never seen em.

Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.

I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.

And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...

The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.

Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.

Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.

Does anyone work on a bunch of local NPM modules wanna describe their workflow for local dev vs deploy?

I’ve got mine but it feels a little trashy. It’s basically one npm script to link all the local modules for dev and another which will npm install them in prod - is there a better way without adding more build tools?

is it necessary to have cherry picking a part of git branching/release process?

we have 3 branches : develop, release and master.

currently every dev works on feature as follows : they make a branch out of develop, write code, raise pr against develop, get it reviewed and merge back to develop. later the release feature list is generated, and we cherry pick all the release related commits to release branch, and make a prod build out of release branch. finally, the code is moved to master and rags are generated accordingly.

so the major issue with this process is feature blocking. as of now, i have identified 4 scenarios where a feature should not be released :

1. parallel team blocker : say i created a feature x for android that is supposed to go in release 1.2.1 . i got it merged to develop and it will be cherry picked to release on relase day. but on release day it is observed that feature x was not completed by the ios dev and therefore we cannot ship it for android alone.

2. backend blocker : same as above scenario, but instead of ios, this time its the backend which hasn't beem created for the feature x

3. qa blocker : when we create a feature and merge it to develop, we keep on giving builds from develop branch adter every few days. however it could be possible that qa are not able to test it all and on release day, will declare thaf these features cannot be tested and should not be moved to release

4. pm blocker: basically a pm will add all the tickets for sprint in the jira board. but which tickets should be released are decided at the very late days of sprint. so a lot of tasks get merged to develop which are not supposed to go.

so there's the problem. cherry picking is being a major part of release process and i am not liking it. we do squash and merges, so cherry picking is relatively easy, but it still feels a lot riskier.

for 1 and 2 , we sometimes do mute releases : put code in release but comment out all the activation code blocks . but if something is not qa tested or rejected by pm, we can't do a mute release.

what do you folks suggest?

You know that feeling you get when there is a bug in Prod and to identify the cause, you setup your dev environment with the exact same codebase as Prod. But the bug won't show it's ugly little mug in devo...

Yeah, Fucks me up too... -_-

did on my last project:
1 .Using QA env as dev env
2. Deploy in production not completely tested stuff (90% tested)
3. Run with errors in prod
4. Manual fix in prod
5. Git versioning

- Am a junior dev in an awesome team & exciting project after my apprenticeship and while having just started my part time studies
- Have restructure in company so I land in an other value stream
- Get laid off by new value stream 6 months later (now) because they have a serious budget cut
- Take time to come to terms with situation. I could finally work more on my side projects or focus a bit more on my studies. Hey actually I will have 5 months time to look for something while being paid by the company and they help me brush up my CV. Pretty neat!
- Now my former boss wants me back because of my experience in the project, but only as a production support and not as dev (because budget and they're bleeding with tickets)

Not sure if I should take the offer as it feels safe to have an income and the team is cool. However, it feels a bit like a degradation as prod support sucks in that project and I'd like to code (which wouldn't be possible then).
And as this is still my first company I'm working in, it would make sense to look for something else...

Grrr need to sleep about it... Decision-making isn't exactly my strength.

I feel there aren't enough tutorials on "best practices" when it comes to combining server/client tooling in a monorepo.

Having done so this weekend, the tasks involved were:
* using graphql w/ express to serve requests and expose a "graphiql" ide instance
* differentiating build steps in prod vs. dev
* applying middleware in prod vs. dev
* working with a single heroku dyno

Still missing:
* hot reloading
* my general sense of direction

Ah the joy transitioning from the unrestricted apprentice network to the tightly restricted prod and dev network and environment. U can be sure thst the corporate proxy will give you a dropkick to ure face when trying anything that was released in the last 5 years...

This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...

Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place

When you enter dummy data to the dev db and your boss tells you to use it in prod 👎

Some little piece of shit fairy messed with my code.

Made some modifications in a tamplate file, tested in dev everything worked. Pushed it to prod and in one of its cases it displayed the array key of each element. Checked the code, an that was what written in there in prod and in dev too. However that shit in dev (and previous versions in prod) displayed key[0]->name as intended.

So I thought git blame and check where it went wrong... Guess what... That fucking line is the same age as the file itself....

How that motherfucking shit worked, and what had I had in mind when writing that shitcode is a mistery...

So I'm assigned once again to fix a new someone else created and that seems to be the case whenever there's an issue...

Boss just assigns it to whoever is most likely to be able to investigate it... which is basically me. Other than the little time I can use to develop stuff, I'm usually cleaning up other people's messes.

And these other people are to busy working on new crap to properly explain how their existing code/processes/changes works.

And well the fact that anything breaks in production (that's not due to upstream one off issues) whoever does not think he needs to take responsibility for it.

So everyone else and especially me has to spend time understanding the shit they wrote and fixing it for them.

How do I tell my boss this nicely that we need clearly definitely ownership and whenever a component blows up in prod, the guy that wrote the code fixes it no matter what? Thereby incentivizing him to not write shit code in the first place and be more proactive in making sure it doesn't in the first place since he knows otherwise he's doing overtime to fix it?

Is it just me or is there really no such thing as a dev job where something doesn't blow up due to poorly tested and designed code every other day?

Any one running Symfony on a Docker container in production? I currently try to migrate our dev env to a docker compose setup (from a "monolith" vagrant vm). I'm atually not stuck at a Symfony specific thing, but on a, I guess Docker specific one(?), The issue is, I need to read and write with two users to one folder (in my case the /application/var/cache folder). Since I mount my whole code into the docker container (to use an IDE on the local files), I've got a volume (not mounted to the outside world) for that folder. (As far, as good). Now this folder is owned by root and root is also the user I get when I enter the container. When I then run a cli script, that writes to this folder, every thing works (as it's run by root) and the resulting entries in the cache dir are owned by root. Trouble starts when the php fpm process tries to write stuff in there too (as it's run by www-data).
If I add `USER www-data` (or create a new user foobar and add `USER foobar`) the container exits with status 0

So I guess the question is, is anyone running an Symfony app on Docker in Prod, if so how do you solve this? Or another question would be what is the best practice to do this? Sure on dev I could just `chmod 777` the whole folder or run the php-fpm process as root, but if that thing ever goes to prod, I wouldn't sleep very well...

I was trying to move a Zend app from one server to another once. there were actually 3 apps running on 2 different servers, an idle rabbit server, and the code in prod was vastly different than what was on the repo. the docs the previous dev wrote were literally the about pages for the tech used.

I remember he had a Windows server running something... all the docs said was "for long processes".... there wasn't a single process ever running on it.

I work in a small team. As the senior dev I tens to focus on important tasks that shape the core of the product but some times I can’t divide my self when there are multiple tasks at hand, so I pass some tasks to the an other mid level dev.

So the task was to create an automation in order to CD (continuously deliver) an order from WHMCS of the (git versioned) product to customers UAT, PROD envs.

To get a background this is an old guy with “constricted” experience in PHP/jQuery/Joomla/Wordpress.

So when we were breaking up the tasks he told me he would like to implement this so i gave him the task as i was busy with core features.

I was like what could go wrong? I know he doesn’t know much about CI/CD but he can read right? He will google right? He will search for CI/CD solutions that do this out of the box right? He will design on paper or what ever and do small POCs right? He will design the flow first before starting the implementation right? RIGHT?

So fast forward to today I had a call with him this morning about some DB staff. And he wanted to show me his progress…

His solution is:
(parentheses is my brain)
1. Customer completes WHMCS order (perfect)
2. Web Hook 🪝 action (YES)
3. cpanel gets source and “automatic!” Init, all using pure PHP code ignoring the usage of the current framework (ok… something is missing)
4. cpanel web hooks(?) WHMCS to send email to customer with the envs initial setup page(?)
5. Customer opens link and adds setup info (ok fuck, fuck, fuck)

(Ok stay cool composed, lets ask some questions maybe he thought it all in a cool way I can’t get my mind around)

Me: So how are you gonna get the correct version from the repo to the env and init the correct schema?

Dev: I haven’t thought about it yet.

Me: Are we gonna save each version to a file system then your code is going to fetch them?

Dev: I haven’t really thought about it we will see. But look on customer init user setup I implemented a password strength validation and it also checks if the password is the same.

So after this Pokémon encounter I politely closed teams. Stood up drank some (a lot) coffee ☕️. Put out the washed laundry while reflecting on life’s good things, while listening to classical music 🎼 .

Then I sat on my office chair drank some more coffee, put some linking park starting with in that order:
“Numb” then “What I’ve Done” and ended with “In the end, it does really fucking matter”

I was hoping that I never have to build stupid websites again, but here I am...

The thing I hate the most, is the hassle to have an easy to update dev, staging and prod environment. Fuck wordpress, fuck drupal, fuck joomla.

git pull && composer install && npm install should be all that's needed to get the latest code in an environment.

composer require *** to install plugins. No stupid web interface where users install plugins in production env.

I don't want to create database dumps just because these fuckers think that you should store configurations in the database.

Is there any clean CMS primarily for professional programmers? Or are they all just made for retarded subhumans?

I've having issues trying to form a proper branching strategy for my mean stack app deployment.

Heroku creates staging and prod branches for my web app so I'm a bit confused if I need my own staging branch?

Currently I have this: feature -> dev -> staging -> heroku staging (the staging branch seems useless)

Also, Heroku allows you to promote heroku staging to heroku prod, so there's no point in making master push to heroku prod.

I'm thinking of making my strategy to the following, but wasn't sure of any pitfalls or anything I'm overlooking long term.

feature -> dev -> master -> heroku staging -> manual promote to heroku prod.

Any suggestions?

Whats the fucking purpose of our companys dev test and prod env. Dev always only has a single instance. Sometimes clustered services run as cluster on test. Producing headaches because the clustering behaviour couldnt be seen on a single instance and Prod lacks all the nice deployment tools off dev/test. Fuck thinking you could dev then test and prod without any major reconfiguration and headaches. And all because the Storage costs is RETARDEDLY expensive because the backup EVERYTHING with ridiculess overkill. That results in headaches when requesting new servers. Took an old Workstation from the shelves and made it my vm slave so at least i could reliably deploy to test.. Fuck this process

Why do some employers make such a distinction between learning the tools at university and learning the same tools at the workplace?

Are they backward or old? Don't they know modern, high-quality universities have modern environments that are in fact real life?

Environments with acc-test-prod-dev with gitlab, ci/cd in Scrum teams and the works? Heck, at my uni we even worked at real companies, did internships there for months!

Come on.. to me this 'the tools you learned in school isn't the same experience as real life experience'. Right, these guys must be on some conservative backward model because there is in fact no difference.
I have worked both during my uni internship at a real company (in teams too) as well as irl at real companies and there is no difference, it's the same thing.
I don't care if I've learned to experience git + ReactJS etc during an internship through uni or at a workplace. It's all bureaucracy.

Context: I am leaving my company to work at a data science lab in another one.

My senior dev (with PO hat): we need to gather data from prod to check test coverage. You will like it as you will be data scientist hehehe (actually not funny). You will have to analyze the features, and find relations between them to be able to compare with the existing tests

Me: oh cool, we can use ML to do that!

Him: Nope, we need to di it in the next 3 weeks so we need to do it manually.

Me:... I have quit for something....

Deploys to Production.
Runtime error.
Open Development server and run in Production setting.
Still runtime error.
Fixes Error.
Error fixed on development.

while (hoursWasted < 3) {
Deploy.
Not working on Prod.
Try other fix.
Still not working, but works perfectly in dev machine.
What the fuck
}

Rage
Go take a walk
Realized I might have deployed to the wrong server
Glanced at deployment path
Realized it's at the wrong server
Reconfigure and Deploy
It works.
Fuck.

so yesterday was a fun day. I'm wiring up dbvis toour db life cycle envs. connection to dev, works. move on to Val, connection works. move on to prod, boom, dbvis violates some security protocal that obviously does not exist in dev or Val which locks out the dB acct. I single handedly shutdown production, simply trying to connect with dbvis. smh. what a day!!

So trunk-based is the new approach everyone is using, because it is so cool.
I used gitflow for the last projects with azure devops, set up the pipelines like tipically in 1 week if I had other things to do with the help of the portal clicking through things. PR-s triggered pipelines, everything worked cool.
But then trunk-based got momentum, so I worked with this client where 2 developers worked for !!!3 months!!! to setup trunk-based pipeline. It was not my money, so I did not say a thing. They were using infrastructure as code.
I am all in for automation, but seriously? Then again, another project where a DevOps team took 1 dev-month to setup the pipeline + meetings. And what do you get in the end? So that the same image goes on all environments? Like how many releases do you have for prod in a year. Lets say 24. 24 x 5 minutes of manual work for the release, that is 2 hours. So my question is why would you spend 2 hours of manual work while you can automate it merely in a month? Everyone loves to code, but using the ui on the DevOps portal saves you so much time. I don't get this. Maybe I am getting old :D

My team is pretty small right now. It's myself and two other guys. One lead, who's been here for five years. A senior who we brought on 2 weeks ago. And me, a regular app dev. The lead put his two weeks in last week and has been trying to brain dump as much as he can onto us.

I've been building a list of prioritization to compensate for when he leaves based on what he was saying was the most important. This list has gotten pretty massive after reviewing most of the processes in place.

I was hired mainly to quell new requests coming in and not to maintain our systems, so that's what I did. I didn't examine our prod code base too closely. I wish I had. It's in a sorry state. I'm pretty sure I have about 2 years of tech debt for a crew of two guys constantly working on it.

I've been trying to prioritize based on what gets the most bug fixes and change requests. These apps will see the biggest changes and will undergo the most maintenance.

Since I'm just a regular app dev it feels weird trying to come up with this and try to prioritize this and come up with a plan. It feels like someone else should have. If it needs done then I guess it needs done. I need to be able to collaborate and work with my co worker and be able to plan for what projects are coming next.

If anyone has any suggestions to tackle tech debt please make them. Or if there's any help for managing priorities in a different manner that may prove helpful I'm open. Honestly, I don't want to tackle this completely blind, it feels like a lot.

What are opinions out there on security theatre?

Should developers have access to aws secrets?

Should dev test and prod be on separate vpcs or all in one vpc.

I have worked at banks where this was strictly not allowed.

Can’t wait to hear responses on this one….

What alternative do you use for EF Migrations?
We are in current stage that it's slowing our build time for 30 seconds, and each month we need to clean them and do the same for the whole pipeline (dev, qa, prod)...
It's way uncomfortable to work with it since we have stored procedures and ef doesn't track them...
What alternative did you chose to use?

PUT method works in dev but not in prod for code ignitor. Stuck on this one from a week. Someone HELP 🥲

(Checked htaccess and PUT is enabled)