My mentor/guider at my last internship.

He was great at guiding, only 1-2 years older than me, brought criticism in a constructive way (only had a very tiny thing once in half a year though) and although they were forced to use windows in a few production environments, when it came to handling very sensitive data and they asked me for an opinion before him and I answered that closed source software wasn't a good idea and they'd all go against me, this guy quit his nice-guy mode and went straight to dead-serious backing me up.
I remember a specific occurrence:
Programmers in room (under him technically): so linuxxx, why not just use windows servers for this data storage?
Me: because it's closed source, you know why I'd say that that's bad for handling sensitive data
Programmers: oh come on not that again...
Me: no but really look at it from my si.....
Programmers: no stop it. You're only an intern, don't act like you know a lot about thi....
Mentor: no you shut the fuck up. We. Are. Not. Using. Proprietary. Bullshit. For. Storing. Sensitive. Data.
Linuxxx seems to know a lot more about security and privacy than you guys so you fucking listen to what he has to say.
Windows is out of the fucking question here, am I clear?

Yeah that felt awesome.

Also that time when a mysql db in prod went bad and they didn't really know what to do. Didn't have much experience but knew how to run a repair.
He called me in and asked me to have a look.
Me: *fixed it in a few minutes* so how many visitors does this thing get, few hundred a day?
Him: few million.
Me: 😵 I'm only an intern! Why did you let me access this?!
Him: because you're the one with the most Linux knowledge here and I trust you to fix it or give a shout when you simply can't.

Lastly he asked me to help out with iptables rules. I wasn't of much help but it was fun to sit there debugging iptables shit with two seniors 😊

He always gave good feedback, knew my qualities and put them to good use and kept my motivation high.

Awesome guy!

toxic workplace; leaving

I haven't wanted to write this rant. I haven't even wanted to talk to anyone (save my gf, ofc). I've just been silently fuming.

I wrote a much longer rant going into far too much detail, but none of that is relevant, so I deleted it and wrote this shorter (believe it or not) version instead. And then added in more details because details.

------

On Tuesday, as every Tuesday, I had a conference call with the rest of the company. For various, mostly stupid reasons, the boss yelled at and insulted me for twenty minutes straight in front of everyone, telling me how i'm disorganized, forgetful, how can't manage my time, can't manage myself let alone others, how I don't have my priorities straight, etc. He told the sales team to get off the call, and then proceeded to yell and chew at me for another twenty minutes in front of the frontend contractor about basically the same things. The call was 53 minutes, and he spent 40 minutes of it telling me how terrible I've been. No exaggeration, no spin. The issues? I didn't respond to an email (it got lost in my ever-filling inbox), and I didn't push a very minor update last week (untested and straight to prod, ofc). (Side note: he's yelled at me for ~15 minutes before for being horribly disorganized and unable to keep up on Trello -- because I had a single card in the wrong column. One card, out of 60+ over two boards. Never mind that most have time estimates, project tags, details, linked to cards on his boards, columns for project/qa/released, labels for deferred, released to / rejected from qa, finished, in production, are ordered by priority, .... Yep. I'm totes disorganized.)

Anyway, I spent most of conference call writing "Go fuck yourself," "Choke on a cat and die asshole," "Shit code, low pay, and broken promises. what a prize position," etc. or flipping him off under the camera on our conference-turn-video-call (switched due to connection issues, because ofc video is more stable than audio-only in his mind).

I'm just.
so, so done.

I did nothing the rest of the day on Tuesday, and basically just played games on Wednesday. I did one small ticket -- a cert replacement since that was to expire the next day -- but the rest was just playing CrossCode. (fun game, fyi; totally recommend.)

Today? It's 3:30pm and I can't be bothered to do anything. I have an "urgent" project to finish by Monday, literally "to give [random third party sales guy] a small win". Total actual wording. I was to drop all other tasks (even the expiring cert lol) and give this guy his small win. fucking whatever. But the project deals with decent code -- it's a minor extension to the first project I did for the company (see my much earlier rants), back when I was actually applying myself and learning something (everything) new, enjoying myself, and architecting+writing my own code. So I might actually do the project, but It's been two days and I haven't even opened single file yet.

But yeah. This place is total and complete shit. Dealing with the asshole reminds me of dealing with my parents while growing up, and that's a subject I don't want to broach -- far too many toxic memories.

So, I'm quitting as soon as I find something new.
and with luck, this will be before assface hires my replacement-to-be, and who will hopefully quit as soon as s/he sees the abysmal codebase. With even more luck, the asshole king himself will get to watch his company die due to horrible mismanagement. (though ofc he'll never attribute it to himself. whatever.)

I just never want to see or think about him again.
(nor this fetid landfill of a codebase. bleh.)

With luck, this will be one of my last rants about this toxic waste dump and its king of the pile.
Fourty fucking minutes, what the fuck.

One week, and it turned out to be worse than that.

I was put on a project for a COVID-19 program in America (The CARES Act). The financial team came to us on Monday morning and said they need to give away a couple thousand dollars.

No big deal. All they wanted was a single form that people could submit with some critical info. Didn't need a login/ registration flow or anything. You could have basically used Google Forms for this project.

The project landed in my lap just before lunch on Monday morning. I was a junior in a team with a senior and another junior on standby. It was going to go live the next Monday.

The scope of the project made it seem like the one week deadline wasn't too awful. We just had to send some high priority emails to get some prod servers and app keys and we were fine.

Now is the time where I pause the rant to express to you just how fine we were decidedly **not**: we were not fine.

Tuesday rolls around and what a bad Tuesday it was. It was the first of many requirement changes. There was going to need to be a review process. Instead of the team just reading submissions from the site, they needed accept and reject buttons. They needed a way to deny people for specific reasons. Meaning the employee dashboard just got a little more complicated.

Wednesday came around and yeah, we need a registration and login flow. Yikes.

Thursday came and the couple-thousand dollars turned into a tens of millions. The amount of users we expected just blew up.

Friday, and they needed a way for users to edit their submissions and re-submit if they were rejected. And we needed to send out emails for the status of their applications.

Every day, a new meeting. Every meeting, new requirements that were devastating given our timeframe.

We put in overtime. Came in on the weekend. And by Monday, we had a form that users could submit and a registration/ login flow. No reviewer dashboard. We figured we could take in user input on time and then finish the dashboard later.

Well, financial team has some qualms. They wanted a more complicated review process. They wanted roles; managers assign to assistants. Assistants review assigned items.

The deadline that we worked so hard on whizzed by without so much as a thought, much less the funeral it deserved.

Then, they wanted multiple people to review an application before it was final. Then, they needed different landing pages for a few more departments to be able to review different steps of the applications.

Ended up going live on Friday, close to a month after that faithful Monday which disrupted everything else I was working on, effective immediately.

I don't know why, but we always go live on a Friday for some reason. It must be some sort of conspiracy to force overtime out of our managers. I'm baffled.

But I worked support after the launch.

And there's a funny story about support too: we were asked to create a "submit an issue" form. Me and the other junior worked on it on a wednesday three weeks into the project. Finished it. And the next day it was scrapped and moved to another service we already had running. Poor management like that plagued the project and worked in tandem with the dynamic and ridiculous requirements to make this project hell.

Back to support.

Phone calls give me bad anxiety. But Friday, just before lunch, I was put on the support team. Sure, we have a department that makes calls and deal with users. But they can't be trained on this program: it didn't exist just a month ago, and three days ago it worked differently (the slippery requirements never stopped).

So all of Friday and then all of Saturday and all of Monday (...) I had extended panic attacks calling hundreds of people. And the team that was calling people was only two people. We had over 400 tickets in the first two days.

And fuck me, stupid me, for doing a good job. Because I was put on the call team for **another** COVID project afterwards. I knew nothing about this project. I have hated my job recently. But I'm a junior. What am I gonna say, no?

Doot doot.

My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.

My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.

My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.

what a joke.

Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.

I told him it was going to happen again.

I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.

So stupid.

Welcome back to practiseSafeHex's new life as a manager.

Episode 2: Why automate when you can spend all day doing it by hand

This is a particularly special episode for me, as these problems are taking up so much of my time with non-sensical bullshit, that i'm delayed with everything else. Some badly require tooling or new products. Some are just unnecessary processes or annoyances that should not need to be handled by another human. So lets jump right in, in no particular order:

- Jira ... nuff said? not quite because somehow some blue moon, planets aligning, act of god style set of circumstances lined up to allow this team to somehow make Jira worse. On one hand we have a gigantic Jira project containing 7 separate sub teams, a million different labels / epics and 4.2 million possible assignees, all making sure the loading page takes as long as possible to open. But the new country we've added support for in the app gets a separate project. So we have product, backend, mobile, design, management etc on one, and mobile-country2 on another. This delightfully means a lot of duplication and copy pasting from one to the other, for literally no reason what so ever.

- Everything on Jira is found through a label. Every time something happens, a new one is created. So I need to check for "iOS", "Android", "iOS-country2", "Android-country2", "mobile-<feature>", "mobile-<feature>-issues", "mobile-<feature>-prod-issues", "mobile-<feature>-existing-issues" and "<project>-July31" ... why July31? Because some fucking moron decided to do a round of testing, and tag all the issues with the current date (despite the fact Jira does that anyway), which somehow still gets used from time to time because nobody pays attention to what they are doing. This means creating and modifying filters on a daily basis ... after spending time trying to figure out what its not in the first one.

- One of my favourite morning rituals I like to call "Jira dumpster diving". This involves me removing all the filters and reading all the tickets. Why would I do such a thing? oh remember the 9000 labels I mentioned earlier? right well its very likely that they actually won't use any of them ... or the wrong ones ... or assign to the wrong person, so I have to go find them and fix them. If I don't, i'll get yelled at, because clearly it's my fault.

- Moving on from Jira. As some of you might have seen in your companies, if you use things like TestFlight, HockeyApp, AppCenter, BuddyBuild etc. that when you release a new app version for testing, each version comes with an automated change-log, listing ticket numbers addressed ...... yeah we don't do that. No we use this shitty service, which is effectively an FTP server and a webpage, that only allows you to host the new versions. Sending out those emails is all manual ... distribution groups?? ... whats that?

- Moving back to Jira. Can't even automate the changelog with a script, because I can't even make sense of the tickets, in order to translate that to a script.

- Moving on from Jira. Me and one of the remote testers play this great game I like to call "tag team ticketing". It's so much fun. Right heres how to play, you'll need a QA and a PM.

*QA creates a ticket, and puts nothing of any use inside it, and assigns to the PM.
*PM fires it back asking for clarification.
*QA adds in what he feels is clarification (hes wrong) and assigns it back to the PM.
*PM sends detailed instructions, with examples as to what is needed and assigns it back.
*QA adds 1 of the 3 things required and assigns it back.
*PM assigns it back saying the one thing added is from the wrong day, and reminds him about the other 2 items.
*QA adds some random piece of unrelated info to the ticket instead, forgetting about the 3 things and assigns it back.

and you just continue doing this for the whole dev / release cycle hahaha. Oh you guys have no idea how much fun it is, seriously give it a go, you'll thank me later ... or kill yourselves, each to their own.

- Moving back to Jira. I decided to take an action of creating a new project for my team (the mobile team) and set it up the way we want and just ignore everything going on around us. Use proper automation, and a kanban board. Maybe only give product a slack bot interface that won't allow them to create a ticket without what we need etc. Spent 25 minutes looking for the "create new project" button before finding the link which says I need to open a ticket with support and wait ... 5 ... fucking ... long ... painful ... unnecessary ... business days.

... Heres hoping my head continues to not have a bullet hole in it by then.

Id love to talk more, but those filters ain't gonna fix themselves. So we'll have to leave it here for today. Tune in again for another episode soon.

And remember to always practiseSafeHex

Manager: How come the push to prod didn’t happen?

Dev: We told you at the scrum yesterday. To reiterate, our dev environment was crashing so it’s not safe to push to prod until that is fixed.

Manager: Ok well lets set a goal to fix that and push to prod happens today so that it guaranteed happens.

Dev: That was our goal yesterday and it definitely didn’t happen.

Manager: I AM AWARE OF THAT. The corrective action is that this time compliance with the goal is 100% ABSOLUTELY MANDATORY!!

Dev: We’ll do our best, can’t guarantee anything until we figure out what the nature of what is occurring on dev though.

Manager: NO. I AM THE BOSS. YOU WILL 100% ABSOLUTELY COMPLY WITH THIS. THAT IS AN ORDER. YOU WILL SUCCESSFULLY GET THIS UPDATE OUT TO PROD TODAY. ANYTHING LESS THAN THAT SHALL BE CONSIDERED INSUBORDINATION. I WANT STATUS UPDATES EVERY 15 MINUTES ON WHERE WE ARE AT WITH THIS.

Dev: …

Dev: Can I get you to send me that request in an email?

*Manager leaves the meeting*

// *****************************

Job search is ticking along. It’s tough going though because I currently make ~120k and the best offers I’ve received so far are all ~70k because “You only have 2 years experience so you couldn’t possibly have the skills to be worth 120k. You are are junior level developer and 70K is already overpaying for you. We can pay you more later™. No we will not give you that in writing”. Ah well, the hunt continues.

Friend of mine killed his MacBook with some Softdrink.

Just poured it all over his poor a1502.

He let it dry for a few days, it starts to work again.
Except the battery.

Goes on Amazon and buys a new battery.

New battery doesn't work either and so he tells me about it and I as stupid as I am couldn't resist the temptation to finally work on a MacBook like my "hero" Lois Rossmann does.

So turns out the board is good.
Cleaned it up and basically nothing happened to it.

So what's the deal with "los batlerias"?

The first got hit by liquid, the second had a broken connection to a cell.
That could have happened through my friend, installing it without testing it first, or at the seller, so it being a DOA battery.

Now away from the stupidity of my friend and the situation to the actual source for this rant.

Once something happens to a modern Managed battery, the Battery Management System (BMS) disconnects the voltage from the system and goes into an error state, staying there and not powering anything ever again.
For noobs, it's dead. Buy a new one.

But It can be reset, depending you know how to, and which passwords were set at the factory.
Yes, the common Texas instruments BQ20Zxx chips have default passwords, and apple seems to leav them at default.

The Usb to SMBus adaptors arrived a few days ago and I went to prod the BMS.

There is a very nice available for Windows called BE2works, that I used the demo of to go in and figure out stuff. The full version supports password cracking, the demo not.

After some time figuring out how Smart Battery Systems (SBS) "API" works, I got to actually enter the passwords into the battery to try get into manufacturer and full access mode.

Just to realise, they don't unlock the BMS.

So, to conclude, my friend bought a "new" battery that was most likely cut out of a used / dead macbook, which reports 3000mah as fully charged instead of the 6xxx mah that it should have, with 0 cycles and 0hours used.
And non default access.

This screams after those motherfuckers scaming the shit out of people on Amazon, with refurb, reset, and locked fucken batteries.

I could kill those people right now.

Last but not least,
My friend theoretically can't send it back because I opened the battery to fix the broken connection.
Though maybe, it'll get send back anyway, with some suprise in the package.

Kudos to the devrant team for epic response time!
<50min from bug report on Twitter to workaround on prod!

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Be me, new dev on a team. Taking a look through source code to get up to speed.

Dev: **thinking to self** why is there no package lock.. let me bring this up to boss man

Dev: hey boss man, you’ve got no package lock, did we forget to commit it?

Manager: no I don’t like package locks.

Dev: ...why?

Manager: they fuck up computer. The project never ran with a package lock.

Dev: ..how will you make sure that every dev has the same packages while developing?

Manager: don’t worry, I’ve done this before, we haven’t had any issues.

**couple weeks goes by**

Dev: pushes code

Manager: hey your feature is not working on my machine

Dev: it’s working on mine, and the dev servers. Let’s take a look and see

**finds out he deletes his package lock every time he does npm install, so therefore he literally has the latest of like a 50 packages with no testing**

Dev: well you see you have some packages here that updates, and have broken some of the features.

Manager: >=|, fix it.

Dev: commit a working package lock so we’re all on the same.

Manager: just set the package version to whatever works.

Dev: okay

**more weeks go by**

Manager: why are we having so many issues between devs, why are things working on some computers and not others??? We can’t be having this it’s wasting time.

Dev: **takes a look at everyone’s packages** we all have different packages.

Manager: that’s it, no one can use Mac computers. You must use these windows computers, and you must install npm v6.0 and node v15.11. Everyone must have the same system and software install to guarantee we’re all on the same page

Dev: so can we also commit package lock so we’re all having the same packages as well?

Manager: No, package locks don’t work.

**few days go by**

Manager: GUYS WHY IS THE CODE DEPLOYING TO PRODUCTION NOT WORKING. IT WAS WORKING IN DEV

DEV: **looks at packages**, when the project was built on dev on 9/1 package x was on version 1.1, when it was approved and moved to prod on 9/3 package x was now on version 1.2 which was a change that broke our code.

Manager: CHANGE THE DEPLOYMENT SCRIPTS THEN. MAKE PROD RSYNC NODE_MODULES WITH DEV

Dev: okay

Manager: just trust me, I’ve been doing this for years

Who the fuck put this man in charge.

Story #1: So I took a month of parental leave. And was planning to extend it a little longer to deal with my final exams. I was planning to spend lots of quality time with my wife and newborn son. Little did I know... It turns out that out of 5 OoO weeks I was looking forward I actually had 3 at most. The rest I've spent working remotely as I was insisted to deploy a brand new and poorly tested feature to PROD 2 days before my paternity leave. So I spent 2 weeks debugging things in PROD. Remotely. Needless to say that did suck.

Story #2: After story #1 I've learnt my lesson. This summer I took 3 weeks annual leave to renovate my apartment. I asked to not to be disturbed unless there's an emergency. And an emergency it was. One of our app users had a planned hi-load batch job lasting for 2-3 months. Hundreds of thousands of items had to be created and processed. It turns out the _processing_ algo had some flaws and was acting out. I was called out and asked to assist. I knew this sort of debugging is going to take a lot of my time so this time I put my conditions on the table: I will assist but I'll extend my leave by 1.5 the time I spend working now. They took the deal. Instead of 3 weeks I had 5 weeks of vacation!

I don't care that much about my salary. I prefer to exchange it for my time off hence I didn't ask for compensations.

Bottom line: NEVER EVER underestimate or undersell your time and effort. You are a valuable asset and if the team/client needs you on your day off -- make it count. Your time off is YOUR time. Never forget it.

Me passing time on the weekend

Random call from unknown number

Turns out it's the manager

M: hey , how is your weekend going ...

Me: nothing much ... Whatsup ?

M : yeah well , we wanted to push some minor adhoc fixes as some clients wanted it urgently

The Devops folks need developer support . Can you pitch in and monitor

Me : I'm not aware of what changes are going , i don't think i can provide support

M : don't worry it's minor changes , it's already tested in pre prod , you just need to be on call for 30 mins

Me : ugh okay .. guess 1 hr won't hurt

M: thanks 👍🏽

Me: *logs in

*Notices the last merged PR
+ 400 lines , implemented by junior dev and merged by manager

*Wait , how is this a *minor* release...

*Release got triggered already and the CI CD pipeline is in progress

*5 mins later

*Pipeline fails , devops sends email - test coverage below 50%

Manager immediately pitches in ...

M: hey , i see test coverage is down , can you increase it ?

Me: and how do u suppose I do that ?

M : well it's simple just write UTC for the missing lines ... Will it take time ?

Me : * ah shit here we go again

Yeah it will take time , there are around 400 lines , I am not aware of this component all together

Can you ask junior dev to pitch in and write the UTC for this

*Actually junior dev is out on a vacation with his girlfriend

M : well he's out for the weekend , but
as a senior dev , i expect you to have holistic understanding of the codebase and not give excuses ,
this is a priority fix which client are demanding we need this released ASAP

Me : * wait wat ?

---

I ended up being online for next 3 hours figuring out the code change and bumping up the UTC 🤦🏾

A tip to tech folks from my personal xp.

If you fuck up and make an impacting mistake in your company, like taking PROD down, noone is going to fire you on spot. Assign some more mandatory trainings - maybe. So you'd be more careful next time.

See, it's not worth getting rid of someone who made a mistake. You should be seated down and insisted to fix it. If you don't - then they might consider firing it. If you do fix it [with help or alone] - you become a more valuable asset to the company as you prove you are responsible for your actions and you take it seriously. You show that you can clean up your own shit and you don't need a babysiter next to you.

If you simply make a mistake and they replace you with someone else, that someone else is likely to be unaware of your mistakes and is doomed to repeat them. It's just bad for business.

Ofc if you making mistakes becomes a tendency rather than an exception, it's also a red flag for the business.

Don't get too laxed! And always answer for your shit. Never hide a fuckup - always alarm about it asap so that corrective actions could be taken by respective organs of the company while you are fixing it.
Come up with an action plan, announce it. Estimate resources you need [like help from others] - announce that too. Update concerned parties every half an hour or so about the status. If you find you need anything else while fixing it or you come across some blockers/delays/change of impact - always announce asap. Do avoid false alarms and disinformation.

// inspired by someone's rant today

Today is sprint demo day. As usual I'm only half paying attention since being a Platform Engineer, my work is always technically being "demoed" (shit's running ain't it? There you go, enjoy the EC2 instances.)

One team presents a new thing they built. I'm still half paying attention, half playing Rocket League on another monitor.

Then someone says

"We're storing in prod-db-3"

They have my curiosity.

"Storing x amount of data at y rate"

They now have my attention. I speak up "Do you have a plan to drop data after a certain period of time?"

They don't. I reply "Okay, then your new feature only has about 2 months to live before you exhaust the disk on prod-db-3 and we need to add more storage"

I am asked if we can add more storage preemptively.

"Sure, I say." I then direct my attention to the VP "{VP} I'll make the change request to approve the spend for additional volume on prod-db-3"

VP immediately balks and asks why this wasn't considered before. I calmly reply "I'm not sure. This is the first time I'm learning of this new feature even coming to life. Had anyone consulted with the Platform team we'd have made sure the storage availability was there."

VP asks product guy what happened.

"We didn't think we'd need platform resources for this so we never reached out for anything".

I calmly mute myself, turn my camera back off and go back to Rocket League as the VP goes off about planning and collaboration.

"CT we'll reach out to you next week about getting this all done"

*unmute, camera stays off* "Sounds good" *clears ball*

I didn't scream.. just told him to jump off of terrace..
What ticked me?! He was a support guy..slowest mofo ever..
I was in the middle of fixing major fuckup on prod, when our VPN to client disconnected. I rushed over to support to ask if it is 'just' an expired session (which he was in charge of renewing but constantly fucked up) or if there is some other problem, so I know how to proceed..do I need to contact our sysadmins, client's support guys etc..
He
started
to
slooooooooowly
explain
I
am
not
the
only
one
with
VPN
problems
...
Was that what I asked you?! // he had an annoying habit of slooooowly talking and explaining unrelated things & personal stuff that bothered him & most of the times he chose the most time sensitive period to drone off..
So I cut him of saying, that others were probably not 'tinkering' with production and that I need this back ASAP, so if he could tell me when the session will be renewed or if there is something else problematic..
He said he will check..I didn't move.. he looked at me insurprise, you want me to check *NOW*?! Yeah, it's urgent.. He proceeded very very veeeery slooooowly, taking the support phone../* he was even eating sandwich during that, so only one hand free, typing one letter at a min */
I was finaly notified that the session expired and that he will fix it soon (meaning in 15-20mins o.O which should not take him more than 5).. and was like 'can I do sth else for you'?! Yeah, do the backflip.. you know the rest..

When managers look at my code, it’s shit, it’s over complicated, it’s overly difficult to read, it took too long, it’s too much for a simple ticket, i handled too many edge cases, we’ll never need most of it, why did I bother making it extensible when it’ll never need to change, how dare I use “unless”, why did I bother writing all these comments, why did I update the documentation that nobody reads because it’s outdated, etc. They say I should be more like the legendary devs and push janky code quickly, and complain that I don’t have any flops (problems in prod) like those are a good thing.

When my coworkers look at my code, they say it’s clean, amazingly easy to read, a monster feature that’s somehow still a joy to review and work on, it makes their lives easier, that it does exactly what it should in all cases, that they learned something from reading it, and thank me for the comments and documentation. And marvel that I finished it so well in so little time.

Am I bragging? Not intentionally; I’ve heard these things repeatedly since I started here, and the contrast between the above is so stark.

In reality, the managers are just idiots who were promoted far above their competence, and make everything worse. (Gee, who woulda thought?) It’s just so frustrating.

One of the biggest reality checks you will run into when starting your first dev related job - and which they don't teach you about in school - is that a lot of the time will be spent working with other people's code, and rewriting it into "your own" is rarely an option.

You might be super into making things, but not everyone manages to maintain that same spark while taking over a 15 year old project with fundamental issues that have to be triaged "for now" because you need a hotfix on this other specific thing out in prod before lunch.

There are no gods now. They left the company years ago and nobody knows why they used the windows registry as a user repo.

CEO: if we would not give new features, clients would be bored and would not pay for tool.
me: but don't you think we should fix buggy old code, that would reduce effort and time that we daily invest in prod bugs?

CEO: I'm not saying we should not fix them but we should maintain the balance which is 80-20. 80% of our work would include adding new features.
😑
Next day in morning receives email:
There is a production issue, fix it asap.

😬

Really just an average week.
Just feel I need a bit of venting. (:

@meet: (monday)
- mgr: we need video transcoding and VOD ASAP.
- dev: on what server? It's expensive, especially without a GPU.
- mgr: prod is beefy. Put it there.
- dev: everything else is gonna crawl then.
- mgr: you have till the end of this week.

@demo (Friday)
- dev: k, it's ready.
- mgr: Why is everything slow??!
- dev: transcoding. Expensive.
- mgr: Why do we transcode? Never said I wanted transcode!
Can't we upload to YT?
- dev: ...yes. But will then each customer that wants VOD will need to setup YT studio and provide an endpoint and stream key.
- mgr: OK. But we're now behind schedule because of this and the customers will not be pleased.
- dev: oh, didn't know we're into gaming.
- mgr: ???
- dev: nvm, see you Monday.

...
Later Friday evening
...
*ding* mgr has added 5 new tasks to your list.
*ding* mgr subtracted 30 points from you.
reason: deadline over due.

Ya ya, the usual shenanigans.
Time to mute for the weekend.

Got an email earlier this week. It went something like this:

"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"

Um, wait, excuse me, WHAT?

1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.

I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.

Gather around folks, I'll paint you a nice picture based on a true story, back from my sysadmin days. Listen up.
It's about HP and their Solaris 5.4/6 support.

- Yet another Prod Solaris dinosaur crashed
- Connected to console, found a dead system disk; for some reason it was not booting on the remaining redundant disk...
- Logged an HP vendor case. Sev1. SLA for response is 30 minutes, SLA for a fix is <24 hours
- It took them 2 days to respond to our Prod server outage due to failed system disks (responses "we are looking into this" do not count)
- it took another day for them to find an engineer who could attend the server in the DC
- The field engineer came to the DC 4 hours before the agreed time, so he had to wait (DC was 4-5 hours of driving away from HP centre)
- Turns out, he came to the wrong datacentre and was not let in even when the time came

- We had to reschedule for two days later. Prod is still down
- The engg came to the DC on time. He confirmed he had the FRU on him. Looks promising
- He entered the Hall
- He replaced the disk on the Solaris server
- It was the wrong disk he replaced. So now the server is beyond rebuild. It has to be built anew... but only after he comes back and replaces the actually faulty disk.
- He replaced that disk on the wrong Solaris server

I just love it when our clients decide to make a clone of live production server..then put it immediately online..and don't tell anyone about this.. and then start bitchin how data gets doubled all of a sudden..

Yeah, no shit sherlock.. you have two prod servers for 'hot swapping' and some services may only be running on one at a time.. You even have a manual on how to switch primary to secondary (turn off services on primary first, then turn them on on secondary and all)..or in case primary actually dies, just turn on services on secondary and you're good to go, right?

So how do ya think cloning the one with running services and putting the clone immediately online will work out?! 🤔

God, I thought it was common sense to not do that..but here I am, bitchin about how people fail to RTFM.. :/ or use brain..fuck..

Highlights from my week:

Prod access: Needed it for my last four tickets; just got it approved this week. No longer need it (urgently, anyway). During setup, sysops didn’t sync accounts, and didn’t know how. Left me to figure out the urls on my own. MFA not working.

Work phone: Discovered its MFA is tied to another coworker’s prod credentials. Security just made it work for both instead of fixing it.

My merchant communication ticket: I discovered sysops typo’d my cronjob so my feature hasn’t run since its release, and therefore never alerted merchants. They didn’t want to fix it outside of a standard release. Some yelling convinced them to do it anyway.

AWS ticket: wow I seriously don’t give a crap. Most boring ticket I have ever worked on. Also, the AWS guy said the project might not even be possible, so. Weee, great use of my time.

“Tiny, easy-peasy ticket”: Sounds easy (change a link based on record type). Impossible to test locally, or even view; requires environments I can’t access or deploy to. Specs don’t cover the record type, nor support creating them. Found and patched it anyway.

Completed work: Four of my tickets (two high-priority) have been sitting in code review for over a month now.

Prod release: Release team #2 didn’t release and didn’t bother telling anyone; Release team #1 tried releasing tickets that relied upon it. Good times were had.

QA: Begs for service status page; VP of engineering scoffs at it and says its practically impossible to build. I volunteered. QA cheered; VP ignored me.

Retro: Oops! Scrum master didn’t show up.

Coworker demo: dogshit code that works 1 out of 15 times; didn’t consider UX or user preferences. Today is code-freeze too, so it’s getting released like this. (Feature is using an AI service to rearrange menu options by usage and time of day…)

Micromanager response: “The UX doesn’t matter; our consumers want AI-driven models, and we can say we have delivered on that. It works, and that’s what matters. Good job on delivering!”

Yep.
So, how’s your week going?

Long time ago, back in a day of Microsoft Office 95 and 97, I was contracted to integrate a simple API for a payment service provider.

They've sent me the spec, I read it, it was simple enough: 1. payment OK, 2. payment FAILED. Few hours later the test environment was up and happy crediting and debiting fake accounts. Then came the push to prod.

I worked with two other guys, we shut down the servers, made a backup, connected new provider. All looked perfectly fine. First customers were paying, first shops were sending their products... Until two days later it turned out the money isn't coming through even though all we are getting from the API is "1" after "1"! I shut it off. We had 7 conference calls, 2 meetings, 3 days of trying and failing. Finally, by a mere luck, I found out what's what.

You see, Microsoft, when you invent your own file format, it's really nice to make it consistent between versions... So that the punctuation made in Microsoft Word 97 that was supposed to start from "0" didn't start from "1" when you open the file in Microsoft Word 95.

Also, if you're a moron who edits documentation in Microsoft Word, at least export it to a fucking PDF before sending out. Please.

# PROD

* 10 app instances running
* 1 instance starts burning up 100% cpu
* we ask for a Thread Dump (stack traces)
* we get a TD taken after they manually restarted the instance
* they: "Please investigate. We need this fixed ASAP"
* .....

EVERY FUCKING TIME!!! Not once in recent years have they managed to take a TD correctly. What kind of a retarded monkey do you have to be for this to not sink in for YEARS!

Who tf put those idiot monkeys there in the first place...

Is prompt engineering going to be the next "big thing"?

Very related fun fact: SQL was meant for business people so they can quickly generate reports w/out needing to rely on programmers. (When was the last time your CEO did a DB query on prod I wonder?!)

My manager is instructing my team to add a feature that can only be enabled for users by running an update script in the database.

When I argued that it's not really "complete" if it can't be turned on without someone going into the production database, I was told that not only is it complete, but they plan to have our non-technical customer service enable it for customers if the customer requests it...

Apparently giving everyone and their brother write access to prod is a good idea, but implementing a checkbox is a "waste of time and would cost too much money".

Probably going to float my resume... :-p

I was asked to look into a site I haven't actively developed since about 3-4 years. It should be a simple side-gig.
I was told this site has been actively developed by the person who came after me, and this person had a few other people help out as well.
The most daunting task in my head was to go through their changes and see why stuff is broken (I was told functionality had been removed, things were changed for the worse, etc etc).
I ssh into the machine and it works. For SOME reason I still have access, which is a good thing since there's literally nobody to ask for access at the moment.
I cd into the project, do a git remote get-url origin to see if they've changed the repo location. Doesn't work. There is no origin. It's "upstream" now. Ok, no biggie. git remote get-url upstream. Repo is still there. Good.
Just to check, see if there's anything untracked with git status. Nothing. Good.
What was the last thing that was worked on? git log --all --decorate --oneline --graph. Wait... Something about the commit message seems familiar. git log. .... This is *my* last commit message. The hell?
I open the repo in the browser, login with some credentials my browser had saved (again, good because I have no clue about the password). Repo hasn't gotten a commit since mine. That can't be right.
Check branches. Oh....Like a dozen new branches. Lots of commits with text that is really not helpful at all. Looks like they were trying to set up a pipeline and testing it out over and over again.
A lot of other changes including the deletion of a database config and schema changes. 0 tests. Doesn't seem like these changes were ever in production.
...
At least I don't have to rack my head trying to understand someone else's code but.... I might just have to throw everything that was done into the garbage. I'm not gonna be the one to push all these changes I don't know about to prod and see what breaks and what doesn't break
.
I feel bad for whoever worked on the codebase after me, because all their changes are now just a waste of time and space that will never be used.

There is a company providing a very speciffic service. And it has a core application for that svc, supported by a core app team.
That company also has other services, which are derivations of the core one. So every svc depends on core.

Now that we're clear on that... I was working in a team of one of the subservices. We very strongly depended on core. In fact, our svc was useless if integration w/ core broke down.

The core team had an annoying habbit. They refused to version their webservices and they LOVED to push api updates w/o any warnings. Our prod, test, other envs used to fail bcz of core api changes quite often. Mgmt, IT head was aware of the problem and customers' complaints as well.

So as a result, once core api changes we're all in a panic mode: all prior priorities are lowered and revival of prod is to be our main focus. Core api is not docummented, the changes are not clear, so we have to reverse engineer the shit out of it. We manage to patch our prod up w/ hotfixes, but now we have tech debt. While working on the debt, core api changed again, in test env. Mgmt pushes debt back and reallocates us to hotfix test. Hotfix is 80% done when another core api breaks. Now mgmt asks us to drop wtv we're working on and fix that new break. By the time we're to deploy the hotfix, another api breaks in another env. The mgmt..... You get the picture :)

2 years go by, nothing has changed so far.

My most satisfying bug fix?

I found a core concurrency issue in this gnarly homegrown ORM and reported it to the lead devs, who (very defensively, having written the damn thing) argued that it would never pop up in a prod environment and I was stupid for even bringing it up. Theoretically, this bug could cause pretty much every foreign key to be assigned to the wrong parent, but only if multiple instances of the application were open/running at once. They were so certain it would never happen on live that they explicitly instructed me not to fix it. After all, this bug had been active for many years on a previous project and nobody complained.

Problem was, that previous project was something that only a single user had open most of the time (think: a manager). The new project was something that would be used by multiple people at the same time (think: all the employees). Once we released this new-project-with-old-orm, it didn't take long at all for our customers to start complaining.

After that, they let me fix the bug. :)

Me: “Hey boss, you assigned these things to me that I’m not qualified for and have no experience in. We should really hire someone with the specialized skills in this”

Boss “I agree. It’s a role I desperately think we should have hired for a long time ago”

Me “Ok so about these tickets the-“

Boss “I need you to write up a justification for this role, what kind of work the person would be doing and what budget implications we will incur”

Me “You’re asking me to write a job description for a class of work I’ve already admitted I have no experience or qualifications doing MYSELF?”

Boss “Correct”

Me “and I’m still responsible in the meantime for getting these other tickets done still aren’t I?”

Boss “Yes”

Me “Very well. I’ll email you a recap of this discussion then so we can come back to it later when we start hiring for the role”

(and so my ass is sufficiently covered when I inevitably bring down prod and people start asking why I broke prod)

There's this huuuge project I was a part of for half a year. I was kicked off along with a few dozens of other devs (>60% of total manpower) a while ago for particular company reasons.

Now the remaining devs are oh-so-enjoying their time there..

1. workload has not changed
2. deadlines have not changed
3. no one will have Christmas-NewYear vacation
4. a new k8s-based infra is scheduled to roll out to PROD on Dec 23 (k8s is still far from ready - might need a few more months)

The most fun part is that it's not client's mgmt who has decided for #4 -- it's our own....

Boys.. Girls.. Save yourselves.

I've been staffed on a old ongoing project, first day.
0. Compatibility has to be guaranteed down till IE9... ppf.
1. Front end made in XHTML+JS(jQuery)... bah, ok.
2. XHTML+JS is actually generated by PHP5.4, not a line is actually statically served... beh, funny, ok.
3. PHP files are the output of an XSLT transform of a bunch of XMLs... meh, seriously? Oooook.
4. XMLs are the product of the serialisation of a truck of stateful JavaEE6 DTOs populated magically (undocumented) with data coming from a SQL DB... WTF mode!!!
5. Session logics lives within PHP-land at point 2, front end makes ajax calls here that propagates to another WS out of our control that triggers -somehow- (undocumented) our Java backend at point 4 to generate new XMLs and then reach front end again. Kill me now.

Boss: look... it's too slow for the client, it's too heavy on our servers: fix it. Ah, and we sold 85% test coverage by October. You're the man for the job. (I'm a Node.js fullstacker and right now there's not even a testing scaffold, ofc).
Me: prod is on Linux or Windows?
Boss: RHEL7.
Me: rm -rf / as root. Done.
Boss: I know I know...
Me: ...

I think time has come...

Have a couple I want to air today.

First was at my first gig as a dev, 4-5 months out of school. I was the only dev at a startup where the owner was a computer illiterate psycopath with serious temper tantrums. We're talking slamming doors, shouting at you while you are on the phone with customers, the works...

Anyways, what happened was that we needed to do an update in our database to correct some data on a few order lines regarding a specific product. Guess who forgot the fucking where-clause... Did I mention this boss was a cheap ass, dollar stupid, penny wise asshole that refused to have anything but the cheapest hosting? No backups, no test/dev/staging environment, no local copies... Yeah, live devving in prod, fucking all customers with a missing semi-colon (or where clause).

Amazingly, his sheer incompetance saved my ass, because even if I explained it, he didn't get it, and just wanted it fixed as best we could.

The second time was at a different company where we were delivering managed network services for a few municipalities. I was working netops at that time, mostly Cisco branded stuff, from Voice-over-IP and wifi to switches and some routing.

One day I was rolling out a new wireless network, and had to add the VLAN to the core switch on the correct port. VLAN's, for those who don't know, are virtual networks you can use to run several separated networks on the same cable.

To add a VLAN on a Cisco switch one uses the command:

switchport access vlan add XYZ

My mistake was omitting the 'add', which Cisco switches happily accept without warning. That command however can be quite disruptive as it replaces all of the excisting VLAN's with the new one.

Not a big deal on a distribution switch supplying an office floor or something, but on a fucking core switch in the datacenter this meant 20K user had no internet, no access to the applications in the DS, no access to Active Directory etc. Oh and my remote access to that switch also went down the drain...

Luckily a colleague of mine was on site with a console cable and access to config backups. Shit was over within 15 minutes. My boss at that time was thankfully a pragmatic guy who just responded "Well, at least you won't make that mistake again" when we debriefed him after the dust settled.

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

School gave me 3 DigitalOcean droplets to try out Kubernetes in the cloud, awesome!

Wrote an Ansible script to not only simply install docker and add users but also add kubernetes, nice!

Oh wait, error?! Well I should've known this wasn't going to be easy... ah well no problem. Let's see... Ansible is cryptic as always, it can't connect to the API server? Is it even running?

Let's ssh to the master, ah nothing is running, great. Let's try out kubeadm init and see what happens, oh gosh, my Docker version has not been validated! No problem, let's just downgrade!

How do I do that? Oh I know, change the version in the role! Wait that version doesn't exit? Let's travel to Docker's website and see what versions exist of docker-ce, oh I see, it needs a subversion, no problem.

Oh that errors too? Wait then what... Oh I need a ~ and a ubuntu and a 0 somewhere, my mistake!

Let's run it again! Fails!

Same ssh process, oh wait...

Oh god no...

Kubernetes requires 2 cores and these things only have 1...

Welp, time to ask the teachers to resize my droplet by a small amount tomorrow, hopefully I'll get a new error!

----------------------------------------------

My adventure so far with Kubernetes. I'm not installing it for any serious/prod reason, just for educational purposes. K8s seems like 'endgame' to me, like one of the 'big guys' that big enterprises use so I'm eager to throw stuff at a droplet and see what happens.

Going further down the rabbit hole tomorrow!

Wish me luck :3

(And yes, I could've figured this all out beforehand with documentation, but this is more fun in my opinion)

So I joined this financial institution back in Nov. Selling themselves as looking for a developer to code micro-services for a Spring based project and deploying on Cloud. I packed my stuff, drove and moved to the big city 3500 km away. New start in life I thought!
Turns out that micro-services code is an old outdated 20 year old JBoss code, that was ported over to Spring 10 years ago, then let to rot and fester into a giant undocumented Spaghetti code. Microservices? Forget about that. And whats worse? This code is responsible for processing thousands of transactions every month and is currently deployed in PROD. Now its your responsibility and now you have to get new features complied on the damn thing. Whats even worse? They made 4 replicas of that project with different functionalities and now you're responsible for all. Ma'am, this project needs serious refactoring, if not a total redesign/build. Nope! Not doing this! Now go work at it.
It took me 2-3 months just to wrap my mind around this thing and implement some form of working unit tests. I have to work on all that code base by myself and deliver all by myself! naturally, I was delayed in my delivery but I finally managed to deliver.
Time for relief I thought! I wont be looking at this for a while. So they assign me the next project: Automate environment sync between PROD and QA server that is manually done so far. Easy beans right? And surely enough, the automation process is simple and straightforward...except it isnt! Why? Because I am not allowed access to the user Ids and 3rd party software used in the sync process. Database and Data WareHouse data manipulation part is same story too. I ask for access and I get denied over and over again. I try to think of workarounds and I managed to do two using jenkins pipeline and local scripts. But those processes that need 3rd party software access? I cannot do anything! How am I supposed to automate job schedule import on autosys when I DONT HAVE ACCESS!! But noo! I must think of plan B! There is no plan B! Rather than thinking of workarounds, how about getting your access privileges right and get it right the first time!!
They pay relatively well but damn, you will lose your sanity as a programmer.
God, oh god, please bless me with a better job soon so I can escape this programming hell hole.
I will never work in finance again. I don't recommend it, unless you're on the tail end of your career and you want something stable & don't give a damn about proper software engineering principles anymore.

On Skype.
[tldr: #muhPrivacy;]

You know, people hiring via Skype.
Gaming, seeing family or having long range relationships.

It's become a decent tool.

Then there is the Skype employee.
Opening a court case because in his work time, evaluating Skype calls - ON FUCKING OBSERVING SKYPING PEOPLE - he has to look at too much flesh (as in porn) for his salary level.
Like : the payment category states that you gotta be classed like 1,2 salary categories higher for such work.

So the first instance did not recognise the employees case, because they said its a state thing, or even higher.
Later instance evaluated the employee was right and decreed Microsoft / the NSA (whomever direct employees they are) to properly categorize their employees.
Therefore cost relatively exploded and an algorithm to detect nudity was built.
Wich is operational way earlier than Skypes TOS renewal mid 2018.
That also bans bad language and auto bans given accounts.

Talking about social credit..
in PROC (or prod, as they're known).
And btw: complaining about Google while posting Christmas gatherings on Instagram.. You get what I mean.

Honestly, I don't recall the sources. It's been a while.

I'd really appreciate a little compendium of this for historical reasons.

They will ask: what has brought us here? What is everyone an ultimate right/left/center/agnostic/religious fascist?

And we'll have it on paper. Or papyrus,.. even stone. As I don't know how far mighty people will go for their fortune.

Last time I was here I was in college. 🤓

Hello world, again 🫂
I graduated and now I have a job as a developer for two years now.

So much ranting has been happening and I'm excited to tell you all about it.

PS: I already had my first canonical event (Prod deployment at 5 pm ... Was not fun ... Wasn't me if you're asking)

On my project the customer has re-signed into a contract several times when they have budget to continue work. The first time they got us to build the system was a huge success story because the team was assembled quickly and we did rapid development. Initialize repo to prod in 1.5 months. The customer asked for the same dev team. Strong dev team, a PM that doesn't take shit, and pure agile. Lets call her don't-take-shit PM.

When the customer re-signed the executive decided that she didn't like don't-take-shit PM. So the project manager gets replaced by play-by-the-rules PM who will comply with stupid requests and micromanagement. He isn't a bad PM but he tries to make everyone happy. The amount of management types executive installs on the project is massive, and development team is cut down in major ways. Customer and executive shit rolls down to the development team and we can't get anything done. The customer starts to lose faith because we can't get traction. They start demanding traditional waterfall/SDLC docs. Which causes more delay in the project.

So the executive decides that the PM can take a fall for it to save face for the company. She moves play-by-the-rules PM to another project. He starts handover to a new PM that has a history of being her pushover. The customer hadn't seen him yet so now we have push-over PM.

Play-by-the-rules PM is finally out of the project and instead of moving to a different account the company decides to "lay him off because there is no work". So basically they made him take the fall for the failure while promising reassignment, and instead let him go. This is so unfair..

Meeting with push-over PM yesterday and he shows us his plan. Identical to play-by-the-rules PM's plan that got him axed.We point that out and show him the docs that were made for it. His face clearly communicates "OH SHIT WHAT DID I SIGN UP FOR?"

Central team: No, your team must be doing something wrong. Our pipeline is super-configurable and works for any situation! You just have to read the docs!

Me: Where are the docs?

Central team: Uhh, well, umm... we'll hook you up with a CI/CD coach!

Me: Okay, cool. In the mean time, can you point me at the repo where all the base scripts are?

Central team: Sure, it's here.

Me, some weeks later: Yeah, uhh, the coach can't seem to figure out how to make our Prod deployment work either.

Central team: That's impossible! It's so easy and completely configurable!

Me: Well, okay... but, here's the thing: your pipeline IS pretty "configurable", in the sense that you look for A LOT of variables...

Central team: See! We told you!

Me: ...none of which are actually documented, so they're just about useless to me...

Central team: But, but the coach...

Me: ...couldn't make heads or taisl of it either despite him literally being ON YOUR TEAM...

Central team: Then your project must just be architected wrong!

Me: Well, we're not perfect, so could be...

Central team: Right!

Me: ...but I think it's far more likely that the scripts... you know, the ACTUAL Python scripts the pipeline executes... while it took me DAYS to get through all your levels of abstraction and indirection and, well, BULLSHIT... it turns out they are incredibly NOT flexible. They do one thing, all the time, basically disregarding any flexibility in the pipeline. So, yeah, I'm thinking this is probably one of this "it's you, not me" deals.

Central team: Waaaaahhhhhhhh!!!!!!

Yesterday night, pushed code that work normally to prod server, website down, internal server error, too many connection to MySQL server, tried to fix it for 4 hours, nothing to do, removed the new code, still the same problem, in my head, I told myself that I'm not good at programming (not the first time), send an email to the host, they tell me the problem is from them and they fixed it. And now I know I'm not bad enough.

Me: Why are we spending time building reports for Support? AFAIK they never read or use them.

Boss: Seems they expect you to do it.

Me: Then what exactly are they supposed to be doing? All the issues seem to just escalate back to us.... We should just make sure issues never get into PROD

Boss: I agree, they're always firefighting... we gave them more funding so eventually they should catch up **Me: I highly doubt it, you should just stop hiring monkeys** esp. if we prevent PROD issues.

Me: Yea... we should prevent production issues because someone always has to pay off the (technical) debt and interest rates in PROD are very high

Few years ago as a junior android dev with couple years of self taught experience of working in startups I submitted a simple android app assignment for a junior android dev role. Assignment had only like 8 requirements so I followed them to the letter. That didn't end well.

App was simple just 3 screens. Login screen with username and password input fields, login button.

Had to call a login endpoint after login button was clicked, redirecting to home screen, calling items endpoint, displaying a list of items and when an item was clicked passing item data and redirect to item details screen.

Needless to say big swinging dick senior was not impressed. UI was not perfect, I forgot to display a loading animation when fetching data, didnt handle back button properly.

I agreed with some points but other comments were clearly just nitpicking: his preferred variable naming conventions, his opinions on architecture that was not up to his standard (official google arch at the time was not up to his standard).

He also was mad that app wasn't prepared for release to googleplay (another out of the ass requirement). Like I would prepare a 3 screen app for prod release that he will forget ever existed after 20min of his review.

Lots more of nitpicking, encapsulation this encapsulation that, omg now hes shocked that there are a few warnings after the project is built.

Regardless my self confidence was destroyed at that point and after few more negative experiences I dropped android dev alltogether for a couple years and switched to game dev.

After game dev ran its course I went back to android dev and found a supportive place where I could grow.

Looking back, they were actually hiring atleast a mid level for a junior position but I was grilled as a senior. The guy literally didnt wrote any single positive thing in that review about my code even tho my senior peers said my project was decent back then, its just that I didnt handle a few edge cases and that's all.

I looked up the guy in linkedin, turns out hes a uni dropout who posts all books that he red about software dev in his education section of his linkedin profile. Found a bunch of other narcissistic stuff on his profile. Guy was a fucking idiot. Even if I worked under him it would have probably sucked.

Learned some important lessons I guess. Always get a second, 3rd and 4th opinion and dont take criticism too seriously. Always check what kind of person is providing feedback.

IMHO technical dept is kind of like smoking cigarettes for some decades.

You were told that shit will hit the fan but you do not take proper action. And one day you'll realize that you fucked up (or not, also seen that).

Worked for a company in IT, where we maintained an ERP which was "in progress" for over a decade. The basic implementation was done by people with zero technical understanding. To clarify: not self coded. Software was bought. We are talking about integrating the system.

Therefore, the foundation was like a wet noodle. When I joined that company, I told them that they need to address that. I told them that things will get slower and slower and that shit will hit the fan if no proper actions taken.
Even made a list with flaws I found. With potential risk and actions to take, that could then be measured.

At that time, five people worked in said department (including me).

People did not want to listen. "Would be too expensive to rewrite stuff".

Nothing has changed about the wet noodle, but I tried to fix as many things in a working system as I could. Felt like heart surgery, because changes got implemented and "tested" in prod. No version control, no documentation, everyone implemented things like they felt (no guidelines for consistency).
A lot of small fuckups that summed up over the years.

I left the company after two years because I had the chance to land a job as a dev.

Been around two years now since I left. Now 9 people work in that department with around the same efficiency as us 5 people back then.

The new employees struggle to be productive, because things are just implemented poorly and not maintainable anymore.

Had some dialogs with them some time ago. Everything I told them would happen, actually happened. What a suprise :-|
I will not go into too much detail about all the shit that's going on there, as it would be just too much (and my morning coffe is almost finished).

I think that we all know the difference between "not beautiful, but does the job" and "oh, that will backfire - badly". And I wish that my communication skills increase so that people start listening in future.

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

Holy shit, my first freelance project will be field-tested for the first time tomorrow!

After a few more rounds of fixing bugs, some serious improvements, and some new features additions, I can actually say that I'm proud of the code I've written! It's not perfect, but I definitely like the way it works - AND IT ACTUALLY WORKS!

Knock on wood, hopefully it won't shit the bed tomorrow.

I hate having to deal with our IT service desk. Every time it takes enormous energy to get to the right people and make them understand that no, you are not an idiot, but you actually have a technical issue.

Sure thing they do have a few competent nice folks there too I've gotten to know over time and they indeed have to deal with a ton of dumb non-tech savvy idiots on a daily basis. However, if my job title mentions "software" and "engineer" they should at least assume I'm an idiot in tech. Or something. Every single time I need to open a ticket, even for the simplest "add x to env y", I need to quadruple check that the subject line is moron-friendly because otherwise they would take every chance to respond "nah we can't do that", "that's not us", or "sry that's not allowed". And then I would need to respond, "yes you do:) your slightly more competent colleague just did this for us 2 weeks ago".

Now you might imagine this is on even another level when the problem is complex.

One of our internal apps has been failing because one of the internal APIs managed by a service desk team responds a 500 status code randomly but only when called with a specific internal account managed by another service desk team.
(when I say "managed by", that doesn't mean they maintain it, it just mean they are the only ones who would have access to change something)

Yesterday I spent over a fucking hour writing a super precise essay detailing the issue, proving a million times it's not on our end and that they need to fix it. Now here is an insight to what beautiful "IT service" our service desk provides:

1) ticket gets assigned to a "Connectivity Engineer" lady

2) few hours later she responds and asks me to give her the app and environment IDs and grant her access to those
(naturally everything in my email was ignored including these two IDs)

3) since the app needs to be in prod for the issue, I make a copy isolating the failing part and grant her access to the original "for reference" and the copy to play with

4) few hours later I get an email from the env that some guy called P made changes to the actual app, no changes to the copy
(maybe they immediately fixed the app even though I asked them to only touch the copy)
I also check the env and the live app had been shared with another 2 people giving them editing rights:)

5) another few hours pass and the lady responds that she had been chatting with P (no mention of who tf that guy is) and that P has a suggestion that might work and I should test it, "please see screen shot" for details:

These motherfuckers sent me a fucking screenshot of the env config file where "P has edited a few parameters" that might help. The screenshot had a 16 line part of the config json with a bunch of IDs and Base64 params which HE EDITED LOCALLY.

Again, because I needed a few iterations to realise what I've just witnessed:
These idiots modified some things in the main app (not the copy) for hours. Then came to the conclusion that the config needs some IDs and params updated. They downloaded the config json. Edited it locally. Did not fucking upload it back to the main or test app. Did not test it live. Did not CC in or direct the guy with changes to me. Did not send me the modified config file. Did not even paste the new IDs into the email. But TOOK A FUCKING SCREENSHOT OF THE MODIFIED FILE AND SENT THAT SHIT TO ME. And then had the audacity to ask me to test it when they had access to it and that's literally their fucking job.

I had to compare the fucking screenshot to the live config file and manually type in the changes.

And no, it still doesn't work. And Now I have to get back to them showing it still fails the same way but I just can't deal with these people. Fuck. Was hoping by the time I write it all down it'd be better, and it does feel a bit better, but I still need to get this app fixed. And I can only do it through these... monkeys. I just can't. Talking to these people drains my life energy... I'm just sad.

ideal sprint fallacy.
total days 10 , total hours(excluding breaks ) 8 hrs per day= 80 hrs per dev

code freeze day = day 8, testing+ fixing days : 8,9,10. release day : day 10
so ideal dev time = 7days/56 hr
meetings= - 1hr per day => 49 hrs per dev

- 1 day for planning i.e d1 . so dev time left . 6 days 42 hrs.
-----------
all good planning. now here comes the messups

1. last release took some time. so planning could not happen on d1. all devs are waiting. . devtime = 5 days 35 hrs.

2. during planning:
mgr: hey devx what's the status on task 1?
d: i integrated mock apis. if server has made the apis, i will test them .
mgr : server says the apis are done. whats your guestimate for the task completion?
d : max 1-2 hrs?
m : cool. i assign you 4 hrs for this. now what about task 2?
d : task told to me is done and working . however sub mgr mentioned that a new screen will be added. so that will take time
m : no we probably won't be taking the screen. what's your giestimate?
d : a few more testing on existing features. maybe 1-2 hrs ?
m: cool
another 4 hrs for u. what about task 3?
d : <same story>
m : cool. another 4 hrs for u. so a total of 12 hrs out of 35 hrs? you must be relaxed this sprint.
d : yeah i guess.
m cool.
-------
timelines.
d1: wasted i previous sprint
d2 : sprint planning
d3 : 3+ hrs of meetings, apis for task 1 weren't available sub manager randomly decided that yes we can add another screen but didn't discussed. updates on all 3 tasks : no change in status
d4 : same story. dev apis starts failing so testing comes to halt.
d5 : apis for task1 available . task 3 got additional improvement points from mgr out of random. some prod issue happens which takes 4+ hrs. update on tasks : some more work done on task 3, task 1 and 2 remains same.
d6 : task1 apis are different from mocks. additionally 2 apis start breaking and its come to know thatgrs did not explain the task properly. finally after another 3+ hrs of discussion , we come to some conclusions and resolutions
d7 : prod issue again comes. 4+ hrs goes into it . task 2 and 3 are discussed for new screen additiona that can easily take 2+ days to be created . we agree tot ake 1 and drop 2nd task's changes i finish task 2 new screens in 6 hrs , hoping that finally everything will be fine.

d8 : prod issue again comes, and changes are requested in task 2 and 3

day 9 build finally goes to tester
day 10 first few bugs come with approval for some tasks
day 11(day 1 of new sprint) final build with fixes is shared. new bugs (unrelated to tasks. basically new features disguised as bugs) are raised . we reject and release the build.

day 2 sprint planning
mgr : hey dev x, u had only 12 hrs of work in your plate. why did the build got delayed?

🥲🫡

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

An intern made a very bad impression on the first day.

This was before I become a developer. I was working in commercial art sales. One day, I had an appointment to onboard two new interns together.

Intern 1 shows up and I ask her for her signed confidentiality agreement. The boss had sent it out a week before and told me the interns were bringing the signed paperwork on their first day. I see the surprised look on her face and she says she forgot. She’s lucky I had access to another copy. If I didn’t, things could have gotten pretty awkward if I had to contact my boss, who was out of office. If there’s no signed agreement, I can’t onboard her and I’d have to send her home. The appointment was made with intern 1’s availability in mind, so intern 1 could have spent her time coming to the office for nothing and being turned away because of a stupid mistake she made.

While we wait for intern 2 to arrive, I try to engage in small talk with intern 1. I try to get to know her a little better and I ask “are you still in college/university?” She word vomits that she thought she had graduated, but six months later she hadn’t received her diploma and she called the school and they told her her pre-college credits had not transferred, so she’s finishing those credits now.

Oh, intern, you should have just simplified all this to “I’m finishing up my degree” or “yes, I’m still in college.” This is TMI. You don’t want to give out information about yourself that could put you in a bad light. You need to know to be discreet about yourself. You’re 22 years old. It’s really bad judgement to say this to your supervisor (me) and we’ve only known each other for ten minutes. I’m not your friend, I’m your supervisor. Honestly, I thought the explanation didn’t make sense because she would have found out about the credits when she tried to transfer them and when she applied for graduation. I didn’t prod for more details.

I did have to tell my boss about intern 1 forgetting the paperwork. It’s not something the intern would be reprimanded for, but it is something that’s not a good sign. The paperwork had been sent by the boss a week prior. It’s troublesome that an intern would forget to complete an important task that was sent by the boss. This was never a problem with prior interns.

Boss did freak out because boss thought I onboarded intern 1 without intern agreeing to the confidentiality agreement. Boss hadn’t considered an intern would forget the paperwork and didn’t tell me what to do if this did happen. I reassured boss that I had printed a new copy and had intern 1 sign the agreement.

I didn’t say anything about the word vomit. The content was troubling, but I was concerned this would be gossip and I wasn’t out to sabotage the intern.

Forgetting the paperwork and the word vomit were signs the intern wasn’t reliable. Intern had trouble taking direction even when it was written down. She’d do stupid things like invite her boyfriend to the office for hours and let BF sit at the boss’s desk—boss caught her and boss’s office is visible from our public viewing floor, so visitor did see this too. I suspected she might have an diagnosed learning disability.

In the end, intern didn’t ask for a reference letter. Boss said that if intern asked for one in the future, the answer would be no.

Intern 1 is the reason why I don’t want to be in change of interns ever again even though I’m not in art sales anymore.

Rolled out a new application I built almost entirely by myself 2 days ago... But my dev group is understaffed and has a project manager who is literally the most clueless person I have ever met, so as a result, we don't have a functional/useful dev/test/prod framework and no standards for how to deploy apps. So my past 2 days were comprised of fixing bugs in the live system that could probably have been caught if I had the time and resources to get everything thoroughly tested. It's stable now, but damn our management for being generally idiots. Our motto appears to be "Fuck it, we'll do it live"

I'm still studying computer science/programming, I still have one year to do in order to graduate (Master). I am in a work study program so I'm working for a company half of the time, and I'm studying the other half. It is important to mention that I am the only web developer of the company

When I arrived in the company 9 months ago, I was given a Vue project which had been developed by a trainee a few weeks before my arrival and I was asked to correct a few things, it was mostly about css. Then, I was ask to add a few functionalities, nothing really hard to code, and we were supposed to test the solution in a staging environment, and if everything was ok, deploy it to prod.

However, the more I did what I was asked, the more functionalities I had to implement, until I reached a point where I had to modify the API, create new routes, etc. I'm not complaining about that, that's my job and I like it. But the solution was supposed to be ready when I arrived, it was also supposed to be tested and deployed.

The problem is, the person emitting these demands (let's call him guy X) is not from the IT service, it's a future user of the website in the admin side. The demands kept going and going and going because, according to him, the solution was not in a good enough state to be deployed, it missed too many (un)necessary features. It kept going for a few months.

The best is yet to come though : guy X was obviously a superior, and HIS superior started putting pressure on me through mails, saying the app was already supposed to be in production and he was implying that I wasn't working fast enough. Luckily, my IT supervisor was aware of what was going on and knew I obviously wasn't to blame.

In the end, the solution was eagerly deployed in production, didn't go through the staging environment and was opened to the users. Now, guy X receives complaints because none of what I did was tested (it was by me, but I wasn't going to test every single little thing because I didn't have time). Some users couldn't connect or use this or that feature and I am literally drowning in mails, all from guy X, asking me to correct things because users are blocked and it's time consuming for him to do some of the things the website was doing manually.

We are here now just because things have been done in a rush, I'm still working on it and trying to fix prod problems and it's pissing me off because we HAVE a staging environment that was supposed to prevent me from working against the clock.

On a final note, what's funny is that the code I'm modifying, the pre-existing one needs to be refactored because bits and pieces are repeated sometimes 5 times where it should have been externalized and imported from another file. But I don't know when and if I will ever be able to do that.

I could have given more context but it's 4am and I'm kinda tired, sorry if I'm not clear or anything. That's my first rant

Please tell me why my boss thinks instead of just implimenting a language picker, it's a good idea to:

1) store the browser's language option in the DB only for new users, but only the first time they log in, never again.

2) never give existing users the ability to change language, and just default them to English.

3) deal with all upcoming language change tickets by having devs manually do DB updates ON THE LIVE PROD DB.

I'm screaming.

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

I'm currently between jobs and have a few rants about my previous job (naturally). In retrospect, it's somewhat therapeutic to range about the sheer brainfuckery that has taken place. Enjoy!

First, let me set the scene: legacy B2B web app made with LEMP stack and sencha ext.js 3 + 4 (don't ask) and a lot of madness. Let's call that app "Alpha".

Alpha is a self made CMS build for typical ERP stuff. Yes, a self made CMS: entities are containers, containers have types and fields and values. Like so many legacy PHP apps, it does not have a dedicated FE: the HTML is rendered on the server and then spewed out to the browser.

Easy right? Coding like it's 1999! But there was a twist: Because everything is basically a container, the HTML-templates are saved in the DB. Along with the nessary JS and the CSS. And the translation variables. Why? Because fuck you! That's why. Who needs a git history anyways.

For some reason, Alpha was kinda slow.

There was also an editor, that allowed you to modify templates (web, mail, pdf) on the fly in prod. Because templates contain repeating data (header/footer), one template could contain additional templates. Much confusion. You could change templates via migration (slow, boring) or just ctrl-c/ctrl-v that sucker (fast, much excitement).

Did I mention Alpha was slow?

On with the rant: e-mails! How do they work? Noone knows. How to send mails asynchronous in PHP? Witchcraft is the only possible answer to that riddle. Here is your enterprise™ solution:
1. create mail
2. insert mail into DB
3. WAIT UP TO 59 SECONDS FOR A FUCKING CRON TO SEND MAIL

Why? "Because that way, we can resend mails in case the network is down :)"

Same procedure for the SOAP-API (db-queue + cron). You read that right: all requests to various other systems are processed once a minute.

Alpha slow.

Alpha was only one of several systems. Imagine a bunch of monolithic php apps, interconnected via SOAP, REST and GraphQL like a godamn intergalactic orgy. Image having to debug that cluster fuck.

Let's say there is a bad request. These things happen. No biggie. Remember the db-queue? Let's try to send the bad request a second time! And a third time! Still no luck? How odd. Let's create a specific file in a specific directory: a LOCK-file. Now, "the db-queue is on hold and no request gets processed :)"

Golly gee thanks Alpha.

Anyhow, did you know that MySQL has a join limit of 61 tables?

So I made an update to my React Native app. I changed UI of a couple of screen, added a few animations here and there, refactored how my graphQL resolvers work in the backend(no breaking changes), changed how data gets loaded into the database etc.

It worked in dev so I figured hey let's deploy it. Today is(was because it's now 3am but more on that later) a national holiday so no one goes to work so no one will use my app so I have an entire day to deploy.

I started at 15:00(because i woke up at 13:00 lol). I tested the update once again in dev and proceeded to deploy it to prod. I merged backend to master, built docker images, did migrations on the db, restarted docker-compose with new images. And now for the app. I run ./gradlew assembleRelease and it starts complaining that react-native-gesture-handler is not installed. Ugh, rm -rf node_modules && yarn install. It worked. But now gradlew crashes and logs don't tell me anything. Google tells me to change a bunch of gradle settings but none of them work. Fast forward 5h, it's around 20:00 and I isolated the issue to, again, react-native-gesture-handler. They updated from 2.2.4 to 2.3.0 which didn't fucking compile. 2 more hours passed (now 22:00) and I got v2.3.1 working which fixed the problem in 2.3.0 but made my app crash on startup. YOUR FUCKING LIBRARY GETS 250K WEEKLY DOWNLOADS AND YOU DONT EVEN BOTHER CHECKING IF IT COMPILES IN PROD ON ANDROID?! WHAT THE FUCK software-mansion?

After I solved that, my app didn't crash. Now it threw an error "Type errors: Network Request Failed" every time I fetch my legacy REST API(older parts use rest and newer use graphql. I'll refactor that in the next update). I'll spare you the debugging hell i went through but another 5h passed. Its 3am. My config had misspelled url to prod but good for dev... I hate myself and even more so react-native-gesture-handler.

Backend : *breaks feature in prod by changing api*

Me: Feature is broken in prod. Please fix. I was told that the API will not deprecate the old use case.

Backend: ... Fixing it in the backend will take time. Add support for it in the frontend.

Me: I'm not done with the new feature just yet. And it will take some time to have it reviewed and fully tested. Please fix the API.

Backend: .... Well, make a new PR and add support for it first. The new feature can come later.

Me. (-_-) Okay

Sometimes it feels like I'm a code janitor rather than a frontend intern

Update: for those of you who know cougar woman/ my self proclaimed "work mom" in my previous posts (bitch who keeps stealing my lunch), yesterday she really tipped me over the edge. So I'm fucking hauling ass on my sprint work because I had to take over another team member's tasks (because he "doesn't have time" for it being prod support but all I see him doing is shitposting at his desk) and someone from another team asked the cougar a question about something. she comes up to me all demanding like "HEY you, you got the worksapce open?"
I was like um... no I don't. I'm working on the other application.
and then she's like "Look at my laptop and show him". Okay bitch let me just drop everything I'm doing to help a random person. The fact that she commanded me instead of just asking me pissed me off. Not even a single sign of "please". I'm tired of her truly. She is a snake. Even to her friend on our team. Every time he's out she's like "hey where is _____? huh why isn't he here??? hmmm" in an instigating way to remind everyone that he might've taken off for no reason. When I was too sick to volunteer at an event she organized in the morning on Saturday, she asked me the following Monday, "oh did you drink too much? lmao. a spiteful, grudge holding bitch for sure

Lessions I learned so far from my first big node/npm project with tons of users:
1) If you didn't build something for a while, expect 3 hours of resolving version conflicts for every two weeks since the last build.
2) Even if the tests pass, run the containers on your own machine and make sure that the app doesn't randomly crash before deploying
3) Even if the app seemed to work on your own machine, run the tests again in an environment mimicking prod at most 15 minutes before replacing the running containers.
4) Even if all else indicates that the app will work, only ever deploy if you expect to be available within the 4 hours following a deployment.
5) Don't use shrinkwrap for anything other than locking every version down completely. A partial shrinkwrap will produce bugs that are dependent on the exact hour you built the app _and_ the shrinkwrap file, and therefore no one will ever have seen them other than you.
6) Avoid gyp, and generally try not to interface too much with anything that doesn't run on node. If parts of your solution use very different toolchains, your problems will be approximately proportional to the amount of code. And you'd be surprised just how much code you're running. (otherwise it's more logarithmic because the more code the less likely a new assumption is unique)
7) Do not update webpack or its plugins or anything they might call unless you absolutely need to
8) Containers are cool but the alpine ones are pretty much useless if you have even just one gyp module.
9) There's always another cache. To save yourself a lot of pain, include the build time in every file or its name that the browser can download, and compare these to a fresh build while debugging to assert that the bug is still present in the code you're reading
+1) Although it may look like it, SQLite is far from a simple solution because the code and the bindings aren't maintained. In fact, it'll probably be more time consuming than using a proper database.

saw an SSIS project once used for basically string replacement and file tasks. Asked manager permission to rewrite this Proof of Concept to something more suitable language.
manager: you can't but you have to support it.
me: but i tought it's a PoC
manager: yes it is
me: so what to support?
manager: it's been sold to a customer and in production for 6 months now
So it's a PoC which is sold and moved to prod...
I hate sales and management douches sometimes... like 98% of the time....
(already left the company)

I am forced to work with a client's notoriously slow SOAP api. Slow in this case is 1.5-2s per request.
The api is structured rather... creatively... at the same time. So we have to bombard it with thousands of requests to build our data base with historical SOAP data. Also the data sometimes is a couple of hours late, giving a flat line (all values at 0) until retroactively fixing the output for the same requests.
So to fill one dev data base with a year's worth of historical data (nice to have when testing a dashboard application) we hammer the api with ~20k requests (~1 million if we want to be thorough).
Best thing about that: There is no staging/test api and the prod api seems not to handle lots of requests at the same time very well...
Latest thought: Maybe we could put a varnish cache in front of the SOAP for testing. Better have wrong data, than nothing at all and we don't kill the prod clients every time we ramp up a new instance.
Also that would dramatically decrease the 4.2 hours of data pumping to about 7 minutes after the first run.

Story time;

Major project, multi million budget, huge business and IT coordination, board level status updates, meeting started back in March 2018 for a Go Live of Aug 2019.

Based on draft requirements (and experience) I request the test environment be built for half of the work. Turns out that no one told Server Eng and they are out of space in both dev and prod until Q2 of 2019. We went from Green to Red because a Service Request.

Not in prod today, but was part of a group project that we handed in and which got us an A.

The project was to write a PID controller for a robot that would drive along a track using a sensor to follow markings on the floor. During development we were drawing graphs of the PID parameters and sensor input every tick, which caused a bit of lag but no worries - we'll turn it off for the trial runs.

Imagine our pikachu shock meme when we turned off the graphs and our calibrations were suddenly *way* off since we had been oversteering all along to compensate for the lag.

There wasn't enough time to optimize it before the deadline and using sleeps didn't produce the same "type" of lag, so we just made the graph minimize itself when it opened. To this day I wonder if the professor ever saw it or if we got the A despite it.

I was asked to integrate a piece of industrial software which was pre-alpha in a prod to align with corporate objectives. It would have caused all kinds of production problems and instability. Nobody would even notice that it wasn’t there. I procrastinated for eight months until version 1 was released.

I told my manager it was done. Then he informed me that it had been working the whole time and he got plenty of positive feedback so i should roll back to the old version until we can do a planned upgrade. I nodded in agreement and put off the downgrade.

If a team uses multiple languages and stacks (Have, JS, Python) do you think it's better to have everyone use/constantly switch between them or have dedicated developers for each language (ie. 80% main, 20% others)?

--END QUESTION, ANSWER NOW BEFOREHAND CONTINUING---
---BEGIN RANT---

My boss likes keeping the team "will rounded" so everyone does everything. One month in working in Java, the next with Node web apps. When I switch to node, it takes like a week of "wtf doesn't it work.... what changed, is it a big?" And usually end it"oh right I remember I need to ..."

And also always... "How the fuck do I write tests in {some reading framework} again?"

So feels like everyone is just a generalist and no one is a master/has time to develop mastery. I don't know if it's just me (1/3 Senior developers on the team that has to do everything) or if I'm the only one that complains... Not that it makes a difference... (Only option to really be heard is to resign but I need to somewhere else to work and finding one is hard for personal reasons)

And well this is the biggest reason I would leave the team. No time for mastery, no standardization/shared knowledge (everyone does their own thing but probably not well and no time for testing or documentation; how the fuck does whatever you wrote work, how do we use it, what the fuck did you put in prod that does ... And where the fuck did you put it cuz it's not in ANY of our repos).

I always feel one day soon it will come crashing down and I can say "I told you so" but will then it's too late and I'll be there one cleaning it up... Again

Its time to remind ourselves the wonderful time of the year, where people look back and decide to be better and learn from your mistakes...

For example never test in prod and wake me up with notifications you dumbfucks... Someone has exams to do..

Seriously, I have seen a lot of push notifications lately, its some kind of Letstestinprod-uary?

At Rackspace there are lights on the walls that go off for things like ddos attacks, fire alarm, etc. The being a code rainbow. Meaning "evacuate the building".

Every time we deployed to prod I always joked one day that it would fail so spectacularly that it would cause a code rainbow.

Question to the frontend devs: I'm currently trying to optimize part of our JS. How do sourcemaps influence loading/rendering Speed in the Browser?

In the `webpack.config.js` we use uglify as minimizer for js. It has the option `sourceMap` Set to `true`. Now as I understand it, this caused it to Inline a sourcemap inside the JS file (which leads to a slightly bigger file size and thus longer load time for the generated file...).

My Question: should sorcemaps be enabled for prod (minified) files? Why yes, why no? Do they help the browser parse the JS faster?

Well, this happens time to time...

I'm freelancing as a backend guy. I like to take care of all infrastructure before really starting to build anything, this mostly includes dev/staging/prod environments with some linear promotion strategy. So.. I did this API. Still on staging, proceeding with the development as planned, everything goes according the timeline.

And then.. this happens... At some point PM told frontend guy that it's time for production (without notifying me), so the frontend guy does what "anyone" would do in this case - tells PM to create DNS record for production to point to staging app.

Time passes, I'm still unaware of this. But I'm starting to see some quality entries in the DB, not the usual QA crap. I write to them that they're doing good job and continue with my tasks.

One of the tasks required some major DB change. I could've written migrations script, but since we're not in "production" yet, I just wipe the DB and recreate schema as I need it.

In 10 minutes the furious PM starts shouting that "production" is down and I need to fix it ASAP.

I'm lost, I'm asking questions, I'm slowly understanding what's happening...

So I want to grab some coffee, sat back down, wrote politely that they suck, added a finger emoji and terminated the contract.

Felt like the right thing to do as I definitely don't want to continue within the same "team".

Yesterday whole 12 hours we were working on deployment about a feature X that has deadline yesterday itself.
Everything damn perfectly running on Test env but not on Prod.
We made Prod into Dev/Test/Fucking garabage env. Haha.
I was laughing to myself at same time crying hard in my deep heart.

Business guys chasing PM
PM chasing us
And from morning till night we were in same room. Had lunch, and dinner only went out for toilet and to refil water bottles.

And found that feature Y is not working at same time that is related to our feature X. Fucking we have been wasted hours on it.

One of my devs got so fucked up emotionally that he messed up the code (not his fault) he didnt had his lunch and dinner. Had to console him later that its not his fault. Poor guy not sure whether he slept or not; will find out in few hours.

Anyways reported a bug.
But that bug assigned to us for fixing.
Are you fucking kidding me.
Anyways no choice. Had to do it.
Hope today everything goes good or horribly bad. FYI no deployments on Friday damn we are in stalememt till Monday.

Fuck that bug
Or
May be fuck our stupiditiy while makiing mistakes.

Confession: a very important feature of the website I'm developping wasn't working for a certain time. The boss wasn't aware because he doesn't go on the site, and I only found out last week because I needed to implement a new feature that used the previous one. Problem: the bug was only on production, not on local (and of course we don't have test server).
I took advantage of the absence of my boss today to clear the situation by making all of my tests on prod. I hope no customer tried to pass a command today, but it's finally repaired. I am both proud and shameful.

After a year of using mongo in prod and personal projects I have realised some things. Its really nice early on the project, especially when there are changing requirements and for small projects or proof of concepts.
But when you make commercial software things tend to get more complex and relational. Stakeholders want reporting and even a report building which a document store isn't the best at.
With most projects projects when they get big things get relational and this becomes more and more expensive to handle in terms of compute power and developer time.
I don't doubt mongo has its place, maybe as an secondary specialised data store or if the project is inherently document oriented.
Blog over.

Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?

# The detailed version #

We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.

But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are

So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.

I guess the saying goes: the only people that know how it works is whoever wrote it and God.

I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.

I think in Agile there's supposed to be artifacts during development but never seen em.

Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.

I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.

And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...

The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.

Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.

Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.

is it necessary to have cherry picking a part of git branching/release process?

we have 3 branches : develop, release and master.

currently every dev works on feature as follows : they make a branch out of develop, write code, raise pr against develop, get it reviewed and merge back to develop. later the release feature list is generated, and we cherry pick all the release related commits to release branch, and make a prod build out of release branch. finally, the code is moved to master and rags are generated accordingly.

so the major issue with this process is feature blocking. as of now, i have identified 4 scenarios where a feature should not be released :

1. parallel team blocker : say i created a feature x for android that is supposed to go in release 1.2.1 . i got it merged to develop and it will be cherry picked to release on relase day. but on release day it is observed that feature x was not completed by the ios dev and therefore we cannot ship it for android alone.

2. backend blocker : same as above scenario, but instead of ios, this time its the backend which hasn't beem created for the feature x

3. qa blocker : when we create a feature and merge it to develop, we keep on giving builds from develop branch adter every few days. however it could be possible that qa are not able to test it all and on release day, will declare thaf these features cannot be tested and should not be moved to release

4. pm blocker: basically a pm will add all the tickets for sprint in the jira board. but which tickets should be released are decided at the very late days of sprint. so a lot of tasks get merged to develop which are not supposed to go.

so there's the problem. cherry picking is being a major part of release process and i am not liking it. we do squash and merges, so cherry picking is relatively easy, but it still feels a lot riskier.

for 1 and 2 , we sometimes do mute releases : put code in release but comment out all the activation code blocks . but if something is not qa tested or rejected by pm, we can't do a mute release.

what do you folks suggest?

Today, I'm making a revolutionary change to our code base. I'm finally deprecating a script that lived in this goddamn repo for way too long. There's about 10 copies of the same script in 10 different directories. The script copies all the code into a "dev" folder and then runs "sed s/prod/dev/g" on all the files, and then overwites a bunch of it with some files suffixed with ".dev". Finally, after fighting the so-called architects, the devs and everyone else that seems to have gotten used to the pain this cursed dumpster fire script has caused us, my merge request is now open and ready to go to get rid of this insanity. Now we won't have to deal with as many "surprises" that happen every goddamn time we deploy to production, overwriting all our hard work by accident, and relieve some of my OCD of having the same script in 10 different places in the repository.

Losing faith in Netflix and their awesome open source projects.

Had a hard time trying to install Security Monkey : poor quality quickstart Ubuntu-only, almost no documentation, same instructions for latest (aka dev) and stable (aka prod) version, no depencies list ... oh and the UI display well only on Chrome ..

Then you surrender and just want to check the dockerized version they provide : it doesn't work neither (build fail or back end process just shut down) !!

I'm done ...

It’s certainly a feeling of progress as a dev when you get to using the advanced features of git to rewrite history successfully.

Though to make this a proper rant: holy hell what a ride! I’m glad I had everything backed up somewhere. Somehow I’d went Thanos on the repo. Deleted about half the files at random. Had to fast forward and then rewrite the history via rebase. Dropped a bunch of commits I think I should have squashed. I’m still wondering if I even did the right thing. I think cherrypicking is what I’ll go for next time. My repo now reads 59 commits behind but whatever. All my work got into one commit which is what the dev controlling prod wanted.

- Am a junior dev in an awesome team & exciting project after my apprenticeship and while having just started my part time studies
- Have restructure in company so I land in an other value stream
- Get laid off by new value stream 6 months later (now) because they have a serious budget cut
- Take time to come to terms with situation. I could finally work more on my side projects or focus a bit more on my studies. Hey actually I will have 5 months time to look for something while being paid by the company and they help me brush up my CV. Pretty neat!
- Now my former boss wants me back because of my experience in the project, but only as a production support and not as dev (because budget and they're bleeding with tickets)

Not sure if I should take the offer as it feels safe to have an income and the team is cool. However, it feels a bit like a degradation as prod support sucks in that project and I'd like to code (which wouldn't be possible then).
And as this is still my first company I'm working in, it would make sense to look for something else...

Grrr need to sleep about it... Decision-making isn't exactly my strength.

Every time we have a release, "Release Engineering" stops building our test environments from master. I've been preaching Continuous Deployment for months and here we are with the most broken system. The build actually builds all test environments AND prod from the same branch because "it's easier" for "Release Engineering". So now I have to wait for the code freeze on release and hope RE doesn't fuck up and deploy an untested branch to prod... AND hope we're given enough time to test and debug the next release since we can't right now...

This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...

Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place

Had a definite week from hell... a bunch of prod issues that only I could fix (that's a whole other rant for another day!)... a piece of code totally kicking my ass for days... a hosting environment that was unstable seemingly every time I needed to do something in it (and that killer piece of code could ONLY be properly tested there, naturally!)... a service that my app depends on flaking out with no indication what the problem was and another team responsible for it that is based off-shore so aren't responsive when I need them to be... a metric shit-ton of procedural bullshit dropped on my head... an immense amount of stress due to the lead-up to a prod rollout next month that absolutely CANNOT fail without huge ramifications for the business but not enough help to ensure it gets done.

But, with all that said, I DID manage to get that killer piece of code working late on Friday after slamming my head against the wall for over a week on it (and ultimately re-writing it from the ground-up on Thursday and Friday)... so, the week of hell ended on a high note at least, which is always a Very Good Thing(tm)!

So following my previous post, the issue happened again. And actually for background what I've been telling my boss, for years, we need ELK setup and integrated into all our APIs ASAP.

I think it's a punishable crime if any program is released into prod at a tech company with out real time logging/monitoring built in?

So issue still happening, user sent us the request details. So now need to find the actual now that handles the request and look into it's logs to see the details.

Now he's doing it the hard way.... Just finished took 1hr, and the best answer her can come up with is "I think .... Maybe ..."

And if course this is based on infinite data. He stopped after finding a "probably cause"

I have a script that is like promotion ELK, downloads all looks and parsed then so I can run queries to pinpoint the exact call and which log it's in. And can see what's happening around it.

We'll see what my way find but definitely does not take more than 1hr...

Loading data maybe but that's because it needs to download the logs and parse them all...

On a side note, guess I'm Beck on devrant as I have something to rant about. Though it's the same something that I was wanting about years ago... Monkeys...

Jr a little low while trying to do some stuff. Took down the prod sql server for a while. Some disturbance but not the end of the world but she’s a little low.

I am like: GREAT WORK!!! NOW YOU ARE LEARNING!!! 💪

Time for cake and celebrations!

Should I be even be testing if no one cares.

I keep asking Devs regarding the functionalities of their system for testing and then, realize it's already on Prod even before I test in beta or enable to test in beta.

Do I exist!!!

Now, for the nth time, I have started to test when things are almost there or already there in Prod.

I should keep myself in adrenaline mode always on from now on.

Need to do something worthy.

The worst way to start a new job.

I’m finishing up a thing for someone. It’ll be in prod soon. I’m nervous. I keep picking at it but it’s “done.” It’s just a silly script. But I haven’t written anything in a long time that someone else is gonna actually use. I write things no one else uses to make my life easier.

Remember how I was - against all that was promised - assigned to a time-sensitive front-end (so definitely not my forté) project about a month ago? Remember how I struggled with the choices of how to go about it - switching from F# (Fable) to Rust (Yew) to eventually settling in with Vue and TS?

Yeah, I’m glad I went that way, even though there could’ve probably been better choices out there: my part is done now, even though it’s not quite prod ready yet (close tho), the team who’ll maintain it takes it over now, after I finish dealing with my current minor issue. And damn their front-end guy is GOOD. Makes me feel very inferior in that department. Well, I am. Back to back-end, thank you very much...

But I have an issue here, that bothers me. I’ve produced a codebase that’s obviously written on a tight schedule: no tests, no documentation, a few embarrassing hacks/workarounds and so forth. I actually feel bad for leaving it out of my hands to them in such a state...

I'm so down that i didn't see the red circle with the cross to add a rant...

Why is that? Because several month ago i began a job with all my motivation & optimistic mood.

I was so glad that a compagny payed attention to my profil that it was the best day of my life. I wanted to improve myself and learn!

At this point i did'nt know yet that i will began to work with assholes.

In this fantastic world, designers are kings and you have to do magic to adapt one of their stupid static design on web.
Because the suprem king is the client and designs are validated.
And don't even ask for an fonctionel analysis they will laught at you!

I did everything that i could do to make things work, fast and good. One time i managed the end of a project all by my self (like said once Celine Dion). I maked the work of my colegue who was on holiday because she left with unfinished work. She said to me "it's easy". She liked to say that i maked lost her time because of my questions and that i need to search the answer by myself & work more and more and more. So i worked, day & night because i didn't have enough time. And other thing is that some persons loved to say "if you don't do that someone will need to do that for you"!

I'm a junior developer and i had acces to staging and prod environements and crashed it both several time... I needed to develope in one year the experience of a senior developer.
Every thing is my fault because i need to pay attention to things that i ignore.

Today i'm not glad, i learned a few things but can't remembered it because things went o fast for me and i can't memorized everithing. All i know is that i'm just happy to still be able to get out from bed.

There was a department. Long time ago their work was somewhat complicated: background checks of businesses, websites, ToSes, assuring agreement compliance, some risk management on top. They started as small 3 people team but over the years they were hiring new employees to catch up with the growing customer base. They were still struggling. Few years back we've integrated 3rd party services to help them and, finally, their backlog was gone!

In January they complained about how much more work they have since the merger so I inquired about which process was troublesome, what was the flow, etc., and it turned out to be very... Tinder-like - the issue was the sheer number of cases:

1. open a case,
2. check results in few windows,
3. if green + green + green, move right.
4. else move left.

It was ridiculous, I wouldn't stand for that. I sat for an hour, made some ghosting scripts that followed same business logic and saved results alongside their actual decisions. Last week I compared the two and there was zero difference so I green-lit it with my boss and pushed to prod.

Oh, the happiness on their faces when they heard the news, the disbelief, the tears of joy!

And then it happened. After 4 years of being cautious not to stir the waters I did it again. Yesterday I accidentally replaced 17 people department with 3 scripts. How was I supposed to know it was *all* they were doing??

Just did an elixir job interview for tsg global, 10 hr test, got the email of fuck you email. Do not fuck with this company they will scam you out of your time i should have spent my 10 hrs wiser lesson learned, im going to be a homeless elixir dev with 20 yrs exp. All cuz parasitic companiess like this. I submitted a prod ready solution that was most likely what they wanted built and i did fir free cuz i dont wanna be homeless. I hate life.

So there is this owner team who reviewed my code recently. I don't have much context about the their system and architecture. We try to build our changes with less context and rely in owner team's knowledge for any review gap.
The guy from the owner team missed something in my review and changes went to prod, review already took more that it was expected to take. He took 1 week for small change reviews. Now, not him but with someone else's advice they had to revert.
I wrote a mail shooting to manager, the guy who reverted and the guy who reviewed, asking the reviewer guy to explain why didn't he mentioned about any issues at the time of the review.
I have tried best from my side. But all this, god!!!
Why everything I do has some kind of weird issue. I feel so bad blaming the guy, I just think that, the way I used to feel anxious he must be feeling the same, but what can I do? I don't want to take the blame I don't even see if I can and I shouldn't be. If it was a major issue it should have been raised but he didn't. I feel so bad that I am almost crying, I am feeling that like always I am going to be judged by my team that work is slow and on top of that I can't do anything for the guy I blamed it on.

I don't know, is it my mistake? but I cannot think of anyway I would have known this.

So I'm assigned once again to fix a new someone else created and that seems to be the case whenever there's an issue...

Boss just assigns it to whoever is most likely to be able to investigate it... which is basically me. Other than the little time I can use to develop stuff, I'm usually cleaning up other people's messes.

And these other people are to busy working on new crap to properly explain how their existing code/processes/changes works.

And well the fact that anything breaks in production (that's not due to upstream one off issues) whoever does not think he needs to take responsibility for it.

So everyone else and especially me has to spend time understanding the shit they wrote and fixing it for them.

How do I tell my boss this nicely that we need clearly definitely ownership and whenever a component blows up in prod, the guy that wrote the code fixes it no matter what? Thereby incentivizing him to not write shit code in the first place and be more proactive in making sure it doesn't in the first place since he knows otherwise he's doing overtime to fix it?

Is it just me or is there really no such thing as a dev job where something doesn't blow up due to poorly tested and designed code every other day?

So a long time ago, I posted Ink lyrics from ColdPlay, and gave them a DevOp twist. I will post the full lyrics, as memorabilia and for a chill moment from rants:

Got access rules said "prod/maintainer"
Pulled in your change with my cli
And you wonder when you wake up
Did it build last night?
Oh, oh, oh, oh, oh
Feels like the pipeline, is broken inside

All I know
All I know
Is that I'm host
When company broke
AIl I know
Is that I love to code
So much that it hurts

Got a homebrew but the shell aint right, oooh, oooh
Just want a way of keeping docker alive
All I know, All I know
Is that the post
BreachForums indorsed
All I know
Is that the data is sold
So much that it hurts
I see that Forbes, and New York Times
They said my boss, began to fire
I see my console and Im mashing Ctrl-C
All I know
Is that loved to code
So much it

Wish me luck.

Deploying Blazor app for the first time in prod in about a month.

Did tests. Curent infrastructure can hold around 70 concurent sessions with no problems. (probably more, 70 was the limit in my browser).

I tested each sessiuon with 70.000 line table. (Yep whole 70.000 lines for each session with a virtual scroll).

Shit is fast. Too fast even. I'm waiting for other shoe to drop, but so far in simulated tests it's amazing.

Let's talk in 2 month AFTER prod deploy.

So trunk-based is the new approach everyone is using, because it is so cool.
I used gitflow for the last projects with azure devops, set up the pipelines like tipically in 1 week if I had other things to do with the help of the portal clicking through things. PR-s triggered pipelines, everything worked cool.
But then trunk-based got momentum, so I worked with this client where 2 developers worked for !!!3 months!!! to setup trunk-based pipeline. It was not my money, so I did not say a thing. They were using infrastructure as code.
I am all in for automation, but seriously? Then again, another project where a DevOps team took 1 dev-month to setup the pipeline + meetings. And what do you get in the end? So that the same image goes on all environments? Like how many releases do you have for prod in a year. Lets say 24. 24 x 5 minutes of manual work for the release, that is 2 hours. So my question is why would you spend 2 hours of manual work while you can automate it merely in a month? Everyone loves to code, but using the ui on the DevOps portal saves you so much time. I don't get this. Maybe I am getting old :D

Why is instagram so thoroughly broken and a user experience torture.

I know the standard answer of "As long as the core flows that hold up the popularity work, no one cares much", and yeah, true, but that's a reason for why no one fixes the broken stuff.

What I want to know is why is it so thoroughly broken in the first place? Granted, Facebook isn't the best of places but one would expect at least a certaim level of competency from a team coming from the same organisation that gave us React JS(even if Instagram did not originate there, they have been in the Zuck empire for a while now). Why do such thoroughly messed up UI/UX and features get pushed to prod in a company that has the time, resources, and talent to do things professionally(read: better than the mess that instagram is). Not to mention a fuck ton of missing simple features that would make using it much better experience (JUST LET ME AT LEAST COPY COMMENTS GODFRKINDAMNIT IF ENABLING EDITING COMMENTS WILL COST YOU YOUR FIRSTBORN'S SOUL)

Maybe I am somewhat biased since I use Instagram desktop more than the mobile app, but my point should still stand.

So apparently my code went to prod more or less all right. Phew, the deadline was this week-end. This project have been sitting there for month, they gave me the technical requirement and never bothered to ask the stakeholders about it. When the contract went in, they started to freak out it wasn't usable.

The thing is, this project had way more moving part and trying to threat video in the frontend is not the easiest. But now is REFACTOR TIME.

I dream of getting rid of the browser video api (too flaky), download the bitmap directly and render it in requestAnimationFrame. I call it just-in-time rendering. I think i'd need to put a decoder in aws, I did it already with ffmpeg. I could not manage to put it in streaming mode though, so it was still a bit slow, but i could decode, write and re-encode faster than the video player speed.

What do you lads think? Doable or not? I at least need to general tidy up (this codebase have grown organically without any fucking direction from above, like this project took all my time on the technical side, I did not have time to run after people to get specs), centralizing state, improve monorepo and tooling, perfs,...

Hopefully they understood i cant keep adding whatever feature they want today.

I have been working in the same PR for 2 months. Many metamorphosis of it i would say 😂 I even made it all the way to prod and had to be reverted coz it was causing straight chaos. I have worked on some pretty complex features in the past year, this the smallest yet most complicated ask i have ever received. Orleans really put my patience to test and here i am, right now completely unable to write any code. I am just here planning my vacation in 4 weeks time.

TypeScript + Vue3 is such a bullshit combination.
No wonder they dragged Vue3's launch for years. Coz it's garbage.

Out of beta and in supposed-prod, yet half the time the complex objects disappear when passed around. Pinia is bullshit-ia.

Unrelated TS components fuck up and console logs a "Ayye New Error we didn't know existed, here go to Vue Issue Helper and tell us about it so we get to make Vue3 worse than before" ffs.

What alternative do you use for EF Migrations?
We are in current stage that it's slowing our build time for 30 seconds, and each month we need to clean them and do the same for the whole pipeline (dev, qa, prod)...
It's way uncomfortable to work with it since we have stored procedures and ef doesn't track them...
What alternative did you chose to use?