Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "prod shit"
-
I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.
The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.
We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.
Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.
Needless to say, he was never allowed to touch code on a prod server ever again.8 -
My mentor/guider at my last internship.
He was great at guiding, only 1-2 years older than me, brought criticism in a constructive way (only had a very tiny thing once in half a year though) and although they were forced to use windows in a few production environments, when it came to handling very sensitive data and they asked me for an opinion before him and I answered that closed source software wasn't a good idea and they'd all go against me, this guy quit his nice-guy mode and went straight to dead-serious backing me up.
I remember a specific occurrence:
Programmers in room (under him technically): so linuxxx, why not just use windows servers for this data storage?
Me: because it's closed source, you know why I'd say that that's bad for handling sensitive data
Programmers: oh come on not that again...
Me: no but really look at it from my si.....
Programmers: no stop it. You're only an intern, don't act like you know a lot about thi....
Mentor: no you shut the fuck up. We. Are. Not. Using. Proprietary. Bullshit. For. Storing. Sensitive. Data.
Linuxxx seems to know a lot more about security and privacy than you guys so you fucking listen to what he has to say.
Windows is out of the fucking question here, am I clear?
Yeah that felt awesome.
Also that time when a mysql db in prod went bad and they didn't really know what to do. Didn't have much experience but knew how to run a repair.
He called me in and asked me to have a look.
Me: *fixed it in a few minutes* so how many visitors does this thing get, few hundred a day?
Him: few million.
Me: 😵 I'm only an intern! Why did you let me access this?!
Him: because you're the one with the most Linux knowledge here and I trust you to fix it or give a shout when you simply can't.
Lastly he asked me to help out with iptables rules. I wasn't of much help but it was fun to sit there debugging iptables shit with two seniors 😊
He always gave good feedback, knew my qualities and put them to good use and kept my motivation high.
Awesome guy!4 -
toxic workplace; leaving
I haven't wanted to write this rant. I haven't even wanted to talk to anyone (save my gf, ofc). I've just been silently fuming.
I wrote a much longer rant going into far too much detail, but none of that is relevant, so I deleted it and wrote this shorter (believe it or not) version instead. And then added in more details because details.
------
On Tuesday, as every Tuesday, I had a conference call with the rest of the company. For various, mostly stupid reasons, the boss yelled at and insulted me for twenty minutes straight in front of everyone, telling me how i'm disorganized, forgetful, how can't manage my time, can't manage myself let alone others, how I don't have my priorities straight, etc. He told the sales team to get off the call, and then proceeded to yell and chew at me for another twenty minutes in front of the frontend contractor about basically the same things. The call was 53 minutes, and he spent 40 minutes of it telling me how terrible I've been. No exaggeration, no spin. The issues? I didn't respond to an email (it got lost in my ever-filling inbox), and I didn't push a very minor update last week (untested and straight to prod, ofc). (Side note: he's yelled at me for ~15 minutes before for being horribly disorganized and unable to keep up on Trello -- because I had a single card in the wrong column. One card, out of 60+ over two boards. Never mind that most have time estimates, project tags, details, linked to cards on his boards, columns for project/qa/released, labels for deferred, released to / rejected from qa, finished, in production, are ordered by priority, .... Yep. I'm totes disorganized.)
Anyway, I spent most of conference call writing "Go fuck yourself," "Choke on a cat and die asshole," "Shit code, low pay, and broken promises. what a prize position," etc. or flipping him off under the camera on our conference-turn-video-call (switched due to connection issues, because ofc video is more stable than audio-only in his mind).
I'm just.
so, so done.
I did nothing the rest of the day on Tuesday, and basically just played games on Wednesday. I did one small ticket -- a cert replacement since that was to expire the next day -- but the rest was just playing CrossCode. (fun game, fyi; totally recommend.)
Today? It's 3:30pm and I can't be bothered to do anything. I have an "urgent" project to finish by Monday, literally "to give [random third party sales guy] a small win". Total actual wording. I was to drop all other tasks (even the expiring cert lol) and give this guy his small win. fucking whatever. But the project deals with decent code -- it's a minor extension to the first project I did for the company (see my much earlier rants), back when I was actually applying myself and learning something (everything) new, enjoying myself, and architecting+writing my own code. So I might actually do the project, but It's been two days and I haven't even opened single file yet.
But yeah. This place is total and complete shit. Dealing with the asshole reminds me of dealing with my parents while growing up, and that's a subject I don't want to broach -- far too many toxic memories.
So, I'm quitting as soon as I find something new.
and with luck, this will be before assface hires my replacement-to-be, and who will hopefully quit as soon as s/he sees the abysmal codebase. With even more luck, the asshole king himself will get to watch his company die due to horrible mismanagement. (though ofc he'll never attribute it to himself. whatever.)
I just never want to see or think about him again.
(nor this fetid landfill of a codebase. bleh.)
With luck, this will be one of my last rants about this toxic waste dump and its king of the pile.
Fourty fucking minutes, what the fuck.33 -
“Yeah but you’re not a *real* developer”
Fuck. you.
I wrote 80% of this code base. I do 80% of the tickets/storyboard points. I do all of the QA. My nose is to the grindstone every fucking day honing this craft and sweating my balls off like a blacksmith staring into the red hot kiln while the sores of previous mistakes scream bloody murder from the unrelenting exposure to heat. I saw this amazing industry of opportunity, freedom and self examination and wanted in no matter what it took. I glued myself to every pithy resource I could possibly get my hands on and crawled through the muck and filth of it all until I could keep myself warm with the smallest spark of my own making. I stoked that spark until it became a fire and stoked that fire until I could set entire forests ablaze. I listened to the ungrateful people keeping warm by my combustion saying it “wasn’t hot enough” or “would have been a nicer colour if they did it” or “could have warmed up just fine jogging on the spot”. I made painstaking alterations to my ignition and watched my undeserving benefactors gradually be silenced and begin to sit quietly by the heat. I jumped into that inferno daily, was reduced to ash daily and emerged reborn daily. But you are right! I didn’t get scammed out of $40k+ studying technology in an archaic institution from instructors who don’t give a shit and answering “D all of the above” for 4+ years straight therefor my opinion doesn’t mean shit. Push your bullshit to prod and watch the server come burning out of the cloud as the apocalyptic swarm of angry tickets come flooding in why don’t you? Bet they didn’t teach you that in school. You’ve never poked around inside an open source codebase in your life. They are just a mystery boxes of magic that unless someone holds your hands with finely crafted instructions containing a 50/50 picture to word ratio you throw a hissy fit. Every problem that comes up instead of working to solve it you reflexively point to the first person in the room while thinking with your pea brain how you can possibly scapegoat them into taking the fall for whatever it is that’s come up today you couldn’t possibly understand.
Not a real developer?
Fuck. You.28 -
Going on a vacation, so notify all clients that I won't be available during two weeks.
Client: well we have this huge presentation and here's a list of stuff we absolutely need for it
Me: sure I have a look.
Me: holy shit dude! That's gona take about 2-3 days. I'm leaving soon!
Client: it's realy critical to have them in a week as it's a very important presentation! Is there any way you can make it work? If we can do anything to help, just name it.
Me: well I'll do my best (planed 1 day for such rhings)
Me: *pulling a 15h day*
Me: here, all done budy! Did a 15h but now it's done, so do that presentation!
Client: oh, nice, but it wasn't that urgent
Me: ...
Next day:
Client ssh in to the server, fucks shit up
Client: well I did a thing and now stage and prod is fucked, can you do anything?
Me: (knowing it will take 30min to fix) well... I try my best. Btw. I'll leave in a few hours and won't take my computer, so try not to fuck every thing again, okay? -
Doot doot.
My day: Eight lines of refactoring around a 10-character fix for a minor production issue. Some tests. Lots of bloody phone calls and conference calls filled with me laughing and getting talked over. Why? Read on.
My boss's day: Trying very very hard to pin random shit on me (and failing because I'm awesome and fuck him). Six hours of drama and freaking out and chewing and yelling that the whole system is broken because of that minor issue. No reading, lots of misunderstanding, lots of panic. Three-way called me specifically to bitch out another coworker in front of me. (Coworker wasn't really in the wrong.) Called a contractor to his house for testing. Finally learned that everything works perfectly in QA (duh, I fixed it hours ago). Desperately waited for me to push to prod. Didn't care enough to do production tests afterwards.
My day afterwards: hey, this Cloudinary transform feature sounds fun! Oh look, I'm done already. Boo. Ask boss for update. Tests still aren't finished. Okay, whatever. Time for bed.
what a joke.
Oh, I talked to the accountant after all of this bullshit happened. Apparently everyone that has quit in the last six years has done so specifically because of the boss. Every. single. person.
I told him it was going to happen again.
I also told him the boss is a druggie with a taste for psychedelics. (It came up in conversation. Absolutely true, too.) It's hilarious because the company lawyer is the accountant's brother.
So stupid.18 -
Root: Fleshes out missing data in some factories. Tests affected code and finds the change breaks some specs (but shouldn’t).
Root: Reaches out to spec author.
Root: Messages thundercunt (the ticket’s code reviewer) on slack about the specs and the reaching out. No response.
Root: Works on another ticket while blocked.
Root: Logs off.
Root: Talks with spec author chick in the morning. Decide to pair on specs later.
TC: Still no slack response.
Root: Gives update in standup. Mentions factories and broken specs. Mentions pairing with spec chick.
TC: Still no slack response.
Root: Pulled off tickets in favor of prod issue. Gets ignored by everyone else diagnosing prod issue. Investigates prod issue by herself. Discovers prod issue isn’t from bad code, but bad requirements — code works as requested. Communicates this with details. Gets ignored by people still diagnosing prod issue. Tries again. Gets ignored. Gives up. Works on non-blocked tickets instead.
TC: Still no slack response.
Hours later:
TC: Comments on PR telling me I broke specs (how did I not notice?), that I need to reach out to spec chick and work with her, and that I can’t resolve the ticket until it’s fixed and passes code review.
TC: Still no slack response. (21 hours later at this point)
TC: Logs off. Still no response (25 hours at this point)
———
Ignoring the prod issue for the moment…
I broke specs. No shit.
I need to talk with spec chick. No shit.
I can’t resolve the ticket. No shit!
Bitch, I told you all of this 21 fucking hours prior, and again 3 hours prior during standup. But no, I clearly “don’t communicate” and obviously have no bloody clue what I’m doing, either, so I need everything spelled out for me.
And no, I didn’t resolve the fucking ticket. Why the fuck would I if it still has pending changes? Do you even check? Ugh!
And what the fuck with that prod issue? I’m literally giving you the answer. fucking listen! Stupid cunts.
Why is it all of the women I work with are useless or freaking awful people? Don’t get me wrong, many of the men are, too, but I swear it’s every single one of the women. (Am I awful, too?)
Just. Ugh.
I can’t wait to leave this sewer of a company.
Oddly still a good day, though. Probably because I talked to recruiters and sent out my resume again.rant oh my root gets ignored. root swears oh my root talks in third person root solves a prod issue thundercunt root communicates root wants to leave root gets ignored15 -
Friend of mine killed his MacBook with some Softdrink.
Just poured it all over his poor a1502.
He let it dry for a few days, it starts to work again.
Except the battery.
Goes on Amazon and buys a new battery.
New battery doesn't work either and so he tells me about it and I as stupid as I am couldn't resist the temptation to finally work on a MacBook like my "hero" Lois Rossmann does.
So turns out the board is good.
Cleaned it up and basically nothing happened to it.
So what's the deal with "los batlerias"?
The first got hit by liquid, the second had a broken connection to a cell.
That could have happened through my friend, installing it without testing it first, or at the seller, so it being a DOA battery.
Now away from the stupidity of my friend and the situation to the actual source for this rant.
Once something happens to a modern Managed battery, the Battery Management System (BMS) disconnects the voltage from the system and goes into an error state, staying there and not powering anything ever again.
For noobs, it's dead. Buy a new one.
But It can be reset, depending you know how to, and which passwords were set at the factory.
Yes, the common Texas instruments BQ20Zxx chips have default passwords, and apple seems to leav them at default.
The Usb to SMBus adaptors arrived a few days ago and I went to prod the BMS.
There is a very nice available for Windows called BE2works, that I used the demo of to go in and figure out stuff. The full version supports password cracking, the demo not.
After some time figuring out how Smart Battery Systems (SBS) "API" works, I got to actually enter the passwords into the battery to try get into manufacturer and full access mode.
Just to realise, they don't unlock the BMS.
So, to conclude, my friend bought a "new" battery that was most likely cut out of a used / dead macbook, which reports 3000mah as fully charged instead of the 6xxx mah that it should have, with 0 cycles and 0hours used.
And non default access.
This screams after those motherfuckers scaming the shit out of people on Amazon, with refurb, reset, and locked fucken batteries.
I could kill those people right now.
Last but not least,
My friend theoretically can't send it back because I opened the battery to fix the broken connection.
Though maybe, it'll get send back anyway, with some suprise in the package.9 -
This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.
Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.
And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.
And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."
We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.
And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.4 -
Got a call while on vacations, the main server is down.
*holy shit* I thought.
As I opened my MacBook, the phone kept ringing and slack was going crazy.
“What should I do? Where is the problem?”, the voices in my head said..
I opened the terminal and tried to get a ssh connection.
Sweat was dropping down from my forehead.
“Connected” the terminal said.
“Fuck yeah, the server is up, only the app is not responding”, I thought and opened the log files.
Suddenly, “STOP” I shout at the log files that were appended way too fast.
Then I saw it.
TimeOutExceptions..
I added an index to the modification date column,“ kill -9“ed the process, started again and went back to vacation mode 🙂
And of course I was the office hero for a while💪
For the smart asses, I’m aware that it’s a bad idea to -9 the app process in prod, but it was so overloaded that i was not able to kill it any other way. And we needed that server up again.4 -
I know a lot of you already know/do this, but to those of you who don't - I know it has saved me more than once:
If you use SSMS (SQL Server Management Studio) - when connecting to a server, before you hit the "connect" button, under options, choose a color to represent which server you are connecting to. I personally use stoplight colors: red = prod, yellow = QA, green = dev....this way, it helps you realize what server you are connected to, minimizing our dreaded "oh shit" moments....lol
Hopefully this helps someone ☺6 -
!rant
We just did a massive update to our prod db environment that would implicate damn near all system in our servers....on a friday.
Luckily for us, our DB is a badass rockstar mfking hero that was planning this shit for a little over a year with the assistance of yours truly as backup following the man's lead...and even then I didn't do SHIT
My boy did great, tested everything and the switch was effortless, fast (considering that it went on during working hours) and painless.
I salute my mfking dude, if i make my own company I am stealing this mfker. Homie speaks in SQL, homie was prolly there when SQL was invented and was already speaking in sql before shit was even set in spec, homie can take a glance at a huge db and already cast his opinion before looking at the design and architecture, homie was Data Science before data science was a thing.
Homie is my man crush on the number one spot putting mfking henry cavill on second place.
Homie wakes up and pisses greatness.
Homie is the man. Hope yall have the same mfking homie as I do5 -
"There is a problem with A, could you check it out? It's urgent for the client"
Me: Okay, just open a ticket for it too.
*Working on A"
After about ten minutes:
"Hey, there is a small problem B and it's also urgent for the client, we need you to check it out"
Me: I'm working on A and you are yet to open a tick... (Interrupted)
"But it's urgent and it's a small fix, we can fix it and push it to prod, A can wait for a bit"
(Since when is it a "we"?)
Me: *sigh* fine, lets see what's B is all about...
*After going over problems C to Z*
"Why isn't A ready yet? The client's mad and it was to be ready as of today"
Me: Because you had problems from B to Z and they were all urgent according to you so after each request you asked of me I had to postpone A with you knowing about it.
"But A is for today!"
FUCK YOU, YOU WORTHLESS WASTE OF JIZZ! YOU RAN TO ME FOR EVERY GOD DAMN URGENT PROBLEM YOU HAD FROM OUR SHIT OF A CLIENT AND INSTEAD OF TELLING ME THE PRIORITY YOU JUST THROWN IT ALL IN A RANDOM ORDER!
FUCK YOU! I WISH YOUR FATHER WOULD'VE SHOT YOU OUT THE WINDOW INSTEAD OF INSIDE YOUR MOTHERS CUNT!5 -
Me passing time on the weekend
Random call from unknown number
Turns out it's the manager
M: hey , how is your weekend going ...
Me: nothing much ... Whatsup ?
M : yeah well , we wanted to push some minor adhoc fixes as some clients wanted it urgently
The Devops folks need developer support . Can you pitch in and monitor
Me : I'm not aware of what changes are going , i don't think i can provide support
M : don't worry it's minor changes , it's already tested in pre prod , you just need to be on call for 30 mins
Me : ugh okay .. guess 1 hr won't hurt
M: thanks 👍🏽
Me: *logs in
*Notices the last merged PR
+ 400 lines , implemented by junior dev and merged by manager
*Wait , how is this a *minor* release...
*Release got triggered already and the CI CD pipeline is in progress
*5 mins later
*Pipeline fails , devops sends email - test coverage below 50%
Manager immediately pitches in ...
M: hey , i see test coverage is down , can you increase it ?
Me: and how do u suppose I do that ?
M : well it's simple just write UTC for the missing lines ... Will it take time ?
Me : * ah shit here we go again
Yeah it will take time , there are around 400 lines , I am not aware of this component all together
Can you ask junior dev to pitch in and write the UTC for this
*Actually junior dev is out on a vacation with his girlfriend
M : well he's out for the weekend , but
as a senior dev , i expect you to have holistic understanding of the codebase and not give excuses ,
this is a priority fix which client are demanding we need this released ASAP
Me : * wait wat ?
---
I ended up being online for next 3 hours figuring out the code change and bumping up the UTC 🤦🏾9 -
Oh boy, my riskiest coding decision was certainly that one time when I refactored some 50k lines of critical legacy shit code in 3 days, straight up merged everything into master and then deployed to prod.
Luckily there was only one minor bug I had to fix after that... phew...
(To my defense: I was solo-working on it - the infamous CMS Of Doom™)2 -
That feeling when your client connection is more stable than the connection of a fucking game server... Incompetent pieces of shit!!! BEING ABLE TO PUT A COUPLE OF SPRITES DOESN'T MAKE YOU A FUCKING SYSADMIN!!!
Oh and I sent those very incompetent fucks a mail earlier, because my mailers are blocking their servers as per my mailers' security policy. A rant from the old box - their mail servers self-identify a fucking .local!!! Those incompetent shitheads didn't even properly change the values from test into those from prod!! So I sent them an email telling them exactly how they should fix it, as I am running the same MTA on my mailers (Postfix), at some point had to fix my mailers against the exact same issue as well, and clearly noticed in-game that they have deliverability problems (they explicitly mention to unblock their domain). Guess why?! Because their server's shitty configuration triggers fucking security mechanisms that are built against rogue mailers that attempt to spoof themselves as an internal mailer, with that fucking .local! And they STILL DIDN'T CHANGE IT!!!! Your fucking domain has no issues whatsoever, it's your goddamn fucking mail servers that YOU ASOBIMO FUCKERS SHOULD JUST FIX ALREADY!!! MOTHERFUCKERS!!!!!rant hire a fucking sysadmin already incompetent pieces of shit piece of shit game dev doesn't make you a sysadmin2 -
A tip to tech folks from my personal xp.
If you fuck up and make an impacting mistake in your company, like taking PROD down, noone is going to fire you on spot. Assign some more mandatory trainings - maybe. So you'd be more careful next time.
See, it's not worth getting rid of someone who made a mistake. You should be seated down and insisted to fix it. If you don't - then they might consider firing it. If you do fix it [with help or alone] - you become a more valuable asset to the company as you prove you are responsible for your actions and you take it seriously. You show that you can clean up your own shit and you don't need a babysiter next to you.
If you simply make a mistake and they replace you with someone else, that someone else is likely to be unaware of your mistakes and is doomed to repeat them. It's just bad for business.
Ofc if you making mistakes becomes a tendency rather than an exception, it's also a red flag for the business.
Don't get too laxed! And always answer for your shit. Never hide a fuckup - always alarm about it asap so that corrective actions could be taken by respective organs of the company while you are fixing it.
Come up with an action plan, announce it. Estimate resources you need [like help from others] - announce that too. Update concerned parties every half an hour or so about the status. If you find you need anything else while fixing it or you come across some blockers/delays/change of impact - always announce asap. Do avoid false alarms and disinformation.
// inspired by someone's rant today7 -
When managers look at my code, it’s shit, it’s over complicated, it’s overly difficult to read, it took too long, it’s too much for a simple ticket, i handled too many edge cases, we’ll never need most of it, why did I bother making it extensible when it’ll never need to change, how dare I use “unless”, why did I bother writing all these comments, why did I update the documentation that nobody reads because it’s outdated, etc. They say I should be more like the legendary devs and push janky code quickly, and complain that I don’t have any flops (problems in prod) like those are a good thing.
When my coworkers look at my code, they say it’s clean, amazingly easy to read, a monster feature that’s somehow still a joy to review and work on, it makes their lives easier, that it does exactly what it should in all cases, that they learned something from reading it, and thank me for the comments and documentation. And marvel that I finished it so well in so little time.
Am I bragging? Not intentionally; I’ve heard these things repeatedly since I started here, and the contrast between the above is so stark.
In reality, the managers are just idiots who were promoted far above their competence, and make everything worse. (Gee, who woulda thought?) It’s just so frustrating.19 -
TLDR - you shouldn't expect common sense from idiots who have access to databases.
I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.
Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.
Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.3 -
I did it.
I finally fucked prod.
And had to do open heart surgery on the service to get it unfucked.
Shit happens. Luckily its internal prod only...10 -
I just love it when our clients decide to make a clone of live production server..then put it immediately online..and don't tell anyone about this.. and then start bitchin how data gets doubled all of a sudden..
Yeah, no shit sherlock.. you have two prod servers for 'hot swapping' and some services may only be running on one at a time.. You even have a manual on how to switch primary to secondary (turn off services on primary first, then turn them on on secondary and all)..or in case primary actually dies, just turn on services on secondary and you're good to go, right?
So how do ya think cloning the one with running services and putting the clone immediately online will work out?! 🤔
God, I thought it was common sense to not do that..but here I am, bitchin about how people fail to RTFM.. :/ or use brain..fuck..4 -
Fuck me in the ass, but do it harder then this api just so I can feel some love 😖
it's one of those days where you have to migrate from soap to rest, only the rest api doesn't have the same structure or search parameters as the soap api, so there's this entire fucking application sending requests at a brick wall, and expecting a purple throbbing 12 inch cock of xml to be pushed into an multi dimensional array and pushed through to the views to derange the mess, only you have to create that fucking 12 inch cock from several 2 inch dipsticks that have a different hierarchy, different field names, and merge the shit together with a glue gun...
good thing it's only an unexpected prod problem... right? 🤷♂️
Ah, the woes of a Monday on the legacy app adventures.rant bullshit applications over engineered using a view to build a data set from hell adopt a piece of shit day1 -
When will I fuckin learn that
a) customers lie
b) customers are sloppy
c) customers are wrong
d) customers do not do their work (properly)
e) customers want us to do their (dirty) work
f) possibly all of the freakinly above?! + khm....
They will fuckin aaaalwaaaays say sth is not working after the update..
And I will alwaaaays assume I fucked up something..even if I didn't touch that part of the code/data..
And almost aaaaalways it turns out that the bug they complain about is how the system worked (or didn't work) before the update and/or some fuckup from their side..
Anyhow, I rushed over, grabbed the files went testing in dev..wtf, output is different, mine is ok, theirs is..wtf is that shit?!
Transfer newly built dll to test..same shit as on prod..wtf?! How?!
I assumed they have thing A correctly linked to thing B.. ofc thing A was linked to thing C in their case and in another case (our test) to correct thing B..
I got chillies when grabbing files, that
I should have tripple checked that they didn't fuck up something on the link part, but I just assumed they know what they were doing & that they checked they linked correct files with correct content already, before being pissy that the update fucked up things.. riiiight!! :/
I wanted to find solutions to this fuckup asap so I disregarded my gut feeling..yet again!! Fuuuck!
I've spent too much time trying to find ways to fix a bug that wasn't even a real bug to begin with.. :/
Fuuuuuck!!
So yeah, always treat the customers like they are 3yrs old & have no clue what they are doing & check exactly wtf they were indeed trying to do..it will save you time & nerves..
And note to self: reread this shit daily!! And imprint it in your brain that everything is not always your fault!!11 -
PSA: If you work on a project where code review is expected and there is a migrations framework set up for you, and you choose to self-merge/commit directly to master and update the prod database in the console instead of writing migrations, your coworkers hate you and you can eat shit. That is all.4
-
There is a company providing a very speciffic service. And it has a core application for that svc, supported by a core app team.
That company also has other services, which are derivations of the core one. So every svc depends on core.
Now that we're clear on that... I was working in a team of one of the subservices. We very strongly depended on core. In fact, our svc was useless if integration w/ core broke down.
The core team had an annoying habbit. They refused to version their webservices and they LOVED to push api updates w/o any warnings. Our prod, test, other envs used to fail bcz of core api changes quite often. Mgmt, IT head was aware of the problem and customers' complaints as well.
So as a result, once core api changes we're all in a panic mode: all prior priorities are lowered and revival of prod is to be our main focus. Core api is not docummented, the changes are not clear, so we have to reverse engineer the shit out of it. We manage to patch our prod up w/ hotfixes, but now we have tech debt. While working on the debt, core api changed again, in test env. Mgmt pushes debt back and reallocates us to hotfix test. Hotfix is 80% done when another core api breaks. Now mgmt asks us to drop wtv we're working on and fix that new break. By the time we're to deploy the hotfix, another api breaks in another env. The mgmt..... You get the picture :)
2 years go by, nothing has changed so far.6 -
Man the senior dev where I work produces the most half baked shit solutions but I guess management loves em because he produces results.
Like Holy fuck this whole place just has a raging hard on for Microsoft products. Plus management won't spend any money on dealing with any of the tech debt and our prod solution is just to erect more monoliths.
Someone please end my suffering5 -
The guy who became my manager just pushes to the prod branch.
On a repo where another team clearly set up development and production branches.
This guy has been pushing code like crazy and I always wanted to take my time setting things up properly in our team: TDD, CI/CD, etc.
Because he pushed so much he became my manager and I was seen as unproductive.
Data Science and software development best practices just dont coexist it seems.
Yeah yeah, it's up to me to start introducing good practices, but atm "getting it done" takes priority over the real based shit.4 -
4:55: Everything looks good in prod.
4:56: Deploy new feature after all is well in Dev.
4:57: Prod goes to shit!
4:58: Call wife to tell her I'm not coming home at 5.
4:59: Prod looks fine w/o anyone doing anything.
5:00: Leave work.
5:25: Get yelled at by wife for leaving at 5 after telling her I couldn't.3 -
Have a couple I want to air today.
First was at my first gig as a dev, 4-5 months out of school. I was the only dev at a startup where the owner was a computer illiterate psycopath with serious temper tantrums. We're talking slamming doors, shouting at you while you are on the phone with customers, the works...
Anyways, what happened was that we needed to do an update in our database to correct some data on a few order lines regarding a specific product. Guess who forgot the fucking where-clause... Did I mention this boss was a cheap ass, dollar stupid, penny wise asshole that refused to have anything but the cheapest hosting? No backups, no test/dev/staging environment, no local copies... Yeah, live devving in prod, fucking all customers with a missing semi-colon (or where clause).
Amazingly, his sheer incompetance saved my ass, because even if I explained it, he didn't get it, and just wanted it fixed as best we could.
The second time was at a different company where we were delivering managed network services for a few municipalities. I was working netops at that time, mostly Cisco branded stuff, from Voice-over-IP and wifi to switches and some routing.
One day I was rolling out a new wireless network, and had to add the VLAN to the core switch on the correct port. VLAN's, for those who don't know, are virtual networks you can use to run several separated networks on the same cable.
To add a VLAN on a Cisco switch one uses the command:
switchport access vlan add XYZ
My mistake was omitting the 'add', which Cisco switches happily accept without warning. That command however can be quite disruptive as it replaces all of the excisting VLAN's with the new one.
Not a big deal on a distribution switch supplying an office floor or something, but on a fucking core switch in the datacenter this meant 20K user had no internet, no access to the applications in the DS, no access to Active Directory etc. Oh and my remote access to that switch also went down the drain...
Luckily a colleague of mine was on site with a console cable and access to config backups. Shit was over within 15 minutes. My boss at that time was thankfully a pragmatic guy who just responded "Well, at least you won't make that mistake again" when we debriefed him after the dust settled. -
Did a nasty thing in prod...
Very nasty.
So nasty even default graph tells me something about going yolo in prod.
Maybe it's weed. But feels like that graph tells me not to trry this shit ever again.9 -
So... Heard back from a recruiter today. Lovely lass.
I’d passed over a submission for her tech demo.
The brief was basically just to create a small simple module that calculates shit, nae effort.
But, when the recruiter had me on the phone she said “I know it’s a silly small module but try and run it up like you would a production ready app”.
The job spec and recruiter were keen on me demonstrating TDD, not specific on js version, final runtime, etc. The job was a senior spec at a higher salary range. So it warranted some effort, and demonstrating more than a simple module.
“Okay, cool, nae bother, let’s crack on.”
The feedback in the response from the dev today:
“He’s over-engineered tests, build...”
SUCK MY LEFT TESTICLE YOU FUCKWIT.
Talk to your recruiters, not me.
The feedback included a phrase I never hope to hear from a developer I work with:
“Tests are good but...” 😞
It was a standard 98% test suite from an RGR cycle, no more or less than I’d expect in prod.
The rest of the feedback was misguided or plain wrong. It was useful to see because I know now when they say they have “high standards” they mean: we listen to the dude who put the factory pattern in a JS brief.
Oh shit also: “someone’s done chmod 777” was in there as a sarcastic comment in the feedback. It was his fucking unarchive tool 😞
My response was brief and polite: “cheers for the consideration, all the best, James”
It’s honestly not worth warning them. Or, asking why they’d criticise something they’d asked me to do.
If you want a shitty js module, ask for a shitty js module and no more.4 -
TL;DR;
Idiot hard coded database host on the app... Pushed to prod and suddenly shit wasn't working... Took me 10 minutes to figure out what was going on...
Wrote a passive aggressive git message and commited.
Before updating prod my boss turns around to me and the following took place:
Boss: is there any problem with the server?
Me: yes, someone (i know who was ) hard coded the test db IP and it broke the backend.
Boss: oh, but will it affect the mobile app?
Me: well, it won't work but I'm already pushing the fix.
Boss: no..err.. I mean... Will I have to make any change to the mobile app?
Me inside: wtf dude... For real?! Get your shit together...
Me: no. It good, I already fixed it.
Boss: OK. Thanks
TL;DR;
Moron hard coded dB's host and stupid boss can't get shit together nor ask who did it to take precautions...12 -
"Architect"(A) - Hey, StrucN, we have a bit of a problem on the module you are working on (which the previous "developers" seem to have given it roofies)
Me: Okay, what seems to be the problem?
A: There is a need to add some functionality to it, we need you to ...
Me: I see, well it can be done but it wouldn't be so simple - the module is a mess and the change would need to be well tested
A: I fear the clients deadline is for tomorrow
Me: Well he'll have to wait, rushing it is the worst possible option
A: I'll talk to him about it, thanks
After around half an hour A rushes back
A: Hey I passed a ticket to you about the additions we spoke about, it should be ready for tomorrow
Me: It won't be ready, it's too complex to complete is in such a shirt notice (considering it's already the end of the day and all the changes need to be pushed tommorow to prod)
A: I know *programmer from useless team B* did something similar so as it is close to what we need you should copy it.
My inner voice: FUCK YOU YOU USELESS FUCKING CUNT! THERE SHOULDN'T BE ANY COPY PASTE SHIT FROM SOME UNRELATED MODULE! YOU SHIT STAINED MEAT BAG ALREADY DID SUCH A SIN IN THE PAST AND I HAD TO FIX ALL OF IT. THE MODULE SHOULDN'T SUFFER ANY MORE AS IT IS ALREADY A GODDAMN RAPE VICTIM!
WHERE DID PROPER PROFESSIONALISM WENT? WHY IS IT THE INDUSTRY FILLED WITH STUPID WANNA BE "ARCHITECTS" WHILE OTHER MORE COMPETENT FOLK SHOULD ALWAYS BE IGNORED BECAUSE IT'S ALWAYS SHOULD BE READY FOR TOMMOROW?!
For fucks sake I miss my old Architect, he could really understand the essence of program development3 -
On my project the customer has re-signed into a contract several times when they have budget to continue work. The first time they got us to build the system was a huge success story because the team was assembled quickly and we did rapid development. Initialize repo to prod in 1.5 months. The customer asked for the same dev team. Strong dev team, a PM that doesn't take shit, and pure agile. Lets call her don't-take-shit PM.
When the customer re-signed the executive decided that she didn't like don't-take-shit PM. So the project manager gets replaced by play-by-the-rules PM who will comply with stupid requests and micromanagement. He isn't a bad PM but he tries to make everyone happy. The amount of management types executive installs on the project is massive, and development team is cut down in major ways. Customer and executive shit rolls down to the development team and we can't get anything done. The customer starts to lose faith because we can't get traction. They start demanding traditional waterfall/SDLC docs. Which causes more delay in the project.
So the executive decides that the PM can take a fall for it to save face for the company. She moves play-by-the-rules PM to another project. He starts handover to a new PM that has a history of being her pushover. The customer hadn't seen him yet so now we have push-over PM.
Play-by-the-rules PM is finally out of the project and instead of moving to a different account the company decides to "lay him off because there is no work". So basically they made him take the fall for the failure while promising reassignment, and instead let him go. This is so unfair..
Meeting with push-over PM yesterday and he shows us his plan. Identical to play-by-the-rules PM's plan that got him axed.We point that out and show him the docs that were made for it. His face clearly communicates "OH SHIT WHAT DID I SIGN UP FOR?"1 -
A guy who had the same nationality as the enterprise we were working for was promoted from JUNIOR js developer to UX/UI coordinator for the entire department just because he was 2 year older than me (26 vs 28). Literally he was a junior dev and went to that.
One day he was accusing me of writing a piece of code which led prod to downtime. I was in the office, he was in another country with our manager and technical director next to him and we were talking over internal conference system. I shown git history + his name + his code and he was saying ‘that’s not true!!!’.
I couldn’t resist and I began to yell something like ‘You fucking fuck piece of shit cocksucker...’ for 5 minutes. Since that day i was the god on my project for UI/UX side.
Even now he is in the same place on the same position...
PS: more stories to come with this guy6 -
Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?
Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!
Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..
And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..3 -
Rant from a previous gig I just remembered that reignited my fury lol
Suddenly, CSV exports became massively critical to our product's success. "They were always part of the plan, if we don't have them the product is a failure". Plot twist, they were NOT always part of the plan. And our backend is not at all designed for querying the combinations of data you're asking for.
Nevermind we've been entirely focused these last few months on making the new user experience as slick as possible because "our customers want cake, not meat and potatoes". Forget the fact that, in order to meet the deadlines, my team coupled the backend a little too much with the needs of the frontend because otherwise integrations took too long. We NEED fucking CSV exports of everything you can fucking imagine.
No. Fuck you. If you want it, it's gonna take at least 2 engineers and a month, and according to you we only have a few weeks of runway. No, I'm not compromising jack shit, this is the reality we live in. This is going to go nuclear in production if we don't do it right. Either give us the month and bankrupt the company, or fucking drop it.
Or...you could go cry to the frontend team for solutions. And convince them to page through ALL of the data and generate CSVs in the fucking browser. Sure, it sort of works in QA with the miniscule amount of data we have there, but how'd that work out for you in prod?
Jesus fucking christ why are you people such incompetent morons, and how the fuck did you become executives??2 -
ughh, studying about various software project management methodologies and lifecycles is so boring.
every different model looks similar, saying :
you got an idea?
- check weather its needed and if its practically / financially possible
- get investment and resources,
- design,develop, test, release
- repeat .
why name them waterfall or spiral or rad or agile or shit?
and we know how project go in reality: "fuck its 2 days to release and 5 features left? push to prod, make breaking features, leave the tests and release"7 -
IMHO technical dept is kind of like smoking cigarettes for some decades.
You were told that shit will hit the fan but you do not take proper action. And one day you'll realize that you fucked up (or not, also seen that).
Worked for a company in IT, where we maintained an ERP which was "in progress" for over a decade. The basic implementation was done by people with zero technical understanding. To clarify: not self coded. Software was bought. We are talking about integrating the system.
Therefore, the foundation was like a wet noodle. When I joined that company, I told them that they need to address that. I told them that things will get slower and slower and that shit will hit the fan if no proper actions taken.
Even made a list with flaws I found. With potential risk and actions to take, that could then be measured.
At that time, five people worked in said department (including me).
People did not want to listen. "Would be too expensive to rewrite stuff".
Nothing has changed about the wet noodle, but I tried to fix as many things in a working system as I could. Felt like heart surgery, because changes got implemented and "tested" in prod. No version control, no documentation, everyone implemented things like they felt (no guidelines for consistency).
A lot of small fuckups that summed up over the years.
I left the company after two years because I had the chance to land a job as a dev.
Been around two years now since I left. Now 9 people work in that department with around the same efficiency as us 5 people back then.
The new employees struggle to be productive, because things are just implemented poorly and not maintainable anymore.
Had some dialogs with them some time ago. Everything I told them would happen, actually happened. What a suprise :-|
I will not go into too much detail about all the shit that's going on there, as it would be just too much (and my morning coffe is almost finished).
I think that we all know the difference between "not beautiful, but does the job" and "oh, that will backfire - badly". And I wish that my communication skills increase so that people start listening in future.8 -
#whenProdBreaks
$data = ["some","predefined","data","set"];
// :/ this suddenly broke
//$response = $this->makeSoapRequest($data);
/**
Due to prod failure, Hot-swap soap for rest - don't ask how we took the same input, spun the shit out of the response and recreated the same data structure that the soap request made, but it works... and that's all you will ever have to know.
**/
$response = $this->makeRestRequest($data);
//process the response
$this->process($response); -
If I ever get the chance to start my own company or pick out the dev team I work with... I will never, ever, ever work with anyone who doesn't understand how to abstract out a problem. Our prod DB is a mess and our API is a mess because we left it to some dudes who are dumb as shit and doing everything ad-hoc. Now their ad-hoc shit doesn't work anymore and I get to clean it up. 😫😞FML
-
Holy fuckin shit. Fuck java updates.
Today we deployed to production with a java app. For whatever reason it didn't work, just throwing SSL errors left and right.
Same app works on dev and qa
There's only one tiiiiny difference: Java 8 141 on dev/qa, Java 8 171 on prod
Guess what happened in 171 ... they updated the CA
rip 4h debuggin5 -
Pressing ctrl+s in sql dev when checking packages.. you accidentally press space or sth and later ctrl s it compiles.. then shit gets real when you accidentaly lock up everything on prod db..2
-
FUCK WORDPRESS
FUCK WORDPRESS
FUCK WORDPRESS
FUCK WORDPRESS
FUCK WORDPRESS
Who the fuck thought "Oh lets put all caches in the options table". This is so incredible dumb. The options table has nearly 8GB and is just unusable. The query for the "real options" takes about 3s. To export this fucking shit I had to block the prod server for 20 minutes to get a db dump. I want to womit in the mouth of the fucking idiot who did this.6 -
I hate having to deal with our IT service desk. Every time it takes enormous energy to get to the right people and make them understand that no, you are not an idiot, but you actually have a technical issue.
Sure thing they do have a few competent nice folks there too I've gotten to know over time and they indeed have to deal with a ton of dumb non-tech savvy idiots on a daily basis. However, if my job title mentions "software" and "engineer" they should at least assume I'm an idiot in tech. Or something. Every single time I need to open a ticket, even for the simplest "add x to env y", I need to quadruple check that the subject line is moron-friendly because otherwise they would take every chance to respond "nah we can't do that", "that's not us", or "sry that's not allowed". And then I would need to respond, "yes you do:) your slightly more competent colleague just did this for us 2 weeks ago".
Now you might imagine this is on even another level when the problem is complex.
One of our internal apps has been failing because one of the internal APIs managed by a service desk team responds a 500 status code randomly but only when called with a specific internal account managed by another service desk team.
(when I say "managed by", that doesn't mean they maintain it, it just mean they are the only ones who would have access to change something)
Yesterday I spent over a fucking hour writing a super precise essay detailing the issue, proving a million times it's not on our end and that they need to fix it. Now here is an insight to what beautiful "IT service" our service desk provides:
1) ticket gets assigned to a "Connectivity Engineer" lady
2) few hours later she responds and asks me to give her the app and environment IDs and grant her access to those
(naturally everything in my email was ignored including these two IDs)
3) since the app needs to be in prod for the issue, I make a copy isolating the failing part and grant her access to the original "for reference" and the copy to play with
4) few hours later I get an email from the env that some guy called P made changes to the actual app, no changes to the copy
(maybe they immediately fixed the app even though I asked them to only touch the copy)
I also check the env and the live app had been shared with another 2 people giving them editing rights:)
5) another few hours pass and the lady responds that she had been chatting with P (no mention of who tf that guy is) and that P has a suggestion that might work and I should test it, "please see screen shot" for details:
These motherfuckers sent me a fucking screenshot of the env config file where "P has edited a few parameters" that might help. The screenshot had a 16 line part of the config json with a bunch of IDs and Base64 params which HE EDITED LOCALLY.
Again, because I needed a few iterations to realise what I've just witnessed:
These idiots modified some things in the main app (not the copy) for hours. Then came to the conclusion that the config needs some IDs and params updated. They downloaded the config json. Edited it locally. Did not fucking upload it back to the main or test app. Did not test it live. Did not CC in or direct the guy with changes to me. Did not send me the modified config file. Did not even paste the new IDs into the email. But TOOK A FUCKING SCREENSHOT OF THE MODIFIED FILE AND SENT THAT SHIT TO ME. And then had the audacity to ask me to test it when they had access to it and that's literally their fucking job.
I had to compare the fucking screenshot to the live config file and manually type in the changes.
And no, it still doesn't work. And Now I have to get back to them showing it still fails the same way but I just can't deal with these people. Fuck. Was hoping by the time I write it all down it'd be better, and it does feel a bit better, but I still need to get this app fixed. And I can only do it through these... monkeys. I just can't. Talking to these people drains my life energy... I'm just sad. -
Found this line in code that's been in prod for 10 years
"int kak = object.getValue()"
There is no contextual reason for it to be called kak, which means shit in Afrikaans14 -
New normal. New app to build.
- Still have to maintain older systems in parallel
- That leaves 1 week for developing the new app
- Slept 2 hours a day, coding coding coding
- Tested the shit out of the app because.. hey, its to help the customers' safety and health... I don't mind staying up late
- Finished the app in 5 days, code is now on prod
- Could barely look myself at the mirror because I look like shit
- Btw the app requires an external device as an input, the existing device works flawlessly based on my testing
- We need more devices
- Clueless manager bought new model instead. He assumed everything is fine, no testing is required
- Tested the app with new device model, doesn't work
- Deadline closing in
- Thanks, there goes my sleep
- THANK YOU1 -
Holy. Shit. Tests. I am testing. All week. Nothing but tests. I am one with the spec. You know what I realised today? Tests are a bit like life. Life is just one big spec suite that takes 75 years to run. Except there's no prod environment at the end of the DMT tunnel waiting for our green pass docket to say 'take me Lucifer, you absolute visionary: I'm ready'. We're all just a spec with no application. We're doomed. Nothing matters. I need to lie down4
-
Tl;Dr Im the one of the few in my area that sees sftping as the prod service account shouldn't be a deployment process. And the ONLY ONE THAT CARES THAT THIS IS GONNA BREAK A BUNCH OF SHIT AT SOME POINT.
The non tl;dr:
For a whole year I've been trying to convince my area that sshing as the production service account is not the proper way to deploy and/or develop batch code. My area (my team and 3 sister teams) have no concept of using version control for our various Unix components (shell scripts and configuration files) that our CRITICAL for our teams ongoing success. Most develop in a "prodqa like" system and the remainder straight in production. Those that develop straight in prodqa have no "test" deployment so when they ssh files straight to actual production. Our area has no concept of continuous integration and automated build checking. There is no "test cases", no "systems testing" or "regression testing". No gate checks for changing production are enforced. There is a standing "approved" deployment process by the enterprise (my company is Whyyyyyyyyyy bigger than my area ) but no one uses it. In fact idk anyone in my area who knows HOW to deploy using the official deployment method. Yes, there is privileged access management on the service account. Yes the managers gets notified everytime someone accesses the privileged production account. The managers don't see fixing this as a priority. In fact I think I've only talk to ONE other person in my area who truly understands how terrible it is that we have full production change access on a daily basis. Ive brought this up so many times and so many times nothing has been done and I've tried to get it changed yet nothing has happened and I'm just SO FUCKING SICK that no one sees how big of a deal this. I mean, overall I live the area I work in, I love the people, yet this one glaring deficiency causes me so much fucking stress cause it's so fucking simple to fix.
We even have an newer enterprise deployment. Method leveraging a product called "urban code deploy" (ucd) to deploy a git repository. JUST FUCKING GIT WITH THE PROGRAM!!!!..... IT WAS RELEASED FUCKING 12 YEARS AGO......
Please..... Please..... I just want my otherwise normally awesome team to understand the importance and benefits of version control and approved/revertable deployments2 -
Was recently asked if our team wanted to change from Java to Kotlin so I looked over the feature lists but don't see much that's impressive.
One argument was less boilerplate code but I think they're are libraries like Lomboq that can write that stuff for you.
The other was smart type casting and type inference and to that I'm like this sounds like giving monkeys a hammer.
Our JS codebase looks like shit... And our Java app just crashed in prod.
Getting a ton of text messages this morning and thankfully I'm on vacation still...
The error is not caused by NPE... but how some or logic spammed the db..
A new language isn't going to fix this.... And a team that writes this sort of shit logic clearly shows they are incapable of learning a new one probably...
They are already script kiddies... Don't need them to become babies...6 -
Just delayed a push to Prod. because of this motherfucking shit:
https://gitlab.com/gitlab-org/...
It never occurred to me to even look. FUCK.15 -
Who the fuck invented the glorified pile of shit people call laravel? Is this actually used in PROD for anything else than load testing a monitoring server by creating loads of error messages?
OOP exists for a reason, not to create bazillions of classes with static methods.
Dump that shit ffs!6 -
Working with LEGAL is the worst!
They say yes to all reviews and once it's in prod they want to completely change the whole thing cause they think every user reads the Terms & Condition!
To be honest I deployed 5 times. I have not read that shit once!1 -
Microsoft :
"YOLO All Devops pipeline are now .net 8"
Me, lunching PROD release : "Whatever, I don't use any advanced shit"
Pipeline : "Can't connect to your nuget soure (private) : 401"
Me : What ? You just connected to it in prevous step in pipeline !
Azure : Yah, but prev st ep was using different syntax so it worked
Me : Ok, here you go, same syntax
Azure : Still nop.
Me : WHY ?!!!
Azure : I won't tell you. 401 !!
Me : Fuck you, I 'll rewrite this step from scrath !
30 mins later
Azure : Naaaan, stil 401.
Me : HOW ?!!! You literally use this nuget feed in 2 prev steps with no issues.
Azure : Fuck you
Me : fuck you too.
15 mins later
me to me... Hmm, I've already restored all packages. They "should" stay where they are..
Me : Take that azure : Ignore restoring for this step"
Azure : Oh yeah, that works ! All compiled !
WTF ?!!!! And why they release cha ges without even sending a mail. because fuck us apparently.
God I hate devops.... -
RAAAAAAH fuck fuck fucking shit!!! Fuck jest Typescript "on the fly" compilation esModuleInterop typeroots, missing definitions jest ts-ignore and xtest everywhere, manual npm linking with different pkg mgrs & pub to a private registry, building docker images locally and doing tag management across git, docker & kubernetes then cross fingers that prod which has 0 common setup with local & test somehow works, open architecture "tickets" and wait months before they resolve, then repeat ad infinitum. How the fuck can I be productive when I need to be all over the place all the time and deal with these meta-code shenanigans. I just wanna code, damned3
-
The mobile application my company is developing is beginning to fail in a prod environment because the third party tool we purchased to sync our 3 databases in the background isn't working as expected, so I have been assigned the task of rewriting the entire application. I chose to do it in react-native/redux which I have never heard of until two weeks ago, and I have never enjoyed programming so much in my life. Shit just clicks and works the first time more often than not. Android Studio had me banging my head against my desk daily. Kudos to these technologies 👌1
-
just got off work,... ANOTHER DAY SPEND TRYING TO FIGURE SOMEONE ELSES PROD ISSUE....
Doesn't help when my PM's like did you figure it out yet, did you figure it out yet?
Why am I cleaning up other people's shit....1 -
I have a confession: I produced a shit ton of wacky code...
The business guy in my team is super fucking bitchy about deploying to prod, while my codebase is a total mess. I could refactor it to use a much cleaner module, but I had to do it because we're running out of time. I spent half a day trying to refactor while failing miserably.
//FIXME: I will come back... or not.1 -
This project is just a complete clusterfuck... But nvm. We had to integrate a third party service pushing data into our system. Btw the service wasnt even working correctly. But that is just the tip of the iceberg. Its friday around lunch time. Message appears "what is the status of the integration?" Yeah havent started working on it. Last info was service is not stable. I doubt that this will be done this week. Next message from PO: "We will all push hard to get this done today and deploy to prod." Why? Because this dumbasses said to the customer this will be deployed eod. And by we you mean the devs once again doing overtime. Has this shit stopped? No. Like for the last two weeks its like we promised the customer xyz to be deployed tomorrow. Not a single dev was asked how long it takes to add this3
-
So one of my colleague does not put fucking merge and commit rules in his piece of shit project, and somehow I gotta help maintaining it. I wish I was admin to put some order in this bullshit.
Some other dumb fuck decided to copy paste a slack thread containing code and commit it in the master branch because he does not understand what the fuck he is doing. Prod is now going to shit lol.
This legit just gave me cancer. -
Some little piece of shit fairy messed with my code.
Made some modifications in a tamplate file, tested in dev everything worked. Pushed it to prod and in one of its cases it displayed the array key of each element. Checked the code, an that was what written in there in prod and in dev too. However that shit in dev (and previous versions in prod) displayed key[0]->name as intended.
So I thought git blame and check where it went wrong... Guess what... That fucking line is the same age as the file itself....
How that motherfucking shit worked, and what had I had in mind when writing that shitcode is a mistery... -
Had a definite week from hell... a bunch of prod issues that only I could fix (that's a whole other rant for another day!)... a piece of code totally kicking my ass for days... a hosting environment that was unstable seemingly every time I needed to do something in it (and that killer piece of code could ONLY be properly tested there, naturally!)... a service that my app depends on flaking out with no indication what the problem was and another team responsible for it that is based off-shore so aren't responsive when I need them to be... a metric shit-ton of procedural bullshit dropped on my head... an immense amount of stress due to the lead-up to a prod rollout next month that absolutely CANNOT fail without huge ramifications for the business but not enough help to ensure it gets done.
But, with all that said, I DID manage to get that killer piece of code working late on Friday after slamming my head against the wall for over a week on it (and ultimately re-writing it from the ground-up on Thursday and Friday)... so, the week of hell ended on a high note at least, which is always a Very Good Thing(tm)!2 -
This is a continuation of my previous rant about admob being not very informative when it comes to invalid traffic and the resulting restriction in ad delivery.
I then wanted to use admob mediation to hang in facebook ads. My app is written with Xamarin.Forms.
So first I needed to make some facebook configuration - create an account, let my app review, create some ad placements and other shit. I came to the point where I had to put in a link to my privacy policy and the link could not be accepted due to some SSL fuckup -.-'
I then found out that there is an issue with my SSL Chain. With the help of whatsmychaincert.com I solved that issue. Little side note here: I have limited knowledge of that stuff and my cousin helped me set up my homepage so I had no idea what I was doing. Did a snapshot and luckily I did not needed that as everything worked :)
This took me around half an hour just so I can paste the fucking link to activate my app in facebook developer portal.
After that I made the whole mediation configuration shit - not an issue as google documented this quite well but it took some time.
Now comes the shitty part. To use admob mediation you need adapters to the other ad network. I found a nuget package with exactly what I needed just to find out that it is outdated. So I pulled the repo and saw that this thing is an aar binding library. Never did that stuff so I read some docs again. Updated the package and consumed it in my app.
The google docs then said "Use this mediation test shit to check if you did everything correct before going prod" - aar binding nr. 2 (but I am now familiar with that :P). This thing then told me that facebook ads could not be loaded because the SDK version is outdated -.-' SDK version comes from another nuget package which is referenced by the first aar thingie. I tracked that thing back to a repo where I found out that they are indeed totally behind. So I downloaded the aar, made a binding lib and bound that to my first aar binding lib as that depends on this.
Put that all back in my app - tested mediation and fucking finally after 6 hours everything comes together! all lights are green and things work.
Sorry if this is not quite a rant but it was quite a journey and I just had to share it. -
The new AWS CodeCommit console is SOOOOOO SLOOOOOOOOOOOOOOOW holy shit.
Multiple seconds to update the page each time you change folder before you can see what's inside.
Who the hell thought shiping that to prod would be a good idea?2 -
FUCK EVERYONE right now. Stupid business with vague information, stupid dev team making SHITTY code. STUPID AUTO CORRECT TRYING TO CENSOR ME!1
-
Being the smartest guy on the team so spend most of the time cleaning everyone else's shit up in prod because they can't recognize what they wrote as shit.
O wait..... Hypothetical.... Hm...
I guess if you replace the proverbial shit with actual shit that would be worse... Smartest guy being a janitor...2 -
Background: We switched from just simple old PHP and JS using notepad++ to PHPStorm and its infinite configurables, Symfony 4, Twig, Composer, Doctrine, Yarn, NPM, Bootstrap, ( thank the stars we didn't try to add Docker in with all this ), any other junk I'm missing here? Then upgraded to Symfony 5.
Symfony's autowiring: madness behind the curtains. I get frustrated about when and where I can just magically inject these dependencies or use config variables, you know, like the ones you define in service.yaml. Hmmm, "service".yaml. In a controller you can say getParameter() but in a service you have to inject the parameter, FROM THE "SERVICE".yaml!!! Autowiring drives me nuts. Ok, so we can supply dependencies using the constructor, that's great! Within a controller you never have to instantiate the object you're passing to the constructor (autowiring handles that). That's cool, weird when we you try to trace it for the first few times, but nice I guess. Feels like half-assin' it. What bugs me here is that it only works in controllers... I guess out of the box.. i'm not even sure. To get that feature to work for services you have to make some yaml edits. Right?Maybe? Some of the Symfony tutorials have you code up some junk then trash it. Change config then wipe that out and do X instead... so I have no idea what "out of the box" for Symfony really is.
Found this cool article that describes my frustrations in better terms and seems like a good resource to learn about autowiring. I need to continue my yaml wizardry classes. https://alanstorm.com/symfony-autow...
.....And on to YAMLs, or CSS, or JS or any other friggin' change you make to a file anywhere... Make a change, reload page, nothing... nope you have to do some hidden cheat combo of yarn dostuff -> cache:clear -> cache:warmup -> cache:cache:the:cache ... I really really hate this crap. Maybe I'm too old school for all this junk. It was simple with pure PHP. Edit code, push file, reload page, and oh look it changed! Done. So happy! Ok, Ok. Occasionally the js or css might get cached by the browser and you have to ctrl/f5 or Shift/f5 .. one of those. With this framework there's just so much more that you have to remember to do get some new feature of your site loaded.
Now, I totally get wanting to use some type of entity framework, but I feel like my entire world turned backwards. Designing tables using something like MySQL Workbench made sense. I can see all the columns and datatypes right there as i'm building them. From what I've experienced now with Symfony/Doctrine is you have to make and entity, get a shit-ton of question lobbed at you and if it's a relation field you have to really have a clear idea of the cardinality up front. Then we migrate that to the database. Carefully read through the SQL if you really really just want to use migrations:migrate in Prod. That alter table could cost you some some downtime if your table is large.
Some days man.... -
Fuck my company, sincerly.
So Im crunching my ass off, to make product, there is +- fuckton of changes that for example require refactoring flow of certain things, restructure of how shit work, Im +- 2nd weekend now, and most heavy features are cleared.
I work till late. constantly I have someone with stupid shit like calls, indeed Im needed for that stuff but also, that slows down progress of this project. Just sake of example friday 18:00 I had call (I work till 16:00) about new minor and frankly easy feature. Today, morning 8:30 one call, than 13:00 long call, Ive done the feature, didn't push it to alpha. yet though.
Now during that call that started 13:00 I get yelled on that all ordered features aren't on prod yet (I throw them to alpha becouse manual tests must be done as standard here).
Dude what the motherfuck. Im literally wearing my ass off to deliver your stupid product becouse I know its critical for company but it does not mean I can do it all in one fucking night.
F**k off and shut your mouth up and let me work for f**k sakes.
Ah also, stop f**king remotely micromanage me you little piece of sh*t.
Thanx for allowing me to vent out,
Peace.2 -
So I'm assigned once again to fix a new someone else created and that seems to be the case whenever there's an issue...
Boss just assigns it to whoever is most likely to be able to investigate it... which is basically me. Other than the little time I can use to develop stuff, I'm usually cleaning up other people's messes.
And these other people are to busy working on new crap to properly explain how their existing code/processes/changes works.
And well the fact that anything breaks in production (that's not due to upstream one off issues) whoever does not think he needs to take responsibility for it.
So everyone else and especially me has to spend time understanding the shit they wrote and fixing it for them.
How do I tell my boss this nicely that we need clearly definitely ownership and whenever a component blows up in prod, the guy that wrote the code fixes it no matter what? Thereby incentivizing him to not write shit code in the first place and be more proactive in making sure it doesn't in the first place since he knows otherwise he's doing overtime to fix it?
Is it just me or is there really no such thing as a dev job where something doesn't blow up due to poorly tested and designed code every other day?3 -
This got me fucked up. Listen yo.
So we have this issue on our modal right. The issue keeps poppin. It's a hotfix because its in prod. So my senior and I were on it. After a few hours, I showed him the part of the code that is buggy. It's 50 lines of code of nested if-else, else-if. And so we're still fighting it. He redid everything since we're using angular2 he did a subject, behavior-subject all that bs and I was still trying to understand what's the bug, because it's happening on the second click and so I did my own thing and found the cause bug and showed it to him, its this:
setTimeout( () => {}, 0)
the bootstrap-modal doesn't allow async inside it (I dont why, its in the package). So he explained to me why it's there. So I did my own thing again and find a workaround which I did, a one-line of angular property, showed it to him he didn't accept it because we'll still have to redo it with subjects and he was on it. I said ok. Went back to my previous issue. The director came in and ask for a fixed, my senior came up to me and told me to push my fix. Alright no problem. So we good now. Went back to our thing bla bla bla, then got an email that we will have a meeting, So we went, bla bla bla. The internal team wants a support for mobile, senior said no problem bla bla bla, after the meeting he approaches me and said (THIS IS WHERE IT GOT FUCKED UP) we wont be supporting bootstrap4 anymore because of the modal issue and since we're going to support mobile and BOOTSTRAP4 grid system is NONINTUITIVE we are moving to material design because the grid system is easier. I was blown away man. we have more than 100 components and just because of that modal and mobile support shit he decided to abandon bootstrap. Mater of fact its the modal its his code. I'm not expert in frontend but I looked at the material design implementation its the same thing other than the class names. OHHH LAWD!3 -
I'm always amazed at how people tend to prefer a certain pain instead of an uncertain relief... Batshit crazy...
One cousin talking about his abusive relationship: "Could be worst.... At least he doesn't beat the children"
A colleague talking about his failed marriage: "It's not so bad, we just have to avoid each other."
And you'll find the same shit with management. "The prod is only on D.O.S. twice a day...", " Every deploy is a hell as shit hit the fan, but after a week it's the usual"...
Crazy how "change" afraid people -
Wish me luck.
Deploying Blazor app for the first time in prod in about a month.
Did tests. Curent infrastructure can hold around 70 concurent sessions with no problems. (probably more, 70 was the limit in my browser).
I tested each sessiuon with 70.000 line table. (Yep whole 70.000 lines for each session with a virtual scroll).
Shit is fast. Too fast even. I'm waiting for other shoe to drop, but so far in simulated tests it's amazing.
Let's talk in 2 month AFTER prod deploy.4 -
// Rant 1
---
Im literally laughing and crying rn
I tried to deploy a backend on aws Fargate for the first time. Never used Fargate until now
After several days of brainwreck of trial and error
After Fucking around to find out
After Multiple failures to deploy the backend app on AWS Fargate
After Multiple times of deleting the whole infrastructure and redoing everything again
After trying to create the infrastructure through terraform, where 60% of it has worked but the remaining parts have failed
After then scraping off terraform and doing everything manually via AWS ui dashboard because im that much desperate now and just want to see my fucking backend work on aws and i dont care how it will be done anymore
I have finally deployed the backend, successfully
I am yet unsure of what the fuck is going on. I followed an article. Basically i deployed the backend using:
- RDS
- ECS
- ECR
- VPC
- ALB
You may wonder am i fucking retarded to fail this hard for just deploying a backend to aws?
No. Its much deeper than you think. I deployed it on a real world production ready app way.
- VPC with 2 public and 2 private subnets. Private subnets used only for RDS. Public for ALB.
- Everything is very well done and secure. 3 security groups: 1 for ALB (port 80), 1 for Fargate (port 8080, the one the backend is running on), 1 for RDS postgres (port 5432). Each one stacked on top and chained
- custom domain name + SSL certificate so i can have a clean version of the fully working backend such as https://api.shitstain.com
- custom ECS cluster
- custom target groups
- task definitions
Etc.
Right now im unsure how all of this is glued together. I have no idea why this works and why my backend is secure and reachable. Well i do know to some extent but not everything.
To know everything, I'll now ask some dumbass questions:
1. What is ECS used for?
2. What is a task definition and why do i need it?
3. What does Fargate do exactly? As far as i understood its a on-demand use of a backend. Almost like serverless backend? Like i get billed only when the backend is used by someone?
4. What is a target group and why do i need it?
5. Ive read somewhere theres a difference between using Fargate and... ECS (or is it something else)? Whats the difference?
Everything else i understand well enough.
In the meantime I'll now start analyzing researching and understanding deeply what happened here and why this works. I'll also turn all of this in terraform. I'll also build a custom gitlab CI/CD to automate all of this shit and deploy to fargate prod app
// Rant 2
---
Im pissing and shitting a lot today. I piss so much and i only drink coffee. But the bigger problem is i can barely manage to hold my piss. It feels like i need to piss asap or im gonna piss myself. I used to be able to easily hold it for hours now i can barely do it for seconds. While i was sleeping with my gf @retoor i woke up by pissing on myself on her bed right next to her! the heavy warmness of my piss woke me up. It was so embarrassing. But she was hardcore sleeping and didnt notice. I immediately got out of bed to take a shower like a walking dead. I thought i was dreaming. I was half conscious and could barely see only to find out it wasnt a dream and i really did piss on myself in her bed! What the fuck! Whats next, to uncontrollably shit on her bed while sleeping?! Hopefully i didnt get some infection. I feel healthy. But maybe all of this is one giant dream im having and all of u are not real9 -
So I've been working with a Ruby DSL my colleague wrote for our rails app that builds app flows represented by data using migrations, which are consumed and rendered by the frontend. So data-driven UI.
It's very solid in prod, so we're running with it, but it can be hard to work with because everything is built using migrations - for example, the one signup flow we have spans across 7 migrations that add/change/remove components in the flow, change decision logic, etc.
I'm building a particularly complex one and can't decide which development method is better. I can either
1. write the flow in one huge migration, then change as needed - keep rolling back, resetting and testing until it works, or
2. increment changes and additions in multiple migrations across multiple pull requests, such that the final product spans across about 10-12 smaller migrations
Which one?
Both are super icky to me but I'm leaning toward 1. At least all of the shit would be in one place and would make sense without needing to switch between 10-12 files to see where shit is being defined, changed, etc. because it reads chronologically.3 -
Brilliant rant from Redditor OK6502 in a thread about a "tech screen" being used to get free labor:
Usually when something like this uses the words complex tech stack it means you're going to have to deal with shitty server code distributed over a mix of Azure and AWS nodes and a lone Linux server running under someone's desk, an infuriating configuration hell with no safeguards for keeping dev and prod isolated, a hodge podge of different scripting languages (why not make scripts in pero that call power shell which then calls more perl? Should work right?) and random but critical shit checked into 3 different SVN, stuff stashed on people's shares that will never be checked even though you can't do your homework b without it, usually copied from someone else's share who left the company 3 years ago, no QA process to speak of (while claiming to be agile, somehow) and a front end that is maintained by one exhausted junior dev who inherited a mess of 20 different js frameworks that all load at the same time with every single click, somehow.
The full thread is really worth reading:
https://reddit.com/r/...