Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "prod bugs"
-
Me : So how's the deadlines here?
Coworker : There are none.
Me : ?
Coworker : if they are unrealistic, we push non-working code. Prod comes up with bugs, and we get a new sprint to resolve those bugs.
Me: ╰[ ⁰﹏⁰ ]╯10 -
This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.
Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.
And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.
And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."
We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.
And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.4 -
CEO: if we would not give new features, clients would be bored and would not pay for tool.
me: but don't you think we should fix buggy old code, that would reduce effort and time that we daily invest in prod bugs?
CEO: I'm not saying we should not fix them but we should maintain the balance which is 80-20. 80% of our work would include adding new features.
😑
Next day in morning receives email:
There is a production issue, fix it asap.
😬10 -
WHY THE FUCK ARE YOU ASSIGNING PROD BUGS WHEN I'M ON A FUCKING VACATION ?!?
Oh wait I wrote that code...
Welp6 -
These fucking imbecile devs left me a bunch of bugs in a 150k LOC legacy hell...
Too bad it took some 8 months for them to finally surface in prod. God damned...
Technically, this whole codebase is one giant abyss filled to the brim with bugs.6 -
FUCKING FUCK ANGULAR!!!!
LIKE FUCK IT IN THE ARSE AND BURN THE MOTHERFUCKER WHILE LAUNCHING A MISSILE ON IT TO BE SURE!
(ノ≧∇≦)ノ ミ ┸━┸
So I am making something on angular and I got everything running in ng serve(development environment) , after handling all issues and showing it to my boss man he approves and asked to put it up on prod for a demo , doesn’t sound like an issue , I make the prod build on cli and BAM! 16 errors ? No issues right?, I’ll just google the issue. Googles.... there aren’t no clear solutions to it as the angular version keeps changing and nobody knows what broke it, I mean people have the issue,but like 100 reasons that can cause it,
HOLY LORD RELEASE A NEWER VERSION AFTER MENDING THE OLD ONE
But nooooooo!
Angular Dev:We fucked this one, lol what should we do boss man?
Angular boss man: lol just leave it, we need to build the new version with newer bugs,
P.S. I like angular, but it’s like a underdeveloped framework, too many issues and too many changes2 -
When bugs are seen by the client and boss therefore asks me "did you know about this bug?", what I'd really like to answer is:
"well shit, no! I would have solved that or at least told you about it, don't you think? what kind of fucking question is that?"
But then I just answer "no, lemme check"2 -
You know how my 'about' here says I create features from bugs?!? Well if you didn't before, now you do.. :P
Anyhow, today the most bizzare thing happened.. Customer played 'reverse uno card' on me..
Meaning?!
They reported a feature missing on uat env, when in fact I fixed a bug they have on prod..
Not sure how I should feel about this, but it sure made my day! (: I just hope they will not open a bug report for this missing bug..4 -
We had 1 Android app to be developed for charity org for data collection for ground water level increase competition among villages.
Initial scope was very small & feasible. Around 10 forms with 3-4 fields in each to be developed in 2 months (1 for dev, 1 for testing). There was a prod version which had similar forms with no validations etc.
We had received prod source, which was total junk. No KT was given.
In existing source, spelling mistakes were there in the era of spell/grammar checking tools.
There were rural names of classes, variables in regional language in English letters & that regional language is somewhat known to some developers but even they don't know those rural names' meanings. This costed us at great length in visualizing data flow between entities. Even Google translate wasn't reliable for this language due to low Internet penetration in that language region.
OOP wasn't followed, so at 10 places exact same code exists. If error or bug needed to be fixed it had to be fixed at all those 10 places.
No foreign key relationships was there in database while actually there were logical relations among different entites.
No created, updated timestamps in records at app side to have audit trail.
Small part of that existing source was quite good with Fragments, MVP etc. while other part was ancient Activities with business logic.
We have to support Android 4.0 to 9.0 of many screen sizes & resolutions without any target devices issued to us by the client.
Then Corona lockdown happened & during that suddenly client side professionals became over efficient.
Client started adding requirements like very complex validation which has inter-entity dependencies. Then they started filing bugs from prod version on us.
Let's come to the developers' expertise,
2 developers with 8+ years of experience & they're not knowing how to resolve conflicts in git merge which were created by them only due to not following git best practice for coding like only appending new implementation in existing classes for easy auto merge etc.
They are thinking like handling click events is called development.
They don't want to think about OOP, well structured code. They don't want to re-use code mostly & when they copy paste, they think it's called re-use.
They wanted to follow old school Java development in memory scarce Android app life cycle in end user phone. They don't understand memory leaks, even though it's pin pointed by memory leak detection tools (Leak canary etc.).
Now 3.5 months are over, that competition was called off for this year due to Corona & development is still ongoing.
We are nowhere close to completion even for initial internal QA round.
On top of this, nothing is billable so it's like financial suicide.
Remember whatever said here is only 10% of what is faced.
- An Engineering lead in a half billion dollar company.4 -
Rolled out a new application I built almost entirely by myself 2 days ago... But my dev group is understaffed and has a project manager who is literally the most clueless person I have ever met, so as a result, we don't have a functional/useful dev/test/prod framework and no standards for how to deploy apps. So my past 2 days were comprised of fixing bugs in the live system that could probably have been caught if I had the time and resources to get everything thoroughly tested. It's stable now, but damn our management for being generally idiots. Our motto appears to be "Fuck it, we'll do it live"1
-
A couple of years ago I was working on a fairly large system with a complex (by necessity) access control architecture.
As is usually the case with those projects, it's awkward for developers to repro bugs that have to do with a user's accesses in production when we are not allowed to replicate production data in test, let alone locally.
We had a bug where I ended up making myself a new row in the production database for a thing I could have access to without affecting real data to repro it safely. I identified the bug so I could repro it in dev/test and removed the row and ensured everything worked normally, whew scary.
Have you ever walked into the office one day, and everyone is hunched over in a semicircle around one person's workstation, before one turns around to look at you and says - after a pause - "... ltlian?.."
Turns out I had basically "poisoned the well" with my dummy entity in a way where production now threw 500 for everyone BUT me who had transitive access to this post-non-entity. Due to the scope of the system, it had taken about a day for this to gradually propagate in terms of caching and eventual consistencies; new entities coming in was expected, but not that they disappear.
Luckily I had a decent track record for this to be a one-off. I sometimes think about how I would explain testing in prod and making it faceplant before going home for the day, other than "I assumed it would be fine". I would fire me.3 -
ideal sprint fallacy.
total days 10 , total hours(excluding breaks ) 8 hrs per day= 80 hrs per dev
code freeze day = day 8, testing+ fixing days : 8,9,10. release day : day 10
so ideal dev time = 7days/56 hr
meetings= - 1hr per day => 49 hrs per dev
- 1 day for planning i.e d1 . so dev time left . 6 days 42 hrs.
-----------
all good planning. now here comes the messups
1. last release took some time. so planning could not happen on d1. all devs are waiting. . devtime = 5 days 35 hrs.
2. during planning:
mgr: hey devx what's the status on task 1?
d: i integrated mock apis. if server has made the apis, i will test them .
mgr : server says the apis are done. whats your guestimate for the task completion?
d : max 1-2 hrs?
m : cool. i assign you 4 hrs for this. now what about task 2?
d : task told to me is done and working . however sub mgr mentioned that a new screen will be added. so that will take time
m : no we probably won't be taking the screen. what's your giestimate?
d : a few more testing on existing features. maybe 1-2 hrs ?
m: cool
another 4 hrs for u. what about task 3?
d : <same story>
m : cool. another 4 hrs for u. so a total of 12 hrs out of 35 hrs? you must be relaxed this sprint.
d : yeah i guess.
m cool.
-------
timelines.
d1: wasted i previous sprint
d2 : sprint planning
d3 : 3+ hrs of meetings, apis for task 1 weren't available sub manager randomly decided that yes we can add another screen but didn't discussed. updates on all 3 tasks : no change in status
d4 : same story. dev apis starts failing so testing comes to halt.
d5 : apis for task1 available . task 3 got additional improvement points from mgr out of random. some prod issue happens which takes 4+ hrs. update on tasks : some more work done on task 3, task 1 and 2 remains same.
d6 : task1 apis are different from mocks. additionally 2 apis start breaking and its come to know thatgrs did not explain the task properly. finally after another 3+ hrs of discussion , we come to some conclusions and resolutions
d7 : prod issue again comes. 4+ hrs goes into it . task 2 and 3 are discussed for new screen additiona that can easily take 2+ days to be created . we agree tot ake 1 and drop 2nd task's changes i finish task 2 new screens in 6 hrs , hoping that finally everything will be fine.
d8 : prod issue again comes, and changes are requested in task 2 and 3
day 9 build finally goes to tester
day 10 first few bugs come with approval for some tasks
day 11(day 1 of new sprint) final build with fixes is shared. new bugs (unrelated to tasks. basically new features disguised as bugs) are raised . we reject and release the build.
day 2 sprint planning
mgr : hey dev x, u had only 12 hrs of work in your plate. why did the build got delayed?
🥲🫡5 -
Lessions I learned so far from my first big node/npm project with tons of users:
1) If you didn't build something for a while, expect 3 hours of resolving version conflicts for every two weeks since the last build.
2) Even if the tests pass, run the containers on your own machine and make sure that the app doesn't randomly crash before deploying
3) Even if the app seemed to work on your own machine, run the tests again in an environment mimicking prod at most 15 minutes before replacing the running containers.
4) Even if all else indicates that the app will work, only ever deploy if you expect to be available within the 4 hours following a deployment.
5) Don't use shrinkwrap for anything other than locking every version down completely. A partial shrinkwrap will produce bugs that are dependent on the exact hour you built the app _and_ the shrinkwrap file, and therefore no one will ever have seen them other than you.
6) Avoid gyp, and generally try not to interface too much with anything that doesn't run on node. If parts of your solution use very different toolchains, your problems will be approximately proportional to the amount of code. And you'd be surprised just how much code you're running. (otherwise it's more logarithmic because the more code the less likely a new assumption is unique)
7) Do not update webpack or its plugins or anything they might call unless you absolutely need to
8) Containers are cool but the alpine ones are pretty much useless if you have even just one gyp module.
9) There's always another cache. To save yourself a lot of pain, include the build time in every file or its name that the browser can download, and compare these to a fresh build while debugging to assert that the bug is still present in the code you're reading
+1) Although it may look like it, SQLite is far from a simple solution because the code and the bindings aren't maintained. In fact, it'll probably be more time consuming than using a proper database.3 -
I just submitted my professional goals this year.
As I was writing them a lot of them actually sound like things my manager should be doing to increase code quality n agility n reduce the # bugs that end up in prod... -
Not sure what's better on Friday:
Fix the bugs and push it to prod so manager will be happy.
Or sit there like a dumb fuck and wait for Monday before touching the code2 -
i have seen bugs getting into prod but not the local language getting into prod. what language is this?15
-
Just before deploying this to prod found this bug... discover this bug with bunch of print statements7
-
Background: We switched from just simple old PHP and JS using notepad++ to PHPStorm and its infinite configurables, Symfony 4, Twig, Composer, Doctrine, Yarn, NPM, Bootstrap, ( thank the stars we didn't try to add Docker in with all this ), any other junk I'm missing here? Then upgraded to Symfony 5.
Symfony's autowiring: madness behind the curtains. I get frustrated about when and where I can just magically inject these dependencies or use config variables, you know, like the ones you define in service.yaml. Hmmm, "service".yaml. In a controller you can say getParameter() but in a service you have to inject the parameter, FROM THE "SERVICE".yaml!!! Autowiring drives me nuts. Ok, so we can supply dependencies using the constructor, that's great! Within a controller you never have to instantiate the object you're passing to the constructor (autowiring handles that). That's cool, weird when we you try to trace it for the first few times, but nice I guess. Feels like half-assin' it. What bugs me here is that it only works in controllers... I guess out of the box.. i'm not even sure. To get that feature to work for services you have to make some yaml edits. Right?Maybe? Some of the Symfony tutorials have you code up some junk then trash it. Change config then wipe that out and do X instead... so I have no idea what "out of the box" for Symfony really is.
Found this cool article that describes my frustrations in better terms and seems like a good resource to learn about autowiring. I need to continue my yaml wizardry classes. https://alanstorm.com/symfony-autow...
.....And on to YAMLs, or CSS, or JS or any other friggin' change you make to a file anywhere... Make a change, reload page, nothing... nope you have to do some hidden cheat combo of yarn dostuff -> cache:clear -> cache:warmup -> cache:cache:the:cache ... I really really hate this crap. Maybe I'm too old school for all this junk. It was simple with pure PHP. Edit code, push file, reload page, and oh look it changed! Done. So happy! Ok, Ok. Occasionally the js or css might get cached by the browser and you have to ctrl/f5 or Shift/f5 .. one of those. With this framework there's just so much more that you have to remember to do get some new feature of your site loaded.
Now, I totally get wanting to use some type of entity framework, but I feel like my entire world turned backwards. Designing tables using something like MySQL Workbench made sense. I can see all the columns and datatypes right there as i'm building them. From what I've experienced now with Symfony/Doctrine is you have to make and entity, get a shit-ton of question lobbed at you and if it's a relation field you have to really have a clear idea of the cardinality up front. Then we migrate that to the database. Carefully read through the SQL if you really really just want to use migrations:migrate in Prod. That alter table could cost you some some downtime if your table is large.
Some days man.... -
Not exactly related to the topic but the exact thing is chilling the fuck out .
I always was anxious and was completely paranoid about minor bugs in my application during prod deployments(that is when I didn't know about testing utils and so on) , till the point that I couldn't fix a minor bug in the CSS and I puked 5 times over.
It was rough times but then I got over it and it really helped me alot.
I know bugs are like really not the kind of things you'd want to see in any application but it will arise in every application :3 -
so a good thing happened. after struggling with our current TL for whole last year, one SSE was promoted to TL and the team got split into 2. now our team has the new TL which is strict but a much more responsible lead and a good friend.
and in a striking change of culture, she has askedus to define our own KPIs rather than using the pre default KPIs. our predefined KPIs were weird :
- number of sprint spillovers >> to minimise
- number of POCs , learning sessions done >> min 2 in year
- number of prod bugs caused >> to minimise
-instancee of coding standards miss >> to minimise
i kind of excelled in all , yet got an 86/100 rating. previous TL was an asshole , so that also contributed to a lower rating without reasoning.
but since now i have the opportunity, what do you suggest should be ideal KPIs for a software engineer 1?1 -
Me: 'here we go, code working completely as intended, tested and without bugs.'
Senior after reviewing code: 'apart from the formatting errors, I'd also do this piece of coding in a different manners'
*Comments exchange in pull request*
Me: 'well this seems more like a change the whole logic request rather than a small improvement, I'll keep it like this and resolve it like suggested on a future opportunity'
Still in prod.