Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "last minute fix"
-
Worst dev team failure I've experienced?
One of several.
Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.
I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.
The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.
3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.
3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.
4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).
About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.
3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.
About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.
Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.
A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.
Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."11 -
As usual finished the task just an hour before demo meeting. That hour is for transportation. Obviously I didn't test nor rehearse.
As usual, in to 2 mins of demo and greeted by error page.
As usual
1) stay the fuck calm
2) this features was already demo-ed and fixed and went fine few weeks ago
3) what the fuck happen now
4) stay the fuck calm, smile.
5) "ah please give me one minute, I forgot to clean up some stuff while working on new features"
6) shit shit. read the error message and log
7) oh I did refactor some files last week. Reorganized the files and folders for better structure and easier understanding. Thought I corrected every occurrences. Obviously I missed few.
8) ssh to the server while screen is still showing on projector
9) dig into the file quick
10) stay the fuck calm
11) fix
12) refresh
13) sorry all good, so I was saying ....
Well finally it's done for today and going back to office. After all it went ok. 👌2 -
Team quarterly capacity planning:
- Confluence document created with a big table (+100 rows) by product / business. Each row is something that needs to be worked on for the coming quarter.
- Row 1 could be an Epic with 15 tickets attached. Row 2 could be adding a single log to our analytics. No consistency.
- For each row, we create a separate confluence document with the "technical details". 75% of the time these remain blank. 1% of the time there is something useful, the rest its a slightly longer version of the description from the bigger document.
- Each row gets a high level estimate by the leads. 50% of the time without sufficient background info to actually do get it accurate.
- These are then copied into the teams excel spreadsheet, where it will calculate if we are over/under capacity.
- We will go backwards and forwards between confluence and excel until we are "close enough" to under capacity without being too much.
- Once done, we then need to copy them into the org/division's excel spreadsheet. This document is huge, has every team on it and massive 50pt text saying "Do not put a filter on this document".
- Jira tickets + Epics will now be created for each one, with all the data be copied over by hand, bit by bit, by product. Often missing something.
- Last week, at the end of this process for Q2 (2 weeks late), 6 of the leads were asked to attend a 30 minute meeting to discuss how to group the line items together because we had too many for the bigger excel spreadsheet.
- This morning I was told business weren't happy with one of our decisions to delay one line item. Although they were all top priority (P0), one of them was actually higher than that again (P-1?) and we need to work it back in.
... so back to step 1
- Mid way through Q2, a new document will be created for Q3. Work items that didn't make the cut will be manually copied from one to the other. 50/50 whether anything that didn't get done on time in Q2 will make its way to the Q3 doc.
- "Tech excellence" / "Tech debt" items (unit/UI tests, documentation, logging, performance, stability etc) will never be copied over. Because product doesn't understand them and assumes therefore that they are unimportant.
==================
PS: I'd like to say this was a rare event for Q2, but no. Q4 and Q1 were so bad, we were made assurances from the director of engineering that he would fix this process for Q2. This is the new and improved process (I shit you not) that has resulted in nothing tangible.7 -
My code just broke during a presentation and I wasn't there to fix it. My CS professor got embarrassed in front of a bunch of future students.
I had made a last-minute change right before going on vacation... Never doing that again!4 -
PM: Ok Android, i've reviewed the latest build, you are good to release. Waiting on iOS's build to test.
Me: ... are we not holding all builds until we hear back from backend about that bug?, as we likely have to change something on our side?
PM: Which bug?
Me: ... the only one we discussed yesterday in the team meeting.
PM: How many customers is it affecting?
Me: that we know of, one ... the CEO of our company
PM: oh that one, yeah were not doing that anymore.
Me: WHAT? i've been waiting all day / night to hear back. Why are we not doing this?
PM: ¯\_(ツ)_/¯ ... Everything is too messy at the minute anyway, the release plan is changing every day. Need to keep it back in line.
Me: ... the plan has changed exactly once. We had a plan at the start of last week for the last release, we changed it YESTERDAY to include 2 critical bug fixes. The only issue with the plan changing is nobody telling us these aren't bugs anymore
PM: We can discuss tomorrow in the team meeting.
CEO: oh hey guys, yeah we pulled that bug fix. Its not really a bug, more like a missing feature. No way it will get done before xmas. Going to live with the way it works for now and fix it properly next year.
Me: Ok, fair enough, but we really need to be told these decisions.
CEO: sure, sorry, didn't think anyone was blocked by this. What was the blocker?
Me: ..... you asked me yesterday to get this bug fix in the build ... you asked for the final build to be made today so we can go through the app store review. As we all discussed yesterday, today is kind of the last day we can really do this.
CEO: ok, its late, we can discuss this tomorrow in the team meeting.
Me: ..... ..... ..... ..... sure7 -
Today my raspberry pi media center bricked itself (at least it won't boot properly. Than I thought I just format the SD card and reinstall everything. But than my windows pc won't boot properly because it's still running on old hdd and I suck at building PCs. Than I tried my ThinkPad with antergos and remembered that it is also dead because the last update broke something. And now I'm trying to boot my windows at least into safe mode and my ThinkPad to boot from the live stick to chroot and fix it. Still waiting since 15min for any progress.
Now is my old Oneplus One with an outdated nightly of a custom Rom my only working connection to the web.
I'm starting to think that waiting for the last minute to fix problems might not the best way for me.10 -
"four million dollars"
TL;DR. Seriously, It's way too long.
That's all the management really cares about, apparently.
It all started when there were heated, war faced discussions with a major client this weekend (coonts, I tell ye) and it was decided that a stupid, out of context customisation POC had that was hacked together by the "customisation and delivery " (they know to do neither) team needed to be merged with the product (a hot, lumpy cluster fuck, made in a technology so old that even the great creators (namely Goo-fucking-gle) decided that it was their worst mistake ever and stopped supporting it (or even considering its existence at this point)).
Today morning, I my manager calls me and announces that I'm the lucky fuck who gets to do this shit.
Now being the defacto got admin to our team (after the last lead left, I was the only one with adequate experience), I suggested to my manager "boss, here's a light bulb. Why don't we just create a new branch for the fuckers and ask them to merge their shite with our shite and then all we'll have to do it build the mixed up shite to create an even smellier pile of shite and feed it to the customer".
"I agree with you mahaDev (when haven't you said that, coont), but the thing is <insert random manger talk here> so we're the ones who'll have to do it (again, when haven't you said that, coont)"
I said fine. Send me the details. He forwarded me a mail, which contained context not amounting to half a syllable of the word "context". I pinged the guy who developed the hack. He gave me nothing but a link to his code repo. I said give me details. He simply said "I've sent the repo details, what else do you require?"
1st motherfucker.
Dafuq? Dude, gimme some spice. Dafuq you done? Dafuq libraries you used? Dafuq APIs you used? Where Dafuq did you get this old ass checkout on which you've made these changes? AND DAFUQ IS THIS TOOL SUPPOSED TO DO AND HOW DOES IT AFFECT MY PRODUCT?
Anyway, since I didn't get a lot of info, I set about trying to just merge the code blindly and fix all conflicts, assuming that no new libraries/APIs have been used and the code is compatible with our master code base.
Enter delivery head. 2nd motherfucker.
This coont neither has technical knowledge nor the common sense to ask someone who knows his shit to help out with the technical stuff.
I find out that this was the half assed moron who agreed to a 3 day timeline (and our build takes around 13 hours to complete, end to end). Because fuck testing. They validated the their tool, we've tested our product. There's no way it can fail when we make a hybrid cocktail that will make the elephants foot look like a frikkin mojito!
Anywho, he comes by every half-mother fucking-hour and asks whether the build has been triggered.
Bitch. I have no clue what is going on and your people apparently don't have the time to give a fuck. How in the world do you expect me to finish this in 5 minutes?
Anyway, after I compile for the first time after merging, I see enough compilations to last a frikkin life time. I kid you not, I scrolled for a complete minute before reaching the last one.
Again, my assumption was that there are no library or dependency changes, neither did I know the fact that the dude implemented using completely different libraries altogether in some places.
Now I know it's my fault for not checking myself, but I was already having a bad day.
I then proceeded to have a little tantrum. In the middle of the floor, because I DIDN'T HAVE A CLUE WHAT CHANGES WERE MADE AND NOBODY CARED ENOUGH TO GIVE A FUCKING FUCK ABOUT THE DAMN FUCK.
Lo and behold, everyone's at my service now. I get all things clarified, takes around an hour and a half of my time (could have been done in 20 minutes had someone given me the complete info) to find out all I need to know and proceed to remove all compilation problems.
Hurrah. In my frustration, I forgot to push some changes, and because of some weird shit in our build framework, the build failed in Jenkins. Multiple times. Even though the exact same code was working on my local setup (cliche, I know).
In any case, it was sometime during sorting out this mess did I come to know that the reason why the 2nd motherfucker accepted the 3 day deadline was because the total bill being slapped to the customer is four fucking million USD.
Greed. Wow. The fucker just sacrificed everyone's day and night (his team and the next) for 4mil. And my manager and director agreed. Four fucking million dollars. I don't get to see a penny of it, I work for peanut shells, for 15 hours, you'll get bonuses and commissions, the fucking junior Dev earns more than me, but my manager says I'm the MVP of the team, all I get is a thanks and a bad rating for this hike cycle.
4mil usd, I learnt today, is enough to make you lick the smelly, hairy balls of a Neanderthal even though the money isn't truly yours.4 -
Best part of being a dev? Rock star status when things are going well: had to get a fix out by 18:00, boss walks by as I'm watching something on YouTube and tells me the client called and wanted to be sure we'd have it done in time, told him not to worry and went back to my video. Pushed the fix at 17:30, and tomorrow I'll be in the same position: every last minute fix delivery only grows my reputation for getting critical last-minute fixes out in time... As skeptical as my boss was at the time when he walked past and saw me on YouTube he had to he polite because he knows how crucial I am to this project
I think tomorrow I'm going to work on my own project and slack off, no one will be able to tell me different2 -
I work for a bank and every production release date it's a chaos... Like, for real, devs running to get their stories approved by the testing team and last minute scope changes that, if not made, would make the whole app fail (real shitty management as you can see).
Longstoryshort, a dev didn't finished one of his stories and create 7 major bugs with another... Today that was my breakfast, took me 4 hours and get it all done and approved... We didn't make the release tho, but I scored some major points with this.
Funny thing, tomorrow I'm telling my PM I'll leave the company for a better job, so that will be their breakfast.6 -
Error, coffee, fix, bug, coffee, fix, 1am!
Time for bed, last minute compile..
Error.
Bugger this, I'm going to be a dam hooker!7 -
About a year ago, I started a new position as a Full Stack Java Developer. When I started my employer got me a brand new, shiny, Asus laptop. As I prefer Linux (mint) to perform my magic I had to whipe Windows 10 and reinstall it. It turned out that my new shiny laptop was in fact so shiny that Linux (mint) didn't support/contain all the necessary drivers (yet), especially the network/bluetooth drivers and the gfx's drivers turned out a bit of a pain.. Over the year things slowly got better with every new kernel update that came in. However, due to me trying to fix things before those updates, Linux also had become somewhat unstable.
So ... last week I took some time to re-install that laptop and also take the opportunity to upgrade from Linux mint 18 to Linux mint 19 ... or so I thought ... Linux mint 19 was running (very) hot to the point where the laptop would shutdown due to the MOBO's thermal protection mechanims kicking in. ... Ok ...maybe Linux mint 19 was not such a good choice .... let's see if Ubuntu 18.04 is an option ... Nope ... Linux would lock up within a minute after booting up ... no mouse, no keyboard ... nothing. .... *sigh* ... let's (re)install Linux Mint 18.3 again ... and behold, I can start performing magic again.
Linux, it can be such a pain at times. I still prefer it, but running into all those 'weird' things on my laptop when reinstalling, I have to admit I have seriously considered 'just' installing windows 10 again and be done with it. Luckily I could also remind myself of what a pain Windows is to do serious docker/java development in comparison to Linux which gave me the strength to keep going ... :)6 -
Manager: client is breathing down our necks for that sockets interface. Wrap up any last minute changes, fix the documentation and get everything posted!
Me:*does all the things.*
That was two days ago. Client hasn't even logged in to download the software and the documentation.1 -
Oh man, its been forever since I've had an actual rant.
so my work ethic is to the point where it's all last minute. My eduction is all last minute. Personal problem, and don't know how to fix that. but it's just getting out of hand.
tbh, I'm at the point of considering dropping uni like this is no joke. maybe transfer to a cheaper because the financials are no good either.
I also need a new job because the place I'm at is no good. here a few things about it:
1) Its Industrial, not really tech related
2) the dudes expect ME to GO TO THEM and ask for help. Not how I roll
3) not the best atmosphere -- I don't really like the 4 total employees, including myself
4) nearly minimum wage
the pros?
1) I learn about my car
2) I can use the shop to fix my car
3) Free stuff (for example, a projector and lunch everyday
4) We're getting a server (soon?)
5) I buy computers for them, they pay me
But seriously, my grades in school are slipping (nowhere dangerous yet) and I am too stressed. At least I'll be getting in more dev work
Moreover, I want to get in some actual learning with Swift, but I can never manage to make time. Plus, games are a thing that I do, also family and friends, also religion is a thing, also work and school, also sleep. No time? Me neither.
Like the organization of this rant? Me too.4 -
I'm a student but I've been working backend and frontend for about a year and a half now. I know just enough to be frustrated whenever a teacher says the words "group" and "project". Anyway, there was an assignment due yesterday for my group, and it was working more or less perfectly, then at the last minute, this guy gets on and tries to "fix" an issue that the teacher said specifically we didn't have to solve.
HE. BROKE. EVERYTHING. And then he pushed straight to master. Ironically, the program still ran, it appeared to be encrypting and decrypting correctly, but he basically removed one of the algorithms we were supposed to implement. I think the professor will give us a better grade than we deserve but still...2 -
Ended up with 16.04 and 18.04 ubuntu in dual boot. (see last rant).
Fixed graphics issues on 18.04. The fix envolved setting stuff in grub. Ok.
Now we apparently need 16.04 again and can kich the 18.04.
np. Let me just boot my 16.04
oh look who has graphics issues now!
It boiled down to the grub settings. so Depending on if I need 18.04 or 16.04 I need to fiddle with grub. great.
And now we don't neet 18.04 at all. So I tried to do something not in the last minute once, and it backfired completly. Wasted 5h on this shit this week. Fk this.2 -
!rant
When a very last minute change is required and you panic a bit, thinking it will take days to fix... but you manage to fix it before lunchtime because the API is well made.1 -
One of my Computer Science modules this year revolved around completing a team project, and one person in the team basically fucked it up for all of us in the last minute.
We had to create a simple task management app for a fictional company, the university did not care about how the program looked and all that mattered was if the app is functional or not. The app relied heavily on a database, so all we basically had to do was get, modify, and add data from a database. Now this person did his part of the programming, but with an outdated database model and did not even test his code as he said MySQL wasn't working on his home computer.
2 days before the final deadline is when we decided to merge everything together in the git repo (as that's when the rest of us finished our tasks), and that's when we found out none of his code worked. We then spent the next 48 hours with little sleep to try our best to fix everything, but unfortunately due to his tasks carrying a majority of the complexity of the program we couldn't fix it all in time and we ended up losing roughly 50% of the marks.
This all probably could have been avoided if one person in the team did look at his git branch properly, but this person was the programming lead of the project and didn't ask for any help at any point until the last moment when we merged everything together. Oh well though, at least I've learnt better for the next team project that I do2 -
I was working on CakePHP as part of class project. I had to demonstrate AJAX. So I created some text files filled with random Wikipedia articles. On drop down item change, my AJAX handler would fetch the content of corresponding file and return it back.
But the problem was, it wasn't returning just the file content, but the whole HTML document. So the calling page would get another set of header, menu, footer and all! There was no time to fix layout for AJAX calls as it was added at last minute before the presentation. So, I just hid the duplicate menu etc using CSS and went for the presentation. It passed with flying colours.
So, if you can't fix something, just hide it! 😂