Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Worst dev team failure I've experienced?
One of several.
Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.
I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.
The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.
3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.
3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.
4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).
About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.
3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.
About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.
Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.
A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.
Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."11 -
A project got pushed live before it was finished, the final piece was supposed to enter site visitors into a draw to win a very expensive prize, but the first few thousand visitors hit the unfinished placeholder page...telling them they'd won said prize. Got noticed pretty quickly, not before a few million had to be claimed in business insurance to pay several thousand 'winners' off. The finger-pointing in the aftermath was quite fun to watch.
QA is essential. -
the team gave access to the database to another team, and one of their guys did a delete cascade by mistake thinking it was theirs... everyone was scrambling for days trying to get a backup, thankfully there was one3
-
Weeeell…
I have been in a team where every week they had this long and complex task of refactoring everything and changing lots of assets (~2 days of work every 2 weeks) cause the senior tards refused to use a script for it. Told them to use the goddamn backend or a script… the answer? “But that would take at least two days of work! Maybe even three! (As a one time job)” Math you ducking ducks! The second time you use it you are in a time profit!2 -
Ok apparently I forgot rants can only be edited within the first 5 minutes, I thought it was 30, and you can't rant 2 times in 2 hours so I'll have to wait before posting this.
So, I'm doing a Genetic Algorithms class, something I liked since I was 15 yo and didn't know shit about coding, but I loved the carykh videos about it. (here is part 1: https://youtu.be/GOFws_hhZs8 )
The yearly class consisted of 3 little projects to be able to do the final exam and an investigation project to pass the subject without a final exam.
We had to make teams, and I got together with 5 more people.
I have a lot to say about these 5 people, but the only thing I'll say is that I was the most experienced programmer among the 6 of us, if they had any experience at all. Mind this is a third cycle class.
We were allowed to use any technology, as long as we wrote the important algorithms by hand, of course.
The development of the first project was such a mess, that one of the members left the subject.
While developing the second one, we were given the topic for the investigation project; fractals.
It took a lot for us to find an application of fractals where we could use genetic algorithms. Once we found it, fractal antennas, we had to learn about antennas, so we interviewed professionals, and such. We ended up learning to evaluate antennas.
We also found a site that used some parameters to generate fractals, we had the parameterization.
We just had to code it. It was July and we just had to code it by October.
We were 5 people, and "we" were so busy writing the little projects, we fucking couldn't finish the investigation project.
We just had to write the proper algorithms and GUI specifics, without even having to write boilerplate (we used the first project as a template), and they still took so much that we didn't have time for the important project.
That sucked, because I had been coding and investigating in many weekends, I spent countless hours on them, I had to pause development on other projects for these ones; and after all that we have to do the (very shitty) final exam.
Since May, the average people together "working" on the different projects was 2.6. And 100% of the time, I was one of them.
We tried to speed up things in the last months but even with the deadline on us and the project not even started, there was no time we all got to work together.
Dude projects don't just get made, someone has to develop them.
It's so sad we had the project ready to be made and 5 people couldn't finish it. There was so little to do to pass and yet these people couldn't.
I guess it's my bad too. I wish I could rush the project in a couple of weeks, but unfortunately the guy with a job and 8 other subjects can't.
You can find the project in my GitHub. I'll do a requiem of what it was to be one of these days, after I catch up with all I left aside for this subject...3 -
MY GENETIC ALGORITHMS INVESTIGATION PROJECT. I WANTED TO RANT ABOUT THAT SO THIS WEEKLY RANT IS PERFECT. I can't write the whole rant rn, stay tuned.5
Top Tags
Weekly Rant
View