Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Assigned to a new project team..

Using git, in a creative way. So.. "master" is "dev" branch, usually. Everyone can push their branch to dev server .. so it's "dynamic for us". Production branch is whatever, as long as the branch has the release version. Sometimes, the release comes from "master".. that mean "dev" in normal geek..

That's just Git. The source code is a saturated spagetti of Entity framework and Caliburn. It is littered with antipatterns, especially basebean. Holy Christmas and Easter that baseclass do a lot of stuff that has no place as a base class ..

Fucking frameworks, I'm gonna start to evangelize frameworks as the no1 antipattern.

MS SQL as the main DB, but is dumped to json FILES through a scheduled task to increase read performance on web.

There is a soap endpoint to expose the json files, fml..

I am assuming I was placed here to improve stuff, I have never in my life seen anything like this before.

There is a special place in hell for this repository

Who the hell returns a SOAP response with a CDATA node containing a text string that is a concatenation of XML document?

Someone hates me...

Tomorrow and the whole next week, another SOAP adventure is waiting for me...shit...

It was the last year of high school.

We had to submit our final CS homework, so it gets reviewed by someone from the ministry of education and grade it. (think of it as GPA or whatever that is in your country).

Now being me, I really didn’t do much during the whole year, All I did was learning more about C#, more about SQL, and learn from the OGs like thenewboston, derek banas, and of course kudvenkat. (Plus more)

The homework was a C# webform website of whatever theme you like (mostly a web store) that uses MS Access as DB and a C# web service in SOAP. (Don’t ask.)

Part 1/2:

Months have passed, and only had 2 days left to deadline, with nothing on my hand but website sketches, sample projects for ideas, and table schematics.

I went ahead and started to work on it, for 48 hours STRAIGHT.

No breaks, barely ate, family visited and I barely noticed, I was just disconnected from reality.

48 hours passed and finished the project, I was quite satisfied with my it, I followed the right standards from encrypting passwords to verifying emails to implementing SQL queries without the risk of SQL injection, while everyone else followed foot as the teacher taught with plain text passwords and… do I need to continue? You know what I mean here.

Anyway, I went ahead and was like, Ok, lets do one last test run, And proceeded into deleting an Item from my webstore (it was something similar to shopify).

I refreshed. Nothing. Blank page. Just nothing. Nothing is working, at all.

Went ahead to debug almost everywhere, nothing, I’ve gone mad, like REALLY mad and almost lose it, then an hour later of failed debugging attempts I decided to rewrite the whole project from scratch from rebuilding the db, to rewriting the client/backend code and ui, and whatever works just go with it.

Then I noticed a loop block that was going infinite.

NEVER WAIT FOR A DATABASE TO HAVE MINIMUM NUMBER OF ROWS, ALWAYS ASSUME THAT IT HAS NO VALUES. (and if your CPU is 100%, its an infinite loop, a hard lesson learned)

The issue was that I requested 4 or more items from a table, and if it was less it would just loop.

So I went ahead, fixed that and went to sleep.

Part 2/2:

The day has come, the guy from the ministry came in and started reviewing each one of the students homeworks, and of course, some of the projects crashed last minute and straight up stopped working, it's like watching people burning alive.

My turn was up, he came and sat next to me and was like:

Him: Alright make me an account with an email of asd@123.com with a password 123456

Me: … that won't work, got a real email?

Him: What do you mean?

Me: I implemented an email verification system.

Him: … ok … just show me the website.

Me: Alright as you can see here first of all I used mailgun service on a .tk domain in order to send verification emails you know like every single website does, encrypted passwords etc… As you can see this website allows you to sign up as a customer or as a merc…

Him: Good job.

He stood up and moved on.

YOU MOTHERFUCKER.

I WENT THROUGH HELL IN THE PAST 48 HOURS.

AND YOU JUST SAT THERE FOR A MINUTE AND GAVE UP ON REVIEWING MY ENTIRE MASTERPIECE? GO SWIM IN A POOL FULL OF BURNING OIL YOU COUNTLESS PIECE OF SHIT

I got 100/100 in the end, and I kinda feel like shit for going thought all that trouble for just one minute of project review, but hey at least it helped me practice common standards.

What the hell am I!? I wonder if you guys can help me...

I've been programming most of my life but I've never actually been a developer by title or job role. I thought maybe if I list what I do and have done someone here could help? I'm sure there are more of you in a similar boat.

- C# and VB dev for some quick DBMS projects to help me understand and mine databases and create a nice simple view for project teams to show findings from the data to help make certain decisions.
- Automating a lot of my colleagues work with Python and if very restricted then just VBA macros in Excel and MSP. This did also include creating tools to gather data during workshops and converting the data for input into other systems.
- Brought Linux to the office with most team members now moving over to Linux with the peace of mind to know that though they do need to try solve their own problems, I can help if need be.
- Had to learn AWS and then implement an autoscaling and load balanced data center installation of a few Atlassian toolsets.
- Creating the architecture diagrams documentation needed for things like the above point.
- Having said that, also have ended up setting up all the Jira/Confluence etc. servers we use and have implemented so far whether cloud (Azure/AWS) or on prem and set up scripts to automate where possible.
- Implemented an automated workflow view in SharePoint based on SP list data and though in an ASPX page, primarily built in JS.
- Building test systems in PHP/JS with Laravel and Angular to help manage integration between systems. Having quite a time right looking into how to build middleware to connect between SOAP and REST API's, the trouble caused more by the systems and their reliance on frameworks we're trying to cut out of the picture.
- Working on BI and MI and training a team to help on the report creation so that I can do the fun creative stuff and then set them to work on the detail :)

Actually it seems safe to say that it seems that though I've finally moved into a dev office (beforehand being the only developer around) I seem to be the one they go to when a strategic solution is needed ASAP and the normal processes can't be followed (fun for someone with a CompSci degree and a number of project management courses under the belt... though I honestly do enjoy the challenges)

But I always end up Jack of all but master of, well hopefully some at least. let's not even get started on the tech related hobbies from circuit design and IoT to Andoid / iOS and game dev and enjoying a bit of pen testing to make sure we're all safe at work and at home.

As much as I don't like boxes, I'm interested to know if there is in fact a box for me? By the way, the above is just a snapshot of my last two years minus the project management work...