Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

I realize I've ranted about this before, but...

Fuck APIs.

First the fact that external services can throw back 500 errors or timeouts when their maintainer did a drunk deploy (but you properly handled that using caching, workers, retry handlers, etc, right? RIGHT?)...

Then the fact that they all speak a variety of languages and dialects (Oh fuck why does that endpoint return a JSON object with int keys instead of a simple array... wait the params are separated with pipe characters? And the other endpoint uses SOAP? Fuck I need to write another wrapper class around the client...)

But the worst thing: It makes developers live in this happy imaginary universe where "malicious" is not a word.

"I found this cloud service which checks our code style" — hmm ok, they seem trustworthy. Hope they don't sell our code, but whatever.

"And look at this thing, it automatically makes database backups, just have to connect to it to DigitalOcean" — uhhh wait...

"And I just built this API client which sends these forms to be OCR processed" — Fuck... stop it... there are bank accounts numbers on those forms... Where's that API even located? What company?

* read their privacy policy *

"We can not guarantee the safety of your personal data, use at your own risk [...] we are located in Russia".

I fucking hate these millennial devs who literally fail to get their head out of the cloud.

Somehow they think it's easier to write all these NodeJS handlers and layers around some API, which probably just calls ImageMagick + Tesseract on the other side.

If I wasn't so fucking exhausted, I'd chop of their heads... but they're like hydra, you seal one privacy breach and another is waiting to be merged, these kids just keep spewing their crap into easy packages, they keep deploying shitty heroku apps... ugh.

😖

Start a development job.
Boss: "let's start you off with something very easy. There's this third party we need data from. They have an api, just get the data and place it on our messaging bus."
Me: "sure, sounds easy enough"

Third party api turns out to have the most retarded conversation protocol. With us needing a service to receive data on while also having a client to register for the service. With a lot of timed actions like, 'send this message every five minutes' and 'check whether our last message was sent more than 11 minutes ago'.
Due to us needing a service, we also need special permissions through the company firewall. So I have to go around the company to get these permissions, FOR EVERY DATA STREAM WE NEED!

But the worst of it all is... This whole api is SOAP based!!

Also, Hey DevRant!

Hi,
I'm not a ranty person so I never actually thought I'd post anything here but here it goes.

From the beginning.
We use ancient technologies. PHP 5.2, Symfony 1.2 and a non RFC complient SOAP with NO documentation.
A year ago We've been thrown a new temporary project. An VOIP app for every OS.
That being iOS, Android, MAC, PC, Linux, Windows mobile. With a 3 month deadline. All that thrown at 4 PHP developers. The idea being that They'll take it, sign the delivery protocol, everyone happy. No more updates for the app needed. They get their funds they needed the app for and we get paid.

Fast forward to today...

Our dev team started the year with great news that We'll most likely have to create a new project. Since the amount of new features would be far greater than current feature set, we managed to finally force our boss to use newer technologies (ie. seperate backend symfony4 PHP7+/frontend react, rest api and so on). So we were ecstatic to say the least. With preestimates aimed at a minimum 3 month development period. Since we're comfortable with everything that needs to be done.
Two days later our boss came to me that one of our most annoying clients needs a new feature. Said client uses ancient version written on a napkin because They changed half of the specification 2 weaks before deadline in a software made not by a developer but some sysadmin who didn't know anything. His MVC model was practically VVV model since he even had sql queries in some views. Feature will take 3 days - fixing everything that will break in the meantime - 1-2 months.
F*** it, fine. A little overtime won't kill me.
Yesterday boss comes again... Apparently someone lost a delivery protocol for a project we ended that half a year ago. Whats even better at the time when we asked for hardware to test we never got any. When we asked about any testing enviornment - nothing. The app being SEMI-stable on everything is an overstatement but it was working on the os'es available at the time. Since the client started testing now again, it turns out that both Android app does not work on 8.1/9 and the iOS app does not work on ios12. The client obviously does not want to pay and we can do little with it without the protocol, other than rewriting the apps.
It will take months at least since all of those apps were written by people that didn't know neither the OS'es nor the languages. For example I started writing the iOS one in swift. Only to learn after half of the development time, that swift doesn't like working by C Library rules and I had to use ObjC also. With some C thrown in due to the library. 3 unknown languages, on an unknown platform in 3 months. I never had any apple device in my hand at that time nor do I intend to now. I'm astonished it worked out then. It was a clusterf**k of bad design and sticking everything together with deprecated apis and a gum. So I'll have to basically fully rewrite it.

If boss decides we'll take all those at the same time I'll f***ing jump of a bridge.

Programming against a SOAP api, developed by the client in Visual FoxPro. Their whole system is in the language.

10 seconds API calls are "normal".

Luckely our backend rarely needs to talk to it 🎉

Strap in...

- Previous employer
- 3rd party partner firm
- integration link between both over SOAP
- Both sides riddled with poor code and messed up political structures (partner firm CEO is an investor in my employer)
- Doing a deployment to update to https (I know)
- Keep http endpoint live
- Other side starts shitting itself
- Diagnose
- Not us
- feelsgoodman.tiff
- Get angry email
- Explain not us
- Back and forth
- Tell client it’s “irrelevant” on https issue, it’s their side that’s gone wrong
- Get angry reply with boss cc’d about how nothing is “irrelevant” for the client
- We all had to have a make up meeting and meal
- Client was calm and reasonable, all agreed we just snapped and it wouldn’t happen again
- 2 weeks later
- Their system shits itself again and suddenly we’re on the hook
- BA on my team (smarmy little bastard) constantly fucking me off
- Get so close to actually screaming and hitting him

So yeah. I don’t tend to hold that a job is more important to me than my dignity.

I have and will never hold my tongue for the sake of a job, I’m not gonna put up with people shouting / belittling / backstabbing etc.

TLDR;
How much do you earn for your skill set in your country vs your cost of living?
BONUS;
See how much I & others earn.

Recently I became aware of just how massive the gap in developers earnings are between countries. I'd love to calculate a fixed score for income vs cost of living.

I know this stuff is sensitive to some so if you prefer just post your score (avg income p/m after tax / cost of living).

I'm not shy so I'll go first:

MY RATES
Normal Rate (Long term): $23
Consulting / Short term: $30-$74
Pen Test: $1500 once off.
Pen Test Fixes: consulting rate.
Simple work/websites: min $400+
Family & Friends: Dev friends are usually free (when mutually beneficial). Family and others can fuck off, even if they can pay (I pass their info to dev friends with fair warning).

GENERAL INFO
Experience: 9 years
Country: South Africa
Developer rareness in country: Very Rare (+-90 job openings per job seeker).
Middle class wage in country: $1550 p/m (can afford a new car, decent apartment & some luxuries like beer/eating out).
Employment type: Permanent though I can and do freelance occasionally.
Client Locality: Mostly local.
Developer Type: Web Developer (True web dev - I do anything web related from custom HTTP servers to sockets, services, advanced browser api's, apps & more).

STACKS / SKILLSETS

I'M PROFICIENT IN:
python, JavaScript, ASP classic, bash, php, html, css, sql, msql, elastic search, REST, SOAP, DOM, IIS, apache

I DABBLE WITH:
ASP.net, C++, ruby, GO, nginx, tesseract

MY SPECIALTIES:
application architecture, automation, integrations, db's, real time data, advanced browser apps/extensions (webRTC, canvas etc).

SUMMARY
Avg income p/m after tax: $2250
Cost of living (car+rent+food): $1200
Score: 1.85

*Note: For integrity when calculating my cost of living I excluded debt repayments and only kept my necessities which are transport, food & shelter.

I really hope you guy's post your results, it would be great to get an idea of which is really the worst / best country to be a developer in.

It was the last year of high school.

We had to submit our final CS homework, so it gets reviewed by someone from the ministry of education and grade it. (think of it as GPA or whatever that is in your country).

Now being me, I really didn’t do much during the whole year, All I did was learning more about C#, more about SQL, and learn from the OGs like thenewboston, derek banas, and of course kudvenkat. (Plus more)

The homework was a C# webform website of whatever theme you like (mostly a web store) that uses MS Access as DB and a C# web service in SOAP. (Don’t ask.)

Part 1/2:

Months have passed, and only had 2 days left to deadline, with nothing on my hand but website sketches, sample projects for ideas, and table schematics.

I went ahead and started to work on it, for 48 hours STRAIGHT.

No breaks, barely ate, family visited and I barely noticed, I was just disconnected from reality.

48 hours passed and finished the project, I was quite satisfied with my it, I followed the right standards from encrypting passwords to verifying emails to implementing SQL queries without the risk of SQL injection, while everyone else followed foot as the teacher taught with plain text passwords and… do I need to continue? You know what I mean here.

Anyway, I went ahead and was like, Ok, lets do one last test run, And proceeded into deleting an Item from my webstore (it was something similar to shopify).

I refreshed. Nothing. Blank page. Just nothing. Nothing is working, at all.

Went ahead to debug almost everywhere, nothing, I’ve gone mad, like REALLY mad and almost lose it, then an hour later of failed debugging attempts I decided to rewrite the whole project from scratch from rebuilding the db, to rewriting the client/backend code and ui, and whatever works just go with it.

Then I noticed a loop block that was going infinite.

NEVER WAIT FOR A DATABASE TO HAVE MINIMUM NUMBER OF ROWS, ALWAYS ASSUME THAT IT HAS NO VALUES. (and if your CPU is 100%, its an infinite loop, a hard lesson learned)

The issue was that I requested 4 or more items from a table, and if it was less it would just loop.

So I went ahead, fixed that and went to sleep.

Part 2/2:

The day has come, the guy from the ministry came in and started reviewing each one of the students homeworks, and of course, some of the projects crashed last minute and straight up stopped working, it's like watching people burning alive.

My turn was up, he came and sat next to me and was like:

Him: Alright make me an account with an email of asd@123.com with a password 123456

Me: … that won't work, got a real email?

Him: What do you mean?

Me: I implemented an email verification system.

Him: … ok … just show me the website.

Me: Alright as you can see here first of all I used mailgun service on a .tk domain in order to send verification emails you know like every single website does, encrypted passwords etc… As you can see this website allows you to sign up as a customer or as a merc…

Him: Good job.

He stood up and moved on.

YOU MOTHERFUCKER.

I WENT THROUGH HELL IN THE PAST 48 HOURS.

AND YOU JUST SAT THERE FOR A MINUTE AND GAVE UP ON REVIEWING MY ENTIRE MASTERPIECE? GO SWIM IN A POOL FULL OF BURNING OIL YOU COUNTLESS PIECE OF SHIT

I got 100/100 in the end, and I kinda feel like shit for going thought all that trouble for just one minute of project review, but hey at least it helped me practice common standards.

Requests to a soap server were failing randomly. In order to contact the API provider, I tried to provide an curl example with the same payload and the error response. Yet when sending the payload over curl, the request worked just fine. When my application was building the request, it failed.

What. The. Fuck.

I checked and double-checked the request body and headers. They were identical.

Of course, no error response was returned by the API provider and, of course, they could not tell me how what error I caused in my request.

So I created a basic dummy server, installed wireshark and compared the payload when sending a request from my application and from curl to my dummy server.

It turns out: curl, if called in a certain way, automagically strips out newlines. The soap client kept them.

So that that shitty soap server crashed due to newlines in the message body!

Stripping out the newlines was rather easy.

Shame on you, your house, and entire family for letting it crash due to them!

Working with a SOAP endpoint. I know it is some .NET server due to the style of stacktrace on exceptions. Nice, a framework where I can expect some type safety granted by static types. I build some xsl to transform the SOAP wsdl files into classes and structs to interact with the endpoint. Works out perfectly.

Plottwist!

Elements which are defined in the xsd/wsdl with maxOccur=unbounded and minOccur=0 should represent a simple collection of this type. Therefore does my implementation expect a collection of this type. But no. The shipped SOAP client in my stack ignores the definition and simply deserializes the SOAP response into T and not a collection of T.

Where the duck are the types when they are defined all over the place?

Me: there seems to be a problem in the Web Sphere app server...I would recommend u change it to weblogic

Client( IT division head of his company): is it compatible with websphere soap..??

Me: soap is generic, websphere is just an app server

Client: no but we have been told to use only websphere soap, is weblogic having that..??

Me: soap is protocol, app server is changeable..

Client: no we want only websphere soap.

Me:....(trying to find the nearest exit)

Today I spent 9 hours trying to resolve an issue with .net core integration testing a project with soap services created using a third party soap library since .net core doesn't support soap anymore. And WCF is before my time.

The tests run in-process so that we can override services like the database, file storage, basically io settings but not code.

This morning I write the first test by creating a connected service reference to generate a service client. That way I don't need to worry about generating soap messages and keeping them in sync with the code.

I sent my first request and... Can't find endpoint.

3 hours later I learn via fiddler that a real request is being made. It's not using the virtual in-process server and http client, it's sending an actual network request that fiddler picks up, and of course that needs a real server accepting requests... Which I don't have.

So I start on MSDN. Please God help me. Nope. Nothing. Makes sense since soap is dead on .net core.

Now what? Nothing on the internet because above. Nothing in the third party soap library. Nothing. At this point I question of I have hit my wall as a developer.

Another 4 hours later I have reverse engineered the Microsoft code on GitHub and figured out that I am fucked. It's so hard to understand.

2 more hours later I have figured out a solution. It's pure filth..I hide it away in another tooling project and move all the filth to internal classes :D the equivalent of tidying your room as a kid by shoving it all under the bed. But fuck it.

My soap tests now use the correct http client with the virtual server. I am a magician.

Many months into the project, we discovered that the client doesn't have a REST API for the data we needed from their side available at all - it's a legacy SOAP service.

I somehow got our Node.JS backend to do SOAP calls and use the XML results.

I'm not sure if I should feel proud or dirty at this point.

I tried SOAP for the first time today. It wasn't bad. I had a client for the service built automatically instead of writing my own.

Why is SOAP dead?

almost one entire day debugging why Soap Client gives some strange errors to find out that the 3rd party service we are calling is returning plain XML instead of a valid Soap message :(

4 hours wasted due to a rather strange "scope poisoning" bug... it was really fucking strange.
Had to move the 5 lines of code, above where the error happened, to a private function.
The 5 lines just called an XLSX library which read a given file and returned an array. I don't know how but the thing somehow messed with my Soap client call one line below. The culprit must have been some IO resource handle which didn't get destroyed when in the same scope due to persisting object references..