Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Worst dev team failure I've experienced?
One of several.
Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.
I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.
The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.
3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.
3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.
4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).
About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.
3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.
About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.
Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.
A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.
Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."11 -
My second year of high-school, we started having class in computer science. I was really looking forward to it cause I always wanted to learn programming.
On first sight it appeared that the professor which taught the class knew something, he looked like a genuine geek with those dorky glasses, briefcase and pants like Steve Urkel, but after couple of his lessons you could see he had no real dev experience and just basic understanding of programming in theory. He was more reading stuff from the book than he was trying to explain them to students and give some real world examples.
So it was just one these days, everybody got back from vacation, it's hot outside, the guy is just reading sentences from his book, half of students talk with each other and other half doesn't give a fuck about him or his class. Pretty sure I was the only one trying to listen to him and learn something from his recitals.
All of a sudden he notices the atmosphere in the classroom, slams the book shut, gives out couple of F-s to the loudest students and yells out loud "NONE OF YOU IN THIS ROOM WILL EVER ACCOMPLISH ANYTHING IN YOUR LIFE, BARE ALONE IN PROGRAMMING"
At first I felt like shit, but soon after that I started thinking "who the hell are you to tell me what I could or will accomplish in my life". Couple weeks later I've bought myself a first book in programming and started learning C++ late at night since I understood that I won't learn anything about programming in that school. Two years later I was correcting this same professor with his claims on a whiteboard in front of a whole class.
Today, seven years after his words I'm a developer living in foreign country with what I could say somewhat a solid experience and understanding of how both software and web are build, while that same professor still recites to his pupils difference between assembly and object code, while praying nobody asks him where and how these are used. For maybe a quarter of my paycheck. So much about his psychic powers..4 -
I already wrote a rant about this yesterday, but since I'm a sysadmin trying to convert to dev.. I dunno, maybe it's not a bad idea to muddy the waters a bit and talk about why not to be a sysadmin.
Personally I think it's that the perceived barrier to entry is just too high, while it isn't. You don't need a huge Ceph cluster and massive servers when you're just starting out. Why overbuild an appliance like that if it's gonna start out at maybe 5 requests a minute?
Let's take an example - DNS servers! So there's been this guy on the bind-users mailing list asking how to set up a DNS server on 2 public servers, along with a website. Nothing special I guess - you can read the thread here: https://0x0.st/ZY-d. Aside from the question being quite confusing, there was advice to read RFC's, get a book, read the BIND ARM, etc etc. And the person to deny this? No one less than Stephane Bortzmeyer, one of the people who works for nic.fr (so he maintains the .fr TLD) and wrote some of those RFC's as part of the DNSOP working group in the IETF. As for valid reasons to set up a DNS server? Could just be to learn how the DNS works, or hell even for fun. As far as professional DNS servers go.. this (https://0x0.st/ZYo9) is the nugget that powers the K root server, one of the 13 root servers that power the root zone of the internet, aka the zone apex. 2 RJ45 connections, and a console connection. The reason why this is possible is the massive recursor networks that ISP's, Google DNS, Cloudflare DNS, Quad9, etc etc provide. Point is, you don't need huge infrastructure to run a server!
Or maybe your business needs email. How many thousands of emails per second are you gonna need to build your mail server against? How many millions will you need to store? If your business has 10 employees and all of those manage about 10k emails total.. well that's easy, 100k emails total. Per second? Hundreds of emails per second per employee? Haha, of course not. Maybe you'll see an email a minute at most. That is not to say that all email services are like this - it is true that ISP's who offer email to their customers, and especially providers like Microsoft and Google do need massive mail servers that can handle thousands of emails per second. But you are not Microsoft or Google. So yeah, focus on the parts of email that are actually hard.. and there is plenty.
Among sysadmins you have this distinction between "professional" sysadmins and homelabbers. I don't mind the distinction itself but I think both augment each other. If you've started out by jumping into a heap of legacy at an established company, you will have plenty of resources, immediately high complexity, and probably a clusterfuck right away. But you will have massive amounts of resources. If you start out with a homelab, you will have not many resources, small workloads, and something completely new for you to build and learn with. And when running a server like that, you'll probably find that the resources required are quite small, to provide you with your new services. My DHCP servers take 12MB memory each. My DNS servers hover around the 40MB mark. The mail server.. to be fair that one consumes around 150. But if you'd hear the people saying that you need huge servers.. omg you need at least a TB of RAM on your server and 72 cores, massive disks and Ceph!1!
No you don't. All that does is scaring people away and creating a toxic environment for everyone. Stop it.1 -
So with my current windows heavy workload it is with a heavy heart that I commit to learning powershell. Wish me luck6
-
TLDR, need suggestions for a small team, ALM, or at least Requirements, Issue and test case tracking.
Okay my team needs some advice.
Soo the powers at be a year ago or so decided to move our requirement tracking process, test case and issue tracking from word, excel and Visio. To an ALM.. they choice Siemens Polarion for whatever reason assuming because of team center some divisions use it..
Ohhh and by the way we’ve been all engineering shit perfectly fine with the process we had with word, excel and Visio.. it wasn’t any extra work, because we needed to make those documents regardless, and it’s far easier to write the shit in the raw format than fuck around with the Mouse and all the config fields on some web app.
ANYWAY before anyone asks or suggests a process to match the tool, here’s some back ground info. We are a team of about 10-15. Split between mech, elec, and software with more on mech or elec side.
But regardless, for each project there is only 1 engineer of each concentration working on the project. So one mech, one elec and one software per project/product. Which doesn’t seem like a lot but it works out perfectly actually. (Although that might be a surprise for the most of you)..
ANYWAY... it’s kinda self managed, we have a manger that that directs the project and what features when, during development and pre release.
The issue is we hired a guy for requirements/ Polarion secretary (DevOps) claims to be the expert.. Polarion is taking too long too slow and too much config....
We want to switch, but don’t know what to. We don’t wanna create more work for us. We do peer reviews across the entire team. I think we are Sudo agile /scrum but not structured.
I like jira but it’s not great for true requirements... we get PDFs from oems and converting to word for any ALM sucks.. we use helix QAC for Misra compliance so part of me wants to use helix ALM... Polarion does not support us unless we pay thousands for “support package” I just don’t see the value added. Especially when our “DevOps” secretary is sub par.. plus I don’t believe in DevOps.. no value added for someone who can’t engineer only sudo direct. Hell we almost wanna use our interns for requirements tracking/ record keeping. We as the engineers know what todo and have been doing shit the old way for decades without issues...
Need suggestions for small team per project.. 1softwar 1elec 1mech... but large team over all across many projects.
Sorry for the long rant.. at the bar .. kinda drunk ranting tbh but do need opinions...
Top Tags
Weekly Rant
View