21
PaperTrail
174d

Best code performance incr. I made?

Many, many years ago our scaling strategy was to throw hardware at performance problems. Hardware consisted of dedicated web server and backing SQL server box, so each site instance had two servers (and data replication processes in place)

Two servers turned into 4, 4 to 8, 8 to around 16 (don't remember exactly what we ended up with). With Window's server and SQL Server licenses getting into the hundreds of thousands of dollars, the 'powers-that-be' were becoming very concerned with our IT budget. With our IT-VP and other web mgrs being hardware-centric, they simply shrugged and told the company that's just the way it is.

Taking it upon myself, started looking into utilizing web services, caching data (Microsoft's Velocity at the time), and a service that returned product data, the bottleneck for most of the performance issues. Description, price, simple stuff. Testing the scaling with our dev environment, single web server and single backing sql server, the service was able to handle 10x the traffic with much better performance.

Since the majority of the IT mgmt were hardware centric, they blew off the results saying my tests were contrived and my solution wouldn't work in 'the real world'. Not 100% wrong, I had no idea what would happen when real traffic would hit the site.

With our other hardware guys concerned the web hardware budget was tearing into everything else, they helped convince the 'powers-that-be' to give my idea a shot.

Fast forward a couple of months (lots of web code changes), early one morning we started slowly turning on the new framework (3 load balanced web service servers, 3 web servers, one sql server). 5 minutes...no issues, 10 minutes...no issues,an hour...everything is looking great. Then (A is a network admin)...
A: "Umm...guys...hardly any of the other web servers are being hit. The new servers are handling almost 100% of the traffic."
VP: "That can't be right. Something must be wrong with the load balancers. Rollback!"
A:"No, everything is fine. Load balancer is working and the performance spikes are coming from the old servers, not the new ones. Wow!, this is awesome!"
<Web manager 'Stacey'>
Stacey: "We probably still need to rollback. We'll need to do a full analysis to why the performance improved and apply it the current hardware setup."
A: "Page load times are now under 100 milliseconds from almost 3 seconds. Lets not rollback and see what happens."
Stacey:"I don't know, customers aren't used to such fast load times. They'll think something is wrong and go to a competitor. Rollback."
VP: "Agreed. We don't why this so fast. We'll need to replicate what is going on to the current architecture. Good try guys."
<later that day>
VP: "We've received hundreds of emails complementing us on the web site performance this morning and upset that the site suddenly slowed down again. CEO got wind of these emails and instructed us to move forward with the new framework."

After full implementation, we were able to scale back to only a few web servers and a single sql server, saving an initial $300,000 and a potential future savings of over $500,000. Budget analysis considering other factors, over the next 7 years, this would save the company over a million dollars.

At the semi-annual company wide meeting, our VP made a speech.

VP: "I'd like to thank everyone for this hard fought journey to get our web site up to industry standards for the benefit of our customers and stakeholders. Most of all, I'd like to thank Stacey for all her effort in designing and implementation of the scaling solution. Great job Stacy!"
<hands her a blank white envelope, hmmm...wonder what was in it?>

A few devs who sat in front of me turn around, network guys to the right, all look at me with puzzled looks with one mouth-ing "WTF?"

Comments
  • 1
    Nice

    But fuck the VP
  • 6
    Stand up, walk forward, and say:
    "GJ Stacy! Without you, We could have done it much sooner!"
  • 3
    @iiii > "But fuck the VP"

    and it wasn't like a one-person adventure. Myself, other web devs making the code changes, couple of network admins herding cats, 'Stacey' had little to no involvement in anything a long the way.

    In fact, she would pop-in randomly and say things like "How big is this data you're sending? 1K? Too much...make it smaller. Use compression." <then leave the convo>
  • 2
    @PaperTrail yeah, guessed that she's a leech
  • 3
    Someone else taking the credit for work that we do.

    Classic corporate culture, am i right?

    *Goes to a corner to cry*
  • 3
    @iiii > "guessed that she's a leech"

    Sometimes, and a master at the corporate politics game. Almost like a super power.

    If you were on her team, you were protected, no matter what. 'Stacey' would, figuratively, gut anyone who she thought would/could threaten anyone on her team or make her team look bad. Her favorite tactic would be make small talk with VPs and 'poison the well' with things like "I've heard PaperTrail isn't a team player, nobody likes him..." etc. In some ways, a positive, in others, not so much.

    There were many toxic and inept devs that we should have fired, but she did a great job in protecting her castle. She could proudly say "In the past 20 years, I never had to fire anyone on my team!". Impressive to other mgrs at her level and above, but to us lowly pigeons, we see the crap falling from the eagles.
  • 1
    run..... why are you in a place where your efforts aren't valued?
  • 2
    @jassole > "why are you in a place where your efforts aren't valued?"

    Those events occurred several years ago and it falls/fell into my 80/20 rule. If I can tolerate 20% of the nonsense, it's fine. I know 100% I could be dealing with something much worse anywhere else.

    Most of the time, its pretty good around here. Only get the random "WTF is going on!" moments every few months.
  • 0
    Oy, @PaperTrail
    Yo garbage truck's on fire
Add Comment