Me 5 years ago : "Guys we are gonna have a perf..."

CEO : "Not now, we need to deliver that functionality asap"

Me 3 years ago : "Guys, performances/scalibiluity will hit us like a trucK'

CEO : "Not nowem a new functionality needs to be done ASAP"

Me 1 year ago : "We are gonna be hit by a tank. We won't even understand what happens"

CEO : "I'm sure we can manage"

2 Days ago : Plateform quasiment down, response time in MIONUTES instead of milliseconds, database on fire.

CEO : "WHAT THE FUCK !!! GO FIXC I ASAP WE CANNOT HAVE THAT SHIT HAPPEN".

This is a brief summaru of working in a startup.

#2 Worst thing I've seen a co-worker do?

Back before we utilized stored procedures (and had an official/credentialed DBA), we used embedded/in-line SQL to fetch data from the database.

var sql = @"Select
FieldsToSelect
From
dbo.Whatever
Where
Id = @ID"

In attempts to fix database performance issues, a developer, T, started putting all the SQL on one line of code (some sql was formatted on 10+ lines to make it readable and easily copy+paste-able with SSMS)
var sql = "Select ... From...Where...etc";

His justification was putting all the SQL on one line make the code run faster.
T: "Fewer lines of code runs faster, everyone knows that."

Mgmt bought it.

This process took him a few months to complete.

When none of the effort proved to increase performance, T blamed the in-house developed ORM we were using (I wrote it, it was a simple wrapper around ADO.Net with extension methods for creating/setting parameters)

T: "Adding extra layers causes performance problems, everyone knows that."

Mgmt bought it again.

Removing the ORM, again took several months to complete.

By this time, we hired a real DBA and his focus was removing all the in-line SQL to use stored procedures, creating optimization plans, etc (stuff a real DBA does).

In the planning meetings (I was not apart of), T was selected to lead because of his coding optimization skills.
DBA: "I've been reviewing the execution plans, are all the SQL code on one line? What a mess. That has to be worst thing I ever saw."
T: "Yes, the previous developer, PaperTrail, is incompetent. If the code was written correctly the first time using stored procedures, or even formatted so people could read it, we wouldn't have all these performance problems."

DBA didn't know me (yet) and I didn't know about T's shenanigans (aka = lies) until nearly all the database perf issues were resolved and T received a recognition award for all his hard work (which also equaled a nice raise).

Boy do I hate office politics...

A client asked our company to fix perf issues on their product. Our coleagues had been picked for the job [being led by another 3rd-party, as per client's request]. Aaand they dropped the ball. The deadline is in 2 weeks, nothing is working.

Mgmt engaged us to put out the fire, but strictly at the scope the other guys were working in.

On the first day of testing we've revealed an elephant-sized perf issue that's as easy to fix as brainlessly changing 4 values in config. And that elephant is masking all the other perf issues.

We got a firm NO for config changes as that is out of the defined scope. And we're asked to continue testing.

I mean, the elephant is THAT huge that any further testing is moot - all other bottlenecks are hidden behind it. And just changing those 4 values would reduce the resources required by a magnitude of ~10.

But that's out of scope...

Client is desperate, lost and honestly asking us, pros in the field, for help.. We know how to help.. It takes 10 seconds to apply the fix..

But our mgmt forbids us to step out of the scope :/

as a result we have to pretend to be dummies hardly knowing what to do and hide the truth from the customer they so desperately want.

This is frustrating. And wrong. And imo unprofessional

Like most people I needed some extra cash during uni, so I proceeded to learn CSS + Photoshop (yeah, I know). Followed by PHP and WordPress.

It can be a very shitty platform until you realize that you can stop combining plug-ins from all over the place with dubious code quality and roll your own.

Anyhow I kept at it until I was able to join a niche company doing a quite popular caching plug-in for WP (yeah, W3 Total) when I suddenly became *very* interested in anything and everything performance.

This landed me a very cozy consulting gig in the Nordics - they were using WP for an elephant-traffic website and had run into a myriad of perf issues.

Fixing them and breaking the monolith awarded me with skills in nodejs, linux, asynchronous caching among others.

I was soon in charge with managing the dev boxes for the entire team, and when the main operations dude left, I was promoted to owning the entire platform. (!) Tinkering with Linux for most of my life really came in handy here. (remember Debian potato?)

Used saltstack + aws cloudformation to achieve full parity between all environments. Learned myself some python and all various tips and tricks which in the end amounted to 90% reduction in time-to-first-byte and considerable cost savings.

By the end of the 2yr contract I had turned myself into a fullstack systems engineer and never looked back.

Lawyers not getting along resulted in us having to abandon NewRelic, so I got to learn and deploy the ELK stack as a homegrown replacement, which was super-fun.

Now I work in the engineering effectiveness department of a Swedish fintech unicorn where all languages under the Sun are an option (tho we prefer Python), so the tech stack is unlimited. Infinite tools and technologies, but with strong governing principles and with performance always in mind so as to pick the right tool for the job.

It's like that childhood feeling when you've just dumped a ton of Lego on the floor and are about to build something massive.

I guess the morale here is however disappointed you feel by your current stack - don't. Always strive to make things better, faster, more decoupled, easier to test, etc. and always challenge yourself to go outside the comfort zone.

## 4 years ago:
- Principal Architect: We are using IO1 storage type. What if we used GP2?
- Perf team: IDK, let's test it!
*we run tests*
- Perf team: results are OK, but we're exhausting Burst IO capacity, effectively hard-limiting number of tests we can run per day
- PArch: ahhh, I see. Then Gp2 is a no-go.

## 3 years ago
*PArch quits. New one is hired*
- PArch2: We are using IO1 storage type. What if we used GP2?
- Perf team: We've already tested that a while ago, results were THIS and THAT
- PArch2: I see. Let's test it again anyway
- Perf team: *wtf???*
*we run the same tests, we get the same results*
- PArch2: I see, so GP2 is a no-go.
- Perf team: *you think....? How did that thought never cross our minds, we wonder...*

## 2 years ago
*new DBA is hired*
- DBA2: We are using IO1 storage type. What if we used GP2?
- Perf team: We've already tested that a while ago, results were THIS and THAT
- PArch2: We've already tested that a while ago, results were THIS and THAT
- DBA2: I see. Let's test it anyways. I've read somewhere that GP2 might be a better bet
- PArch2: you might be right, let's do that
- Perf team: *wtf???*
*we run the same tests, we get the same results*
- DBA2: I see, so GP2 is a no-go.
- Perf team: *you think....? How did that thought never cross our minds, we wonder...*

## 1 year ago
*DBA manager left; new one was hired*
- MGMT_dba2: We are using IO1 storage type. What if we used GP2?
- ........

Should we even bother bringing up the history.....?

Less rant, more mildly interesting Java trivia.

Integer i0 = 3; Integer i1 = 3;
Integer i2 = 300; Integer i3 = 300;

i0 == i1 is true as expected
i2 == i3 is actually false

Java caches -128 to 127 Integer objects for faster perf so when you're inside that range, the objects are indeed the same, but because == checks object equality, the Integer outside of the range is not cached and had to be initialized, so i2 and i3 are two different objects.

You can totally break some tests this way :)

Client: *uses Oracle on AWS RDS*

Ora: *licence about to expire*

Client: we need a cheaper and equally performant solution.

AWS: we have an outstanding Aurora db! Its performance is flawless!

Client: shut_up_and_take_my_money.png

Engineers: *migrating to Aurora mysql*
Engineers: fine-tuning the db for 2 months, adapting app code.
Aurora: *shits its pants during test ramp-up, delivers ~70% of Ora perf*

==========

Client: we need to do better!

AWS: try Aurora Postgresql!

Engineers: *migrating to Aurora postgresql*
Engineers: fine-tuning the db for 1 month.
Aurora: *shits its pants during test ramp-up, delivers ~40% of Ora perf*

AWS: let me see...
AWS DBAs: *fine-tuning the db for another 2 months*
Aurora: *survives ramp-up, delivers ~70% Ora perf*

AWS DBAs: your application is wrong.
We: Ora didn't complain about that...

AWS DBAs: umm.. Err.. Then your load test is wrong
We: ora wasn't complaining about that either...

AWS DBAs: errr.. Ummm.. Oh, I've got it! Your queries aren't optimized!
We: we cannot change queried - they are hardcoded by our framework's vendor

AWS DBAs: well there you go! Your queries aren't optimized! It's your fault, not aurora's

yyeeahhhh... Riight...

From my xp, aws support and aws engineers ALWAYS try to put the blame on a client. Always. Even when they're obviously wrong.

Rant!

Been working on 'MVP' features of a new product for the past 14 months. Customer has no f**king clue on how to design for performance. An uncomfortable amount of faith was placed on the ORM (ORMs are not bad as long as you know what you are doing) and the magic that the current framework provides. (Again, magic is good so long as you understand what happens behind the smoke and mirrors - but f**k all that... coz hey, productivity, right?). Customer was so focussed on features that no one ever thought of giving any attention to subtler things like 'hey, my transaction is doing a gazillion joins across trizillion tables while making a million calls to the db - maybe I should put more f**king thought into my design.' We foresaw performance and concurrency issues and raised them way ahead of the release. How did the customer respond? By hiring a performance tester. Fair enough - but what did that translate into? Nothing. Nada. Zilch. Hiring a perf tester doesn't automagically fix issues. The perf tester did not have a stable environment, a stable build or anything that is required to do a test with meaningful results. As the release date approached, the customer launched a pilot and things started failing spectacularly with the system not able to support more than 15 concurrent users. WTF! (My 'I told you so' moment) Emails started flying in all directions and the hunt for the scapegoat was on (I'm a sucker for CYA so I was covered). People started pointing in all directions but no one bothered to take a step back and understand what was causing the issues. Numero uno reason for transaction failure was deadlocks. We were using a proprietary DB with kickass tooling. No one bothered to use the tooling to understand what was the resource in contention let alone how to fix the contention. Absolute panic - its like they just froze. Debugging shit and doing the same thing again and again just so that management knew they were upto something. Most of the indexes had a fragmentation of 99.8% - I shit you not. Anywho, we now have a 'war room' where the perf tester needs to script the entire project by tonight and come up with some numbers that will amount to nothing while we stay up and keep profiling the shit out of the application under load.

Lessons learnt - When you foresee a problem make a LOT of noise to get people to act upon it and not wait till it comes back and bites you in the ass. Better yet, try not to get into a team where people can't understand the implications of shitty design choices. War room my ass!

I fucking hate people who report somebody else's work as their own successes so much.

I've written a fair amount of perf tests for our project so far (actually I'm like the only person doing that). Some fucker from another team asks me if I could write one more. I agree, because why not. I spend a few hours, making sure to cover all cases and commit the test. Then the same fucker runs it and reports it as HIS PERFORMANCE MEASUREMENTS.

0 credit given to me. Fuck you, I just wanted to be helpful and you used this.

I'm still quite young and tend to fall for shit like this, but getting more and more grumpy because of those people.

garbage collectors' lifestyle matters!

Ever eyeballed the abyss of your memory leaks? Shit, garbage collectors deserve a raise.

Unsung heroes, janitorialing thru that VM like a dung beetle, silently fucking up your perf so you can do that delicious spaghetti. Indiana-jonesing the fuck out of that memory trash can and euthanizing all that disgusting heap of pointers hanging, dangling, like... well, like garbage.

At the very least they're deterministic, unlike that Markov chain we all had the displeasure of fucking up. Amen? Amen! 🙌🏻

You gotta wonder, though, what goes through their nuggin. Do they reminisce about the potential of that half-ass-written class? Do they weep for the elegance of a forgotten function bottlenecking their job? Nah, probably just counting down the nanoseconds till their next full GC cycle. Aaah, like cold beer in Saturday barbecue.

So next time your program miraculously avoids a memory error, take a moment, put your hands up in the air and say a prayer to your garbage collector.

Silently covering for your fuckups

We'd just finished a refactor of the gRPC strategy. Upgraded all the containers and services to .Net core 3, pushed a number of perf changes to the base layer and a custom adaptive thread scheduler with a heuristic analyzer to adjust between various strategies.

Went from 1.7M requests/s on 4 cores and 8gb ram to almost 8M requests/s on the same, ended up having to split everything out distributed 2 core instances because we were bottlenecking against 10gb/e bandwidth in AWS.

People should have mandatory lessons in vector processing.

In canteen, after lunch, there were 4 places you could place your trays. But only small, one-way corridor, for one person at a time to get there.

Every person picked the first place and while they were placing the tray, people behind them had to wait. Huge line started to form. If they, instead, always picked the last empty place, all tray places would be occupied for longer and the processing speed could increase almost 4 times.
Textbook vector processing example.

Just had one of those days.... totally learned A LOT about how to properly profile a Linux box using Brendan Gregg's( Netflix ) USE method. Only problem is... I didn't make any actual progress on my application project itself. 😩😩😩😭😭😭

Ffs, HOW!?!? Fuck! I need to get this rotten bs out.

RDS at its max capabilities from the top shelf, works OK until you scale it down and back up again. Code is the same, data is the same, load is the same, even the kitchen sink is the same, ffs, EVERYTHING is the same! Except the aws-managed db is torn down and created anew. From the SAME snapshots! But the db decides to stop performing - io tpt is shit, concurrency goes through the roof.

Re-scale it a few more times and the performance gets back to normal.

And aws folks are no better. Girish comes - says we have to optimize our queries. Rajesh comes - we are hitting the iops limit. Ankur comes - you're out of cpu. Vinod thinks it's gotta be the application to blame.

Come on guys, you are a complete waste of time for a premium fucking support!

Not to mention that 2 enhanced monitoring graphs show anythung but the read throughput.

Ffs, Amazon, even my 12yo netbook is more predictable than your enterprise paas! And that support..... BS!

We're now down to troubleshooting aws perf issues rather than our client's....

Not dev, but a perf-eng confidence boost.

Our company was hired by a client to onboard perf-testing process and do some perf-related go-live stuff. Basically, make sure the app meets the SLAs.

Our company mobilized some internal resources for the task. The had 3-4 months. 2 months later they realized they won't pull it off. What a shame...

When the threat of dropping the ball and losing the client and recommendations became very real, they engaged us. Half the time, half the resources, a worried and annoyed client who now wants to control the whole initiative.

During the first 2 meetings we get the general idea of what they have, what they want. We take some time to prepare a plan to make it on time. The client argues our plan, mostly because one of the main points was mocking downstream dependencies [integrations]. He asks, then demands to do it all with live integrations. We explain why this is an incredible risk and why we should do it the proposed way. He disagrees.

Alright then... Maybe he knows smth we don't. Let's do it the risky way...

A month later test results are far from the target. I did my best with app de-bottlenecking and fine-tuning. But since the live integrations do not deliver, they hide other bottlenecks. The initiative is stuck.

Finally, the client agrees to do it with mocking. But now there's no time left as it will take almost a month to prepare mocks...

The client agrees we should have done it our way from the start. They postpone the go-live and we carry out our testing and tuning the right way.

That was one expensive and long "I told you so". But it boosted our [perf team's] confidence to the top and beyond :)

don't tell us how to do our job, unless you do want extra expenses

Changed the Ora DB instance type to one having 2x fewer CPUs. That alone increased the db perf ~4x+

Recently I've come to notice that I keep using my gut feeling every day in perf investigations [and it pans out]. Until now I used to think that "gut feeling" in IT and everywhere else is a synonim for "guessing". Turns out it's not at all that simple.

Damn, that's exciting! Love that spidey-sensey-feeling!! I hope it means I'm getting quite good at what I do.

I'm moving to PHP.

No, seriously. PHP devs were treated like “you're the tech guy, I don't care, make it work” for so long that PHP deps library has everything. If you need to do an unusual task like slowing download speed to 64 kbps, there is a lib for that. Caching is one lib away. Yes, libs themselves are subpar, but they do the job.
Performance? I never had any perf issues in my apps. DB is always the bottleneck, and I know databases.
Frameworks? I don't care about them.
Also, I'll always find PHP devs on the market.

Shut the fuck up with your elitist rust crap. PHP is a nuclear-resistant cockroach that will outlive you, your stupid language and everything you wrote in it. My PHP code will be running fine after every line of code you ever wrote in rust/python/java/scala/whatever fancy language you like is no longer in use.

Yes, I talked shit about PHP in the past. I was neither pragmatic nor mature. Many things changed since. For starters, I'm a CTO now. Hating PHP was easy and socially acceptable. Talk shit about PHP, get internet points — that's how it always worked.

No more. PHP is the king.

Kafka lead after a perf test - we have amazing performance, processing 43 records a second😄
Me - WTF!!!! 🤬

In my experience, any BE dev or old architect/lead programmer that says they “can do frontend” does shit like writing Ajax calls in script tags directly in the html. They are the ones who add style attributes directly in html. They are the ones who google how to center a div and they still use float positioning because all of them are old, arrogant BE devs who get caught in a single framework who convince themselves they are an expert. They can’t give any good UX advice. They don’t know how to use a screen reader. They don’t know what WCAG means. They don’t constantly keep up to date on what browsers are supporting and what’s being released in the unstable versions. They don’t know what a web component is. They don’t know what a closure is. They don’t know anything about optimizing web perf metrics. They couldn’t tell you what web crawlers look for. They couldn’t tell you anything about design principles and anti-patterns. They don’t know how to manage a web application that will be seen by millions AND keep it nice, shiny, and refactorable on the code side. What do they really fucking know? how to write an MVC app? How to connect APIs and integrate code that other people wrote? I do full stack all day and writing anything not-client-facing is super easy.

Take that stick out of your ass and get over yourself you asshole. You haven’t written anything close to amazing even though you constantly act like you’re a god-tier programmer and your shit doesn’t stink.

Hit the books like the rest of us you fuck.

The Frontend is anything but fucking easy.

Do people still use redis when you already use postgres? How is pg perf if you are writing like 10,000s of row data/second. I am slightly outdated....

Bad perf is fine, our users are old and used to slow software.

BUT FUCKING WHY ARE YOU PAYING MY FIRM TONS OF MONEY TO REWRITE IF ITS TO BE JUST AS SLOW?!?!?!

Fucking jQuery in Polymer 0.5.

When polymer 0.5 was released, things seemed incredibly easy at first, but when you need to do some complex things, the abstraction layer provided by web components are not of much help. Babel wasn't there too, so I ended up using scope hacks to access event listeners (var self = this). Worse, I have to use jQuery because many of things are downright tedious or fucked up back then, including myself.

Now, React is here; No jQuery, no hacks, no web component polyfills, no unsolvable perf bugs, no scope hacks, no 10sec loading time, no regrets.

Responses are faster in non-prod perf testing environment than in prod.

4 hours to compile a version of perf that demangled function names.

Thanks, Ubuntu...

Hey guys, I need your honest opinion. What do you think about service provider office location? Situation is that Im looking for an office for me and two employees. Found one good location which is perf in the inside: has a private modern office, meeting rooms, skype rooms, nice kitchen and etc. However its ugly af from the outside: U need to get inside this hospital looking soviet building and take an elevator to the 8th floor to see all the modern stuff. While price seems a bargain, Im kinda afraid of how we could come across potential clients who would visit us for a meeting. As a potential client how would you judge service provider (in this case android dev company) which has nice offices in the inside but ugly af building from the outside?

Working all day long on a new feature to resolve a perf issue but realised at 6pm only that everything was pointless. I love my job

Remember guys, perf matters...

Help needed: what type of goals you'd set for the upcoming perf review when you're less than a few months in a company?