Me: GET /sleep
Baby: 307 Temporary Redirect
Baby: 204 No Content
Me: 200 OK

Me: GET /sleep
Baby: 307 Temporary Redirect
Baby: 413 Payload Too Large
Me: 102 Processing
Me: 200 OK

Me: GET /sleep
Baby: 307 Temporary Redirect
Baby: 444 Connection Closed Without Response
Me: 200 OK

Me: GET /sleep
Baby: 307 Temporary Redirect
Baby: 444 Connection Closed Without Response
Me: 429 Too Many Requests

Me: GET /sleep
Baby: 307 Temporary Redirect
Me: 101 Switching Protocols
Me: 408 Request Timeout
GF: 102 Processing

Me: GET /sleep
Sleep: 404 Not Found
Me: 406 Not Acceptable

(Morning)
Me: 501 Not Implemented

Response time of different people on Whatsapp:

Best Friend: 5-10 sec
Friend: 1-2 min
Best Friend(Girl): 3-5 min
Girl Friend: 1-2 day 😢
Client(when me solving bug): 0.00005 sec

Client(when me asking payment): *Blocked*

😕😕😕😕😕

Our team was having a problem with very slow response times from a 3rd party web service they were contacting to get some device stats. No issues on the other end, but it had already been weeks. They ask me to take a look at it.

I take a few days, do a couple of benchmarks and tests and I isolate the section of code responsible. Turns out, the method they were calling would timeout if the device was offline. We ask the vendor, and they confirm this. They tell us to call methodX to check if it was up.

After having done that, lookup now only took seconds. They were annoyed that it wasn't documented but was just glad it was fixed.

Boss: we have to fix this bug.
Me: It is not a bug ..the server takes more time to send the response which cause the timeout issue . we may need to change the implementation to increase the performance to send the response quickly. It will take some time
Boss: okay can we fix this by today
Me: ya if we increase timeout to 20 seconds the issue is fixed
Boss: No we want the server to send the response quickly and we need the fix now

I worked for the weekend to fix it finally......Guess what ....the change dint go live since the scenario was not valid and will never likely to happen in production

Still dealing with the web department and their finger pointing after several thousand errors logged.

SeniorWebDev: “Looks like there were 250 database timeout errors at 11:02AM. DBAs might want to take a look.”

I look at the actual exceptions being logged (bulk of the over 1,600 logged errors)..

“Object reference not set to an instance of an object.”

Then I looked the email timestamp…11:00AM. We received the email notification *before* the database timeout errors occurred.

I gather some facts…when the exceptions started, when they ended, and used the stack trace to find the code not checking for null (maybe 10 minutes of junior dev detective work). Send the data to the ‘powers that be’ and carried on with my daily tasks.
I attached what I found (not the actual code, it was changed to protect the innocent)
Couple of hours later another WebDev replied…

WebDev: “These errors look like a database connectivity issue between the web site and the saleitem data service. Appears the logging framework doesn’t allow us to log any information about the database connection.”

FRACK!!...that Fracking lying piece of frack! Our team is responsible for the logging framework. I was typing up my response (having to calm down) then about a minute later the head DBA replies …

DBA: “Do you have any evidence of this? Our logs show no connectivity issues. The logging framework does have the ability to log an extensive amount of data regarding the database transaction. Database name, server, login, command text, and parameter values. Everything we need to troubleshoot. This is the link to the documentation …. If you implement the one line of code to gather the data, it will go a long way in helping us debug performance and connectivity issue. Thank you.”

DBA sends me a skype message “You’re welcome :)”

Ahh..nice to see someone else fed up with their lying bull...stuff.

The moment you realize that you have successfully beaten reality with your unit-tests...

There are unit-tests for ...
... the api returning a 408 Http StatusCode when an internal request times out.
... the react app take this status-code and fires an action to display a specific error message for the user.

Every bit of code runs just fine.

Deploy this hell of an app on the server. Dandy Doodle.

Do a smoketest of the new feature.
FAIL!
Chrome starts to crumble during runtime. The api Request freezes.
Firefox takes the 408 api response but fails to interpret it in react app.

So I began to wonder, what the hell is going on.

Actually I recognized that I had the glorious idea to return a clientside error code in a serverside api response.
Glorious stupidity :/

Finally I fixed the whole thingy by returning an 504 (Gateway timeout) instead of 408 (Clientside timeout)

Cheers!

While waiting for internet to load on crappy bandwidth, the wait can burn you out just as easily. Including response timeouts...

Wanted to try a new alerting based on a new Prometheus metric we added. To trigger an alert we killed the dev stage db of the service. Alert didn't get triggered. The reason was that the metrics endpoint suddenly needs exactly 60s for a response if the db is killed and prometheus timeout is 20s.
And to top it off, this behavior happens for each service we developed (that has a db) .
Well at least the new alerting already helped find a bug.

This had just happened, I was trying to increase the default timeout of an nginx running in a container for a proxy pass and always got a 504-gateway timeout response. I was setting proxy_connect_timeout, proxy_send_timeout, proxy_read_timeout, send_timeout, keepalive_timeout, etc. and nothing worked, after two hours of adding and removing lines of configuration (and waiting 1 minute for every time I tried a request), then I realized I have a local nginx for redirect server names to local ports (the container), that nginx was the one that actually responds with the 504 error, after that I tried a request with the port of the container ALL WORKED!!!!