Be me

Have a company wide incident happen during on call

Say you're rolling back a change in a service that might have been the cause

Have someone laugh and say that change was just a new endpoint and completely unrelated

Be embarrassed

Have a senior director point out the code change that was the issue

Embarrass yourself in front of the entire company (it impacted everyone)

But hey atleast it wasn't my change

Na it is not embarassing, you did a good guess in order to mitigate the problem.

@Linux thank you, atleast changes can be unrolled back I guess even if reverted

Maybe that someone wasn't laughing at you; maybe they were laughing at the situation. Don't feel guilty.

@kamen I wish I could post an image of the slack thread comment but it was a "this is just adding an endpoint and has nothing to do with the change :insert laughing emoji:"

On the other hand, its quite natural to suspect the latest change before something breaks.

But you should check the change before rolling back, the change could be like this irrelevant but it could also be a fix for a bigger problem.

You should never blindly roll back changes if you have any other options

@Voxera Yeah agreed, could have been anything

@pandasama And to clarify, suspecting the last change and checking it out would be my first call if I did not have more info.

But as for actually roll back, unless I know its the cause or I have ruled out all other likely suspects I would hesitate.

I have seen to often rollbacks done in cases where they would not solve the problem and only cause longer and more wide spread down time while occupying resources better spent on actual error checking.

@Voxera Yeah don't want to damage things further, you're right