73

The network starts slowing down, transactions start to fail across the 450+ stores, the website starts to spit 500 errors what is going on?

Queue a frantic running around the office working out what was going wrong... Calls from all 3 data centres, nothing is going in or out of the network.

Notice the network admin come back to his desk, his eyebrows raise and he looks left and right before unplugging his laptop ethernet from one of the server access points

The network rushes back to life, everything is fine.

That particular network mapping tool is now banned for use on production.

Comments
  • 18
    Impressive to sink such a big setup.

    I say there must not have been enough separation in the network :)

    I made a very similar thing once but I only managed to close down the local network at the office when the monitoring program started to advertise it self as the default gateway to get all traffic.

    Problem was it failed to then route it on to the real gateway.

    But it never affected production system as they where on a different subnet isolated from the office :)
  • 12
    @Voxera normally it should be fine, someone plugged into the core router directly to run the mapping tool.
  • 3
    @seraphimsystems Straight to the mother lode.... Ouch.
  • 9
    @seraphimsystems well, if your going to screw up, why not do an excellent job at it :P
  • 6
    @Voxera I think he was costing the company his annual salary every 5 minutes considering the whole company was basically offline.
  • 10
    @Voxera we had a similar situation. Tech took down the whole center (~1000 employees, mainly Ops doing monitoring, processing tickets and working on a production devices across multiple countries) because he was playing with his GNS3 to get ready for CCNP.

    Guess he got more networking hands on experience than on any Cisco course because that panic and adrenaline rush can't be taught.
  • 2
    Another one who verified that a really stupid idea is indeed really stupid by collecting empircal evidence...
  • 1
    @seraphimsystems No no no no... He was *teaching* the company a very expensive lesson and didn't take a single penny for it, just took it upon himself as a part of his job description. Basically a hero.
  • 0
    the CTO of a company I worked for was able to loop the entire network (affected everyone, even folks in Canada) just by plugging in a randomly appeared LAN cable in a telephone.
    so.. yeah..
  • 0
    @rootofskynet that happened more than once on my former job.

    All phones had two ethernet ports so that they could be connected between the computer and the network.

    The idea was both to save on ethernet ports as most offices where designed with just one per desk, but also, by routing all traffic it could interact with a desktop software on the computer without extra setup.

    But if those ports where ever both connected to the network … instant loop back without throttling.
Add Comment