36
PaperTrail
296d

Worst hack/attack I had to deal with?

Worst, or funniest. A partnership with a Canadian company got turned upside down and our company decided to 'part ways' by simply not returning his phone calls/emails, etc. A big 'jerk move' IMO, but all I was responsible for was a web portal into our system (submitting orders, inventory, etc).

After the separation, I removed the login permissions, but the ex-partner system was set up to 'ping' our site for various updates and we were logging the failed login attempts, maybe 5 a day or so. Our network admin got tired of seeing that error in his logs and reached out to the VP (responsible for the 'break up') and requested he tell the partner their system is still trying to login and stop it. Couple of days later, we were getting random 300, 500, 1000 failed login attempts (causing automated emails to notify that there was a problem). The partner knew that we were likely getting alerted, and kept up the barage. When alerts get high enough, they are sent to the IT-VP, which gets a whole bunch of people involved.
VP-Marketing: "Why are you allowing them into our system?! Cut them off, NOW!"
Me: "I'm not letting them in, I'm stopping them, hence the login error."
VP-Marketing: "That jackass said he will keep trying to get into our system unless we pay him $10,000. Just turn those machines off!"
VP-IT : "We can't. They serve our other international partners."
<slams hand on table>
VP-Marketing: "I don't fucking believe this! How the fuck did you let this happen!?"
VP-IT: "Yes, you shouldn't have allowed the partner into our system to begin with. What are you going to do to fix this situation?"
Me: "Um, we've been testing for months already went live some time ago. I didn't know you defaulted on the contract until last week. 'Jake' is likely running a script. He'll get bored of doing that and in a couple of weeks, he'll stop. I say lets ignore him. This really a network problem, not a coding problem."
IT-MGR: "Now..now...lets not make excuses and point fingers. It's time to fix your code."
IT-VP: "I agree. We're not going to let anyone blackmail us. Make it happen."

So I figure out the partner's IP address, and hard-code the value in my service so it doesn't log the login failure (if IP = '10.50.etc and so on' major hack job). That worked for a couple of days, then (I suspect) the ISP re-assigned a new IP and the errors started up again.

After a few angry emails from the 'powers-that-be', our network admin stops by my desk.
D: "Dude, I'm sorry, I've been so busy. I just heard and I wished they had told me what was going on. I'm going to block his entire domain and send a request to the ISP to shut him down. This was my problem to fix, you should have never been involved."

After 'D' worked his mojo, the errors stopped.

Month later, 'D' gave me an update. He was still logging the traffic from the partner's system (the ISP wanted extensive logs to prove the customer was abusing their service) and like magic one day, it all stopped. ~2 weeks after the 'break up'.

Comments
  • 9
    That was the most stupid ass managememt team meeting.... No one suggested a firewall block???
    Use Fail2ban, or firewall rule to block the entire subnet.
  • 4
    @magicMirror > "No one suggested a firewall block???"

    No one from that dept was invited. I'm not a network guy, I assume when 'D' found out, that's what he did.
  • 3
    @PaperTrail There were IT guys in the room. DAFUQ those guys are paid for? sheesh.
  • 6
    @magicMirror > "There were IT guys in the room"

    No offense to them...well, maybe a little, I was probably the most qualified IT guy in the room. When I suggested that this was something 'D' should do, the manager immediately assumed I was trying to 'pass the buck'.

    He was big on 'teamwork', which in his mind meant any problem could be solved by anybody. Database timeout? Don't call the DBA...*you* fix the problem! I can't count how many times I was 'called to the office' when, for example, I gathered diagnostic info, send to a DBA (hey, looks like the index isn't optimized) and get dinged for not taking the initiative and solving the performance problem myself.
  • 2
    @PaperTrail Thats.... 🤦‍♂️. Not How Teams Works. Not even a little bit.
    I feel your pain.
  • 2
    This Rant is so well written that I could imagine how it all went
  • 1
    > you defaulted on the contract

    Putting Jake in the right?
  • 0
    I like how it's all your fault ... :grimacing:
Add Comment