Going on hour TEN of soft-launch weekend. Lots of problems with all the new procedures, and now a problem with a DNS entry that's taking forever to get resolved. Still have failover testing to do before users start their testing, and all of this means I've been awake for about 27 hours straight at this point.

And I've got this to look forward to again in a week since next weekend is go-live... presumably we will have learned a lot of lessons from this and it will be much quicker and smoother, but honestly, I'm a cynic, so I'm not gonna assume that'll be the case.

This kinda sucks, is the main thesis here.

  • 2
    Ha, soft launch. "We have no confidence, but if we don't do something we will have to explain why we missed an arbitrary date set by someone with 0 engineering experience."
  • 0
    - dns
    - 27 hours
  • 0
    @SortOfTested Honestly, I think it's more like: "Soft launch: we made the process of deploying an app SO fucking complicated and SO fucking onerous that if we don't force you to try it before the real deal there's NO GODDAMN WAY you'll jump through all the hoops we've put up and make it for real."
  • 0
    Wound up taking over 30 hours, and we had to abort doing the required failover test (going to re-schedule for during the week) 'cause we just ran out of time and people actually needed to get sleep. SO many problems... some were just dumb mistakes to be honest, but those got resolved pretty quickly and easily... but a lot of them were procedural issues because the whole process is new to all of us and it, to put it bluntly, pretty ridiculous. Then when you've got like half a dozen different teams and the way they all want to be communicated with boils down to "eh, just open a Jira ticket and we'll get to it, uhh, probably never", bad docs, what a nightmare. We DID ultimately get the thing up and running (minus that one test), but man, I feel like I stormed Normandy beach by the end of it. We definitely learned a lot from the experience, that's the good news, so at least it's reasonable to think go-live will go smoother... but you know, I don't think I'd place any bets on it either.
Add Comment