Seeing ALL the members of my team finally coming into their own. One person tackled our entire not-at-all-simple CI/CD setup from scratch knowing nothing about any of it and, while not without bumps in the road, did an excellent job overall (and then did the same for some other projects since he found himself being the SME). Two of my more junior people took on some difficult tasks that required them to design and build some tricky features from the ground-up, rather than me giving them a ton of guidance, design and even a start on the basic code early on (I just gave them some general descriptions of what I was looking for and then let them run with it). Again, not without some hiccups, but they ultimately delivered and learned a lot in the process and, I think, gained a new sense of self-confidence, which to me is the real win. And my other person handled some tricky high-level stuff that got him deep in the weeds of all the corporate procedures I'd normally shield them all from and did very well with it (and like the other person, wound up being an SME and doing it for some other projects after that). It took a while to get here, but I finally feel like I don't need to do all the really difficult stuff myself, I can count on them now, and they, I think, no longer feel like they're in over their heads if I throw something difficult at them.

A few critical bugs slipped into production this year, with a few requiring some after-hours heroics to deal with (and, unfortunately, due to the timing, it all fell on me). Of course, that just tells us that next year we really need to focus on more robust automated testing (though, in reality, at least one of the issues almost certainly would not - COULD NOT - have been caught before-hand anyway, and that's probably true for more than just one of them). We had avoided major issues the previous three years we've been live, so this was unusual. Then again, it's in a way a symptom of success because with more users and more usage, both of which exploded this year, typically does come more issues discovered, so I guess it tempers the bad just a little bit.

  • 1
    Having ability to rollback makes possible to eliminate the problem quickly. And fixing it in whatever needed time.

    That allows to have just middle testing. Instead of super duper v testing
  • 1
    @darkwind Agreed, and rollback definitely is always an option available to us without much hassle from a technical perspective.

    But, it tends to only be an option if the issue is noticed quickly.

    Unfortunately, in these cases, it took a day or two, or three even, to be discovered (despite the fact that they were quite significant issues, just happens they only came up in limited circumstances, but circumstances that made them dealbreakers. That, coupled with the fact that there were some new features in these releases that the business really needed - they couldn't afford to lose them due to critical processing deadlines - made rollback an option that was only considered for a few minutes before it was dismissed.

    The only good news is that the fixes were relatively easy in all cases (after the difficult analysis that is), so we got them out in all cases I believe the day after each issue was reported, same day in one case IIRC, so good turnaround times at least.
Add Comment