I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…

I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.

I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.

We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.

Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.

My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.

Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.

At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.

Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?

Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.

Amazingly, no one was fired.

Me: The dev agency didn’t follow best practices. They only implemented front end validation on the form. The form submits to a public endpoint, so bots don’t have to go through our site to submit the form. That’s why our database is still filled with $1 donation transactions. I honestly recommend telling this to the dev agency and request that you not be charged for the extra work needed to do this right.

Manager: They charge $95/hr and they’re billing for 8 hours already.

[Aside: The agency’s task was to implement a $10 minimum on the form, do some text changes, and deploy.]

Me: I would expect work to be done according to accepted best practices. It’s really a half done job.

Manager: But they were very helpful when we had that payment processing emergency. They stayed late to help us. We shouldn’t push this in case we need their help again. Can you do the backend validation? [We are in US and agency is in Lithuania.]

Me: 🤬😩😑🤐[To myself: This wouldn’t have happened if the fundraising team hadn’t panicked and would only wait until I came back from my one day of PTO.]

When you’re fundraising, it’s AI
When you’re hiring, it’s ML
When you’re implementing, it’s linear regression
When you’re debugging, it’s printf()

Stakeholder: The orders aren’t importing from our order systems into our fundraising systems correctly. Gifts are showing last year’s codes. Do I restart the system?!

Me: Dude, I don’t know. My only responsibility here was to update the website’s config with the new codes you gave me. I did that. The correct codes for the orders are being sent to the orders system and that system is receiving the orders correctly. I know nothing about the configuration between the orders and fundraising systems.

Plus, y’all should have done testing before you ok’d my deploy. Don’t go looking to me to know someone else’s job. Y’all should have especially done testing since the person who’s the SME resigned months ago and there’s been no one to replace him yet.

Fundraising? Here is some short tips..

So I've been thinking about how to get people in my high school to join a team that I am creating for MLH. I also need to figure out how to plan and host the hackathon as well as fundraise for it. I'm trying to get all of this done be March of next year so that everything runs smoothly(hopefully). My school district is full or super rich kiddos so fundraising should be easy. Anyways do you guys have any tips for trying to get people into a team and organizing/fundraising for a hackathon?