My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

TLDR;
Wrote a slick scheduling and communication system allowing me to assign photography resources based on time and location.

I'll tell you a little secret ... I'm not actually a dev. I'm a photographer, pretending to be a dev.

Or ... perhaps it's the other way around? (I spend most of my time writing code these days, but only for me - I write the software I use to run my business).

I own a photography studio - we specialize in youth volleyball photography (mostly 12-18 year old girls with a bit of high school, college and semi-pro thrown in for good measure - it's a hugely popular sport) and travel all over the US (and sometimes Europe) photographing.

As a point of scale, this year we photographed a tournament in Denver that featured 100 volleyball courts (in one room!), playing at the same time.

I'm based in California and fly a crew of part-time staff around to these events, but my father and I drive our booth equipment wherever it needs to go. We usually setup a 30'x90' booth with local servers, download/processing/cashier computers and 45 laptops for viewing/ordering photographs. Not to mention 16' drape and banners, tons of samples, 55' TVs, etc. It's quite the production.

We photograph by paid signup only - when there are upwards of 800 teams/9,600 athletes per weekend playing, and you only have four trained photographers, you've got to manage your resources!

This of course means you have to have a system for taking sign those sign ups, assigning teams to photographers and doing so in the most efficient manner possible based on who is available when the team is playing. (You can waste an awful lot of time walking from one court to another in a large convention center - especially if you have to navigate through large crowds - not to mention exhausting yourself).

So this year I finally added a feature I've wanted for quite some time - an interactive court map. I can take an image of the court layout from the tournament and create an HTML version in our software. As I mouse over requests in one window, the corresponding court is highlighted on the map in another browser window. Each photographer has a color associated with them. When I assign requests to a photographer, the court is color coded with the color of the photographer. This allows me to group assignments to minimize photographer walk time and keep them in a specific area. It's also very easy to look at the map and see unassigned requests and look to see what photographer is nearby.

This year I also integrated with Twilio and setup a simple set of text shortcuts that photographers can use to let our booth staff know where they are, if they have memory cards that need picking up, if they need water/coffee/snack, etc. They can also move assignments on their schedule or send and SOS for help if it looks like they aren't going to be able to photograph a team.

Kind of a CLI via the phone. :)

The additions have turned out to be really useful and has made scheduling and managing the photographers much easier that it was in the past.

10 years of repeating cycles of the following:

#interview
them: yeah, this is a gamedev position, c#, unity, prototyping, maybe some hololens r&d

me: cool! exactly what i was looking for, as i said a few times, i can't do php anymore, it literally causes me literal deppression.

them: don't worry, we have people for thaz, but we have nobody for c# and unity, with some art skills feel as well as you do.

me: great, glad we're on the same page. i'm taking the job! <3

them: great! oh btw, there's this enterprise intranet app in php that needs some additions, can you please do them?

me: ... what did we talk about during my interview?

them: yeah, but it's just gonna be a short thing, don't worry.

me: ...well...ok, i think i can do that.

*3 to 6 months still on the same, or the next, php enterprise bullshit app. i'm totally exhausted in all ways possible, stressed literally permanently, dreading every day, every new ticket, every meeting every contact with everyone, not able to give a shit about what i do anymore, thinking about suicide*

them: you lazy incompetent fuckup, you're fired!

* i stop communicating and coming out of my room for anything else than toilet, and shopping. stop communicating with my friends, with anyone, anxiety and exhaustion caused by even the thought of talking to anyone about anything, or doing anything, is usually unbearable. i spend 3 to 8 months like this, just sleeping, drinking, watching youtube, sometimes playing games but even that "activity", or rather even the thought of that "activity" is often exhausting. after that time, i kind of recuperate emotionally and mentally, start looking for another unity+c# gamedev job, find it, apply,
goto #interview

I'm the only programmer in the company. Sometimes it's exhausting doing everything. I'm currently working on 3 iOS apps, a web app, building out the APIs and managing the databases at the same time.

Full fucking stack. Every day.

days were nothing seems to work and you end up solving the problem of the problem of the problem of the problem... so frustrating 😥

I miss psychological safety. I'll define it as the willingness to be vulnerable to criticism and the belief that contrary opinions are embraced and judged on their merit.

When I first entered the startup scene my manager had exceptional candor. He had no qualms talking about how kids and personal projects caused his investment in his work to wax and wane.

He always made time to talk to me when I was frustrated and made me feel like he truly listened to what I had to say, even if he didn't act on it.

At the time, I attributed the safety to the company culture created by the CTO. The startup failed and eventually, I found my way to that CTO's next startup.

Completely different experience. I find myself in despair as I hear "I'm more senior and therefore am right and don't have time or interest in your ideas" blatantly stated.

When I disagree with people, I try to ask clarifying questions to identify where the divergence occurs. Sometimes I'm surprised and learn something new, sometimes my questions prompt reconsideration.

With the CTO (now CEO), we go in circles where he squirms, deflects, and outright refuses to respond to my questions. He cancels 75% of 1:1's and when we do talk he suggests that if I disagree I "should introspect which of my beliefs is holding me back from embracing his superior way of doing things"

Multi-hour slack wars suck the life out of anyone trying to ask questions. It's so exhausting to ask questions it's often cheaper and faster to wallow in despair for an hour and hack something together than descend into people shouting preferences at each other and shaming me for not already knowing the answer.

Perks, pay, and tech-stack are all cool. It feels selfish to be unhappy because I can't innovate or challenge the status quo. Having tasted that safety though, I'm left with an unquenched thirst that grows stronger with every conflict.

Sometimes I feel most of my mornings are just dealing with my newest co-worker and helping them do the same things every morning

They are not freaking retaining information and it's like any major issue they run into gets pushed on me. Before they started I liked my job, my coworkers, and the meetings weren't as exhausting. But now I still like my job. But my enjoyment is declining the more I have to explain something multiple times or a simple common sense issue gets forwarded to me

Hello everyone, someone else is stressing the huge amount of technology, especially tools, frameworks that go from one day to the next, and this need to know and adapt them to your vertical or third party projects? ..... for me It's exhausting, sometimes it's really hard to focus ....

It's fuckin midnight, after an exhausting day. Instead of working on a company, why don't each one of us just create a blog site, then add some ads, we share our url here and we'll gonna visit each shared url to generate traffic so we can generate revenue.

Oh if only it could work that way. These what if scenarios sometimes coming to my mind at night. Sorry for the interruption, keep scrolling.