My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

None...

My work desk is ordered chaos. It's an unspectacular white desk.

To the left paper for taking notices... And a collection of notes highly unordered.

In the top left corner of the desk the router.

Middle (front to back): Color sorted post it's with my backlog and all the stuff that I need to keep track off.

Keyboard, Monitor

Right (front to back): mailboxthingy, post it block, utensilo (made of bamboo)

That's the working desk. Like the "shining knight in armor"....

Next to the working desk is the play desk, a large kitchen table with a 5mm thick black rubber (keep your dirty thoughts for yourself ;)) as insulation and scratch prevention.

There is usually utter chaos.

Banana PI R2, test router, old hardware, hardware storage cabinet, screw and other small part cabinet....

And to the right is a large chest of drawers where I cram everything in that doesn't belong anywhere.

I'm pretty boring I guess.

Except you are kinky for old hardware and lots of weird stuff noone uses nowadays...

Then you're in heaven I guess.

Since day 0, I have been fond of computers. One of my first plush was called "DataDog" and looked like a CRT screen with dog ears around. According to my mum I was "addicted" to it.

At year 2, my dad was arranging some music on some software while I was watching him on his lap. Quick jump to the present: nowadays and since 10 years I run my own home studio with three guitars, two keyboards, one bass, three monitors, a microphone, an amp and a cabinet... coincidence? I think not!

Fast forward 5 years later (so I'm 6-7 years old), and I was playing with the legendary pinball game on Win95, as well as Flight Simulator. Then I was hogging mum's laptop to play settlers II (<3 that game), I eventually got my computer, and got into Quake III Arena being aged 10 (and had to tell my mum that game was safe for my age haha - I eventually removed the blood effects).

The Quake 3 Arena chapter is interesting: it got me into router configuration as I wanted to open a port through the router to host my own dedicated games with friends, it got me into DNS configuration (I was running a no-DNS client that allowed friends to join me through a DNS while having a dynamic IP) and eventually... to modifying .cfg files to tune my server as I wanted it. No programming here but a nice intro into :)

Then I hated the fact everybody would point their finger at me and say "geek" - I was only 13, fragile, sensitive, and I wanted everything but a bad image on me.

Meanwhile I continued on getting interested in hardware and configure my own computers, and investing myself into music production.

Then, university. "What do you want to study?" I thought of everything but IT, fleeing the image of a "geek". Turns out it was a waste of time, and at 21 yo I got into web development (well, just html and css), then learned a bit of PHP, finally got a specialized 2-year training and now here I am!

I was bound to be in IT either way since day 0, and funny fact, I've used every windows edition since Win95.