My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

A room full of mostly old male stressed out engineers sat in chairs, and the presenter said:

"So who watched Judging Amy last night?"

The presenter went on to express her surprise that nobody in the room had seen last night's episode of Judging Amy.... and wasn't going to drop the topic.

The meeting, if it ever had any, now had no chance of going anywhere good.

By the end of the meeting someone would walk out and "retire" shortly there after, and it certainly wasn't going to be the presenter....

Backstory:

The company built on the IBM model of sell pricey custom hardware (granted it worked really well) and sell expensive support contracts wasn't doing as well as it had hoped. Granted it was still doing better than most of its neighboring companies, but it was clear that with the .com bust the days of catered lunches every day were over.

The company had grown fat and everyone knew that while the company had a good enough product(s) to survive, there weren't enough lifeboats for everyone to survive.

In the midst of this an HR department that took up nearly 20% of the office space at HQ felt it needed to justify its existence / expenses.

They decided to do this in the same way they always had, by taking funding from other departments, this time not by simply demanding more direct budgets for themselves.... they decided to impose mandatory 'training' on other departments ... that they would then bill for this training.

When HR got wind that there were some stressed out engineers the solution was, as it always is for HR.... to do more HR stuff:

They decided to take these time starved engineers away from their jobs, and put them in a room with HR for 4 days. Meanwhile the engineer's tasks, deadlines and etc remained the same.

Support got roped into it too, and that's how I ended up there.

It would be difficult to describe the chasm between HR and everyone else at that company. This was an HR department that when they didn't have enough cubes (because of constant remodeling in the HR area under the guise of privacy) sat their extra HR employees next to engineering and were 'upset' that the engineers 'weren't very friendly and all they did was work'.

At one point a meeting to discuss this point of contention was called off for some made up reason or another by someone with a clue.

So there we all sat, our deadlines kept ticking away and this HR team (3 people) stood at the front of the room and were perplexed that none of these mostly older males in this room had seen last night's episode of Judging Amy.

From there the presentation was chaos, because almost the entire thing was based on your knowledge of what happened to poor stressed out Amy ... or something like that.

We were peppered with HR tales of being stressed out and taking a long lunch and feeling better, and this magical thing where the poor HR person went and had a good cry with her boss and her boss magically took more off her plate (a brutal story where the poor HR person was almost moved to tears again).

The lack of apparent sympathy (really nobody said much at all) and lack of seeming understanding from the crowd of engineers that all they should do is take a long lunch, or tell their boss to solve their problems ... seemed to bother the HR folks. They were on edge.

So then they finally asked "What are your stressers?" And they picked the worst possible person they could to ask, Ted.

Ted was old, he prickly, he was the only one who understood the worst ass hell of assembly that had been left behind.

Ted made a mistake, he was honest with folks who couldn't possibly understand what he was saying. "This mandatory class is stressing me out. I have work to do and less time because of this class."

The exchange that followed was kinda horrible and I recall sitting behind Ted trying to be as small as possible as to not be called on. Exactly what everyone said almost doesn't matter.

A pedantic debate between Ted and the HR staff about "mandatory" and "required" followed. I will just sum it up that they were both in the wrong for how they behaved for a good 20 minutes...

Ted walked out, and would later 'retire' that week.

Ted had a history and was no saint. I suspect an email campaign by various folks who recounted the events that day spared ted the 'fired' status and he walked with what eventually would become the severance package status quo.

HR never again held another 'training', most of them would all finally face the axe a few months later after the CEO finally decided that 'customer facing, and product producing' headcount had been reduced enough ... and it was other internal staff's time for that.

The result of the meeting was one less engineer, and everyone else had 4 days less of work done...

Met a Project Manager (at a friend's party) who had transitioned to a PM role from a developer role (most probably he wrote shitty code)

Smartass PM to me (after I told I code for living) : I really pity poor programmers and I feel sorry for them, the work they do, the effort they put in l, it's just now worth it

Me : yes you are right if we don't code PM are just not worth it, I understand it's a skill to talk about deadlines and features and what not, but the Pre-requisite is that some one would code it first. Also coding is not that anyone can do, I do it because I enjoy it, I m just not meant for superficial talks and I love building things, that's y I do it..

Smartass PM : (dumbstuck)

After half an hr of bullshit conversation...smartass PM has realized it by now that in Silicon Valley (where we live) it's much cooler to be a developer than being a PM (he has recently moved from east coast)...

PM to me : I just live on stack-overflow
Me thinking : Really !!

People should not compare their career paths, every one has their interest and personality

After months and months of slaving away, I quit my start-up job and feel completely amazing- here's what happened:

Met a classmate in grad school and he talked about starting his own company and he had full funding and etc. After graduation, moved to the new city where the job was located.

There were all these promises of us being co-workers and working on cool things and many other promises made. Soon after starting the job, most of these promises we're just smoke and mirrors.

Started working day in day out. Worked from 8am-9pm most days and worked on weekends too. Treated me like a I was a dog, talked down to me, gave unrealistic deadlines, pressured me with attitude and threats of losing my job. Hell, they thought they were the smartest person to touch the earth basically- example being that they mixed jQuery with VueJS in our Django template.....who the F*** does that. Another thing being that they had issues with me soft deleting records since they wanted them completely hard deleted and we had gotten into a giant argument about that fml.

What led to me leaving the job was that I had gotten sick one of the weeks, and I still showed up to work. Each day I was gradually getting sicker and sicker. Still tried my best to get work done. Saturday morning I get the most passive aggressive and bitchy text from my co-worker. "if you don't complete blah blah blah by Monday, we are going to have issues. Then on Monday you will work on blah blah blah". They blew the fuse with me. They would always punish me for being sick or taking a vacation. I'm not a dog, not a machine, I'm a f****** person. Went into his office when the work week started and gave my resignation on the spot and felt like it was the best decision I've ever made.

Now I just feel like a giant toxic cloud has disappeared from my life. I did walk away with so much experience and knowledge but now I just feel extremely burnt out from programming. Is this what I even wanna do anymore?

Few lessons I learned along the way:
1. If it sounds too good to be true, it probably is
2. Free lunches aren't worth it
3. Unlimited PTO doesn't really mean unlimited- there's always stipulations
4. Start-up life isnt as cool as they say- don't take TV portrayals as the real thing
5. Your mental health is extremely important
6. It's okay to admit to yourself that you're burnt out
7. Take a break
8. STARTUPS ARE NOT FOR EVERYONE

This is just my experience and what I learned, so telling my story. Phew, feels so good to get that off my chest

I was hired as a junior game developer and I had a senior who was working alongside me on a client’s project.

Within one month they moved my senior to another project and made me the project manager of the project that I was working on.

Worst fucking experience ever. Client’s were unresponsive for a week sometimes and pretty strict when it came to deadlines. Needless to say my employer lost the client and I was made the junior developer again.

Working hard to meet crazy deadline to finish last update before new product announcement to make it look better. Our CEO blabs about new top secret product at some conference throwing away all marketing efforts up to date and putting marketing team into panic mode. Result? They moved the announcement date without discussing it with development. Result? Our efforts and overtimes wasted and we are announcing product before it is ready. End result? I'm pissed so I wrote angry e-mail to our CEO. Wondering what will happen now :-) But with unfinished announced product and crazy deadlines they need me a lot more than I need them.

The place I currently work at has got this culture of ignoring developers.

Deadlines get made by 3rd parties and project managers who don't have the technical nounce or experience of our system to make a call on deadlines.

Demos of products are arranged without a discussion with developers as to whether said component will be ready on that date.

3rd parties make decisions about future architecture, offer to assist, then disappear for days on end, to only come back and make out as though they've not been holding us up.

Upper management take no interest, don't listen to the people they pay to do a job.

Currently just moved a PHP web app into a multi tenant scalable EBS environment, but apparently it's not worth asking our view on technical aspects of the business before the shit hits the fan.

Lies to clients about documentation and policies, for example, claims from Sales we have a DR and BCP plan, client called is out, they sent a 2 paragraph A4 document to the client claiming it was our DR and BCP plan without talking to anyone technical, including myself who has years of DR experience. Embarrassing.

Could go on, but rant over.

We basically don't unit test at work. I write some tests for my code and honest to God people complain I'm wasting time saying a test bed and manual tests are good enough. We don't write test beds for about half of our production code and rely on integration tests for the rest. We only test release builds which have been symbol stripped, I get handed a crash report with no stack trace that I'm unable to reproduce and expected to stay late to fix it for some arbitrary internal deadline.

I've since moved to R&D where basically I'm left to do my own thing so it's better.

We don't project manage. Project leads take time estimates and double them so management might cut them some slack. This doesn't matter because management made up time estimates before the project started. Last project I was on had a timeline of 3 months and took a year.

We have released broken products. Not that any of the above really matters, our software products have made about 50k revenue in 2 years. There are 6 people on software. Fortunately hardware has made about 3 mill. That said our hardware customers are getting frustrated with us as we keep fucking up, shipping broken products and missing deadlines.

I've been working there about a year and a half and will be looking for a job at the end of the current project.

I joined devRant about when I was most pissed off with my job, my rant frequency has definitely gone down since I moved over to R&D.

If any of you have been following my last few rants, you'll know I've been working on a project with a particularly difficult client, trying to meet wholly unrealistic deadlines with only one other developer.

The situation has reached the climax. The client had a call with our project manager and boss on Monday to discuss things. Despite them still not having paid a single bill since October, they've demanded the release date be moved to the 6th April. Apparently we'd agreed to release on this date, despite making no such promises, the (optimistic) deadline we were working towards has always been, since it was set about 2 weeks ago, the 16th April.

Apparently AWS migration won't take as long as we think it will, because the designers that do the CSS for this project say so, despite knowing nothing about the architecture of the requirements of the system once live (like if backups are required and what of).

The bottom line is that client is ending development with us the day after the project goes live to give it to their own in-house team. If they want us to work more after the date, they have to buy blocks of days.

To make things better, a large part of new functionality relies on an external API we can't even begin to do learning tests with, let alone integrate due to back-office errors on their end. They've had since Friday to give us our token, yet here we are.

Something tells me my holidays booked for for the first week of April are going by the wayside.

Most hacky things I've ever done:

A windows scheduled task that kicks off a massive as fuck ETL job, riddled with errors. Damn thing had a mind of its own and only ran whenever it felt like it. Client was happy, deadlines were met, boss moved me to another task.

One of my really good friends met me today and she's become a project manager in a big firm and has been asked to manage a project and get it delivered.
A bit of history first. She and I graduated from the same college and got into different firms as software developers. We and our dev friends used to bitch about how PMs don't know anything and don't give a fuck about the quality of project. She moved on to pursue MBA. Fast forward 4 years.. it's today and we are in my apartment having dinner. She's going on and on about how bad the project is, how mess of the project this has become and how she doesn't care at all about the project and doesn't know anything but wants devs to finish the job no matter what. She knows the deadlines are aggressive but has directions to shit all over the devs and get the work done. So she's just doing that.. giving them an earful and asking for daily updates and questioning them about delays without even knowing what the project is about.

When I try to tell her that it's not the right thing to be doing, she's proudly admitting that's she's only going to manage the project for a while and doesn't really care what happens to it.

I have lost some respect for her now.. :(

So.. wondering what people's thoughts are on this.

So I started working for a company late last year, so far I've done four projects (one was a test), the contract and hourly rate was sorted out and all happy. However, in the three months I've worked there the terms of payment has been changed two times (both times within 30 days).

First (originally) was being paid for x amount of hours any hours over that moves on to an hourly rate.

Second change moved over to a fixed rate per month.

And currently I'm now just on an hourly rate.

The justification for the change in terms was "the current terms isn't working out for us" - as far as I'm aware no-one elses terms have changed. However the quickness of the term changes here do concern me.

Now I do have some personal issues (disability for one) as well as other issues so I don't do things faster than other other devs, however, I get the job done and I stick to deadlines (and yes the company is aware of my personal issues).

In terms of current projects I'm only doing tweaks to current projects so my hours at the moment per week are slow (I have been asked about new projects) but I've heard nothing whatsoever on start dates etc.

Just an FYI the current work done, I've had no issues whatsoever.

So I would like some thoughts on what you'll do if you were in this situation as personally I can think of two reasons but I would rather see what other people think here.

So, if anyone remembers my previous rants (or otherwise), the app I'm working on is finally going to release. We release in about a week and a half, despite having at minimum 10 days work + testing (not including all testing of aws migration that we still haven't fully completed) to even have an MVP.
This is all because the clients have been pushing a release date (16th July) in marketing. Oh, and the icing on the cake? Our boss agreed for a submission to app stores (back-end pending) on Friday (yes, in like 3 days) instead of the agreed next Wednesday date.

Side note: the client is coming over to our office for Thursday and Friday. Why? We are unsure.

How do you deal when you are overpromising and underdelivering due to really shitty unpredictable codebase? Im having 2-3 bad sprints in a row now.

For context: Im working on this point of sale app for the past 4 months and for the last 3 sprints I am strugglig with surprises and edgecases. I swear to god each time I want to implement something more complex, I have to create another 4-5 tickets just to fix the constraints or old bugs that prevent my feature implementation just so I could squeeze my feature in. That offsets my original given deadlines and its so fucking draining to explain myself to my teamlead about why feature has to be reverted why it was delayed again and so on.

So last time basically it went like this: Got assigned a feature, estimated 2 weeks to do it. I did the feature in time, got reviewed and approved by devs, got approved by QA and feature got merged to develop.

Then, during regression testing 3 blockers came up so I had to revert the feature from develop. Because QA took a very long time to test the feature and discover the blockers, now its like 3 days left until the end of the sprint. My teamlead instantly started shitting bricks, asked me to fix the blockers asap.

Now to deal with 3 blockers I had to reimplement the whole feature and create like 3 extra tickets to fix existing bugs. Feature refactor got moved to yet another sprint and 3 tickets turned into like 8 tickets. Most of them are done, I created them just to for papertrail purposes so that they would be aware of how complex this is.

It taking me already extra 2 weeks or so and I am almost done with it but Im going into really deep rabbithole here. I would ask for help but out of other 7 devs in the team only one is actually competent and helpful so I tried to avoid going to him and instead chose to do 16 hour days for 2 weeks in a row.

Guess what I cant sustain it anymore. I get it that its my fault maybe I should have asked for help sooner.

But its so fucking frustrating trying to do mental gymnastics over here while majority of my team is picking low hanging fruit tasks and sitting for 2 weeks on them but they manage to look good infront of everyone.

Meanwhile Im tryharding here and its no enough, I guess I still look incompetent infront of everyone because my 2 weeks task turned into 6 weeks and I was too stubborn to ask for help. Whats even worse now is that teamlead wants me to lead a new initiative what stresses me even more because I havent finished the current one yet. So basically Im tryharding so much and I will get even extra work on top. Fucking perfect.

My frustration comes from the point that I kinda overpromised and underdelivered. But the thing is, at this point its nearly impossible to predict how much a complex feature implementation might take. I can estimate that for example 2 weeks should be enough to implement a popup, but I cant forsee the weird edgecases that can be discovered only during development.

My frustration comes from devs just reviewing the code and not launching the app on their emulator to test it. Also what frustrates me is that we dont have enough QA resources so sometimes feature stands for extra 1-2 weeks just to be tested. So we run into a situation where long delays for testing causes late bug discovery that causes late refactors which causes late deliveries and for some reason I am the one who takes all the pressure and I have to puloff 16 hour workdays to get something done on time.

I am so fucking tired from last 2 sprints. Basically each day fucking explaining that I am still refactoring/fixing the blocker. I am so tired of feeling behind.

Now I know what you will say: always underpromise and overdeliver. But how? Explain to me how? Ok example. A feature thats add a new popup? Shouldnt take usually more than 2 weeks to do my part. What I cant promise is that devs will do a proper review, that QA wont take 2 extra weeks just to test the feature and I wont need another extra 2 weeks just to fix the blockers.

I see other scrum team devs picking low hanging fruit tasks and sitting for 2 weeks on them. Meanwhile Im doing mental gymnastics here and trying to implement something complex (which initially seemed like an easy task). For the last 2 weeks Im working until 4am.

Im fucking done. I need a break and I will start asking other devs for help. I dont care about saving my face anymore. I will start just spamming people if anything takes longer than a day to implement. Fuck it.

I am setting boundaries. 8 hours a day and In out. New blockers and 2 days left till end of the sprint? Sorry teamlead we will move fixes to another sprint.

It doesnt help that my teamlead is pressuring me and asking the same shit over and over. I dont want them to think that I am incompetent. I dont know how to deal with this shit. Im tired of explaining myself again and again. Should I just fucking pick low hanging fruit tasks but deliver them in a steady pace? Fucking hell.