Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "offsite"
-
Too much stupid today..
Client: "Our server is almost full, fix it"
Me: "Okay, you can upgrade the VM, or I can see if theres anything that can be cleaned up?"
C: "Clean it up"
Me: *Finds 5 extra MySQL DB's that amount to 50-60GB in total*
Me: "Well I can remove some of your extra databases, and some of the local backups, as they're also backed up offsite. They're currently unused and dated back to the beginning of 2017, everything is out of date anyways."
C: "No"
Me: "Okay, is there a reason?"
C: "No"
Me: "Okay, so we'll upgrade the VM?"
C: "No"
Me: "I'm sorry, but those are really your two options"
C: "Just fix it"
...................
...................
...................
What in the ducking fuck !@#$#%?
If you're so fuggin smart, then why do they pay me?
I'm clearly stupid and unable to read your mind.21 -
"I should really make better offsite backups"
"Right, this service doesn't do client side encryption"
"Oh this one doesn't have a Linux client"
"OK this one only sets up a single directory you can dump shit into"
"Wtf this one charges more than a high class escort girl"
Whatever... I'm sure my house won't burn down.15 -
This is my most ridiculous meeting in my long career. The crazy thing is I have witnessed this scenario play out many times during my career. Sometimes it sits in waiting for a few years but then BOOM there it is again and again. In each case the person that fell into the insidious trap was smart and savvy but somehow it just happened. The outcomes were really embarrassing and in some cases career damaging. Other times, it was sort of humorous. I could see this happening to me and I never want it to happen to you.
Once upon a time in a land not so far away there was a Kickoff Meeting for an offsite work area recovery exercise being planned for our Oklahoma locations. Eleven Oklahoma high ranking senior executives were on this webinar plus three Enterprise IT Directors (Ellen, Jim and Bob) who would support the business from the systems side throughout the exercise.
The plan was for Sam Otto, our Midwest Director of Business Continuity to host this webinar. Sam had hands-on experience recovering to our third party recovery site vendor and he always did a great job. He motivated people to attend the exercise with the coolest breakfasts and lunches you could imagine. Donuts, bagels, pizza, wings, scrumptious salads, sandwiches, beverages and desserts. He was great with people and made it a lot of fun.
At the last minute Charles 'Don't Call Me Charlie' Ego-Smith, the Global Business Continuity Senior Vice President, decided to grand-stand Sam. He demanded the reins to the webinar. Pulled a last-minute power-play and made himself the host and presenter. You have probably seen the move at some point in your career. I guess the old saying, 'be careful what you wish for' has some truth to it - read on and let me know if you devRanters agree...
So, Charlie, I mean Charles, begins hosting the session and greets all of the attendees. Hey, good so far! He starts showing some slides in the PowerPoint presentation and he fields a few questions, comments and requests from the Oklahoma executives. The usual easy to handle requests such as, 'what if we are too busy to do recover all systems', 'what if we recover all of our processes from home', 'what if we have high profile visitors that month?' Hey you can't blame them for trying. You are probably thinking to yourself, 'been there - heard that!' But luckily our experienced team had anticipated the push-back. Fortunately, Senior Management 'had our backs' and committed that all processes and systems must participate and test - so these were just softball requests, 'easy-peasy' to handle. But wait, we are just getting started!
Now the fireworks begin. Bob, one if the Enterprise IT directors started asking a bunch of questions. Well, Charles had somewhat of a history with Bob from previous exercises and did not take kindly to Bob's string of questions. Charles started getting defensive and while Bob was speaking Charles started IM'ing. He's firing off one filthy message after another to me and our teammate Sam.
'This idiot Bob is the biggest pain in the ass that I ever worked with'; 'he doesn't know shit', 'he never shuts the f up', 'I wanna go over to his office and kick his f'in ass...!'
Unfortunately...the idiot Charles had control of the webinar and was sharing his screen so every message he sent was seen by all of the attendees! Yeah, everyone including Bob and the Senior Oklahoma executives! We could not instant message him to stop as everyone would have seen our warnings, so we tried to call Charles' cell phone and text him but he did not pick up. He just kept firing ridiculously embarrassing dirty IM messages and I guess we were all so stunned we just sat there bewildered. We finally bit the bullet and IM'ed him to STOP ALREADY!!! Whoa, talk about an embarrassing silence!
I really felt sorry for Bob. He is a good guy. Deservedly, Charlie 'Yes I am going to call you CHARLIE' got in big time hot water after the webinar with upper management. For one reason or another he only lasted another year or so at our company. Maybe this event played a part in his demise.
So, the morale is, if you use IM - turn it off during a webinar if you are the host. If you must use it, be really careful what you say, who you say it to and pray nothing embarrassing or personal is sent to you for everyone to see.
Quick Update - During the past couple of months I participated on many webinars with enterprise software vendors trying to sell me expensive solutions. Most of the vendors had their IM going while doing webinars and training. Some very embarrassing things came flying across our screens. You learn a lot reading those messages when they pop-up on the presenters' screen, both personal and business related. Some even complaints from customers!
My advice to employees and vendors is to sign-out of IM before hosting a webinar. Otherwise, it just might destroy your credibility and possibly your career.5 -
I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…
I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.
I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.
We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.
Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.
My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.
Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.
At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.
Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?
Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.
Amazingly, no one was fired.4 -
So, apparently we had this important meeting with a client offsite this morning, I was "told" yesterday but nobody thought of creating a calendar event or sharing emails about it. I forgot, I don't even know the address or the hour.
My boss and this sneaky front ender came by at 11am joking about me not remembering.
Me: wow, I forgot. If it isn't on the calendar I don't even know it exists.
Turned to my screen and got back to work.
Fuck this shit.1 -
26 or so hours up now. And I've got a few stories to tell :) feel free to refresh your cup of coffee and take a seat.
Last few days I've been going into this odd place called intown.irl to get in touch with its inhabitants. An odd place I have to say. But in some cases quite rewarding, even got a MILF home with me and into bed at some point. Anyway...
3 days ago I think it is now? Thursday evening I took my laptop to this local bar where I had this issue about dihydrogen monoxide with one of the bartenders earlier (you'll find that rant on those keywords). Still wanted to visit it regardless though, as I met that first woman there earlier that approached me. Unfortunately I didn't see her there that day.
Some bald guy who was clearly drunk approached me. Many people were already giving curious looks at this laptop I brought to the bar. I finally tuned it up with the stickers from FOSDEM.. I'll put a picture of it in the comments. My theme was one of privacy (central), distributions and Google's open source initiative (which aligns with the keychain token I got from them as well). But of course.. that guy.. he thought that a pimped/riced laptop obviously meant that I was a hacker.
Guy went to the toilet.. went back.. and suddenly grabbed my laptop and turned it towards him. Boy was I never more smugly satisfied that those rubber pads on the bottom are quite resilient. Could've almost damaged my screen by trying to grab it like that. But it's a CCFL display.. so high voltage. If it were to become broken.. worth it. 😈
On it at the time was a terminal, pinging Google (had network issues at that bar, to the point where one of the - I think - staff members got up to me and offered the WiFi password and got to talk with me.. more on that later), and my usual Linux desktop along with the Arch anime wallpaper with the quote of Da Vinci.. simplicity is the ultimate sophistication. Of course the guy saw the terminal.. and probably reaffirmed.. yep, that's a hacker. At least he wasn't too wrong about the general term.. but the hat.. most likely he was wrong on that one.
Guy left with this question.. "you are a hacker, aren't you."
I replied to him: "No sir. I'm not a hacker. I've got no idea what you're talking about."
Guy kept looking at me weirdly for the whole night to come.
Back to that companion guy though. Mac user, yada yada.. but he told me about his backup solution. Apparently - I shit you not - he has not only the photos on his local device, he's also frequently backing them up in Time Machine (which I was really curious about whether it uses mirroring or snapshots.. he couldn't tell, lmk if you do) but not only that.. he was storing another offsite backup in that very bar, in case his house went on fire.
Now that is a proper backup scheme!!! If only more people were like that.
Seriously though.. that bald guy who took my laptop just like that... I just let it slide for that one time, but I tend to treat my machines as an extension of my very self. I think that was a very uncalled for move. Asshole...
How would you have reacted to such a thing? And.. maybe that's why we technologists don't get outside too often? Fucking everything is hacking these days if it's not Knopkes and Blinkenlights… Not every shell is a h4xx0ring console for h3kk1ng de fasbuk…9 -
The mad dash to crochet all the gifts before the offsite continues.
One of my teammates really loves this cowboyger emoji https://slackmojis.com/emojis/...1 -
I really felt like a badass one time when I managed to recover all projects on our dev server after a full meltdown of the HDD.
We had no recent backups, because our backup server was down for a few months, and our (at the time small) company was in a tight spot on finances, and couldn't get a replacement.
The problem was that the HDD on the backup server failed, but we were storing all projects also on the dev server, along with our local git repos (no GitHub at the time for us), but then the dev server HDD also broke, and I used every piece of data recovery software I found trying to recover the data, until one actually managed to read the raw data from the HDD and store it as a virtual drive, that I then used to try and build another partition index and it actually worked!
Lost about 10% of the data, but that was enough, as i managed to recover all the git repos and databases...
I don't even remember the tools that got the job done in the end, but that was one hell of a week, and at the end I felt like a true IT God!
True story!
PS: 2 weeks later we had a new backup server, another offsite backup solution and a GitHub account for the company. Was delayed on salary in order to manage it (me and the CEO both agreed to give our pay for one month to get them), but worth it!1 -
NewLifeNewHope update No.2 / Day 5
My Server Is finally opened and i installed 2x8 TB WDC Harddrives and SATA 128 SSD. I know this server is freaking garbage, but i got this PC for free, and have somewhat good-ish upgrade path, so heres the spec :
-Gigabyte H110m-S2 LGA1151
-Intel i3-6300
-8GB DDR4 single channel RAM
-128 GB SSD
-2x8TB Harddrives
-TP-Link 1000mbps NIC
so the plan is to make this server as the Main Repository -- yes no offsite backup plan for now -- and also i want to make this server as an Email Server and for hosting my company's website. I've already asked for static IP from my ISP and will take effect tommorow.
I need help for choosing the Operating System (i like centOS) for my server. and to setting this Server to work like what i planned but i don't know where to start, Any help and/or References will be great !14 -
Ransomware has the effect to destroying data and my will to live.
I come in to work yesterday to see everyone panicking. We got hit by WannaCry. The boss is with IT dealing with it and sends all of us home. No biggie, since all of my work data is backed up on an offsite server.
I come back into work to still see everyone in a frenzy. I think they are still overreacting to the attack, so I go to my corner in the office. I boot up to see the Ubuntu logo, which is a huge problem, not for me of course, but for everyone else who can barely work Windows. I barge in on my boss and ask him why the fuck is Ubuntu on the PC's.
He replies, "I looked it up, Linux is not affected by viruses".
What the fuck is wrong with people? After I tell him to revert back to Windows because he's being a fucking retard, he goes ahead to send all of us home.
I can only hope he doesn't install Zorin or Elementary because I will quit.6 -
This Christmas I transitioned into a new job. At the old job I was the only kubernetes-guy, so since they no longer have any developers who are confident working with that, they decided to go with LAMP-stack.
The data from dev-kubernetes-server was backed up by some guy and moved to an offsite-server, or so they told me. Turns out, he had backed up the kubectl-config-file, and not the databases. Now everything is wiped. Sure glad we still have that config-file!
Of course, since that was only our dev-server, there was nothing too important there, except for all the documentation. The only other backup? On my laptop, which I turned in to them, and is now wiped and used by one of the sales-guys.
Now I’m being called in at least twice a day, since I was their goto-guy for almost anything backend-related. Feels great, after they spent a couple of months attempting to rewrite everything in pure PHP (with a strict no-dependency policy for some reason). Fml.2 -
The annual end of year meeting that takes a 3 day offsite so we can list 21 new enterprise projects to be completed the upcoming year - hehe 😅
-
Wrote a good backup program, no one cares. I mean, its great if you want to create fast offsite backups! And it is simple to use, and has a pretty dedicated developer working on it all day long (Me)... Well, of course you would want to check it out, so here you go https://github.com/paulkramme/...7
-
Finally back at the HQ and away from Offsite Hell after 18 months!!!!!!!!!!!! Real internet! Coffee on tap! Community of practice meetings! COOODDDEEEE!!!! Also back to devrant. Goodbye Indian devs from hell, j/k they still suck life out of my day with their deprecated ways.
Side note:
Switched to Unreal Engine from Unity recently and my god it is amazing! I definitely prefer being able to use C# with unity vs c++ with unreal but the blueprint system is a great visual programming system.
Unit testing is my new side chick. She wants me to leave my wife; I'm considering it.
Unrelated: Read Dead Redemption 2 is amazing.1 -
I think celebrated too early. Lol. So we moved our servers offsite to a data centre this weekend. Everything was done yesterday and we started testing stuff late last night and we are today. Everything was going great and now, they’re beginning to fail again.
All I’m asking for now is to pray for my team and I before the office opens tomorrow. 😁 -
One offsite team are not adding dependencies to the build script and are puzzled when it doesn't compile on my machine....
-
If you had to evacuate your office due to a fire without your laptop and the office was destroyed could you continue to work? Do you have offsite recovery laptops?
Developers usually have flexibility where they can work from but sometimes do not have backup machines available and configured.
As a side question - would your other critical processes - accounting, HR, sales be able to continue to keep your business going in a disaster - or would they be like deer in headlights?6 -
Does anyone have recommendations for a good offsite backup solution that’s like student-level cheap (or preferably free) that allows you to keep control over your data (I don’t want to upload my stuff to google drive or the like for privacy and government surveillance reasons as I’m sure @linuxxx would understand).9