Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "no deploy friday"
-
"Don't deploy on Friday" is a public admittance that your company either has no CI/CD pipeline, or that all your devs are retarded rhesus monkeys who only wipe their ass if the product manager wrote it as a spec.
If the saying was: "Don't port your whole API to GraphQL on a Friday", or "Don't switch from MySQL to Postgres on a Friday", I would agree.
But you should be able to do simple deploys all the time.
I deployed on Christmas & New Year's eve. I've deployed code while high on LSD, drunk-peeing 2 liters of beer against a tree after a party. I've deployed code from the hospital while my foot was being stitched up. On average, we deploy our main codebase about 194 times a week.
If you can't trust your deploys, maybe instead of posting stupid memes about not deploying on Fridays, you should fix your testing & QA procedures.46 -
Worst thing you've seen another dev do? Long one, but has a happy ending.
Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.
The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.
Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.
At 5:00PM on a Friday, deploys, goes home.
Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)
After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)
We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.
Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.
Well, since I was responsible for logging, I was next in the hot seat.
For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."
Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.
A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.
Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).10 -
Currently on an internship, PHP mostly, little bit of Python and the usual web stuff, and I just had the BEST FUCKING DAY EVER.
Wake up and find out I'm out of coffee, oh boy here we go.
Bus leaves 10 minutes late, great gonna miss my train.
Trains just don't wanna ride today, back in a bus I go, what's normally a 10 minute train travel is now a 90 minute bus ride.
Arrive at internship, coffee machine is broke, non problem, I'll just lose it slowly.
NOW HERE COMES THE FUCKING GOOD PART!!
Alright, so I'm working on a CMS that can be used just about on any device you want, mobile or desktop, it's huge, billion's of rows of scientific data. Very specific requirements and low error margins. Now, yesterday I was really enjoying myself here until today, Project manager walks in, comes to my desk and hands me a Samsung Gear S3, an Apple watch and some cheap knockoff. He tells me that before the Friday deploy, THE ENTIRE CMS SHOULD WORK ON THOSE WATCHES!
I mean, don't get me wrong, I like a challenge but it's just not right, I mean, I'm still not sure what the right way to handle tables on phones is, but smart watches, just no. Besides that, I've never worked with any Apple devices, let alone WatchOs, nor have I worked with Android Wear.
Also, Project Manager is a total dickhead, he's the kinda guy that prefers a light theme, doesn't clean up his code, writes 0 documentation for an API, 1 space = tab, pure horror.
So after almost flipping my desk, I just called my school coach to announce I'm leaving this internship. After a brief explanation he decides to come over, and guess what, according to the Project Manager I wasn't supposed to do that, I was supposed to test if it would be possible.
FUCKING ASSFUCKFACE9 -
So I worked on getting a server ready for about 30 hours last week to be ready for a deploy on Monday Night (last night). Not only did I work on it for 30 hours, we had two other architects and a senior engineer working on it too. We got everything done Friday and it was ready to go with a simple cutover on Monday night.
The only thing left to do was deploy a link change Monday night on the existing landing page. My part was the backend servers and application that had the complicated SSO system and the other part was just a link to get to the SSO. I asked the person responsible for deploying the landing page's link if he was ready about a dozen times. He kept saying he was deploying X (the code name for the project deploy) and that is all he was doing.
Now jump to that night. They have decided that a single landing page wasn't enough and they were going to deploy a full CMS. Well no one knew what the hell was going on and they didn't realize that the landing page was hosted externally on another host. After arguing for two hours they delayed the deployment for multiple days. 24 hours later they are still trying to figure out the CMS on a host.
30 hours and four senior engineer's time wasted to get everything done for the deadline all to be canceled because of on jackass's lack of planning. WTF2 -
I had a huge epiphany on Friday... not all developers enjoy coding.
Discovered when they brought down 2 of our environments, well told them what was wrong with the changes in their code that caused the environments to break, gave them links directly to the file in the gitlab repo that needed to be updated, and...
They fucking went home. The change would’ve taken all of about 30-45 seconds to update and they fucking left.
This person’s team lead come storming in pissed off because her manager is furious about 2 environments going down and preventing everyone else from being able to deploy their changes.
We provide the exact same details to the team lead about what needs to be changed, and advise that her team member took off....
30 mins later, her manager is storming up to us (devops/sre) livid as hell.
Explain the situation for a third time... manager is like, why can’t you guys fix it?
Look here you dense motherfuckers, we can fix the code. We can be the plumbers that clean up your shit. But what value do you gain as a developer if you don’t understand how the systems work and you keep pushing shit in?
Made the changes, fixed the environments, done right? Wrong.
The original developer made more changes not knowing what would happen and thoroughly fucked the environments again.
This dumb-fucking dumpster fire of a dude then sends us a slack message. “It’s down again, can you fix it?”
Our manager steps in and tells us to send him a link to the logs and have him fix it himself!
Thank goodness we have a badass manager.
Send logs, send repo file links (again), and send line numbers in the logs to try and help just a bit more. Dude goes almost the whole day without fixing it, environments are down, other devs are pissed, we throw this dude to the wolves. His manager starts to head over and was about to talk with my team lead when our manager steps out of his office and tells him the in’s and out’s of the situation and that our job isn’t to play log parser/error fixer for the developers. This dude that’s breaking the environments needs to be the one to fix the issue and his team lead should be aware of the problems and should have been able to correct his errors before it ever came to us.
The amount of hand-holding we do is ridiculous.
(Disclaimer, this one guy making some mistakes doesn’t sound too bad, but this is actually a common occurrence for like 40% of all of our developers)
We literally have interns still in college running circles around some of our full time devs. I know I’m not a developer, but for anyone that’s new-ish to developing, when you see shit like that please don’t lose hope. Those ass-hats got into programming purely for a paycheck, not because of passion.
Stick with it and your greatness will know no bounds 👍
As for you craptastic dipstick lickers, FUCK YOU!!! Go back to school and learn how to give a damn.4 -
Long rant ahead.. 5k characters pretty much completely used. So feel free to have another cup of coffee and have a seat 🙂
So.. a while back this flash drive was stolen from me, right. Well it turns out that other than me, the other guy in that incident also got to the police 😃
Now, let me explain the smiley face. At the time of the incident I was completely at fault. I had no real reason to throw a punch at this guy and my only "excuse" would be that I was drunk as fuck - I've never drank so much as I did that day. Needless to say, not a very good excuse and I don't treat it as such.
But that guy and whoever else it was that he was with, that was the guy (or at least part of the group that did) that stole that flash drive from me.
Context: https://devrant.com/rants/2049733 and https://devrant.com/rants/2088970
So that's great! I thought that I'd lost this flash drive and most importantly the data on it forever. But just this Friday evening as I was meeting with my friend to buy some illicit electronics (high voltage, low frequency arc generators if you catch my drift), a policeman came along and told me about that other guy filing a report as well, with apparently much of the blame now lying on his side due to him having punched me right into the hospital.
So I told the cop, well most of the blame is on me really, I shouldn't have started that fight to begin with, and for that matter not have drunk that much, yada yada yada.. anyway he walked away (good grief, as I was having that friend on visit to purchase those electronics at that exact time!) and he said that this case could just be classified then. Maybe just come along next week to the police office to file a proper explanation but maybe even that won't be needed.
So yeah, great. But for me there's more in it of course - that other guy knows more about that flash drive and the data on it that I care about. So I figured, let's go to the police office and arrange an appointment with this guy. And I got thinking about the technicalities for if I see that drive back and want to recover its data.
So I've got 2 phones, 1 rooted but reliant on the other one that's unrooted for a data connection to my home (because Android Q, and no bootable TWRP available for it yet). And theoretically a laptop that I can put Arch on it no problem but its display backlight is cooked. So if I want to bring that one I'd have to rely on a display from them. Good luck getting that done. No option. And then there's a flash drive that I can bake up with a portable Arch install that I can sideload from one of their machines but on that.. even more so - good luck getting that done. So my phones are my only option.
Just to be clear, the technical challenge is to read that flash drive and get as much data off of it as possible. The drive is 32GB large and has about 16GB used. So I'll need at least that much on whatever I decide to store a copy on, assuming unchanged contents (unlikely). My Nexus 6P with a VPN profile to connect to my home network has 32GB of storage. So theoretically I could use dd and pipe it to gzip to compress the zeroes. That'd give me a resulting file that's close to the actual usage on the flash drive in size. But just in case.. my OnePlus 6T has 256GB of storage but it's got no root access.. so I don't have block access to an attached flash drive from it. Worst case I'd have to open a WiFi hotspot to it and get an sshd going for the Nexus to connect to.
And there we have it! A large storage device, no root access, that nonetheless can make use of something else that doesn't have the storage but satisfies the other requirements.
And then we have things like parted to read out the partition table (and if unchanged, cryptsetup to read out LUKS). Now, I don't know if Termux has these and frankly I don't care. What I need for that is a chroot. But I can't just install Arch x86_64 on a flash drive and plug it into my phone. Linux Deploy to the rescue! 😁
It can make chrooted installations of common distributions on arm64, and it comes extremely close to actual Linux. With some Linux magic I could make that able to read the block device from Android and do all the required sorcery with it. Just a USB-C to 3x USB-A hub required (which I have), with the target flash drive and one to store my chroot on, connected to my Nexus. And fixed!
Let's see if I can get that flash drive back!
P.S.: if you're into electronics and worried about getting stuff like this stolen, customize it. I happen to know one particular property of that flash drive that I can use for verification, although it wasn't explicitly customized. But for instance in that flash drive there was a decorative LED. Those are current limited by a resistor. Factory default can be say 200 ohm - replace it with one with a higher value. That way you can without any doubt verify it to be yours. Along with other extra security additions, this is one of the things I'll be adding to my "keychain v2".11 -
When your week has been so busy and exhausting you remember at 1PM Friday you have a deadline for Monday morning and force yourself to do a weeks worth of work in 4 hours and deploy it on a Friday without QA testing!
To future me - I apologise for Monday’s headaches. -
Today I learned that bugs in Proxmox aren't bugs because they're not *exactly* within the scope of le fancy PVE web UI.
Today I also learned that running Samba on the PVE host is stupid. No real reasons but let's assume security. Well it's decently secured, has good passwords, and the killer is.. it isn't even fucking accessible to the internet! And even if it was, privilege separation is no secret to me.
But clearly I'm an idiot for even thinking about running Samba on PVE. Well guess what?! PVE is aimed at sysadmins that want to deploy a virtualization server. It's not a big stretch to imagine that those sysadmins might be halfway competent and want to run external services on the PVE host, is it.
But apparently it is. I'm an idiot and bugs aren't bugs anymore. Go fucking kill yourself, motherfuckers in the ##proxmox IRC channel. I really hope that your servers will go down on Friday when you're on call. Fucking cunts 😑
Edit: IRC chatlog @ https://clbin.com/nU9Fu13 -
So last week I really fucked up
I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.
So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.
Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"
FUUUUUUUUUUUUCK!!!
Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.
Needless to say, I had a fairly good weekend.
Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert1 -
So i just saw a post about not pushing to production on Friday's and that reminded me of something.
A team we are working with (we needed help on something so we got another company to send us a team and do it for us) recently push to production on a Friday at around 5pm.
We didn't know that they had done it until a shit ton of errors started happening and we had no idea why (it was working so far, none of us did anything, so what the fuck is going on).
To make things better, one of my colleagues tried calling them and they wouldn't answer the phone. They knew what they did, so they went home before we could notice it and we had to correct their mistake.
We had everyone calling us saying they had gotten an error, and they needed to get home but couldn't because of it (it messed up a process that only happens at the end of the day before people go home. I can explain more in the comments if anyone is confused, or just wants to know).
Eventually everything was solved, but that was a very stressful ending of the week.
So yeah, don't deploy on Friday's please and thank you7 -
A top food chain client wants a feature Fx
and has a deadline on Friday.
We are still working on it and already estimated hours and set deployment on Monday.
(No deployments on Friday)
And the business/sales guy comes up with new deadline to submit it at Friday morning.
And was only discussing with one of my team member already working on it. And i knew there is more hours required for testing and need to deployment pre deployment phase (staging of dev)
I was over hearing the conversation between them and I got pissed off and jumped in and said Not Possible at all.
He tries to argues about giving something to him. I said we can give it to you but will not garauntee anything. Now project manager jumps in. PM and my team already know that we will be delivering on Monday.
He arguing that if the Fx is not ready then I will call client developer to office to test it directly on my team members laptop.
I said, No way. We are not ready yet and havent finished yet. Major work will be on Thursday and on Friday we will be testing till end of the day.
PM explains him blah blah stuff.
He calms down and says no worries we will check the status on Friday afternoon amd roll out something to Client.
PM, developer and I looked each other and I said, sure will deploy but will not garauntee anything. He goes back to his desk.
Seriously.
WE ALREADY ESTIMATED F* MAN HOURS AND WILL BE READY ON MONDAY MEANS MONDAY DONT F* BUILD MORE PRESSURE ON US. F* SALES2 -
This project is just a complete clusterfuck... But nvm. We had to integrate a third party service pushing data into our system. Btw the service wasnt even working correctly. But that is just the tip of the iceberg. Its friday around lunch time. Message appears "what is the status of the integration?" Yeah havent started working on it. Last info was service is not stable. I doubt that this will be done this week. Next message from PO: "We will all push hard to get this done today and deploy to prod." Why? Because this dumbasses said to the customer this will be deployed eod. And by we you mean the devs once again doing overtime. Has this shit stopped? No. Like for the last two weeks its like we promised the customer xyz to be deployed tomorrow. Not a single dev was asked how long it takes to add this3