4+ years ago, in an interview, I was asked if I was familiar with keeping backups of my code on Google Drive.

When I asked them to explain what that is, they said that after a deployment, they make a ZIP file of the project and keep it on Google Drive.

When I asked about using GitHub/BitBucket they said they don't know what that is and neither do they intend on using it.

So yeah .....

PM (on slack): "we’re about to deploy to production".
Me: "ok"
… I keep on working on a task / remain available for any post deployment issues …
PM (5 minutes later on slack): "deployment broke production! We need to handle this NOW!"

My dev colleague has already called it a day, but I’m still online

Me: "ok I don’t have access to prod, can you describe what’s going on? I can’t reproduce on any other environment"

PM: …

10 minutes go by

Me: "anybody there?"

PM: …

45 minutes later, I realize PM is offline

The following day:
PM: "ok we got prod running again" (turns out it was client’s fault for not updating a config we as devs can’t access)
PM: "but we’re REALLY UPSET! You guys need to be available to intervene for any issues following deployment to production! At least one of you should be available!"

Me: "but, but…" 🫠

I'm a tech lead for a digital agency.

Digital agencies are universally known for being shite. Why? Because they typically push through sub-optimal code with very little testing over tiny deadlines for maximum profit. Maybe I've just had bad experiences but this is the 5th digital agency that I've worked at that does this bollocks.

I am currently sitting on a Teams call at 8:39pm because the fuckwit project/account managers are unable to face up to the big scary client and ask them terrifying questions like "Is this bug a blocker for the deployment?" or "We don't have enough time to fix/change these things, can we delay another day?". They just assume that A - We will work into the evening, and B - that all the issues are P1 and that we should all 'pull together' as 'team players' to get this done in time.

No, Me and my team have to work into the evening for seemingly free because these pricks can't do their jobs properly.

The funniest thing of all? When I speak to the CTO about overtime payment he tries to make me feel bad about "we don't typically pay for overtime..."

Fuck Everyone.

Time to find a new contract.

I knew this guy was pure evil.
Deployment on friday night !
And that too before Christmas.
Thank god Santa doesnt use Tesla

I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…

I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.

I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.

We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.

Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.

My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.

Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.

At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.

Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?

Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.

Amazingly, no one was fired.

So my department is "integrating CI/CD"

Right now, there's a very anti-automation culture in the deployment process, and out of our many applications, almost none have automated testing. And my groups is the only one that uses feature branching - one of the few groups that uses branching at all beyond "master, dev"

So yeah... You could see how this is already ENTIRELY fucked from the very beginning.

First thing they want to do is add better support for a process... Which goes directly against CI/CD.

The process is that to deploy to production (even after it is manually approved by manager), someone in another department needs to press a button to manually deploy. This, as far as I can tell, is for business rule reasons rather than technical ones.

They want us to improve that (the system will stay exactly the same with some streamlined options for said button pressers)

I'm absolutely astounded at the way our management wants to do something but goes in exactly the opposite direction. It's like the found an article of what CI/CD was and then took notes on exactly what not to do.

Half way through a 2 hour deployment, and a fucking test fails.
Yes, it takes 2+hours to run the tests.

Rerun in test environment: pass
Rerun in prod: pass
Rerun changed test in prod: fail..

Why, why you got to hate me for?
I love it when production is the place where config gets changed.

I should just quit. I am not paid enough to deal with this pissing contest.
Reviewer:
Need to add instructions (on readme) for installing pnmp, or if possible, have the top-level npm i install it (lol).
Also, it looks like we are no longer using lerna? If that's right, let's remove the dependency; its dependencies give some security audit messages at install.
Me:
it's good enough for now. Added a new ticket to resolve package manager confusions. (Migrate to pnpm workspaces)
Reviewer:
I will probably be responsible for automating deployment of this (I deployed the webapp on cloudflare pages and there is no work that needs to be done. "automating deployment" literally means replacing npm with pnpm). I disagree that it's good enough for now.

Imagine all readmes on github document how to install yarn/pnpm.

Lesson learned:
If you think an OOP static site developer can't handle modern JS framework, you are probably right.

I can't even deal with this. We just deployed a new update to our system, and everything was going smoothly. And then, out of nowhere, we started getting a bunch of error messages and user complaints.

Why do these things always happen? We spent hours trying to figure out the source of the problem, and it turns out it was because we didn't do enough testing before the deployment. Are you kidding me?

I know that testing can be time-consuming, but seriously, this is ridiculous. It's frustrating when something like this happens, especially when we're under a tight deadline. And to make matters worse, we had to roll back the deployment and start all over again. I just want to throw my computer out the window.

Uuuugghhh!

Buffer usage for simple file operation in python.

What the code "should" do, was using I think open or write a stream with a specific buffer size.

Buffer size should be specific, as it was a stream of a multiple gigabyte file over a direct interlink network connection.

Which should have speed things up tremendously, due to fewer syscalls and the machine having beefy resources for a large buffer.

So far the theory.

In practical, the devs made one very very very very very very very very stupid error.

They used dicts for configurations... With extremely bad naming.

configuration = {}
buffer_size = configuration.get("buffering", int(DEFAULT_BUFFERING))

You might immediately guess what has happened here.

DEFAULT_BUFFERING was set to true, evaluating to 1.

Yeah. Writing in 1 byte size chunks results in enormous speed deficiency, as the system is basically bombing itself with syscalls per nanoseconds.

Kinda obvious when you look at it in the raw pure form.

But I guess you can imagine how configuration actually looked....

Wild. Pretty wild. It was the main dict, hard coded, I think 200 entries plus and of course it looked like my toilet after having an spicy food evening and eating too much....

What's even worse is that none made the connection to the buffer size.

This simple and trivial thing entertained us for 2-3 weeks because *drumrolls please* none of the devs tested with large files.

So as usual there was the deployment and then "the sudden miraculous it works totally slow, must be admin / it fault" game.

At some time it landed then on my desk as pretty much everyone who had to deal with it was confused and angry, for understandable reasons (blame game).

It took me and the admin / devs then a few days to track it down, as we really started at the entirely wrong end of the problem, the network...

So much joy for such a stupid thing.

Central team: No, your team must be doing something wrong. Our pipeline is super-configurable and works for any situation! You just have to read the docs!

Me: Where are the docs?

Central team: Uhh, well, umm... we'll hook you up with a CI/CD coach!

Me: Okay, cool. In the mean time, can you point me at the repo where all the base scripts are?

Central team: Sure, it's here.

Me, some weeks later: Yeah, uhh, the coach can't seem to figure out how to make our Prod deployment work either.

Central team: That's impossible! It's so easy and completely configurable!

Me: Well, okay... but, here's the thing: your pipeline IS pretty "configurable", in the sense that you look for A LOT of variables...

Central team: See! We told you!

Me: ...none of which are actually documented, so they're just about useless to me...

Central team: But, but the coach...

Me: ...couldn't make heads or taisl of it either despite him literally being ON YOUR TEAM...

Central team: Then your project must just be architected wrong!

Me: Well, we're not perfect, so could be...

Central team: Right!

Me: ...but I think it's far more likely that the scripts... you know, the ACTUAL Python scripts the pipeline executes... while it took me DAYS to get through all your levels of abstraction and indirection and, well, BULLSHIT... it turns out they are incredibly NOT flexible. They do one thing, all the time, basically disregarding any flexibility in the pipeline. So, yeah, I'm thinking this is probably one of this "it's you, not me" deals.

Central team: Waaaaahhhhhhhh!!!!!!

I've been working as a programmer for 16 years now, and would say I'm not inexperienced, so it's frustrating to feel like a noob after months at a new company when they have poor internal documentation, hundreds of repos with default readme, pretty much no use of docker, sub standard equipment and use their own weird software for deployment. It's hard to meet expectations under these conditions.

i was hired to join a team of old devs (40+) in an unnamed European country "yay goodbye 3rd world it's time to enjoy the quality of life" assist with enhancing already existing software and creating new solutions.
prior to my arrival most things were slow and super buggy, looking at the code base it shouldn't be a surprise, amateur hour everyone, logic implemented that is not needed, comment driven development, last time code review was done back in 1996. lots of anti patterns.
i swear there is a for loop that does nothing but it loops through a 100+ elements list, trunk based development with tfs since git is "not really needed"
test projects are not there.
>enter me an educated fool, with genuine passion for the craft and somehow a decent amount of knowledge.
>spent the last year fixing stuff educating people on principles and qualities.
> countless hours of training and explaining. team is showing cooperation, a new requirement comes in to develop with react.
> tear my ass creating reusable shit and self explanatory code with proper naming etc using git with feature branching, monday is first deployment day.
> today a colleague was working on an item submit a pull request and self approve it
> look at the code..... WTF the dumb fuck copied and pasted the whole code from different kendo components but somehow managed to refractor the name to test component, commented out all the code that he didn't use did the api call directly from the component, has 2 useeffects that depends on the a fucking text box changes for no reason, no redux implementation, the acceptance criteria is not achieved, and it doesn't work it just look right.
> first world country shit cannot scold, cannot complain, lead by example.
>asked him why you did this, the response was yeah probably i shouldn't have done that, i really didn't understand anything in the training but didn't want to waste time!!!!
> rest of the team created a different styled disaster with different flavors they don't even name their shit the same way.

fellow developers I'm stuck in a spaceship with a bunch of imposters, seriously i never cried in my entire life now I'm teary and on the verge of a break down.
talk with management "improving needs time" and offers me to join a yoga session to release the stress as if reaching nirvana would deliver shit on monday.
i really don't know what do is this a rant, is this a cry for help, I'm not sure, any advice is welcomed.

Boss: How long will it take to finish the project?

Me: (Gives date for finishing dev and deployment.)

Boss: great, sound reasonable.

Me: ...

Boss: wait. Aren't you on vacation the two weeks after that.

Me: jup.

Boss: yeah we are not doing that again. Client can wait another two weeks.

Before someone says no Boss/PM is like that, he was/is a developer too so discussing deadlines and efforts is usually pretty relaxed since he knows our codebase and how long it takes to do things.

Love my new job but fuck they are way behind in any kind of modernization.

Just saw a demo over zoom where someone was showing the team how to change the margin on an error page.

They literally changed the HTML directly in prod using the VIM. So first of all no web modernation because there was no build, no deployment, not git, no pipeline - NOTHING!

This project went from 40 people to around 200 in 6 mos. You can't have all these people in prod just making fucking changes.

If this job did not pay 110k a year I would bail.

I really hate sales people. My stakeholder wants to buy an address verification service but is hesitant to purchase now because the dev time needed would be substantial. Now the sales rep has planted seeds of doubt in my SH and SH thinks I grossly overestimated the labor I quoted.

Sales rep is all “major corporations have installed this in a weekend.” 🤬🤬🤬 Major corporations also have more than one developer and probably aren’t dealing with a website that has a dozen address forms that all work differently. Oh, and I DON’T WORK WEEKENDS MOFO.

My SH originally requested a labor estimate for installing the AVS on all address forms and that’s what I delivered. My audit revealed a dozen different forms. I’m working with a legacy code base that’s been bandaged together and maintained by an outside dev agency. The only thing the forms have in common is reusable address fields. They all work differently when it comes to validating and submitting data to the server and they all submit to different api endpoints. At least a quarter of those forms are broken and would need to be fixed (these are mostly admin-facing). I also had to provide an estimate on frontend implementation when I have no idea what they want the FE to look like.

My estimate was 5-8 weeks for implementation AND testing. I wrote up my findings and clearly explained the labor required, why it was needed, and the time needed. All was fine until the sales rep tried to get into SH’s head.

My SH is now asking for a new estimate and hoping for 1-2 weeks of labor, which is what will SH to buy the AVS. Then go to the outside dev agency you used to work with and ask for a second opinion. I’m sure they’d also tell you at least month if not more for testing, implementation, and deployment because you have a DOZEN FORMS you want to add this to. 1-2 weeks is only possible for a single form.

My manager doesn’t work in the same coding language I do, but he read my documentation and supports my original estimate.

I honestly want to ask my SH if this sales rep is giving a very good price for the AVS. If not, are there other companies in the mix? Because right now you have a sales rep that’s taking you for a ride and trying to pressure you all so he can get another notch in his belt for getting another “major corporation” as his account. I don’t think it’s a good idea to be locked in with a grimy sales rep.

I hate web dev because I know enough to know how it "should" be done, know that it's not done that way, but also don't know enough to do it that way quickly.

Or why I spent the last 2 days scripting deployment of a website that costs 9$ a year to run, when I couldn't find a deployment script anywhere else.

Hate these managers.

I started working on a company 6 months back. When i joined i was told such amazing things about this place. I was given a job to develop a tool for a client. I did that alone. Now that the final deployment is done i am not needed there any more. I can start looking for outside work. WTF why would you hire a guy if you want him to work for only 3-4 months when you can hire a consultant or someone on contract or mention that in beginning. Fuck you even i want to leave this place now as soon as possible but still have to see their stupid faces for next 2 weeks.

Im deploying a Machine Learning Model to production. We dont have an automated deployment pipeline for the models, so we do it manually exposing the models through a rest api.

I asked for the model artifact to the DS, they didn't have permissions to download files from Databricks.

I asked their manager for the artifact. He told me that he has the permissions, then bullshitted me with something about the formal process, some shit about proper permissions handling, and that they do no have a standard process for sharing files right now so i should wait.

I was like "bro, share the artifact with me to unblock my work, then stablish your process, i dont care". He said no, and just after that he started a thread involving half of the middle management and data engineers asking for feedback on how to stablish a process for sharing databricks files. Just Wtf.

I got pissed, i reach out to his superior (good friend of mine), that was on vacation btw, and i told him the situation. He opened slack and humiliated him so bad, that i almost felt bad for the manager jajajajaja.

I grabbed my model artifact and got out of there instantly.

I got a contract with this schools to build a student portal,

I do all the needful and the project whatever guy insists that I use their current shared hosting to host this MERN stack application.

first of all, cPanel is my least favorite place when it comes to deploying, I actually dont do deploying I just hand it over to whoever is the IT guy there.

I discovered there's no provision for nodejs in their current plan, I go through all the stress of contacting the shitty customer support and the process of squeezing out useful information from them.

I'm only doing this because the project whatever has refused to pay me until their site is deployed. throughout the process of creating this project I had setup continous deployment on heroku and netlify and I had to beg this guy to look at the changes and review them.

well, today I asked the former guy that built the current site for the login details to the schools dashboard on the hosting providers site and he says he used his personal details for it, according to him projects from other organizations are there too.

I swear I'm going to loose my shit, freelancing sucks

So I started working at a large, multi billion dollar healthcare company here in the US, time for round 2,(previously I wasn't a dev or in IT at all). We have the shittiest codebase I have ever laid eyes on, and its all recent! It's like all these contractors only know the basics of programming(i'm talking intro to programming college level). You would think that they would start using test driven development by now, since every deployment they fix 1 thing and break 30 more. Then we have to wait 3 months for a new fix, and repeat the cycle, when the code is being used to process and pay healthcare claims.

Then some of my coworkers seem to have decided to treat me like I'm stupid, just because I can't understand a single fucking word what they're saying. I have hearing loss, and your mumbling and quiet tone on top of your think accent while you stop annunciated your words is quite fucking hard to understand. Now I know english isn't your first language and its difficult, I know, mine is Spanish. But for the love of god learn to speak the fuck up, and also learn to write actual SQL scripts and not be a fucking script kiddie you fucking amateur. The business is telling you your data is wrong because you're trying to find data that exists is complex and your simple select * from table where you='amateur with "10years" experience in SQL' ain't going to fucking cut it. Learn to solve problems and think analytically instead of copy fucking pasta.

How can so many devs have NO FUCKING CLUE what DNS is or how it works

FFS the amount of times ive had to tell devs to clear their dns cache when the browser tells them right in their fucking face a nxdomain error after deployment of a new test system and they just come screaming in confusion why its not working is entirely too high

When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.

Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.

When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.

Speaking of which, there's more than six microservices per developer!

Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?

When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:

1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.

Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).

Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.

When confronted/advised about these issues the response from the CTO/CEO
varies:

(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".

Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.

There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".

It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?

Never! Deploy! To production! On Thursday! *banging my head against the wall*
Now I need to revert some things manually on production ON MY DAY OFF 🤦‍♂️🤦‍♂️

Long time no see devRant. This rant is dedicated to an MQTT implementation we use. Mosquitto, mqtt.js - FUCK YOU.

I spent the last fucking 30+ hours trying to find why the bloody fuck the stupid server / client won't connect to the shitty mqtt broker. From changing all possible config, enabling & disabling specific code nothing abso-fucking-lutely works.

But then it will randomly decide to connect to the fucking broker, not causing any issues at all. And each fucking day when I wake up again and think to myself: oh today I can actually leave when it is still somewhat bright outside - NOPE. Because guess what? The fucking shitty abomination doesn't work anymore.

I just love these types of problems that are almost impossible to debug because the only logs you get is: "SERVER disconnected". It's impossible to get a proper reason out of this shit show, it's just turned into randomly guessing what the error could be (and especially where it could be).

And each time I got it to work, tested it and let the testing team know that they can start testing it will just stab me in the back and be like "fuck you, I'm not working any more". Luckily it's not like the deadline is next week... otherwise work is great, trust me.

Lessions I learned so far from my first big node/npm project with tons of users:
1) If you didn't build something for a while, expect 3 hours of resolving version conflicts for every two weeks since the last build.
2) Even if the tests pass, run the containers on your own machine and make sure that the app doesn't randomly crash before deploying
3) Even if the app seemed to work on your own machine, run the tests again in an environment mimicking prod at most 15 minutes before replacing the running containers.
4) Even if all else indicates that the app will work, only ever deploy if you expect to be available within the 4 hours following a deployment.
5) Don't use shrinkwrap for anything other than locking every version down completely. A partial shrinkwrap will produce bugs that are dependent on the exact hour you built the app _and_ the shrinkwrap file, and therefore no one will ever have seen them other than you.
6) Avoid gyp, and generally try not to interface too much with anything that doesn't run on node. If parts of your solution use very different toolchains, your problems will be approximately proportional to the amount of code. And you'd be surprised just how much code you're running. (otherwise it's more logarithmic because the more code the less likely a new assumption is unique)
7) Do not update webpack or its plugins or anything they might call unless you absolutely need to
8) Containers are cool but the alpine ones are pretty much useless if you have even just one gyp module.
9) There's always another cache. To save yourself a lot of pain, include the build time in every file or its name that the browser can download, and compare these to a fresh build while debugging to assert that the bug is still present in the code you're reading
+1) Although it may look like it, SQLite is far from a simple solution because the code and the bindings aren't maintained. In fact, it'll probably be more time consuming than using a proper database.

I did not think that making a serverless Discord bot would be such a learning experience. The code itself was easy. The hard part was the infrastructure, because I decided to automate it all with Terraform and deploy it on AWS.

Before this project, I had no idea how API Gateways worked. Now I still have very little idea how they work but I managed to build one anyway. Eventually. And then I had to figure out how to automate the deployment of a lambda layer and function that would both still be managed in the Terraform state, with any code changes triggering a rebuild and update for the resource.

And then I had to untangle a dependency mess because API Gateways have some weird issues where two resources that have no explicit dependencies on each other will throw an error if they don't deploy in the right order.

And then I went the wrong way with Github actions trying to conditionally chain multiple workflows together before I realized I could just put multiple jobs with conditions in a single workflow.

And now after all that work over the course of 2 days, I have a bot that does this:

A deployment that happens on a Sunday morning 😐

The universe has taken a cactus.

It proceeded to gift the cactus with a toxin that greatly enhances the stimulus of pain.

After the universe watched it's miraculous creation it decided to shove it up so far my arse that my gag reflex turned on and I puked a lot of cactus.

Didn't sleep well, weekend hardware migration finish, today an old server got moved.

Some part, most likely the redundant PSU, had a short circuit - decided to take the switches out... Which are the only non redundant hardware...

There was only one critical system in the whole rack, that was one redundant firewall.

Guess what happened..... Naaaa?

*drum roll*

For whatever reason, the second firewall didn't kick in, so large part of internal network unreachable as VPN was on the firewall.

:thumbsup:

That's not cactus level yet.

Spontaneously a large part of the work at home crew decided to call, cause getting an email wasn't enough.

So while all the phones were ringing and we had the joyful fun to carefully take apart a whole rack to check for possible faulty wiring / electric burns / hardware damage and getting firewall up and running again...

Some dev decided to run a deployment (doable as one of the few working at the company at the moment -.-).

I work from home, but we had a conference phone call running the whole time so I could "deescalate" and keep others up-to-date. So me on headphone with conference call, regular phone for calls, while typing mails / sms for de-escalation.

Now we're reaching cactus level, cause being tortured by being annoyed out of hell by all telephone ringing, the beeping of UPS (uninterruptible power supplies), the screaming of admins from the server room and the roaring of air coolers…

Suddenly said dev must have stood in the midst of the chaos… and asked for help cause "the deployment broke, project XY is offline"...

I think it was the first time since years that I screamed at the top of my lungs.

Bad idea (health issues)… but oh boy was it a pleasure to hear my own voice echo through the conference speaker and creating an echoic sound effect.

It was definitely worth coughing out my loungs for the next hour and I think it was the best emotional outburst ever.

I feel a bit sorry for the dev, but only a tiny bit.

After the whole rack thing, the broken deployment fixing and the "my ears are bleeding and I think I will never be able to talk again" action...

We had to roll out several emergency deployments to fix CVEs (eg libexpat).

This day was a marvelous shit show.

I will now cry myself to sleep with some codein.

Just launched a successful deployment today. Took 4 months to build everything. It's probably not a big deal but I'm just really happy today that everything worked right away, and almost no deployment hiccups at all.

Only one issue popped up, but come to find out it's a particular thing about the Prod environment and nothing to do with my code.

Gonna go celebrate now, before more work comes in. (hey, can you refactor this for me?)

we want you to be
- full stack developer (you do everything front end, back end)
- dev ops/SRE (you can sort out the deployment CI/CD pipeline, cloud platform services AWS/GC/Azure whatever)
- architect (you can design the software as well)
all in 1, you gotta be multiple roles/departments
good luck getting this experience on the job (hell in a startup is not for everybody and certainly not for me)

also why the fuck companies who aren’t startups ask for this idk

not sure if i missed any roles/competencies so far , don’t forget you need like >=3 years of experience possibly in every field for entry roles and more for anything higher than that

Posted in DevOps discussion board (teams channel):

“Program x isn’t behaving the same way that it does on production. Can you please take a look?”

..a little background: we have a deployment scheduled for today and this issue was found during regression testing.

The issue found is that when a file is clicked on it disappears from the screen, and then isn’t opened…

The file is not on prem, and doesn’t get uploaded to a server that our DevOps team owns…

So why on earth would this development team be asking DevOps to look into a bug that is most likely a code related issue? 😆

Is this a common occurrence for anyone else?
A Bug is found, and the first thought is that the code isn’t the issue?

Waiting 3 min for our deployment pipeline's "build step" to finish downloading all 1.08GB of source code because business won't prioritise tech led tickets (to use shallow cloning in this case) 😞

Feeling so apathetic rn 😝

DEBUG=False

1. Updated kubernetes shit image
2. For hours cant figure out why shits showing v1 instead of v2
3. Thought it was caching shit
4. Ran --no-cache command to fix this shit
5. Wasted half a day to debug this shit
6. Turns out in kubernetes deployment yaml there was imagePullPolicy set to IfNotPresent, instead of Always. The shit wasnt pulling the v2 because the image (v1) was already present. This shit blows my mind

!rant, but story

Update: I now work as a 1st lvl support

After 5 to 11 months I'll work in deployment and after that in development.

Unclogging a workflow that stretches > dozen porjects, from building to analysis of build (static / lint check etc.)… Deployment.

It's frustrating.

Touching one thing and everything falls apart.

Thus small changes, fixing the rest of workflow, testing ...

Repeat .

Going now since 1 1/2 weeks. Possibly another or two weeks more.

It is soooooooo boring tedious annoyingly frustrating slowmotion shit

-.-

Recruiter: We are looking for a full-stack expert. You have taken multiple apps from conception to deployment, and have experience and opinions on the best technologies to use and why. You should be comfortable implementing new features from scratch, making changes to existing features and writing complex migrations on production data.

Dev: lol

Finally finished the model deployment and it fuckin works!!!

belle bouteille viens ici que j’te caresse ~~~

Wtf is wrong with AWS CDK? I add one EC2 instance and it deletes all my queues and api resources, then fails to create my EC2, does some fucking rollback then attempts and fails at some fucking rollback clean up.
So it nuked my entire fucking stack because why?
Because FUCKING WHY JEFF. WHY IS YOUR ROCKET SHAPED LIKE A PENIS AND WHY IS YOUR SHITTY FUCKING DEPLOYMENT SYSTEM FUCKING WITH ME LIKE THIS.
I can’t do this anymore. I’ve been doing this for months now. I really don’t know whether to laugh or have a mental break down.

Complete Disaster Kunt. That’s what I’m calling this shit from now on. I just don’t get how it can fuck itself up so much

How do you do your CI/CD pipeline? Sorry if this is a dumb question. Just wondering how the tests and deployment usually runs. Is it on a per team basis? Is it the whole release getting deployed to Test many times per day? What happens if too many automated tests fail or there is not enough coverage, does it abort the deployment? If so, how can every team get delayed by every issue - is that actually a good policy?

My pipeline is very slow and requires a team of 12 people working in shifts to complete it. I’m not an expert but I know it does a lot of steps and never completes without manual intervention. I would like to help but I’m not sure how bad it is.

I had a pretty good year! I've gone from being a totally unknown passionate web dev to a respected full stack dev. This will be a bit lengthy rant...
Best:
- Got my first full time employment dev role at a company after being self-taught for 8+ years at the start of the year. Finally got someone to take the risk of hiring someone who's "untested" and only done small and odd jobs professionally. This kickstarted my career, super grateful for that!
- Started my own programming consulting company.
- Gained enough confidence to apply to other jobs, snatched a few consulting jobs, nailed the interviews even though I never practiced any leet code.
- Currently work as a 99% remote dev (only meet up in person during the initialization of some projects.) I never thought working remotely could actually work this well. I am able to stay productive and actually focus on the work instead of living up to the 9-5 standard. If I want to go for a walk to think I can do that, I can be as social and asocial as I want. I like to sleep in and work during the night with a cup of tea in the dark and it's not an issue! I really like the freedom and I feel like I've never been more productive.
- Ended up with very happy customers and now got a steady amount of jobs rolling in and contracts are being extended.
- I learned a lot, specialized in graph databases, no more db modelling hell. Loving it!
- Got a job where I can use my favorite tools and actually create something from scratch which includes a lot of different fields. I am really happy I can use all my skills and learn new things along the way, like data analysis, databricks, hadoop, data ingesting, centralised auth like promerium and centralised logging.
- I also learned how important softskills are, I've learned to understand my clients needs and how to both communicate both as a developer and an entrepeneur.
Worst:
- First job had a manager which just gave me the specifications solo project and didn't check in or meet me for 8 weeks with vague specifications. Turns out the manager was super biased on how to write code and wanted to micromanage every aspect while still being totally absent. They got mad that I had used AJAX for requests as that was a "waste of time".
- I learned the harsh reality of working as a contractor in the US from a foreign country. Worked on an "indefinite" contract, suddenly got a 2 day notification to sum up my work (not related to my performance) after being there for 7+ months.
- I really don't like the current industry standard when it comes to developing websites (I mostly work in node.js), I like working with static websites (with static website generators like what the Svelte.js driver) and use a REST API for dynamic content. When working on the backend there's a library for everything and I've wasted so many hours this year to fix bugs and create workarounds related to dependencies. You need to dive into a rabbit hole for every tool and do something which may work or break something later. I've had so many issues with CICD and deployment to the cloud. There's a library for everything but there's so many that it's impossible to learn about the edge cases of everything. Doesn't help that everything is abstracted away, which works 90% of the time but I use 15 times the time to debug things when a bug appears. I work against a black box which may or may not have an up to date documentation and it's so complex that it will require you to yell incantations from the F#$K
era and sacrifice a goat for it to work properly.
- Learned that a lot of companies call their complex services "microservices". Ah yes, the microservice with 20 endpoints which all do completely unrelated tasks?

We can say I'm so called senior in my company. Not exactly a senior because I have like 2.5 years of commercial experience, but everyone in the company has much less.

As I'm kind of technology enthusiast, I have a lot of personal experience with Linux, Docker, I'm basically quite familiar with a lot of stuff both in theoretical and practical manner. That's why often I'm helping my less experienced friends to solve some issue, help with docker, help with deployment, and I'm really happy to do so.

Recently I was helping to deploy something ASAP on environment no one is familiar with and it took me couple of hours to do so, as it required changes in a lot of places. Even our team lead was happy about it.

tl;dr

I've just got email from my boss, that I should focus less on helping others. Not even a "thanks, good job". After I literally helped this company to get their shit together

Situation - I am responsible for refactoring and performance improvements in a company with several teams. This means I gotta do static analysis on code, run compliance tools and make changes in code or in the deployment pipeline, make sure the cloud is configured properly etc.,

Here is the catch when it comes to working on a ticket- the Azure team does not give my team permissions to make the necessary changes in the cloud. The Azure team won't pick up the ticket and do it themselves either.

Instead, we take the ticket, read the docs, take a guess on what's right or wrong. Then proceed to inform the Azure team who then go on to make that change. It is very hit or miss and often the ticket comes back to us and we do the same process again. Sometimes I have to spin up resources on my personal Azure account to tinker with settings to see which knobs are there for making changes to a resource.

Either pick up a ticket and work on it yourself, or give us azure with sufficient rights for us to be able to make the change. This midway status is infuriating, super unproductive and painful for us. Is this common? I am so frustrated.

Is it possible to get through a work week without hearing about a surprise requirement when the deployment is supposed to be one week from now?

Bonus points: One week from now, deployment is delayed due to more whatever shit they forgot to take care of in the biz side

#Suphle Rant 9: a tsunami on authenticators

I was approaching the finish line, slowly but surely. I had a rare ecstatic day after finding a long forgotten netlify app where I'd linked docs deployment to the repository. I didn't realise it was weighing down on me, the thought of how to do that. I just corrected some deprecated settings and saw the 93% finished work online. Everything suddenly made me happier that day

With half an appendix chapter to go, I decided to review an important class I stole from my old company for clues when I need to illustrate something involved using a semblance of a real world example (in the appendix, not abstract foo-bar passable for the docs)

It turns out, I hadn't implemented a functionality for restricting access to resources to only verified accounts. It just hasn't been required in the scheme of things. No matter, should be a piece of cake. I create a new middleware and it's done before I get to 50 lines. Then I try to update the documentation but to my surprise, user verification status turns out to be a subset of authentication locking. Instead of duplicating bindings for both authentication and verification, dev might as well use one middleware that checks for both and throws exceptions where appropriate.

BUT!

These aspects of the framework aren't middleware, at all. Call it poor design but I didn't envisage a situation where the indicators (authentication, path based authorisation and a 3rd one I don't recall), would perform behaviour deviating from the default. They were directly connected to their handlers and executed after within the final middleware. So there's no way to replace that default authentication scheme with one that additionally checks for verification status.

Whew

You aren't going to believe this. It may seem like I'm not serious and will never finish. I shut my system down for that day, even unsure how those indicators now have to refactored to work as middleware, their binding and detachment, considering route collections are composed down a trie

I'm mysteriously stronger the following day, draw up designs, draft a bunch of notes, roll my sleeves, and the tsunami began. Was surprisingly able to get most of previous middleware tests passing again before bed, with the exception of reshuffled classes. So I guess we can be optimistic that those other indicators won't cause more suffering or take us additional days off course

Spent a couple of weeks on writing a cronjob which updates a certain value in the application config, and spend the last few months on testing it in different environments to make sure it does not fail in production. Ran the deployment script, and the damn cronjob fails because of ssl certificate on production. fuck me

So I’m tryina put together a resume and I see a lot of dev résumé talking about increased revenue from x to y or user base from z to w.

What I don’t understand is while yes the user base did increase from z to w it wasn’t just you that caused it. The entire company was working hard at it to get there so why are u claiming that it was you. And apparently recruiters love these kpis that u make out of your ass.

Should I give in and just put them in there anyway?

I worked in a startup so my job didn’t really have a definition of what to be told to do and do I had to deal with building front end with vue to figuring out how to automate our deployment flow and it’s really hard to quantify my performance like sure 3000 tickets solved but u don’t know what portion of them were full blown features what portion was just a one liner.

Sure make helm charts depend on other charts that are only accessible inside the production k8s cluster, who would want to have a local deployment of what they are working on anyway

Anybody had experience with Hashicorp Nomad for smaller deployments (personal in this case)? Thinking of using it after having done it too many times with complex Kubernetes and Docker... 😅 Also don't wanna run this thing bare bones as i have a whole bunch of services to deploy together.

hey, so i have recently started learning about node js and express based backend development.

can you suggest some good github repositories that showcase real life backend systems which i can use as inspiration to learn about the tech?

like for eg, i want to create a general case solution for authentication and profile management : a piece of db+api end points + models to :

- authenticate user : login/signup , session expire, o auth 2 based login/signup, multi account login, role based access, forgot password , reset password, otp login , etc

- authorise user : jwt token authentication, ip whitelisting, ssl pinning , cors, certificate based authentication , etc (

- manage user : update user profile, delete user, map services , subscriptions and transactions to user , dynamic meta properties ( which can be added/removed for a single user and not exactly part of main user profile) , etc

followed by deployment and the assoc concepts involved : deployment, clusters, load balancers, sharding ,... etc

----

these are all the buzzwords that i have heard that goes into consideration when designing a secure authentication system for a particular large scale website like linkedin or youtube. am not even sure how many of these concepts would require actual codelines and how many would require something else.

so wanted inspiration from open source content to learn about it in depth, replicate and create new better stuff if possible .

apart from that, other backend architectures like video/images storage system, or just some server for movie, social media, blog website etc would also help.

Just built out my first app using Cloudflare Workers, Typescript, and DurableObjects. Holy shit, this is nice stuff.

It's taken little to no time to build out:

* JSON API written in Typescript
* JWT verification against my OAuth backend (SAML support too)
* CI Automated Deployments including unit tests
* DurableObject support
* 3rd party HTTP calls + caching (built in to the framework!) to reduce network latency and hiccups.
* Cron-like tasks on each stored object so they can awaken the app on a schedule and update themselves as necessary
* Rapid deployment to new environments

The local testing with coordinated "miniflare" is dreamy too.