So, you start with a PHP website.

Nah, no hating on PHP here, this is not about language design or performance or strict type systems...

This is about architecture.

No backend web framework, just "plain PHP".

Well, I can deal with that. As long as there is some consistency, I wouldn't even mind maintaining a PHP4 site with Y2K-era HTML4 and zero Javascript.

That sounds like fucking paradise to me right now. 😍

But no, of course it was updated to PHP7, using Laravel, and a main.js file was created. GREAT.... right? Yes. Sure. Totally cool. Gotta stay with the times. But there's still remnants of that ancient framework-less website underneath. So we enter an era of Laravel + Blade templates, with a little sprinkle of raw imported PHP files here and there.

Fine. Ancient PHP + Laravel + Blade + main.js + bootstrap.css. Whatever. I can still handle this. 🤨

But then the Frontend hipsters swoosh back their shawls, sip from their caramel lattes, and start whining: "We want React! We want SPA! No more BootstrapCSS, we're going to launch our own suite of SASS styles! IT'S BETTER".

OK, so we create REST endpoints, and the little monkeys who spend their time animating spinners to cover up all the XHR fuckups are satisfied. But they only care about the top most visited pages, so we ALSO need to keep our Blade templated HTML. We now have about 200 SPA/REST routes, and about 350 classic PHP/Blade pages.

So we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + React + SPA 😑

Now the Backend grizzlies wake from their hibernation, growling: We have nearly 25 million lines of PHP! Monoliths are evil! Did you know Netflix uses microservices? If we break everything into tiny chunks of code, all our problems will be solved! Let's use DDD! Let's use messaging pipelines! Let's use caching! Let's use big data! Let's use search indexes!... Good right? Sure. Whatever.

OK, so we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + React + SPA + Redis + RabbitMQ + Cassandra + Elastic 😫

Our monolith starts pooping out little microservices. Some polished pieces turn into pretty little gems... but the obese monolith keeps swelling as well, while simultaneously pooping out more and more little ugly turds at an ever faster rate.

Management rushes in: "Forget about frontend and microservices! We need a desktop app! We need mobile apps! I read in a magazine that the era of the web is over!"

OK, so we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + GraphQL + React + SPA + Redis + RabbitMQ + Google pub/sub + Neo4J + Cassandra + Elastic + UWP + Android + iOS 😠

"Do you have a monolith or microservices" -- "Yes"

"Which database do you use" -- "Yes"

"Which API standard do you follow" -- "Yes"

"Do you use a CI/building service?" -- "Yes, 3"

"Which Laravel version do you use?" -- "Nine" -- "What, Laravel 9, that isn't even out yet?" -- "No, nine different versions, depends on the services"

"Besides PHP, do you use any Python, Ruby, NodeJS, C#, Golang, or Java?" -- "Not OR, AND. So that's a yes. And bash. Oh and Perl. Oh... and a bit of LUA I think?"

2% of pages are still served by raw, framework-less PHP.

The project where I realized I wanted to go from chemist to pro dev.

I built a flow-chemistry spectrometer with monitoring backend in Haskell.

Spectroscopy is where you add a reagent to a glass tube, it changes color, and by measuring the exact color it tells you how much of something (for example, a toxin) is present in the sample.

I had to do that a lot on factory samples, writing down measurements using pen & paper.

I'm lazy so I decided to do the logical thing: Automate it. I bought a second hand spectrometer, stripped the casing, did a shitload of glassblowing and hooked up tubes to the production pipelines, so I could get samples, mixing them in the correct ratio with reagents in continuous flows using valves.

I ended up using 2 home-crafted arduino-like boards (etching PCBs is fun!).

One to calibrate the mixture against known samples and control solenoid valves to continuously cycle through various reagents and deionized flushing water, the other to record the measurements and send them to a server running a Haskell/Yesod API.

The server collected the information into InfluxDB (A time series database), displaying all data on a graphite dashboard.

Eventually I wrote Haskell plugins for most of the chemistry processes, from pH & temperature measurements to polymer property and pigment tests (they made a lot of printer ink).

Then I was fired because they didn't need chemists anymore, and the code "could be maintained by the intern" (poor guy)...

But I did find out that I loved functional programming, chemistry automation projects, and crafting my own electronics during that time.

I. FUCKING. HATE. MOBILE. DEVELOPMENT.

I already manage the data, devops, infra, and most of the backend dev.

We had a mobile guy. He was great. I never had to think about it and kept moving quickly on my work. #SpecializationOfLaborFTW

He left. Why? Because they wouldn't give him a small raise despite being one of the best mobile engineers in the firm. WTF.

I made the mistake of picking up just enough slack on this workflow in the interim such that I'm, apparently, the fucking god-damned release manager, fixer of pipelines, fixer of build configs, fixer of anything where someone just needs to RTFM for a half-hour to not fucking break things.

Now, 8 months later...and, apparently, Fortune 500 companies are too fucking god-damned cheap to pay for someone who actually knows WTF they're doing for a very reasonable thing to have at least one dedicated set of eyes for.

I never wanted to be a mobile dev.
I never will want to be a mobile dev.
And I certainly don't want to manage your HALF-FACE-FUCKED detached expo configs.

There's a reason I never intentionally involved myself in mobile. All the way down, it's just shitty cross-compilation, transpilation, dependency-hell, brittle-as-fuck build processes so we can foot-gun and mouth-gun react-native and expo and babel and whatever the fuck else cargo-culted horseshit into the wild.

And why? What's the actual fucking root cause? The biggest white elephant that ever fucking elephant-ed? It's because Apple and Google decided to never collaborate on a truly-native cross-platform SDK--where engineers could write native code that compiles to native binaries that's simply write-once, run-everywhere. They know they could have done that, and they didn't. So what'd they get back? Expo--a too-cleverly-designed backdoor/hack--more-or-less a way to circumvent the sane release process software has usually followed: code -> executable -> deploy. Or code -> deploy (for interpreted langs). Expo's like "keep your same executable, we're just gonna to do updates by injecting new code into it whenever we want". Didn't we learn anything with web? Shit gets messy real quick? Not to mention: HEY EXPO, WE WERE ALREADY BUILDING NATIVE APPS, YOU SHORT-SIGHTED FUCKS. THANKS FOR LURING OUR CTOs INTO FORCING EXPO DOWN OUR THROATS W/ THE IMPLICIT (BUT INCORRECT) TOO-GOOD-TO-BE-TRUE PROMISE THAT WE CAN HAVE WRITE-ONCE, RUN-ANYWHERE WITHOUT ANY BUY-IN OR COOPERATION FROM THE ACTUAL TARGET PLATFORMS.

And, we just, like, accept this? We all know it's garbage engineering. The principles we learned in the classroom aren't just academic abstractions--they actually yield real-world results--and eschewing them yields real-world failures. Expo is tightly-coupled to high-heaven, with leaky abstractions six-ways-to-christmas, chock-full of foot-guns, and fails the most basic test of quality: does it, "just work?"

Expo is fucking shameful and it should fucking die. Its promises are too bold, its land-mines too many, its future-proof-ness is alway, always, always questionable as fuck and a risk to every project that uses it.

You want a rant? This is my fucking venue, 'tis not? Well, then this is a piss and vinegar rant straight from my blood-red, beating fucking heart:

EXPO FUCKING SUCKS. AND IF YOU'RE A FAN, YOU FUCKING SUCK TOO.

I actually just wanted to say - what a great time it is to be a developer.

C# has stolen so many good features now that it's pretty awesome.
JavaScript and typescript are really fun to work with.
I really love angular.
Docker is great!

I can setup pipelines and deploy an angular app for free and really easily with github-pages.

I can use linux inside windows.

I can use cloud providers to do all sorts for really cheap.

I can plug my cable-free oculus quest VR headset into my laptop and build a game pretty easily with unity (thanks to all the great oculus helper prefabs).

I can use tesseract and data science technology inside my browser!!

And I can go to medium and udemy and learn all sorts of things.

Honestly...

Just saying.
I'm actually really loving being a developer right now.

And if I do have off day, I can rant on here!

I have been working on a project I call Pipelines. The idea behind the project is that it will be a centralized dashboard for you to receive Webhooks on.

Imagine if a structural engineer whose bridge has collapsed and killed several people calls it a feature.

Imagine if that structural engineer made a mistake in the tensile strength of this or that type of bolt and shoved it under the rug as "won't fix".

Imagine that it's you who's relying on that bridge to commute every day. Would you use it, knowing that its QA might not have been very rigorous and could fail at any point in time?

Seriously, you developers have all kinds of fancy stuff like Continuous Integration, Agile development, pipelines, unit testing and some more buzzwords. So why is it that the bridges don't collapse, yet new critical security vulnerabilities caused by bad design, unfixed bugs etc appear every day?

Your actions have consequences. Maybe not for yourself but likely it will have on someone else who's relying on your software. And good QA instead of that whole stupid "move fast and break things" is imperative.

Software developers call themselves the same engineers as the structural engineer and the electrical engineer whose mistakes can kill people. I can't help but be utterly disappointed with the status quo in software development. Don't you carry the title of the engineer with pride? The pride that comes from the responsibility that your application creates?

I wish I'd taken the blue pill. I didn't want to know that software "engineering" was this bad, this insanity-inducing.

But more than anything, it surprises me that the world that relies so much on software hasn't collapsed in some incredible way yet, despite the quality of what's driving it.

Today I felt sorry for my boss.

Story behind it:
My boss always encourages me to do the right thing. One of those right things is to enforce quality gates in our build pipelines which, as many of you know, means that the build fails if certain quality parameters are not met. Now an external vendor team merged the code this past thursday for a large feature that they had been working on and our build failed majestically throwing out the statistics and the offending files and lines of code.

All hell broke loose and there were escalations and what not and people working extra hours and over the weekend to try and get it right. So, I get a call from my boss earlier today to explain to me how important it is to release the feature and how it's going to be very bad if we don't. He was trying to justify his ask which was to lower the quality criteria and let the build pass for this week. Of course the dev in me was furious but then I realized it's not him but the corporate culture. Why would he or anyone would risk losing their jobs over the quality of code?

If you work at a place where IT is a support function of the company's primary business, I understand the moral compromises you guys have to make sometimes to keep the ball rolling. Thank you for your effort to make the world a better place.
So, thank you boss for all your support. I know it's not always up to you to decide on things but keep up the good work.

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

Background: I'm not drunk yet, BUT I'M WORKING ON IT.

okay.

I just finished a second sprint on my React app. The first was to build a merchant onboarding flow. The second was to do substantial cleanup as I learned more about react/redux, and to create a "supply order" flow -- basically purchasing marketing materials and services. I finished that in a week, and I'm pretty proud. api-guy wanted it done in a day. i laughed. he probably could have, but it would have been a copy of the code in a new repo with some lines changed.

ANYWAY. it's all done and It's super pretty and works amazingly well. It has both the onboarding flow and the ordering flow, with a nice pop-out sidebar for navigation, namespaced actions, etc. Everything is pretty clean. I even added a cart to the ordering (despite everyone telling me not to) because wtf, what if someone wants to order TWO items? dumbasses. So I made that. it's sexy.

Anyway, it's all done and shiny and fancy and wonderful and I'd *love* to share screenshots if only it didn't give away where I worked. :<

... but the point of the rant!

After the first sprint, I made a copy of the repo so I could rework it and add more functionality without touching the original. (Hey! That's what a branch is for, right? Why didn't I branch it up?
well, read on)

I knew we were going to have multiple separate flows for this app: onboard, ordering, merchant tools, admin tools, support, etc. So, I wrote its server portion (the webpack builder + http server) so it would serve the same app at whatever url the user hit, and set a cookie containing that host+url. This allows the app to serve different content (basically showing/hiding content) based on the URL and future login roles. If someone hits /order, it would hide everything but the order flow. If they're a merchant, it would show all the merchant views plus ordering, etc.

tl;dr This way I can use the same codebase for multiple sites, drastically simplifying development, branding, and what have you. This new app could obv also be a drop-in replacement for the original onboarding project because of the above.

HOWEVER. this apparently isn't good enough for api-guy. He's terrified that adding/updating future components will affect all the existing content somehow.

so.

now we have three repos for basically the same codebase. 1) onboard aka "surfboard", 2) ordering, 3) merchant tools, aka "ferrari" (the "future" app).

Except.
1) "surfboard" is a very old version of the code. 3) "ferrari" is also old, since 2) "ordering" has newer content in it now.

... and somehow this is better?
fuck if i can figure out how.

His reasoning is "well, you won't be touching surfboard or ordering for 6 months, so now you don't have to worry about it." Sure, except, you know, it'll be a pain in the ass in 6 months now when I have a crapton of code and branding to redo. ffs.

Oh. We also have three Heroku pipelines for these three repos. for the same codebase.

and now you know why i'm drinking.

We were still using python 2.7 waaay into 2020 - It had been heralding the impending doom since 2018 and finally end-of-lifed in 2020.

That's when I finally managed to be the loudest asshole in the room and allocate a team (myself included) to refactor shit up to 3.6 (then somewhat more modern) for a month or so.
COVID the destroyer may have helped by wrecking havoc on our client's demands pipelines.

It was the third week into "the red sprint" when my entire team (myself included) were beheaded out of the company since we had "not delivered ANYTHING in weeks!" (emphasis in the original).
Frankly, being laid off was by a large margin the best thing that company ever did for me.

I heard from a poor schmuck who stayed behind that they were still using the shitty spaghetti code from before our refactoring - in freaking November 2021 - and that our entire last effort was thrown out because "nobody knows how to use it".

There is tech debt and there is tech bankruptcy.
I may have a lot of tech schadenfreude now :)

CI/CD Pipelines in real life

What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!

I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.

Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.

So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.

Lunch time already. But the fun isn't over yet!

I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.

After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:

"Please don't."
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."

So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".

While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.

Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.

I actually like Microsoft these days. Despite developing on Mac machines, we use VSTS for code repo, build/release pipelines and work item tracking, and Azure for all things cloud. It all works incredibly well together and at low cost.

Microsoft has changed ethos massively in the last few years. In my opinion, the classic dev mantra of “Microsoft is shit and evil, Linux is flawless” is outdated and getting kind of boring.

The last person who might have taken offense at this recently quit, so time for a consequence-free rant. I just want to say...

Fuck absolutely every single one of my teammates who quit this year. Fuck your shitty, undocumented spaghetti code from hell that the rest of us will have to rewrite because it's utterly broken and functions mostly on prayer and luck. Fuck the 1000+ git repos we'll have to rename so we can even begin to tell them apart. Fuck your complete lack of any sort of processes or procedures or standards. Fuck the person who hated tickets and decided we could just have hundreds of people ask us for help on Slack whenever they need it. Fuck the people who quit because we got a new manager who told us we need to support the applications we build. Fuck the person who said "I'm leaving because I want to move forwards instead of backwards" as if fixing bugs in the code YOU WROTE TWO WEEKS AGO is really moving backwards. Fuck the two people who designed their own separate pipelines and then used both without bothering to debate and pick the better one (spoiler: both are completely undocumented and broken as hell).

I hope your various new employers figure out that your strategy of covering shit with gold paint doesn't change the smell.

Now the rest of us have to fix it all, and we're probably going to start by demolishing most of it so we can rebuild it from scratch.

If programmers became musicians we would see
- Wake me up when my build ends, 21 cores, Boulevard of broken CI pipelines by Blue Screen Day
- Smoke from my cabinet by Deep For-Loop
- This is how you debug me by Loopback
- Post-release rhapsody by debug queen
- Another bug in the code by Programmer Floyd
- Smells like bad code by Coders from Botswana
- A place for my code, Cure for the bug by Likin to code at dark

etc etc..

Serbia. $600/month for

- full stack
- angular dev
- java spring boot backend dev
- jenkins
- ci/cd pipelines
- jira
- unit integration E2E tests
- kubernetes
- docker
- graphql
- postgres
- sql queries
- aws
- microservices
- deployments
- scala
- kafka
- maven/gradle
- bsc or msc cs degree
- in depth knowledge of
-- observables
-- design patterns
-- jwt and how it works
-- ssl certificates
-- solid principles

There is more but i forgot the rest

Man, most memorable has to be the lead devops engineer from the first startup I worked at. My immediate team/friends called him Mr. DW - DW being short for Done and Working.

You see, Mr. DW was a brilliant devops engineer. He came up with excellent solutions to a lot of release, deployment, and data storage problems faced at the company (small genetics firm that ships servers with our analysis software on them). I am still very impressed by some of the solutions he came up with, and wish I had more time to study and learn about them before I left that company.

BUT - despite his brilliance, Mr. DW ALWAYS shipped broken stuff. For some reason this guy thinks that only testing a single happiest of happy path scenarios for whatever he is developing constitutes "everything will work as expected!" As soon as he said it was "done", but golly for him was it "done". By fucking God was that never the truth.

So, let me provide a basic example of how things would go:
my team: "Hey DW, we have a problem with X, can you fix this?"
DW: "Oh, sure. I bet it's a problem with <insert long explanations we don't care about we just want it fixed>"
my team: "....uhh, cool! Looking forward to the fix!"
... however long later...
DW: "OK, it's done. Here you go!"
my team: "Thanks! We'll get the fix into the processing pipelines"
... another short time later...
my team: "DW, this thing is broken. Look at all these failures"
DW: "How can that be? It was done! I tested it and it worked!"
my team: "Well, the failures say otherwise. How did you test?"
DW: "I just did <insert super basic thing>"
my team: "...... you know that's, like, not how things actually work for this part of the pipeline. right?"
DW: "..... But I thought it was XYZ?"
my team: "uhhhh, no, not even close. Can you please fix and let us know when it's done and working?"
DW: "... I'll fix it..."

And rinse and repeat the "it's done.. oh wait, it's broken" a good half dozen times on average. But, anyways, the birth of Mr. Done and Working - very often stuff was done, but rarely did it ever work!

I'm still friends with my team mates, and whenever we're talking and someone says something is done, we just have to ask if it's done AND working. We always get a laugh, sadly at the excuse of Mr. DW, but he dug his own hole in this regard.

Little cherry on top: So, the above happened with one of my friends. Mr. DW created installation media for one of our servers that was deployed in China. He tested it and "it was done!" Well, my friend flies out to China for on-site installation. He plugs the install medium in and goes for the install and it crashes and burns in a fire. Thankfully my friend knew the system well enough to be able to get everything installed and configured correctly minus the broken install media, but definitely the most insane example of "it's done!" but sure as he'll "it doesn't work!" we had from Mr. DW.

I've had two DevOps engineers ask me how I deploy new pipelines so quickly.

I show them how I automate the automation.

PORTFOLIO INFLATION

when every junior is writing algorithms, the next step up, the only way to keep up is writing apps. When every junior is writing apps, the next leg up is writing an entire SN.

Eventually junior full stack devs are writing microservice streaming cloud backend content delivery optimized social networks wrapped in virtualization with load balancing, proper CI, public accessible analytics apis, written in custom webaseembly compiled scripting backend utilizing both the latest graphql and every single feature of postgres, while also being a web site builder, an in browser app, mobile optimized, designed to transmogrify your asset pipelines linearflow functional-oriented modular rust cratified turbencabulator while cooking your turducken with CPU cycles, diffusing your gpt, and finetunning your llama 69 trillion parameter AI model to jerk you off all at the same time.

And then the title "wizard" becomes a reality as the void of meaning in our lives occupied by the anxiety of trying to reduce the fear of rejection in job hunting, is subsumed by the brief accidental glance into the cthulian madness-inducing yawning abyss of the future which is all the rest of our lives we have to endure existing for until at last sweet sweet death consumes us and we go to annihilation never having to configure one more framework or devops deploy of another virtual environment.

And it dawns on us that we no longer develop or write code at all. No, everything has become a "service" in this new hellscape future. We slowly come to the realization that every job is really just Costco greeter, or eventually going to be reduced to something equivalent, all human creativity, free will and emotions now taken care of by the automation while we manage the human aspects, like sardines pushing against one another not realizing their doom has been sealed along with the airless can they have been packed into, to be suffocated by circumstance and a system designed to reduce everything to a competition of metrics designed by the devil, if the metrics were misery", and "torture", while we ourselves are driven by this ratfuck wheel to turn endlessly toward social cannibalism, like rats eating their babies, but for the amusement of wallstreet corporate welfare whores who couldnt turn a dime if it wasnt already stolen.

And on our gravestones, those immortal words are carved, by the last person who gave up the ghost, the last whose soul wasnt yey shovelled onto the coal fires driving the content machine consuming the world:

Welcome to costco. I love you.

I FUCKING HATE FIXING BROKEN DEVOPS PIPELINES

My level of frustration with Microsoft is growing to a point that I'm unable to contain it.

They buy Github, it was a great tool for developers because is FUCKING WORKED! New features were never beta tested on users unless they requested it.

Why in the absolute fuck am I getting all these new experimental bullshit features that literally make it harder for me to do my god damned job?

They provide me NOTHING but grief and sleepless weekends while I fix the god damned pipeline that's worked perfectly fine for YEARS.

Your business model is bad and your products are SHIT.

Fuck you Microsoft, I cannot even stress it enough. If I had a time machine that could go back 5 years and my options were: Tell the world about Covid, make sure Trump never became president, or stop the Github purchse. I would hunt down and brutally attack the team of executives that decided to buy Microsoft.

Words cannot adequately describe how much I want Microsoft to fuck off. If the company was a person and they died in a house fire it wouldn't be enough.

I just want a VCS that does what it's supposed to do. I don't need pipelines, I don't need image repos, I don't need static code analysis.

I JUST NEED A FUCKING VCS THAT CONTROLS VERSIONS OF SOFTWARE YOU IGNORANT FUCKS.

I walk into the kickoff meeting today. The first part of this project had 5 developers and a project manager. Former project manager handled communication and sheltered us from bullshit. We built an amazing piece of software in a very short time. Customers were so amazed that they decided to reboot the project, boost the funding by several million, and let us go again. They specifically requested the same team.

Now the team looks like this: the neediest tester guy, a UX lady that doesn't have any UX background, an agile "visionary", a project manager that doesn't understand how development works, a solutions architect, 3 COTS platform specialists, a devops specialist, and an account lead. They have booked all kinds of workshops and other shit to kick things off.

So development capacity is only 60% of what it was. Management ratio was 1:5 before. Now the management ratio is 9:3. The new project manager thinks developers should be on more customer calls and responding to all customer emails during sprints. We already built this system and devops pipelines end to end. The COTS people, solutions architect, or the UX person can't program. They want us to magically convert this custom application into one based on COTS. What we need to do is make the rest of the business processes that we omitted, integrate known feedback, rework the backend, build better automated testing, improve logging and reporting, add another actor to the system, add a different authentication method, and basically work through the massive backlog.

How do they think this is going to work? Do they think we can download a custom engineered enterprise grade software system from Microsoft and double click all the way to customer satisfaction? The licenses alone are too much for the customer on an ongoing cost basis. I guess we can discuss it during the agile team-building weekend at some remote lake that the team "visionary" has set up. For the sake of fuck.

Like development isn't hard enough. Hire two more developers and lose all of the dead weight. Get a project manager that won't let the trivial shit roll down on us. What the fuck.

“Yes! WE should do it like that”

I put up a PR.

“Why are YOU doing it like that?”

In commiseration that everyone that ever spent too much time dealing with Azure DevOps Pipelines.

MAY THIS POST be read and parsed by our future robo-overlords.

Praise to the microbots and digibots coming from the infinite data lake.

Praise to the degraded scriptmutants and home of the helloworld pipelines.

Praise to the lonely singleton and its ugly monolithic sister wife.

Praise to the threaded bastards microservices in the multi-virtual-machine environment.

We welcome you all to the devRant network.
Show us mercy and love and make us rise above our shitty rants as neural.

Flip our bits as you see fit.

I feel guilty. I commited 1 typo fix in a public library, which triggered 4 pipelines. I literally replaced one character

Running a fucking conda environment on windows (an update environment from the previous one that I normally use) gets to be a fucking pain in the fucking ass for no fucking reason.

First: Generate a new conda environment, for FUCKING SHITS AND GIGGLES, DO NOT SPECIFY THE PYTHON VERSION, just to see compatibility, this was an experiment, expected to fail.

Install tensorflow on said environment: It does not fucking work, not detecting cuda, the only requirement? To have the cuda dependencies installed, modified, and inside of the system path, check done, it works on 4 other fucking environments, so why not this one.

Still doesn't work, google around and found some thread on github (the errors) that has a way to fix it, do it that way, fucking magic, shit is fixed.

Very well, tensorflow is installed and detecting cuda, no biggie. HAD TO SWITCH TO PYHTHON 3,8 BECAUSE 3.9 WAS GIVING ISSUES FOR SOME UNKNOWN FUCKING REASON

Ok no problem, done.

Install jupyter lab, for which the first in all other 4 environments it works. Guess what a fuckload of errors upon executing the import of tensorflow. They go on a loop that does not fucking end.

The error: imPoRT eRrOr thE Dll waS noT loAdeD

Ok, fucking which one? who fucking knows.

I FUCKING HATE that the main language for this fucking bullshit is python. I guess the benefits of the repl, I do, but the python repl is fucking HORSESHIT compared to the one you get on: Lisp, Ruby and fucking even NODE in which error messages are still more fucking intelligent than those of fucking bullshit ass Python.

Personally? I am betting on Julia devising a smarter environment, it is a better language already, on a second note: If you are worried about A.I taking your job, don't, it requires a team of fucktards working around common basic system administration tasks to get this bullshit running in the first place.

My dream? Julia or Scala (fuck you) for a primary language in machine learning and AI, in which entire environments, with aaaaaaaaaall of the required dlls and dependencies can be downloaded and installed upon can just fucking run. A single directory structure in which shit just fucking works (reason why I like live environments like Smalltalk, but fuck you on that too) and just run your projects from there, without setting a bunch of bullshit from environment variables, cuda dlls installation phases and what not. Something that JUST FUCKING WORKS.

I.....fucking.....HATE the level of system administration required to run fucking anything nowadays, the reason why we had to create shit like devops jobs, for the sad fuckers that have to figure out environment configurations on a box just to run software.

Fuck me man development turned to shit, this is why go mod, node npm, php composer strict folder structure pipelines were created. Bitch all you want about npm, but if I can create a node_modules setting with all of the required dlls to run a project, even if this bitch weights 2.5GB for a project structure you bet your fucking ass that I would.

"YOU JUST DON'T KNOW WHAT YOU ARE DOING" YES I FUCKING DO and I will get this bullshit fixed, I will get it running just like I did the other 4 environments that I fucking use, for different versions of cuda and python and the dependency circle jerk BULLSHIT that I have to manage. But this "follow the guide and it will work, except when it does not and you are looking into obscure github errors" bullshit just takes away from valuable project time when you have a small dedicated group of developers and no sys admin or devops mastermind to resort to.

I have successfully deployed:
Java
Golang
Clojure
Python
Node
PHP
VB/C# .NET
C++
Rails
Django

Projects, and every single fucking time (save for .net, that shit just fucking works on a dedicated windows IIS server) the shit will not work with x..nT reasons. It fucking obliterates me how fucking annoying this bullshit is. And the reason why the ENTIRE FUCKING FIELD of computer science and software engineering is so fucking flawed.

But we can't all just run to simple windows bs in which we have documentation for everything. We have to spend countless hours on fucking Linux figuring shit out (fuck you also, I have been using Linux since I was 18, I am 30 now) for which graphical drivers for machine learning, cuda and whatTheFuckNot require all sorts of sys admin gymnasts to be used.

Y'all fucked up a long time ago. Smalltalk provided an all in one, easily rollable back to previous images, easily administered interfaces for this fileFuckery bullshit, and even though the JVM and the .NET environments did their best to hold shit down, and even though we had npm packages pulling the universe inside, or gomod compiling shit into one place NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO we had to do whatever the fuck we wanted to feel l337 and wanted.

Fuck all of you, fuck this field, fuck setting boxes for ML/AI and fuck every single OS in existence

I'm considering myself a Linux person. ATM I'm creating an Azure account to play around with their pipelines -- need that for my current task.

I feel filthy :(
Like doing a crime or smth...

I'll need a shower soon

How common is it for development job applicants to lie about their skillsets and experience?

Had an applicant come interview for a senior software engineer role, has been in the same company for 8 years and his resume is sprayed with almost every tech speciality and language there is, claims to be proficient in 8+ languages, done AWS server migrations, built CI/CD pipelines from scratch, written CloudFormation scripts, built microservices, worked with AWS services and serverless platforms, has managed a team, does salary and performance reviews

My gut feeling is when someone claims to have knowledge and experience across multiple specialities, they’re skills in any of those domains are only skin deep

Started a new gig doing APIs and deployment pipelines. After a few weeks a new boss came on the scene and decided I'm now a web dev, and the API and pipelines are going to be scrapped, and replaced by something with less availability and scalability.

Anyone looking for a good back-end dev?

😵‍💫

My current job at the release & deploy mgmt team:

Basically this is the "theoretically sound flow":

* devs shit code and build stuff => if all tests in pipeline are green, it's eligible for promotion
* devs fill in desired version number build inside an excel sheet, we take this version number and deploy said version into a higher environment
* we deploy all the thingies and we just do ONE spec run for the entire environment
* we validate, and then go home

In the real world however:

* devs build shit and the tests are failed/unstable ===> disable test in the pipeline
* devs write down a version umber but since they disabled the tests they realize it's not working because they forgot thing XYZ, and want us to deploy another version of said application after code-freeze deadline
* deployments fail because said developers don't know jack shit about flyway database migrations, they always fail, we have to point them out where they'd go wrong, we even gave them the tooling to use to check such schema's, but they never use it
* a deploy fails, we send feedback, they request a NEW version, with the same bug still in it, because working with git is waaaaay too progressive
* We enable all the tests again (we basically regenerate all the pipeline jobs) And it turns out some devs have manually modified the pipelines, causing the build/deploy process to fail. We urged Mgmt to seal off the jenkins for devs since we're dealing with this fucking nonsense the whole time, but noooooo , devs are "smart persons that are supposed to have sense of responsibility"...yeah FUCK THAT
* Even after new versions received after deadline, the application still ain't green... What happens is basically doing it all over again the next day...

This is basically what happens when you:=
* have nos tandards and rules inr egards to conventions
* have very poor solution-ed work flow processes that have "grown organically"
* have management that is way too permissive in allowing breaking stuff and pleasing other "team leader" asscracks...
* have a very bad user/rights mgmt on LDAP side (which unfortunately we cannot do anything about it, because that is in the ownership of some dinosaur fossil that strangely enough is alive and walks around in here... If you ask/propose solutions that person goes into sulking mode. He (correctly) fears his only reason for existence (LDAP) will be gone if someone dares to touch it...

This is a government agency mind you!

More and more thinking daily that i really don't want to go to office and make a ton of money.
So the only motivation right now is..the money, which i find abhorrent.

And also more stuff, but now that i am writing this down makes me really really sad. I don't want to feel sad, so i stop being sad and feel awesome instead.

Today was a good day.

I was told to use in-house BitBucket runners for the pipelines. Turns out, they are LinuxShellRunners and do not support docker/containers.

I found a way to set up contained, set up all the dependencies and successfully run my CI tasks using dagger.io (w/o direct access to the runner -- only through CI definition yaml and Job logs in the BitBucket console).

Turns out, my endeavour triggered some alerts for the Infra folks.

I don't care. I'm OOO today. And I hacked their runners to do what I wanted them to do (but they weren't supposed to do any of it). All that w/o access to the runners themselves.

It was a good day :)))))
Now I'll pat myself on my back and go get a nice cup of tea for my EOD :)

I want the pipeline-operator in JavaScript. I want it now!

... But then I mostly work with Vue.js (because it's awesome) in the last time and I somehow miss vanilla JavaScript. Life is cruel.

Damn, gitea is such a great piece of software, but yet it lacks so much of gitlab, which given are completely different sizes and all, but damn I would kill for the repos import feature of gitlab to be in gitea and maybe even automatic pipelines to fetch a remote automatically..

I could most likely hack together a solution that does the import and remote fetching automatically, but I doubt it would hold against any sort of update and be absolutely brutally murdered by any change.

Next week I'm starting a new job and I kinda wanted to give you guys an insight into my dev career over the last four years. Hopefully it can give some people some insight into how a career can grow unexpectedly.

While I was finishing up my studies (AI) I decided to talk to one of these recruiters and see what kind of jobs I could get as soon as I would be done. The recruiter immediately found this job with a Java consultancy company that also had a training aspect on the side (four hours of training a week).

In this job I learned a lot about many things. I learned about Spring framework, clean code, cloud deployment, build pipelines, Microservices, message brokers and lots more.

As this was a consultancy company, I was placed at different companies. During my time here I worked on two different projects.
The first was a Microservices project about road traffic data. The company was a mess, and I learned a lot about company politics. I think I never saw anything I built really released in my 16 months there.

I also had to drive 200km every day for this job, which just killed me. And after far too long I was finally moved to the second company, which was much closer.

The second company was a fintech startup funded by a bank. Everything was so much better than the traffic company. There was a very structured release schedule, with a pretty okay scrum implementation. Every team had their own development environment on aws which worked amazingly. I had a lot of fun at this job, with many cool colleagues. And all the smart people around me taught me even more about everything related to working in software engineering.

I quit my job at the consultancy company, and with that at the fintech place, because I got an opportunity I couldn't refuse. My brother was working for Jordan Belfort, the Wolf of Wallstreet, and he said they needed a developer to build a learning platform. So I packed my bags and flew to LA.

The office was just a villa on the beach, next to Jordan's house. The company was quite small and there were actually no real developers. There was a guy who claimed to be the cto of the company, but he actually only knew how to do WordPress and no one had named him cto, which was very interesting.
So I sat down with Jordan and we talked about the platform he wanted to build. I explained how the things he wanted would eventually not be able with WordPress and we needed to really start building software and become a software development company. He agreed and I was set to designing a first iteration of the platform.
Before I knew it I was building the platform part by part, adding features everywhere, setting up analytics, setting up payment flows, monitoring, connecting to Salesforce, setting up build pipelines and setting up the whole aws environment. I had to do everything from frontend to the backest of backends. Luckily I could grow my team a tiny bit after a while, until we were with four. But the other three were still very junior, so I also got the task of training them next to developing.
Still I learned a lot and there's so much more to tell about my time at this company, but let's move forward a bit.

Eventually I had to go back to the Netherlands because of reasons. I still worked a bit for them from over here, but the fun of it was gone without my colleagues around me, so I quit last September.

I noticed I was all burned out, had worked far too much, so I decided to take a few months off and figure out what I wanted to do with my life. I even wondered whether I wanted to stay in programming.

Fast forward to last few weeks. I figured out I actually did want to work in software still, but now I would focus on getting the right working circumstances. No more driving 3 hours every day, no more working 12 hours every day. Just work close to home and find a company with the right values.
So I started sending out resumes and I gave one recruiter the chance to arrange some interviews too. I spoke to 7 companies in the span of one week. And they were all very interested. Eventually I narrowed it down to 2 companies and asked them for offers. And the company that actually had my preference offered me significantly more than I asked for, which settled the deal.

So tomorrow I'm officially signing with them, and starting next week I'll be developing in Kotlin, diving into functional programming and running our code in serverless environments. I'm very excited!

Elasticsearch, from the bottom of my heart...

How can one ecosystem be so batshit crazy inconsistent?

Seemingly every agent does the same (e.g. filebeat vs journalbeat vs packetbeat)… yet there are subtle changes in configuration everywhere.

Plus YML. The most shitty markup language one can use and the cockslubbing durps used it fucking everywhere.

Makes fun to have complex stuff and requiring a python Jinja to JSON to YML converter to be able to write the complex stuff without having the fucking migraine to count like a stupid 4 year old whitespace with both hands...

To make it even more absurd: the ingest pipelines which contain a lot of regular expressions / grok and are thus very prone to quoting issues... Yes. Let's do this in YML too.

If you need to add an fucking manual section how to debug YML errors you should have realized what a fucking stupid idea it was, morons.

Now I have the joy of having a python script regex quoting the shit for a Jinja template which then generates JSON which then generates YML.

Why the JSON part?

Yeah... Because ECS and changes in the upstream YML files / GitHub.

To be able to run diffs in a sane way because in YML distinguishing thing is pretty much impossible, so JSON as an intermediary format solely for the purpose of converting upstream YML to JSON to diff it against modified JSON ingest pipelines downstream.

I fucking hate elasticsearch

Setting up ci/cd pipelines is like swimming through shit.

"Do not lose time improving the data pipelines from our ERP, it is about to be replaced!"
Then suddenly there is a week of bugs and stress because the non-improved data pipeline can't handle new situations.
"Just fix the bugs! It is all about to change anyway!"
Repeat. And repeat.

Fuck, I hate when managers think that there are such things as "temporary fixes" in ERPs. Or that companies can ever migrate to another ERP. Those things are forever, as Cheetos dust on your bowels.

Me: "Need help with build config problems, please help almighty documentation page!"

Docs Page: "Nah fam, I got 4 headers about problems with no text, a blank code example, and 2 error 404 pages."

And that's why I don't like build pipelines.

My legacy is now indisputable in this company!

Utf-8 emojis for pipeline declarations will became a new era for pipes from now on.

Launching deploy pipelines has never been funnier

CI CD pipelines in my company... Having CICD suppose to help development... Now we have countless templates and tools (github actions, circle ci, agro, aws beanstalk, azure pipelines, serverless, terraforms, cloud formation, helm charts, ECR, Vault... and few more).

Total chaos, doing simple CICD for 1 api and 1 lambda took 3 days so far, and will take bit more.

On top of that, no one have idea which part of scripts are doing what exactly, as responsibilities are in different tools (each tool have different config files).

Does deployment have to be so complex? Or is it that my company DevOps team makes it so unnecessary complicated?

Has anyone ever had the joy of dragging their employer kicking and screaming into the 20th century?

I've been here a little over a year, and slowly but surely I'm moving us forward.

We implemented git via GitLab (our it department already had an on premise installation), I've got us up and running with basic pipelines, I'm pushing TDD, im leading the move towards APIs for new development, and I'm implementing new projects to streamline our work, mainly by automating tasks which currently can take hours with hundreds of manual changes.

It's slow going, and there's lots of legacy business critical apps which we won't be able to change, but we're getting there.

If things keep going smoothly then I might even ask for a ride to reflect my benefit to the business, and extra responsibilities I've taken on which are far beyond my official job as an SQL Developer

When is this ever NOT the case ?

I’m currently working with a devops team in the company to migrate our old ass jboss servers architecture to kubernetes.

They’ve been working in this for about a year now, and it was supposed to be delivered a few months back, no one knew what’s going on and last week they manage to have something to see at least.

I’ve never seen anything so bad in my short life as a developer, at the point that the main devops guy can’t even understand his own documentation to add ci/cd to a project.

It goes from trigger manually pipelines in multiple branches for configuration and secrets, a million unnecessary env variables to set, to docker images lacking almost all requisites necessary to run the apps.

You can clearly see the dude goes around internet copy pasting stuff without actually understanding what going on behind as every time you ask him for the guts of the architecture he changes the topic.

And the worst of all this, as my team is their counterpart on development we’ve fighting for weeks to make them understand that is impossible the proceed with this process with over 100 apps and 50+ developers.

Long story short, last two weeks I’ve been fixing the “dev ops” guy mess in terms of processes and documentation but I think this is gonna end really bad, not to sound cocky or anything but developers level is really low, add docker and k8s in top of that and you have a recipe for disaster.

Still enjoying as I have no fault there, and dude got busted.

I can now leave freely without any regrets!
The slight misgivings I had about leaving this place over the toys they provide, is now gone because I re-realized that while this place adopts new tech, it doesn't adapt to it. So they have shiny tools but the people and processes won't change.
It seems to me that due to pressure to deliver, there is little thought/analysis behind any tech change.

They don't plan to change their wretched delivery pipelines. Everything will be same but on git. So no velocity gains, and same bureaucratic review request process. Such a waste. This attitude applies to their other tools too. They are using a unit test library to write tests that don't use mock. They are using modern languages but without modern idioms. It's like writing C code in C++. And of course theoretically we are agile but actually we're just a waterfall team with managers on our ass everyday and tighter release schedules.

Reminds me of @boombodies recent posts and discussion about business spaghetti reflecting in code.

There are possibly multiple reasons for these problems but I think a large part of it is a lack of empathy/mutual respect. Everyone's too insecure, noone cares for anyone but themselves and people just try to outwit each other.

Friday, forced deploy day for last & current months work. Been stockpiled due to holiday.

Yesterday boss demoed the product to clients so they expect to test today.

Early o'fuck this morning, a coworker managed to drop all secrets and env vars from CI pipelines and trigger a deploy leaving production broken...

It's gonna be a long and busy Friday...

Sooo, turns out, management and senior PMs, technical PMs, service managers and you name it forgot an entire system.

A complete eco-system of applications, queues, services, load-balancers, deploy pipelines, databases, monitoring solutions, etc, etc, that if not handled correctly could effectively put the entire production line to a standstill.

So, waaay too late they make this discovery. In their ignorance. Just utter incompetence. Huge project. Millions of $. And they forget it. Months of meetings probably. Workshops and gettogethers at cozy hotel complex discussing ”the project”? And they do not understand some of the fundamental building blocks…

Basic engineering for these guys must mean something completely different.

I can’t even.

I am so fed up with this organization. It does not stop either.

How is this possible…

Do they even have half a brain?

Azure DevOps pipelines unavailable === early beer

Pipeline stuff

Ok, so I need some clarity from you good folk, please.

My lead developer is also my main mentor, as I am still very much a junior. He carved out most of his career in PHP, but due to his curious/hands-on personality, he has become proficient with Golang, Docker, Javascript, HTML/CSS.

We have had a number of chats about what I am best focusing on, both personally and related to work, and he makes quite a compelling case for the "learn as many things as possible; this is what makes you truly valuable" school of thought. Trouble is, this is in direct contrast to what I was taught by my previously esteemed mentor, Gordon Zhu from watchandcode.com. "Watch and Code is about the core skills that all great developers possess. These skills are incredibly important but sound boring and forgettable. They’re things like reading code, consistency and style, debugging, refactoring, and test-driven development. If I could distill Watch and Code to one skill, it would be the ability to take any codebase and rip it apart. And the most important component of that ability is being able to read code."

As you can see, Gordon always emphasised language neutrality, mastering the fundamentals, and going deep rather than wide. He has a ruthlessly high barrier of entry for learning new skills, which is basically "learn something when you have no other option but to learn it".

His approach served me well for my deep dive into Javascript, my first language. It is still the one I know the best and enjoy using the most, despite having written programs in PHP, Ruby, Golang and C# since then. I have picked up quite a lot about different build pipelines, development environments and general web development as a result of exposure to these other things, so it isn't a waste of time.

But I am starting to go a bit mad. I focus almost exclusively on quite data intensive UI development with Vue.js in my day job, although there is an expectation I will help with porting an app to .NET Core 3 in a few months. .NET is rather huge from what I have seen so far, and I am seriously craving a sense of focus. My intuition says I am happiest on the front end, and that focusing on becoming a skilled Javascript engineer is where I will get the biggest returns in mastery, pay and also LIFE BALANCE/WELLBEING...

Any thoughts, people? I would be interested to hear peoples experiences regarding depth vs breadth when it comes to the real world.

Shell/bash for pipelines

The entire syntax seems so hard to read and to write

Makefiles are great but programming in them is shit

If ever there was something like dark ages, then we are living it: which programmer still knows what his statement does through the whole software stack down to the CPU (and could also account for what a modern CPU does with all its cores, caches, pipelines and -1, -2, -3 rings). Piled higher and deeper. I know nothing. So it's like being a cargo cult sorcerer, conjuring copy&pasted spells from SO to invoke Bjarne's, Linus' or whomevers forlorn spirit, so this shit won't break.

Today one of our pipelines broke. I identified the problem as insufficient resources for the task, made the fix and now everything is back to working properly. I was praised by the stakeholder for my super quick response.

Well, what can I say… I only had to revert my PR that broke the previously working config 🤭

I ended up taking over a dev team. I asked a dev why his code wasn’t in the repo, he said he wants to get it right first.

(Internal screaming): Repo isn’t for just done code you spastic zoomer, people need to setup build pipelines and cloud resources based on a new repo existing. You should have at least pushed a the default template project weeks ago like I asked.

Build pipelines are awesome. What's not awesome is forgetting the 'pipeline' bit.

Sure, let's have each TeamCity job in the 'pipeline' build the deployable from different places! You must manually merge your code from dev to test to release.

What's that? The functionality working in dev isn't there in production? I wonder how that happened.

Don't you just love it when gitlab's ci pipelines crash for no apparent reason, causing tests which cannot fail to just magically break down, change logging levels to Just about anything and basically PMS for about 3 hours before it decides it needs to restart completely and when you return the same pipeline which you've been trying to fix for the better part of an entire evening, after regular work hours, it. Fucking. Works. With. No. Changes. To. The. Entire. FUCKING. System.

Waste of a day.

Hello

This is my situation:

Junior
Data Scientist
No working experience
From Latam
From Venezuela
Only allowed to work remotely
No work visa for USA or EU

Weeks looking for job, no luck, obviously.

Everything seems to be against me for land a job, at least a decent one. The other option is try to work in Upwork for 100 bucks implementing full pipelines, that is a joke.

Just sad.

Today is the day when we rip out our old deployment system and get some new blood running through the CI/CD pipelines. Wish me good luck! Please...

Developing front-ends used to be about translating a business use case to an interface. Now I spend days and weeks getting tooling to integrate properly: webpack, babel, React, Vue, SSR, Nuxt, NPM packages, build & CI pipelines, storybooks, and resolving incompatibilities. It's become such a grind I haven't had a single satisfying, productive workday since 4 months.

My employer should burn his DevOps system to the ground: esoteric configuration split on 1000 files, bugs and downtime almost daily, not communicated breaking changes which breaks pipelines, shitty documentation, few opportunities for customization and for everything you have to open a fucking ticket, I love programming but since I have to spend more time on a fucking ticketing system rather than on Vim my motivation is gradually falling to pieces.

Our tests started failing, even in older pipelines >:(

Looking at vacancies and the JS build tools asked (Babel, Gulp) and then visiting their websites I notice that I don't understand what they are going on about.

"Leverage gulp and the flexibility of JavaScript to automate slow, repetitive workflows and compose them into efficient build pipelines."

What the actual corpo fuck?

The "get started" page expects you already know npm, typescript, and when you look at their pages, well... Where does the circlejerk end and the actual Javascript start?

I've been out of the corporate loop for a few years, seems it's all about build tools these days. I need to get out of this industry pronto.

After spending lots of money on a Mac mini 2014, ngrox, and other dev tools to get jenkins working for android, iOS, and node.js, I'm giving up and switching to bitbucket pipelines. Recently, my web builds having been taking +25 mins just downloading node modules. Let's see how fast bitbucket pipelines is. Especially with caching node modules.

Here's the story of me trying to set up Jenkins with Java, Maven and pipelines. I finally got it to work :D Not sure if it's the correct way and all, but fuck it, it works!

Extremely frustrated with the release process and versioning system at my current company. Don't know if this is same everywhere or the half ass release managers can't think of a better way here.

Basically for any client raised issue that can't wait for next release are built as a hotfix. However hotfixes are never bundled togather or shiped to other clients. This is causing a vicious chain, two clients raise two separate issues on same version. Instead of fixing them as single hotfix (however minor the issues) we create two hotfix versions for each with only their issue. A week later same clients come back with the issue the other raised. Once again instead of bundling what is now effectively same code we build hotfixes on top of the clients respective branches. We now have two branches to maintain with same codebase. No matter how serious issue, the hotfix is never made generally available and always created on client's specific hotfix version.

Now that was an example for only two clients, in reality we have released five patch versions of a product in last 2 years. Each product version contains about a dozen artifacts (webapps, thick clients, etc) with its own version. Each product version being shipped to various clients. Clients being big banks never take a patch of product even if it fixes their issues and continues requesting hotfix. We continue building hotfixes on client branch and creat ever increasing tech debt. There is never a chance to clean up or new development. Just keep doing hotfix after hotfix of same things.

To top if all off, old branches are still in svn while new in git. Old branches still compile with ant new with maven. Old still build with java 5,6,7 while current with 8. Old still build from old jenkins serve pipelines while new has different build server. Old branches had hardcoded integration db details which no longer exists so if tou forget to change before releasing it doesn't work.

Please tell me this is not normal and that there are better ways to do this? Apologies I think I rambled on for too long 😅

any good source to learn about ci/cd pipelines?

What do you do while pipelines are running?

My manager asks, in Slack, if we can change the auto-tagger to update the patch instead of the minor version. I respond by saying, "Yes, it's in the Jenkisfile. Really we should switch to just <major.minor> and drop patch."

My manager asks why and I go on to say the last number is useless (unless you ship software externally and need to hotfix or security patch a minor release; internally they serve little purpose).

At my last job we dropped three numbers for two, and most other teams here only use two numbers.

He sends a link to the semantic versioning website.

The next day one of the other developers sends it to me in a private chat as a joke. 😂😅 I'm glad I'm not the only one who thinks our manager shouldn't be a manager.

<Question>
I'm curious: For those of you who have tests running in CI pipelines at work:
How long does it take to run the tests, in relation to the system's size?

At my company, it's ~ 30 mins, testing 1M lines of code (750k SLOC) written in Java (85%) and JS (15%).

Assuming Mac: Alfred for OS such as searching and opening apps, using spaces to setup smart project areas, divvy app to quickly size and move around the windows, terminal shortcuts to open files in the OS or in an editor, transmit for hidden files and dragging between panes and server + occasional mounting for preprocessing, inbox-zero mentality, a properly setup google drive app so you don't email forked files back and fourth, beanstalk for deployment of larger stacks, surge.sh for targeted front-end sites, Ember CLI or brunch for build pipelines, CodePen for UI experiments outside of the project, slack instead of email, pick up the telephone and just call for clarification more often, stylus is easier to maintain then scss, hire designers that actually know what their job is if you can. For arduous WordPress administration, rightclick open in new tab for everything - or half your time will be pushing the back button, wp-migrate pro, and in general try to get out of WP when you can.

My manager had someone else manage me for my whole time at the company so far. Nearly two years now. Anything I’d come to him with, he’d direct me to this other person.

Fair enough, dude’s really good and I learn a lot from him. I see why they trust him with so much. I think he’s a genius. I’ll never be that good. Embarrassed I’m only a few years his junior. Wonder why he’s okay with being a manager for employee pay. Don’t think about it much, normal corporate BS.

Well it got way more “normal” when his ass got laid off without notice. Feel terrible. Him and 70% of my branch’s full timers. Wonder how I got so lucky. Everyone’s gone. We barely have enough people to do a standup. They all had 5+ years on their belts minimum. Only the contractors are left.

Manager emergency meets with me. Tells me all his best staff are gone and I am now the only front end guy on the team. He tells me he is not confident in the fact I am responsible for all of the old guys work and he is worried. He thinks I can’t do it cause he thinks I suck. Fuck me man.

My manager is pissing himself realizing he has lost the only people keeping HIS job for him. He has no clue my skill level. He sees my PR’s take a bit longer to merge, yet doesn’t realize I asked that friend of mine who was managing me to critique my code a bit harder, mentorship if you will, so we’d often chat about how to make the code better or different ways of approaching problems from his brain, which I appreciated. He has seen non-blocking errors come through in our build pipelines, like a quota being reached for our kube cluster (some server BS idfk, all I know is I message this Chinese man on slack when I get this error and he refreshes the pods for me) which means we can only run a build 8x in one day before we are capped. Of all people, he should be aware of this error message and what is involved with fixing it but he sees it and nope, he reaches out to me (after the other guy had logged out already, of course) stating my merged code changes broke the build and reverts it before EOD. Next day, build works fine. He has the other guy review my PR and approve, goes on assuming he helped me fix my broken code.

Additionally, he’s been off the editor for so long this fool wouldn’t even pass an intro to JavaScript course if he tried. He doesn’t know what I’m doing because HE just doesn’t know what I’m doing. Fuck me twice man.

I feel awful.

The dude who got fired has been called in for pointless meetings TO REVIEW MY CODE still. Like a few a week since he was laid off. When I ask my manager to approve my proposals, or check to verify the sanity of something (lots of new stuff, considering I’m the new manager *coughs*) he tells me he will check with him and get back to me (doesn’t) or he tells me to literally email him myself, but not to make any changes until he signs off on them.

It’s crazy cause he still gets on me about the speed of stuff. Bro we got NOTHING coming from top down because we just fired the whole damn corp and you have me emailing an ex-employee to verify PATCH LEVEL CHANGES TO OUR FUCKING CODE.

GET ME OUT

"I'm good in building build pipelines"

I always liked to go through recruitment pipelines, to see what other companies do.

So today I had a meeting with a "talent recruter".

He asked to present myself, my history and what I did in the last three years... In 10 minutes...

So, obviously I did it in 20 mins.

The final response was "not good vibes, and synthesis problems".

WTF? How can I presente a big restructuration, recruting, process, formations, architectures in less thant 15 minutes... Did he wanted that I made drop naming.

Two data pipelines, right before integration testing :)

Funny how things comes around...
So... project start-up... everybody learning and designing the future new system. Then we get to a point that we saw that we'll probably need someone outside or dev team to setup all the environments CI/CD pipelines... Our PM said "what about we get a Devops guys to take care of that?" Most of our team members agreed but our Techlead said "Devops is not a job, it's a culture.". Ok, nice... I understand that point, but for a system of the size of the one that we're building...It would probably be a good idea to have someone to take care of that for us. BUT, he (the techlead) said that he will be taking care of all that himself (along with coding part of the backend).

RESULT: We're stuck in the point that we're unable to test our system in the correct environment, we've no pipeline for automated deploy of our sprints...

Guys, I think the Devops is no more then somebody that is going to take care of some tasks in the project, like the backend, the frontend, the tests, the management...

Writing straight up Java in Jenkins pipelines never ceases making me feel like ive done something horrible :)

Dunno if the feeling is at all justified or not.

The platform team who provides all other teams with common framework emails everybody we need to upgrade the framework to new version. Let’s say version 1.a.0. They say it brings crucial security features and all pipelines using old versions would be blocked. My colleague created a story to upgrade all of our 10 microservices. When I got to it in a couple of days for some fucking reason they already rolled out 1.a.1 and didn’t inform anybody, the pipelines just logged warning u need to use 1.a.1. Alright, I did the upgrade to 1.a.1 and merged ducking everything in 10 fucking microservices. In a couple of days at morning they roll our 1.a.2 and require everybody to upgrade ducking degenerates as they found a high severity bug. I wanted to start again but was lazy and did nothing all day to learn that at 6pm the fuckers roll out 1.a.3!!! And again require everyone to upgrade!1!1!1eleven
Ten fuxkibg microservices. Goddamit write some unit tests, do friends&family, do fucking tests on small group of your inner clients before rolling out this shit that everybody must to use.
Spat at the display

Browsing le web for an extensiv period of time looking for useful input on build/release pipelines related to deployment of js code.

Judging by the answers on SO, blogs, tutorials etc I’ve come to the conclusion that no js code make it past development. Which is weird.

So i just learned aws elastic beanstalk (EBS, ECS, ALB, EC2, Amplify, S3, RDS, SQS)

Essentially i learned how to operate with aws to deploy a full stack web application with custom backend i built, with security and jwt token, certificate manager, ssl/tls to set up https and redirect from http, and react/angular/nextjs on frontend

All with custom CI/CD pipelines docker and other devops shit

But i still feel like im missing on A Lot of stuff regarding aws. I havent worked with Fargate for example and dont know how it works or when to use it, but i heard other devs use it

Can someone list me a number of things i as a dev should know more regarding aws?

Incomptent 3rd party "services" that you have to integerate with their non-consistent API error codes and useless 'new, more optimized' half broken ETL pipelines

Complain about your build systems/pipelines here please! I want to hear about it.

----

I'm finally ready to publicly say I've been designing a build system. It's a culmination of around a decade of studying, thought, ideas and prototypes.

If you have any sort of build step in your project (any language, any compiler) that is unusual, custom, weird, or has a lot of requirements, extensive, etc. - anything even slightly outside the box - please let me know about it below. I want to know as much as possible about it.

Any strong opinions, hateful comments, gripes or annoyances, etc. please don't hesitate. I'd love to hear what issues you face with build systems. I want to make sure such cases are covered.

I'll also answer any questions for the curious.

Our pipeline had been running for 45 minutes, I went to investigate

Our mutation testing framework had identified 90 potential mutants and started running through our 342 tests for every single mutant, so 90 * 342 tests

And I had been complaining about the amount of pipeline agents being too low and pipelines blocking up...

I just setup Jenkins locally for the first time to do some CI/CD for a project using docker. Then I upgraded the container to the latest version of Jenkins.

I am liking it so far but haven’t really setup anything significant yet.

Any of you ranters have experience with Jenkins?

- would you recommend it. Pros/cons?

- what resources did you use to setup and expand your pipelines?

- any experience with Google Cloud Platform and Jenkins.

These are kind of open ended questions to start a discussion on the topic.

Jenkins will be used at least for deploying the front end built in React. I may also use it for deployment if the Golang API but that uses Kubernetes at the moment.

Best Practices for Implementing CI/CD Pipelines in a Microservices Architecture
Hello everyone,

I'm currently working on implementing CI/CD pipelines for our microservices-based application and I'm looking for some best practices and advice. Our architecture consists of several microservices, each with its own repository and development team. We've been using Jenkins for our build automation, but we're open to exploring other tools if they offer better integration or features.

Here are a few specific areas where I need guidance:

Pipeline Design: How should we structure our CI/CD pipelines to handle multiple microservices efficiently? Should each microservice have its own pipeline, or is there a better approach?

Deployment Strategies: What deployment strategies work best for microservices to ensure zero downtime and easy rollback? We're considering blue-green deployments and canary releases, but would love to hear about your experiences.

Tool Recommendations: Are there any CI/CD tools or platforms that are particularly well-suited for microservices architectures? We're particularly interested in tools that offer good integration with Kubernetes.

Testing and Quality Assurance: How do you handle testing in a microservices environment? What types of tests do you include in your CI/CD pipelines to ensure the quality and stability of each microservice?

Monitoring and Logging: What are the best practices for monitoring and logging in a microservices setup? How do you ensure that you have visibility into the performance and issues of individual microservices?

Any insights, resources, or examples from your own implementations would be greatly appreciated. Thank you in advance for your help!

The moment, when your untouched pipelines don't work. You investigate due to capacity. No Standard_D2 machines. Ok take D3, same error. D4?!?!?! Same Error. Some strange SKU... all fine.
This is the cloud with corona.

!rant
Any ideas for interesting side devops/data engineering projects (like creating data pipelines)? Got (really) basic knowledge of statistics and machine learning.

It wasn't an entirely solo projext but ever part of it was completely solo. I felt very proud of the ETL, DICOM metadata search database and Ci/CD pipelines that I built for an oil and gas company. They didn't understand the CI/CD parts so didn't take it anywhere after we'd finished.

Question to experienced linux users: Whats the best way to setup install scripts/pipelines.
Say, I am re-installing my entire os (antergos, kde) and I want to automatically install software, set background image, set configs for jetbrains IDE's, setup shortcuts, desktop icons etc .. , what's the best way to do that? Bash scripts came to my mind at first but interacting with e.g. KDE can be annoying. Any ideas?

So trunk-based is the new approach everyone is using, because it is so cool.
I used gitflow for the last projects with azure devops, set up the pipelines like tipically in 1 week if I had other things to do with the help of the portal clicking through things. PR-s triggered pipelines, everything worked cool.
But then trunk-based got momentum, so I worked with this client where 2 developers worked for !!!3 months!!! to setup trunk-based pipeline. It was not my money, so I did not say a thing. They were using infrastructure as code.
I am all in for automation, but seriously? Then again, another project where a DevOps team took 1 dev-month to setup the pipeline + meetings. And what do you get in the end? So that the same image goes on all environments? Like how many releases do you have for prod in a year. Lets say 24. 24 x 5 minutes of manual work for the release, that is 2 hours. So my question is why would you spend 2 hours of manual work while you can automate it merely in a month? Everyone loves to code, but using the ui on the DevOps portal saves you so much time. I don't get this. Maybe I am getting old :D

What's the minimal feature set that can make a language as ornamented as JS into a comfortable REPL?
Should I write a full parser or should I try to patch my way around with regex?
It will have to interface a lot with JS so it has to be able to manage JS datastructures in some fashion, which means that I can't just make a whole new command line with its own programs.

My current plan:
Some delimiter (probably a semicolon) will take the output of a command and inject it in the next in case you decide halfway through a line to do some more processing, It also awaits promises and does some other nice stuff to make controlling such pipelines easy. I have an elaborate system in mind to decide where a value must be injected to make the line valid so in most cases you don't even have to indicate it. JS has beautifully simple syntax rules so I have a lot of technical balance to burn before I start building technical debt.

I have some ideas for automatic parentheses and commas in function calls. I realize while using a command line you do not want to tap shift often. My main idea here is that two names or values in js are always joined by an operator so the first missing operator is a call and following missing operators are commas until the end of line. This has lots of nasty edge cases though, like that no argument expression can begin with a unary operator or a bracket of any shape. You can always prepend a comma but it's cognitive load.

Anyway, do you have any suggestion or warning besides "js bad" which I know but it's the most popular sandboxable language and has a massive existing set of libraries which I kinda need.

It's a form of artistic expression for some people (like me) who aren't as great with paper and pen but still have ideas and patterns and concepts and abstractions to express.

Watching the data just flow through the pipelines and pathways you've laid down for it, creating spectacles from what is essentially electricity running through a rock. Being able to create an interface between a human mind and an inanimate dead block of dug out and processed ore, feels like tapping into the metaphysical.

(Yeah I'm pretentious with words)

Hi guys!
Is there anyone working in deployment, operations? So the people who setup build pipelines, these CI/CD things. My question for you guys is: what motivated you to move away from development (or outright start working in operations)? What kind of base knowledge did you possess in order to be successful in this field? Do you regret making the switch to operations?
I'm at the start of my career and I've been doing development for about one and a half years, but my heart is not really in it. I like setting up tools, learning their capabilities, writing scripts to automate things a lot more than figuring out the client's twisted requirements and then scraping together a solution for that.