Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "deploy"
Why do shithead clients think they can walk away without paying us once we deliver the project !!!
So, here goes nothing..
Got an online gig to create a dashboard.
Since i had to deal with a lot of shitheads in the past, I told them my rules were simple, 20% advance, 40% on 50% completion and 40% after i complete and send them proof of completion. Once i receive the payment in full, only then i will hand over the code.
They said it was fine and paid 20%.
I got the next 40% also without any effort but they said they also needed me to deploy the code on their AWS account, and they were ready to pay extra for it, so i agreed.
I complete the whole project and sent them the screenshots, asking for the remaining 40% payment. They rejected the request saying my work was not complete as i had not deployed on AWS yet. After a couple of more such exchanges, i agreed to setup their account before the payment. But i could sense something fishy, so i did everything on their AWS account, except registered the domain from my account and set up everything. Once i inform them that its done and ask for the remaining payment.
The reply i got was LOL.
I tried to login to the AWS account, only to find password had been changed.
Database access revoked.
Even my admin account on the app had been removed. Thinking that they have been successful, they even published ads about thier NEW dashboard to their customers.
I sent them a final mail with warning ending with a middle finger emoji. 24 hours later,
I created a github page with the text " This website has been siezed by the government as the owner is found accused in fraud" and redirected the domain to it. Got an apology mail from them 2 hours later begging me to restore the website. i asked for an extra 10% penalty apart from the remaining payment. After i got paid, set an auto-reply of LOL to thier emails and chilled for a week before restoring the domain back to normal.
Dev : 1
Shithead Client: 025
I left a company once. Was there with one colleague and we had this kind of code review habit that we looked at each other changes befor merging them to the deploy branch. On my last day I made a dancing cat dance in front of our app as a tiny joke for him. He instead of reviewing just pulled this time and deployed the new version on the companies dev server without a look. So the fist time the cat showed up was appareantly in the first meeting after I left and everyone went completely crazy because they thought they got hacked.
I think they never found the hidden rock roll in the app.4
PM (on slack): "we’re about to deploy to production".
… I keep on working on a task / remain available for any post deployment issues …
PM (5 minutes later on slack): "deployment broke production! We need to handle this NOW!"
My dev colleague has already called it a day, but I’m still online
Me: "ok I don’t have access to prod, can you describe what’s going on? I can’t reproduce on any other environment"
10 minutes go by
Me: "anybody there?"
45 minutes later, I realize PM is offline
The following day:
PM: "ok we got prod running again" (turns out it was client’s fault for not updating a config we as devs can’t access)
PM: "but we’re REALLY UPSET! You guys need to be available to intervene for any issues following deployment to production! At least one of you should be available!"
Me: "but, but…" 14
I was only seventeen back then and I was a Java Developer Intern, not knowing much about enterprise oriented coding.
The project leader in our dev team saw a lot of potential and passion in my work, but was convinced I wasn't taught enough to do the right thing.
I was mainly doing shitty mappers and services back then, which were somewhat used but never lasted long and were ditched a few months later, which always bummed me out. I wanted to make an impact on REAL projects that would deploy into production.
So Mister Mentor (GDPR forbid to use the actual name), who was always first to come and last to leave the office, taught me what it means to code for real.
We stayed after 5pm until 7-8pm multiple times a week and he taught me in a deeply understanding and calm way how to:
- Git (SVN)
- Unit Test
And most importantly:
- How to debug like an absolute BOSS
(We even debugged native Java Libraries just for fun to see if we could break them)
Fast-forward a month later and little intern me made his first commit on production.
Without Mister Mentor, I wouldn't be half as good of a developer as I am today.3
So one year ago I was working at this company from the US, me being in Europe, which automatically implies there is several hours of timezone difference.
The eng. manager decided we would have a release tomorrow (decision was made one month earlier), and stuff was being prepped up to make that happen.
In the US the workday was about lunch time and in EU it was one hour before finishing. The manager gets us in a meeting and asks me and another dude to do some testing that would take several hours to do. This testing could have been done several days or weeks earlier.
40 minutes after that meeting I get a private message from the PM asking for the status of the test...
Me: aaa.. well I started it and will continue tomorrow
Manager: wait what? we have launch tomorrow, this testing has to be done by tomorrow
Me: it's the end of the workday here, I got personal errands that I have to attend to
Manager: uhm ok ... I see...
I was just messaging something in the public chat right before calling it a day and the manager writes "thanks for the input, your day is over now", completely out of context to the conversation I was having with whomever.
There was no question of "can you stay extra hours and do this?", there was no "hey, I know your day is over we will pay you premium hours with this amount as according to our contract, could you do this now as we have release tomorrow?" ..no ..just .. "do it!". I automatically assumed that ..hey, maybe he wants to do this during and after the live launch (and yes I do admit my mistake of not asking just to be clear, but I assumed the manager knows that there is a timezone difference ..like it's a no brainer).
I can not tell you the heat sensation I had after that last reply from the manager ... it was completely uncalled for, and unreasonable.
I mean why not make a pre-launch phase where you put stuff on the staging server, and perform all the necessary tests and then when you get all the green lights from testing you then proceed with the actual deploy? ...no ... mention this like right at the end of the day before the launch....
And another thing that scratched my neuronal cortex is, how does he know exactly how long the tests would take?12
I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…
I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.
I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.
We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.
Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.
My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.
Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.
At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.
Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?
Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.
Amazingly, no one was fired.6
Current work project is microservices architecture out of 4 - 8 components.
It is fully Infrastructure as a Code automatized. I just change somewhere code, git pushing
And it automatically invokes Gitlab CI, terraform, ansible, kubernetes helm charts.
Auto checking itself with unit and integration tests in autoredeployed staging env. Then it saves tested results to docker registry and asks for one button verificating click to be rereleased to prod.
I just go for drink or eat food. While all the stuff is happening.
And I am proud that all the infrastructure, backend and frontend I made on my own.
I don't need to remember how to Deploy it. It is all automatized3
Got laid off last week with the rest of the dev team, except one full stack Laravel dev. Investors money drying up, and the clowns can't figure out how to sell what we have.
I was all of devops and cloud infra. Had a nice k8s cluster, all terraform and gitops. The only dev left is being asked to migrate all of it to Laravel forge. 7 ML microservices, monolith web app, hashicorp vault, perfect, mlflow, kubecost, rancher, some other random services.
The genius asked the dev to move everything to a single aws account and deploy publicly with Laravel forge... While adding more features. The VP of engineering just finished his 3 year plan for the 5 months of runway they have left.
I already have another job offer for 50k more a year. I'm out of here!13
So my department is "integrating CI/CD"
Right now, there's a very anti-automation culture in the deployment process, and out of our many applications, almost none have automated testing. And my groups is the only one that uses feature branching - one of the few groups that uses branching at all beyond "master, dev"
So yeah... You could see how this is already ENTIRELY fucked from the very beginning.
First thing they want to do is add better support for a process... Which goes directly against CI/CD.
The process is that to deploy to production (even after it is manually approved by manager), someone in another department needs to press a button to manually deploy. This, as far as I can tell, is for business rule reasons rather than technical ones.
They want us to improve that (the system will stay exactly the same with some streamlined options for said button pressers)
I'm absolutely astounded at the way our management wants to do something but goes in exactly the opposite direction. It's like the found an article of what CI/CD was and then took notes on exactly what not to do.25
I'm having an existential crisis with this client.
We are spending millions of $s every year to make sure the product's performance is perfect. We are testing various scenarios, fine-tuning PLABs: the environment, application, middleware, infra,... And then we provide our recommendations to the client: "To handle load of XX parallel users focusing on YY, yy and Zy APIs, use <THIS> configuration".
And what the client does?
- take our recommendations and measure the wind speed outside
- if speed is <20m/s and milk hasn't gone bad yet, add 2x more instances of API X
- otherwise add 3xX, 1xY and give more CPUs to Z
- split the setup in half and deploy in 2 completely separate load-balanced prod environments.
- <do other "tweaking">
- bomb our team with questions "why do we have slow RTs?", "why did the env crash?", "why do we have all those errors?", "why has this been overlooked in PLABs?!?"
If you're improvising despite our recommendations, wtf are we doing here???
One day I will crack. Hopefully, not sometime soon.3
Just let me be a programmer. Why do I have to learn yaml and deploy an app with 100 lines of code to a kubernetes cluster with literally 0 users?
I don't know if I'm being pranked or not, but I work with my boss and he has the strangest way of doing things.
- Only use PHP
- Keep error_reporting off (for development), Site cannot function if they are on.
- 20,000 lines of functions in a single file, 50% of which was unused, mostly repeated code that could have been reduced massively.
- Zero Code Comments
- Inconsistent variable names, function names, file names -- I was literally project searching for months to find things.
- There is nothing close to a normalized SQL Database, column ID names can't even stay consistent.
- Every query is done with a mysqli wrapper to use legacy mysql functions.
- Most used function is to escape stirngs
- Type-hinting is too strict for the code.
- Do not use a package manger composer because he doesn't have it installed.. Though I told him it's easy on any platform and I'll explain it.
- He downloads a few composer packages he likes and drag/drop them into random folder.
- Uses $_GET to set values and pass them around like a message contianer.
- One file is 6000 lines which is a giant if statement with somewhere close to 7 levels deep of recursion.
- Never removes his old code that bloats things.
- Has functions from a decade ago he would like to save to use some day. Just regular, plain old, PHP functions.
- Always wants to build things from scratch, and re-using a lot of his code that is honestly a weird way of doing almost everything.
- Using CodeIntel, Mess Detectors, Error Detectors is not good or useful.
- Would not deploy to production through any tool I setup, though I was told to. Instead he wrote bash scripts that still make me nervous.
- Often tells me to make something modern/great (reinventing a wheel) and then ends up saying, "I think I'd do it this way... Referes to his code 5 years ago".
- Using isset() breaks things.
- Tens of thousands of undefined variables exist because arrays are creates like $this = 5;
- Understanding the naming of functions required me to write several documents.
- I had to use #region tags to find places in the code quicker since a router was about 2000 lines of if else statements.
- I used Todo Bookmark extensions in VSCode to mark and flag everything that's a bug.
- Gets upset if I add anything to .gitignore; I tried to tell him it ignores files we don't want, he is though it deleted them for a while.
- He would rather explain every line of code in a mammoth project that follows no human known patterns, includes files that overwrite global scope variables and wants has me do the documentation.
- Open to ideas but when I bring them up such as - This is what most standards suggest, here's a literal example of exactly what you want but easier - He will passively decide against it and end up working on tedious things not very necessary for project release dates.
- On another project I try to write code but he wants to go over every single nook and cranny and stay on the phone the entire day as I watch his screen and Im trying to code.
I would like us all to do well but I do not consider him a programmer but a script-whippersnapper. I find myself trying to to debate the most basic of things (you shouldnt 777 every file), and I need all kinds of evidence before he will do something about it. We need "security" and all kinds of buzz words but I'm scared to death of this code. After several months its a nice place to work but I am convinced I'm being pranked or my boss has very little idea what he's doing. I've worked in a lot of disasters but nothing like this.
We are building an API, I could use something open source to help with anything from validations, routing, ACL but he ends up reinventing the wheel. I have never worked so slow, hindered and baffled at how I am supposed to build anything - nothing is stable, tested, and rarely logical. I suggested many things but he would rather have small talk and reason his way into using things he made.
I could fhave this project 50% done i a Node API i two weeks, pretty fast in a PHP or Python one, but we for reasons I have no idea would rather go slow and literally "build a framework". Two knuckleheads are going to build a PHP REST framework and compete with tested, tried and true open source tools by tens of millions?
I just wanted to rant because this drives me crazy. I have so much stress my neck and shoulder seems like a nerve is pinched. I don't understand what any of this means. I've never met someone who was wrong about so many things but believed they were right. I just don't know what to say so often on call I just say, 'uhh..'. It's like nothing anyone or any authority says matters, I don't know why he asks anything he's going to do things one way, a hard way, only that he can decipher. He's an owner, he's not worried about job security.13
It works locally, it works in Dev, it works in Test, but fails to deploy in UAT. Is it a data issue? I don't know, I don't have permissions to see the UAT database. Literally all I know is that this API is returning 500 instead of what it's supposed to return, but only sometimes.
Guess I'll sit here all day and try to solve the problem telepathically as there is literally no way of troubleshooting other than scrolling through the code and hoping that a cartoon lightbulb appears above my head.2
Other team lead: Hi DevOps Team, We need you to deploy this app to production. It's maintainers gave up on it in 2019, but we looked at it and it feels right.
Me: Uhm. That's not going to work. It'll fail the security scan before you can even finish the build in CI.
Other team lead: Yeah, this app is the right thing to do, and we needed it last week, but since that won't work, we'll just use this other very very infant technology that was just born yesterday. It's not stable in production, or on MySQL, or in AWS at all, but it's the other direction we can to go.
Me: What problem are you trying to solve in the first place?
Other team lead: Oh, we need access to the read from the production database.2
I recently tried to apply the same data analytics rationale that I use at work to my personal life. This is not a rant, it is more like an data storytelling of an actual use case I would like some input on.
I set a goal - gotta thin up a bit and calm down my ticker - and got a (almost unreasonably expensive) field expert consultant to yell at me about it for a couple hours.
I unravel the metrics - there is like a million weight-related KPIs and most say nothing at all. I have never seen an non-infrastructure measurable subject that could not be resumed to 2-5 performance metrics. I got overall weight, how well my nine-years-old business suit fits me, heart rate, and day-after relative muscle pain (it will make sense soon).
Then its data-pipeline time. I bought a cheap weight scale and smartwatch, and every morning I input the data in an app. Yes, I try to put on the suit every morning. It still does not fit.
After establishing a baseline, I tried to fit different approaches. Doing equipment-free exercises, going to the gym, dieting. None was actually feasible in the long run, but trying different approaches does highlight the impacts and the handling profile of each method.
Looking at the now-gathered data, one thing was obvious - can't do dieting because it is not doable to have a shopping list and meals for me and another for the family.
Gym is also off the table - too much overhead. I spend more time on the trip there and back than actually there.
And home exercise equipment is either super crappy or very expensive. But it is also the most reasonable approach.
So it is solutions time. I got a nice exercise bycicle (not a peloton), an yoga mat (the wife already had that one) and an exercise program that uses only those two resources. Not as efficient without dieting, not as measurable and broad as the gym, but it fits my workflow. Deploy to production!
A few months pass and the dataset grows. The signal is subtle but has support - it works! The handling, however, needs improvement, since I cannot often enough get with the exercise program. Some mornings are just after some hard days.
I start thinking about what else I can improve in the program, but it is already pretty lean and full of compromises.
So I pull an engineer and start thinking about the support systems and draft profile. What else could be draining my willpower and morning time?
Chores. Getting the kids ready for school, firing up the moka pot, setting the off-brand roomba, folding the overnight-dried clothes, cooking breakfast, doing the dishes, cleaning the toilets. All part of my morning routine. It might benefit from some automation.
Last month I got that machine our elders call "wasteful" and "useless crap lazy entitled Americans invented because they feel oh-so-insulted for simply doing something by hand like everyone always did" - a "dish-washer".
Heh, I remember how hard was to convince my mother-in-law that an remote-controled electric garage door would not make she look like an spoiled brat.
Still to early to call, but I think that the dishwasher just saved me about 25 mins every morning. It might be enough to save willpower for me to do more exercise.
This is all so reflective of all data analytics cases really are out in the wild - the analytics phase seems so small compared to the gathering and practical problem-solving all around. And yet d.a. is what tells you that you are doing the wrong thing all along. Or on what you should work next.3
probably every time I see my tests failing.
Each time I am writing tests I'm convincing myself "it's an investment", "spend 2 hours now to save 2 days later", "unit-tests are good".
And each time I'm chasing away ideas like "perhaps they are right, perhaps writing unit tests is a waste of time..", "this code is simple, it should ever break - why test it??", "In the 2 hours I'll spend writing those UT I could build another feature"
Yes, it is terribly annoying to write tests, especially after writing the production code (code-first approach). Why test code that you know works, right?
But after a few weeks, months or years, when the time comes to change your feature: enhance it, refactor it, build an integration with/from it, etc, I feel like a child who found a forgotten favourite candy in his pocket when I see my tests failing.
It means I did a very good job writing them
It means it was not a waste of time
it means these tests will now save me hours or days of trial-and-error change→compile→deploy→test cycles.
So yeah, whenever I see my tests fail, I feel warm and fussy inside :)3
I f&#king hate it here. I am just eyeing to exit as soon as 1 year of my contractual obligation is over. My employer is a good employer. Provides good benefits but I just can't take the bureaucrazy in here. Just yesterday, had to ask another team to deploy objects on our behalf as they are the schema owner. They did it and asked us to review it today. But how? We don't even have manual access to the schema, because we are not the content owner and security! But that's fine, I can always query the catalog views and check the metadata and should be able to conclude the deployment. Right? NOOOO. Because security! Of what? Column names?
Prev rant: https://devrant.com/rants/5145722/...2
I posted the kintsugi rant recently, about how I reached zen when it comes to software engineering, and let go of all frameworks, including React.
Yesterday I was getting started with Lisp (because simplicity), and posted the rant about how Lisp code was fixed with REPL on a spacecraft 100 million miles away from earth.
Today I was contemplating replacing everything I have on Heroku with a single NGINX VPS, as I don't need containers.
I was also reevaluating the way I deploy apps, replacing everything with simple bash scripts.
I'm learning NGINX now.
…and I just realized it all wasn't some kind of breakthrough that had a reason behind it.
It was my bipolar disorder all along. Having bipolar disorder implies you change your ways, and sometimes your entire world view, once a year. The change cannot be averted or postponed. It will happen.
Bipolar makes you suffer, but, on the flip side, keeps you from becoming stale and set in your ways. Also, bipolar is an undefeatable protection against manipulators and abusive relationships. Yes, you can be manipulated, but the manipulator's framework will last one year tops. Then it will be obliterated together with your world view, and you will start a new life.11
Thursday afternoon. Client gives us the go to deploy the latest release to production.
Friday late afternoon; my colleague - "wait, did we ever deploy"? Me - "uh, nope".
"Alright have a good weekend"
Stakeholder: The orders aren’t importing from our order systems into our fundraising systems correctly. Gifts are showing last year’s codes. Do I restart the system?!
Me: Dude, I don’t know. My only responsibility here was to update the website’s config with the new codes you gave me. I did that. The correct codes for the orders are being sent to the orders system and that system is receiving the orders correctly. I know nothing about the configuration between the orders and fundraising systems.
Plus, y’all should have done testing before you ok’d my deploy. Don’t go looking to me to know someone else’s job. Y’all should have especially done testing since the person who’s the SME resigned months ago and there’s been no one to replace him yet.
In August 2021 I asked my bosses for a raise for my extra work that I done for the last 6 months to create the first 4 microservices in the company and deploy them in production on a new infrastructure that is cloud based.
Today 27/01/2022 they reported how happy they are of me and that they will take my proposal *in consideration*.
Now i am searching for a new job.
Funny part: I am simple guy if they would have given me a NAS from Synology with lot of space as a reward I would actually have been happy.
P.S. jokes on them, i left 4 easter eggs hidden in the app, pipeline, db and user manual.3
So I made an update to my React Native app. I changed UI of a couple of screen, added a few animations here and there, refactored how my graphQL resolvers work in the backend(no breaking changes), changed how data gets loaded into the database etc.
It worked in dev so I figured hey let's deploy it. Today is(was because it's now 3am but more on that later) a national holiday so no one goes to work so no one will use my app so I have an entire day to deploy.
I started at 15:00(because i woke up at 13:00 lol). I tested the update once again in dev and proceeded to deploy it to prod. I merged backend to master, built docker images, did migrations on the db, restarted docker-compose with new images. And now for the app. I run ./gradlew assembleRelease and it starts complaining that react-native-gesture-handler is not installed. Ugh, rm -rf node_modules && yarn install. It worked. But now gradlew crashes and logs don't tell me anything. Google tells me to change a bunch of gradle settings but none of them work. Fast forward 5h, it's around 20:00 and I isolated the issue to, again, react-native-gesture-handler. They updated from 2.2.4 to 2.3.0 which didn't fucking compile. 2 more hours passed (now 22:00) and I got v2.3.1 working which fixed the problem in 2.3.0 but made my app crash on startup. YOUR FUCKING LIBRARY GETS 250K WEEKLY DOWNLOADS AND YOU DONT EVEN BOTHER CHECKING IF IT COMPILES IN PROD ON ANDROID?! WHAT THE FUCK software-mansion?
After I solved that, my app didn't crash. Now it threw an error "Type errors: Network Request Failed" every time I fetch my legacy REST API(older parts use rest and newer use graphql. I'll refactor that in the next update). I'll spare you the debugging hell i went through but another 5h passed. Its 3am. My config had misspelled url to prod but good for dev... I hate myself and even more so react-native-gesture-handler.3
Never! Deploy! To production! On Thursday! *banging my head against the wall*
Now I need to revert some things manually on production ON MY DAY OFF 🤦♂️🤦♂️10
Until today, I had assumed deploying stuff to prod would NOT be one of my responsabilities in this company. Apparently that's not the case.
Had to deploy my code and pray it didn't break anything. Why is this a big deal at all?
Well you see, there is no repository. At all. No git, no svn, not even duplicate folders. No tests, no pipeline. Just a bunch of CPanels.
Had to manually copy files and folders from the development site to the production site and partially copy a database. "Just drag and drop" were the instructions I was given.
As if using CakePHP2, PHP5 and having to parse fucking Excel files wasn't bad enough, now I have to deal with one of the worst ways to deploy code.
Fuck it, I'm switching on the looking-for-job flag on linkedin.5
Continuing on the PR ranting.. That's six PR's in the queue now. The most important one took over a week to sort through because someone had a grand idea to generalize the work and that required a ton of changes. Great to receive that kind of feedback AFTER I've already done the work.. Then we waited on translations because can't miss those! Now the tests are failing. Not caused by me! Someone messed up the E2E tests. That's two weeks in the queue now.
Because one test is failing, that's a block across all the PR's. All of them have now been waiting multiple days in the queue. After someone manages to fix the broken test, all PR's go one by one into the QA testing queue and that takes 1-2 hours per branch to test and deploy to production.
Really love having a cycle time of four days for coloring a button differently! And the constant context switching.
TIL don't rely too much on in memory databases if your client runs development and production environment on the same machine.
Lessions I learned so far from my first big node/npm project with tons of users:
1) If you didn't build something for a while, expect 3 hours of resolving version conflicts for every two weeks since the last build.
2) Even if the tests pass, run the containers on your own machine and make sure that the app doesn't randomly crash before deploying
3) Even if the app seemed to work on your own machine, run the tests again in an environment mimicking prod at most 15 minutes before replacing the running containers.
4) Even if all else indicates that the app will work, only ever deploy if you expect to be available within the 4 hours following a deployment.
5) Don't use shrinkwrap for anything other than locking every version down completely. A partial shrinkwrap will produce bugs that are dependent on the exact hour you built the app _and_ the shrinkwrap file, and therefore no one will ever have seen them other than you.
6) Avoid gyp, and generally try not to interface too much with anything that doesn't run on node. If parts of your solution use very different toolchains, your problems will be approximately proportional to the amount of code. And you'd be surprised just how much code you're running. (otherwise it's more logarithmic because the more code the less likely a new assumption is unique)
7) Do not update webpack or its plugins or anything they might call unless you absolutely need to
8) Containers are cool but the alpine ones are pretty much useless if you have even just one gyp module.
9) There's always another cache. To save yourself a lot of pain, include the build time in every file or its name that the browser can download, and compare these to a fresh build while debugging to assert that the bug is still present in the code you're reading
+1) Although it may look like it, SQLite is far from a simple solution because the code and the bindings aren't maintained. In fact, it'll probably be more time consuming than using a proper database.3
Deploy 2 apps
app #1 sends request to app #2 every minute
app #2 sends request to app #1 every minute
they never sleep
Friday, forced deploy day for last & current months work. Been stockpiled due to holiday.
Yesterday boss demoed the product to clients so they expect to test today.
Early o'fuck this morning, a coworker managed to drop all secrets and env vars from CI pipelines and trigger a deploy leaving production broken...
It's gonna be a long and busy Friday...
Sooo, turns out, management and senior PMs, technical PMs, service managers and you name it forgot an entire system.
A complete eco-system of applications, queues, services, load-balancers, deploy pipelines, databases, monitoring solutions, etc, etc, that if not handled correctly could effectively put the entire production line to a standstill.
So, waaay too late they make this discovery. In their ignorance. Just utter incompetence. Huge project. Millions of $. And they forget it. Months of meetings probably. Workshops and gettogethers at cozy hotel complex discussing ”the project”? And they do not understand some of the fundamental building blocks…
Basic engineering for these guys must mean something completely different.
I can’t even.
I am so fed up with this organization. It does not stop either.
How is this possible…
Do they even have half a brain?
I did not think that making a serverless Discord bot would be such a learning experience. The code itself was easy. The hard part was the infrastructure, because I decided to automate it all with Terraform and deploy it on AWS.
Before this project, I had no idea how API Gateways worked. Now I still have very little idea how they work but I managed to build one anyway. Eventually. And then I had to figure out how to automate the deployment of a lambda layer and function that would both still be managed in the Terraform state, with any code changes triggering a rebuild and update for the resource.
And then I had to untangle a dependency mess because API Gateways have some weird issues where two resources that have no explicit dependencies on each other will throw an error if they don't deploy in the right order.
And then I went the wrong way with Github actions trying to conditionally chain multiple workflows together before I realized I could just put multiple jobs with conditions in a single workflow.
And now after all that work over the course of 2 days, I have a bot that does this:2
Jesus christ I need my VP and CIO to get their hands out of Azure and GCP and just let me work.
Yes, governance and security and IAM are big deals. That's why you have infraops people like me to deal with that.
I'm literally working with one hand tied behind my back because just about every button press or CLI command I need to do my damn job as a professional cloud fluffer requires me to go bother an executive and ask permission to pretty please can I deploy a new container, can you go press the shiny button? No not that one, move your mouse up...up..now UP..ok over lef-no..can I have mouse control? Sigh fine, do you see where it says "Approvers", no that says "Release Pipeline"
Look I actually kinda like this job, I do, in as much as when I have something to do I get left the fuck alone to do it. Meetings are minimal, aside from the odd days when one of our app services decides to yeet itself into the river Styx, there's little distractions.
Yeah, developers do dumb shit but that's probably best left to the notion of job security and never talked about again less they go to HR and complain that the ops guy was very stern and direct and made the developer take some accountability for their work product.
It's so intergalactically stupid that I have to go ask permission just to do ops tasks by the same people barging down my goddamn door asking why the ops task isn't done yet.
"Because you won't give me permissions in GCP to actually DO anything".
Okay. Rant over. Time for lunch. Good meeting, see you all at the holiday party.2
Yesterday whole day ive been trying to deploy an ios app to app store from a flutter project but kept getting "module not found" in build compilation error
I thought to myself am i fucking dumb?
Or maybe i am smart but extremely UNLUCKY in life like always?
Today i googled for this error and one of the top stack overflow answers with a +50 bounty points, first sentence they answered was "this is a very bad and UNLUCKY error, after trying to solve this issue for hours i finally found the solution..."
Let me tell you a story about an independent contractor who was just told the reason I don’t have enough information to build, deploy, access, or test the software I was brought on to maintain…
is that the previous developer is holding all documentation essentially “hostage” in an attempt to squeeze more payment out of the client. What the tap-dancing shit have I gotten myself in to.3
That feeling when u delete 4months of coding work that u didn't deploy on git, with just one line of code ....😔😔😠😤..
I wanted to delete the virtual environment oo...6
One day, the Director of Web Ops (marketing role) submitted a ticket to update the list of product categories on the website’s navigation. Sounds like a simple ticket right? Just some html edits. Nope. Every day for three days, she changes her mind and adds new changes. What should have taken me 10 minutes stretched out to three days. She held up code review of my ticket because she kept making changes.
She had plenty of time to sort out what she wanted. That ticket had been sitting in the To Do pile for two days before I touched it.
She was being an asshole because she knew she could get away with it and I had no recourse: my direct manager was on vacation, the entire dev team was going to be laid off anyway so no one was going to defend us on “trivial” matters, and we were going to enter code freeze soon so she’d just argue it was critical business changes for our critical revenue season.
I suspect she was also just not good at her job. I never met her in person because she was hired during the 2020 pandemic and we were all working remotely. I did see her make a five minute presentation during an all staff meeting…and she didn’t come off too well. Her voice was trembling during her turn to speak…like she was not confident or not prepared.
She knew she was causing chaos but she put on this act of not knowing. She was definitely trained on our dev team’s practices for tickets and deployments. She knows about code review, beta testing, and user acceptance testing that has to happen before a ticket can be deployed.
It happened to be before Thanksgiving weekend 2020. Our deploy was going to happen on Tuesday instead of Thursday because Thursday was a holiday (no one would be working) and Wednesday was a half day.
Tuesday afternoon at 1pm, she messages me and the dev in charge of deploy about more changes! My time is already occupied because our Product Manager went on vacation and dumped a large amount of user acceptance testing on me. I scream at my computer at that point because I realize I’m in the ninth circle of hell. I tell the other dev in a separate message that Web Ops has been making changes EVERY DAY since I picked up that ticket.
Other dev tells her that we have to check with the C-suite executive for engineering because we’re not allowed to make changes to tickets so close to the deploy. This is actually the policy. He also tries to give Web Ops the benefit of the doubt because we’re not deploying on our usual day. He had to do that to so she didn’t feel bad (and so she doesn’t complain about us not working towards the company’s goals).
Other dev had to do the code changes because I was otherwise occupied with user acceptance testing. If I were him, I’d be pissed that I was distracted from concentrating on the deploy so close to the holiday.
Director of Web Ops was actually capable of even more chaos. I ranted about it before. For that dramatization and if you want to go down the rabbit hole, see: https://devrant.com/rants/4811518/...4
Does anyone even use AWS CDK?
I saw something about CDKv2 getting released and immediately made up my mind about it and just want to validate my opinion.
I'm having a hard time thinking of a case where I would need to use yet another layer of bullshit to deploy cloud infra.
It's bad enough with terraform(which I far prefer over cloud formation). But now you can use python or node? What's next, deploying with XSLT?
I'm partially ranting, because I know someone on my team is going to show this as the "new thing" and I'll be stuck maintaining my code...as code--and that really pisses me off.
I'm also legitimately curious on how many of you have run across this being used successfully and for what problem did it solve?10
Question for devs who work in large multi-team environments:
A) What is your code review process like? Does a senior review it once and then it's off to QA or do you have "levels" of approval?
B) If you're launching a feature that depends on another team how are you coordinating it? Do you just talk over a ticket and then hit merge and deploy at the same time or like what's your process like?
C) What CI/CD tool do you use? Also what code hosting platform do you use? Github/GItlab/etc.
D) Are you currently happy with the CI tool you're using? If not what are some common issues you're facing?5
Fucking IT bloggers be telling you about how to deploy scalable hype meta monad 420 doritos bacon containers. Forget those hipsters.
The only developer blog you need is ACM Queue.
I work for a client that introduced a "roadmap" with tasks labeled with task numbers in May this year. And its 1st July and development-wise we are at task #15. And the client havent asked us to deploy anything yet. Not even task #1.
So its been 2 months of us just developing features in our local machines and now its pissing me off.
What would you do in my situation ?4
Ahh, management. They now decided to implement yet another clown role. Release manager…
I am leaving. I just must leave this workplace!
They have so many roles now and it’s getting increasingly difficult to avoid them. I thought we were flying low before to avoid the radar (we use ci/cd, all automated, deploy all days of the week, so we are good. I mean, we go from business need to implement with (some) quality in minutes. Yes, we make mistakes and we fix them rapidly and continuously).
It will be difficult to stay. I really thought I would enjoy it here but management is making it pretty clear that they are not serious about software. They want fancy titles and pretend to work.
Fire em all I say! No one will notice that they are gone. 🤷🏼♂️4
i didn't want to leave home today, i still had a ton of code to deploy before a deadline.
i went out, phone gets stolen. and i end up spending a lot of money i am not supposed to spend.
thankfully, i had my WhatsApp logged in on my desktop PC and the latest update allows you to use the web version even when the device is offline, was able to reconnect with clients and assure them that i haven't disappeared with their money.
I feel numb, it's probably going to feel much worse when i wake up tomorrow, i just kept sleeping for the most of today which is kinda strange but i don't want to think much about it.1
We can say I'm so called senior in my company. Not exactly a senior because I have like 2.5 years of commercial experience, but everyone in the company has much less.
As I'm kind of technology enthusiast, I have a lot of personal experience with Linux, Docker, I'm basically quite familiar with a lot of stuff both in theoretical and practical manner. That's why often I'm helping my less experienced friends to solve some issue, help with docker, help with deployment, and I'm really happy to do so.
Recently I was helping to deploy something ASAP on environment no one is familiar with and it took me couple of hours to do so, as it required changes in a lot of places. Even our team lead was happy about it.
I've just got email from my boss, that I should focus less on helping others. Not even a "thanks, good job". After I literally helped this company to get their shit together4
okay so I'm working on a personal project
a medical and healthcare system
thinking maybe I can kick start a start-up based on this thing...
so been 3 days now trying to find a platform to deploy this thing for free of course just for presentation and demonstration.... and its been a pain
Finally settled for pythonanywhere.com managed to deploy but the deployment can easily drive you crazy if you dont know what you are doing which i had no idea what i was doing (lol) but its an easy think if your project is up on github found that out when i was researching how to deploy
was excited coz pythonanywhere offers a free MySQL server if your application needs a db on the backend
set that up and guess what what...... it doest even connect (lol)
was getting frustrated now and jumped on the search engine and searched for free mysql online db hosts and found this great platform
managed to grate an account, created a db and integrated with my application
then used this online phpmyadmin to check if the application was able to create the db structure on the remote server https://www.phpmyadmin.co/
and the structure was there :)
thot i should share maybe some1 might be wondering how to host their db backed application for free5
Would you consider compression (gzip static files) as a prepare step of the deploy stage or as a part of the build stage?
It's somewhat irrelevant but its bugging me8
My brother works in fintech sector. He had created an Options strategy after months of hard work and deployed the strategy in production via CI/CD.
However, the strategy didn't deploy automatically on Monday and few trades didn't happen leading to loss. His boss came down firing at him as to why strategy was not deployed.
Turns out the IT team had changed the password on Friday evening as per their routine password updates.2
Apparently there's a limit to how many API endpoints you can deploy in a second using CDK. Thanks for not having an error message that says that and wasting two weeks.1
i want to deploy websockets on multiple servers with horizontal scaling. i don't know what to use. redis pub/sub? haproxy? i wanna know your opinions. ❤️6
Anybody had experience with Hashicorp Nomad for smaller deployments (personal in this case)? Thinking of using it after having done it too many times with complex Kubernetes and Docker... 😅 Also don't wanna run this thing bare bones as i have a whole bunch of services to deploy together.