Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
I. FUCKING. HATE. MOBILE. DEVELOPMENT.
I already manage the data, devops, infra, and most of the backend dev.
We had a mobile guy. He was great. I never had to think about it and kept moving quickly on my work. #SpecializationOfLaborFTW
He left. Why? Because they wouldn't give him a small raise despite being one of the best mobile engineers in the firm. WTF.
I made the mistake of picking up just enough slack on this workflow in the interim such that I'm, apparently, the fucking god-damned release manager, fixer of pipelines, fixer of build configs, fixer of anything where someone just needs to RTFM for a half-hour to not fucking break things.
Now, 8 months later...and, apparently, Fortune 500 companies are too fucking god-damned cheap to pay for someone who actually knows WTF they're doing for a very reasonable thing to have at least one dedicated set of eyes for.
I never wanted to be a mobile dev.
I never will want to be a mobile dev.
And I certainly don't want to manage your HALF-FACE-FUCKED detached expo configs.
There's a reason I never intentionally involved myself in mobile. All the way down, it's just shitty cross-compilation, transpilation, dependency-hell, brittle-as-fuck build processes so we can foot-gun and mouth-gun react-native and expo and babel and whatever the fuck else cargo-culted horseshit into the wild.
And why? What's the actual fucking root cause? The biggest white elephant that ever fucking elephant-ed? It's because Apple and Google decided to never collaborate on a truly-native cross-platform SDK--where engineers could write native code that compiles to native binaries that's simply write-once, run-everywhere. They know they could have done that, and they didn't. So what'd they get back? Expo--a too-cleverly-designed backdoor/hack--more-or-less a way to circumvent the sane release process software has usually followed: code -> executable -> deploy. Or code -> deploy (for interpreted langs). Expo's like "keep your same executable, we're just gonna to do updates by injecting new code into it whenever we want". Didn't we learn anything with web? Shit gets messy real quick? Not to mention: HEY EXPO, WE WERE ALREADY BUILDING NATIVE APPS, YOU SHORT-SIGHTED FUCKS. THANKS FOR LURING OUR CTOs INTO FORCING EXPO DOWN OUR THROATS W/ THE IMPLICIT (BUT INCORRECT) TOO-GOOD-TO-BE-TRUE PROMISE THAT WE CAN HAVE WRITE-ONCE, RUN-ANYWHERE WITHOUT ANY BUY-IN OR COOPERATION FROM THE ACTUAL TARGET PLATFORMS.
And, we just, like, accept this? We all know it's garbage engineering. The principles we learned in the classroom aren't just academic abstractions--they actually yield real-world results--and eschewing them yields real-world failures. Expo is tightly-coupled to high-heaven, with leaky abstractions six-ways-to-christmas, chock-full of foot-guns, and fails the most basic test of quality: does it, "just work?"
Expo is fucking shameful and it should fucking die. Its promises are too bold, its land-mines too many, its future-proof-ness is alway, always, always questionable as fuck and a risk to every project that uses it.
You want a rant? This is my fucking venue, 'tis not? Well, then this is a piss and vinegar rant straight from my blood-red, beating fucking heart:
EXPO FUCKING SUCKS. AND IF YOU'RE A FAN, YOU FUCKING SUCK TOO.27 -
Dear Indian Companies,
Why do you hire for a role and then say: "We dont have that role but then we want you to grow up to be a Generalist"!
6 years as a build, release and SCM guy at Moto and Nokia back then, I shifted to this big Indian IT corp coz Nokia was shutting down...
A week into my orientation (which is a crazy weirdness inducing ritual in and of itself), the new manager I'm supposed to be working with comes up and says- "Here's the code repo, there are 2 open jQuery issues, fix them!"
I'm not really sure what to say at this point because jQuery is nice and all but thats not who I am.. I'm the infra / DevOps guy. And this is circa 2012 when DevOps as a term was just hotting up...
Tell me to setup a multi-stage pipeline and automated test cycles, I'll do it drunk, but oh no! bug fixing on a jQuery script? Noooo!!!!! I just dont have the chops for it.
So long story short, I get reported to HR for insubordination - Yeah, Go Figure!
Cue: HR meeting
HR: You wont work?
Me: I cant work on jQuery. I am a sysadmin / devops guy... Give me a project that involves those skills and I'll work.
HR: But we hired you to work on jQuery.
Me: But you did not mention jQuery / UI / UX in the job description - Pulls up email and shows JD for interview which says Symbian, Build, Release, Configuration Management but NOT jQuery.
HR: ....
Me: :-/
HR: But we want you to be a generalist.
Me: #wtf
HR: We want an engineer to be able to do anything he is tasked with!
Me: Can I know my last working date here?
And thats how my career at a glorious IT corporation just went poof!
When I think back on it, I feel good that I chose to do what I wanted to get better at and what I loved working on...
And this is the problem with IT companies in our country - They play with people's aspirations and passions... To the point that all thats left of a software engineer is the looking forward to pay day so he can start the damn cycle all over again.11 -
Fucking 20 hour days. Third one this week.
Been at work since 6am, it is now midnight. Spent the morning fixing bush league code mistakes from "expert" onshore developers, and explaining how-to-wipe-your-ass level concepts to some rude cunt who is absolutely going to take credit for my work after I leave.
Now I'm just waiting on this slow boat scp to finish because the invalids the customer hired to manage their infra can't figure out the 3 minute exercise that is standing up a registry, so the container deployment process is fucking export multiple 500mb Redhat images as a tar and ship it across the cripplenet they call a datacenter. And of course the same badmins don't understand rsync and can't manage to get network throughput in a datacenter with a $300M annual budget over 128kbps. I guess that's fast for whatever jugaad horseshit network they're used to.
I've said it before, but it bears repeating. Fuck IBM. They're a cancer and at this point I question the moral compass of anyone who works for them.7 -
I decided to draw something while checking out 3000+ builds happening in parallel in the school infra
sysadmin life is boring
10 -
Fuck my life...
Okay, so I’m working on a web app with a small group... the app is basically a lead generator for new business in another country. We just need contact details cause they’re a fucker to buy.
Step 1: prototype to the investors, working with the ceo to make this thing look shiny AF.
Goes well as fuck.
CEO: “when can we get this out?”
Me: “it’s basically done mate, get your guys to look at it and we can talk about marketing”
Que a shower of 10 or so bellends with senior in their title going into a room and coming out with:
Bellends: “so on this page we want the user to confirm and accept the contract”
Me: “cool, makes some sense, that’s what it’s already doing.”
Bellends: “afterwards we want to show them the price and have them put in their banking details.”
Me: “Wait, you what when?”
Bellends: “Yeah, well Jenny says we should have as few clicks as possible to get to the final stage and have the customer accept.”
Me: “Jenny’s on fucking crack, moving the contract formation phase to after the contract acceptance stage is not an option”
Bellends: “Oh it’s okay, Andy in legal said that would be okay”
Me: “Andy’s a fucking moron, tell him that online contract formation laws were updated 2014/2015 and you can’t do that anymore”
Bellends: “No, andy’s legal, surely he knows”
Bellends: “We want all of this above the fold”
Me: “OH FUCKING SUCK A DICK YOU ABSOLUTE BAND OF FUCKWADS... which one of you, which one hasn’t looked at a website this millennia!?”
Needless to say I ignored all their shit, got the lead generator out and told the CEO those ten people are certifiably fucking useless.
Bonus round; recent, but “it has to be on internal infrastructure”
“Why? It’s a mobile app sending rest calls to a third party saas.”
“It just has to, we have this thing called the private cloud and w”
“Wait... you what son, priv 🤦🏼♂️ private what mate?”
“Private cloud”
“You... you mean a server rack?”
“Nah we spent £2mn on it, it’s brilliant”
“Hahahaha you fucking dick, you blew £2mn on server infra with fuckall to put on it!?”
“No, no it’s the private cloud”
“Fucking idiot, aye son, where’s the fucking bean stalk you prick!?”
“It has to go on internal infr”
“Shut up, that won’t work”9 -
So, some time ago, I was working for a complete puckered anus of a cosmetics company on their ecommerce product. Won't name names, but they're shitty and known for MLM. If you're clever, go you ;)
Anyways, over the course of years they brought in a competent firm to implement their service layer. I'd even worked with them in the past and it was designed to handle a frankly ridiculous-scale load. After they got the 1.0 released, the manager was replaced with some absolutely talentless, chauvinist cuntrag from a phone company that is well known for having 99% indian devs and not being able to heard now. He of course brought in his number two, worked on making life miserable and running everyone on the team off; inside of a year the entire team was ex-said-phone-company.
Watching the decay of this product was a sheer joy. They cratered the database numerous times during peak-load periods, caused $20M in redis-cluster cost overrun, ended up submitting hundreds of erroneous and duplicate orders, and mailed almost $40K worth of product to a random guy in outer mongolia who is , we can only hope, now enjoying his new life as an instagram influencer. They even terminally broke the automatic metadata, and hired THIRTY PEOPLE to sit there and do nothing but edit swagger. And it was still both wrong and unusable.
Over the course of two years, I ended up rewriting large portions of their infra surrounding the centralized service cancer to do things like, "implement security," as well as cut memory usage and runtimes down by quite literally 100x in the worst cases.
It was during this time I discovered a rather critical flaw. This is the story of what, how and how can you fucking even be that stupid. The issue relates to users and their reports and their ability to order.
I first found this issue looking at some erroneous data for a low value order and went, "There's no fucking way, they're fucking stupid, but this is borderline criminal." It was easy to miss, but someone in a top down reporting chain had submitted an order for someone else in a different org. Shouldn't be possible, but here was that order staring me in the face.
So I set to work seeing if we'd pwned ourselves as an org. I spend a few hours poring over logs from the log service and dynatrace trying to recreate what happened. I first tested to see if I could get a user, not something that was usually done because auth identity was pervasive. I discover the users are INCREMENTAL int values they used for ids in the database when requesting from the API, so naturally I have a full list of users and their title and relative position, as well as reports and descendants in about 10 minutes.
I try the happy path of setting values for random, known payment methods and org structures similar to the impossible order, and submitting as a normal user, no dice. Several more tries and I'm confident this isn't the vector.
Exhausting that option, I look at the protocol for a type of order in the system that allowed higher level people to impersonate people below them and use their own payment info for descendant report orders. I see that all of the data for this transaction is stored in a cookie. Few tests later, I discover the UI has no forgery checks, hashing, etc, and just fucking trusts whatever is present in that cookie.
An hour of tweaking later, I'm impersonating a director as a bottom rung employee. Score. So I fill a cart with a bunch of test items and proceed to checkout. There, in all its glory are the director's payment options. I select one and am presented with:
"please reenter card number to validate."
Bupkiss. Dead end.
OR SO YOU WOULD THINK.
One unimportant detail I noticed during my log investigations that the shit slinging GUI monkeys who butchered the system didn't was, on a failed attempt to submit payment in the DB, the logs were filled with messages like:
"Failed to submit order for [userid] with credit card id [id], number [FULL CREDIT CARD NUMBER]"
One submit click later and the user's credit card number drops into lnav like a gatcha prize. I dutifully rerun the checkout and got an email send notification in the logs for successful transfer to fulfillment. Order placed. Some continued experimentation later and the truth is evident:
With an authenticated user or any privilege, you could place any order, as anyone, using anyon's payment methods and have it sent anywhere.
So naturally, I pack the crucifixion-worthy body of evidence up and walk it into the IT director's office. I show him the defect, and he turns sheet fucking white. He knows there's no recovering from it, and there's no way his shitstick service team can handle fixing it. Somewhere in his tiny little grinchly manager's heart he knew they'd caused it, and he was to blame for being a shit captain to the SS Failboat. He replies quietly, "You will never speak of this to anyone, fix this discretely." Straight up hitler's bunker meme rage.13 -
senior devops in the school IT dep: "gcc is not working, it keeps telling theres no input files when there is, and its failing most of my builds"
are you sure sir? Give it another go
(Meanwhile my friends managed to sneak a "fixed image")
see? it works.
Senior Devops: " I swear it didn't work"
(we giggle)
check the last build's .bashrc
>alias gcc= (the error message e was referring)
Senior Devops: "MOTHERFUCKERS"
Actually its just your infra that we pushed the image to, school infra is unaffected
I mean it was fun while it lasted.4 -
Got laid off last week with the rest of the dev team, except one full stack Laravel dev. Investors money drying up, and the clowns can't figure out how to sell what we have.
I was all of devops and cloud infra. Had a nice k8s cluster, all terraform and gitops. The only dev left is being asked to migrate all of it to Laravel forge. 7 ML microservices, monolith web app, hashicorp vault, perfect, mlflow, kubecost, rancher, some other random services.
The genius asked the dev to move everything to a single aws account and deploy publicly with Laravel forge... While adding more features. The VP of engineering just finished his 3 year plan for the 5 months of runway they have left.
I already have another job offer for 50k more a year. I'm out of here!13 -
We do infra as a code, and one of my coworker worked on the project alone. Few months down the road, when shit hits the fan, he just message me this is not working.
First of all, I did not write that shit, and also I was never part of the conversation during the decision making. So when shit hits the fan what do you expect me to do? Do some black magic and fix it magically???
1 -
Wow! They are incredible!
I keep creating new email filters every week or so, and they keep finding new ways to send me spam!
The best part is - these emails are sent from our internal infra. Judging by the sender it looks like they have created a bot collecting various events and sending them to... everyone.
Much smart. Many useful.
Much working
1 -
I'm having an existential crisis with this client.
We are spending millions of $s every year to make sure the product's performance is perfect. We are testing various scenarios, fine-tuning PLABs: the environment, application, middleware, infra,... And then we provide our recommendations to the client: "To handle load of XX parallel users focusing on YY, yy and Zy APIs, use <THIS> configuration".
And what the client does?
- take our recommendations and measure the wind speed outside
- if speed is <20m/s and milk hasn't gone bad yet, add 2x more instances of API X
- otherwise add 3xX, 1xY and give more CPUs to Z
- split the setup in half and deploy in 2 completely separate load-balanced prod environments.
- <do other "tweaking">
- bomb our team with questions "why do we have slow RTs?", "why did the env crash?", "why do we have all those errors?", "why has this been overlooked in PLABs?!?"
If you're improvising despite our recommendations, wtf are we doing here???
One day I will crack. Hopefully, not sometime soon.3 -
Infra engineers should be war veterans
I literally spent 4-5 days on CI, DevOps and Intune
The worst part is you don't even know what will happen until the CI runs
4 -
There's this huuuge project I was a part of for half a year. I was kicked off along with a few dozens of other devs (>60% of total manpower) a while ago for particular company reasons.
Now the remaining devs are oh-so-enjoying their time there..
1. workload has not changed
2. deadlines have not changed
3. no one will have Christmas-NewYear vacation
4. a new k8s-based infra is scheduled to roll out to PROD on Dec 23 (k8s is still far from ready - might need a few more months)
The most fun part is that it's not client's mgmt who has decided for #4 -- it's our own....
Boys.. Girls.. Save yourselves.3 -
Inherited a simple marketplace website that matches job seekers and hospitals in healthcare. Typically, all you need for this sort of thing is a web server, a database with search
But the precious devs decided to go micro-services in a container and db per service fashion. They ended up with over 50 docker containers with 50ish databases. It was a nightmare to scale or maintain!
With 50 database for for a simple web application that clearly needs to share data, integration testing was impossible, data loss became common, very hard to pin down, debugging was a nightmare, and also dangerous to change a service’s schema as dependencies were all tangled up.
The obvious thing was to scale down the infrastructure, so we could scale up properly, in a resource driven manner, rather than following the trend.
We made plans, but the CTO seemed worried about yet another architectural changes, so he invested in more infrastructure services, kubernetes, zipkin, prometheus etc without any idea what problems those infra services would solve.2 -
[Applies for RedHat OpenShift Beta on April]
this will be lit I said, since I was quite a bug reporter for Eclipse Che/Codenvy
[Took me until november to open up a ticket so they can take me in]
oooh shit this is lit
[Starts an Eclipse Che Workspace]
huh, why isn't it provisioning?
[inb4 OpenShift devs says its both a OSIO infra issue and a known bug in the OpenShift panel]
oof.
[Makes it work using workaround]
Woo this is gon-
[Haha no, Too Many tedirects for you]
REEEEEEEEEE
Conclusion: Openshift.io is a lit platform, it just happened to be very VERY beta.
I like the kubernetes "pods" tho3 -
FUCK IT
After YEARS of research, I couldn't find a single working load testing tool
So this weekend I created my own. With blackjack and hookers.
It's limited to my app, so not reusable, but wow in 1 weekend I got more data and found more infra problems than in the past 3 years.17 -
Sometimes it's better to burn a bridge so you don't even think about crossing it in the future.
See, I left a company some years ago because I didn't see my future in it and all management combined had a collective intelligence of a chicken.
However, I got a call from them a couple of months ago asking me if I could return. The salary was double and the working arrangement seemed fine. On paper. WFH. Flexibile hours...
Since I actually liked the project itself for its technical challenge, I accepted the return offer. What a bad idea that was.
Of course, the things that made me leave for the first time had only gotten worse. Bad leadership, idiot developers in team leader positions. Tech debt higher than Mount Everest. Bad infra that makes you want to off yourself every time you work on it. The whole circus.
Seriously, the "senior" team leader will happily merge code that includes assert(true == true), but hold up a well written MR because he has a personal vendetta with the developer.
Personally, I always check him whenever he starts being an ass. But the poor juniors are in hell. They're terrified.
Now I'm leaving again, but this time I've made sure I can't come back.2 -
My own colo server. My own cloud. My own infra.
Fuck all of the CSP's and their fucking broken TOS and their data privacy violations!
Why do you think Amazon is so cheap? Because when they discover a product/service/software on AWS makes money, they WILL Reverse engineer it, make it and price you out of the market. It's their business model!4 -
Today was a good day.
I was told to use in-house BitBucket runners for the pipelines. Turns out, they are LinuxShellRunners and do not support docker/containers.
I found a way to set up contained, set up all the dependencies and successfully run my CI tasks using dagger.io (w/o direct access to the runner -- only through CI definition yaml and Job logs in the BitBucket console).
Turns out, my endeavour triggered some alerts for the Infra folks.
I don't care. I'm OOO today. And I hacked their runners to do what I wanted them to do (but they weren't supposed to do any of it). All that w/o access to the runners themselves.
It was a good day :)))))
Now I'll pat myself on my back and go get a nice cup of tea for my EOD :)3 -
Our best dev/arch just quit.
C dev lead & a dev staying late chatting.
Lead: am building, takes long
Dev: unit testing, time taking
Ask them y they r building on their latitudes when we got them linux precision with xeons/64gb workstation 1 each ?
Both: I code on latitude.
Build/test times. (pure Java/maven)
Latitudes=an hr or more
Precision=2m to 11m
Jenkins Infra we have =10 mins with test & push. Parallel builds support.
Am suposed to help with an open mind. They now want Mac pro12 -
(short story)
happened a couple of months ago. I was drawing some diagrams on the board -- planning new infra for the app. While explaining all this to the analyst I needed some random number. It was smth like 1470285206. Half an hour later I need another random number. And I again come up with the same exact sequence....
It was both funny and spooky at the same time. Apparently my biological RNG is utterly broken.. Either that or my subconsciousness wants to tell me something about an early morning of 2016 Aug 4.1 -
Dev: Hey infra guys, our backend time to first byte takes 8 seconds! Can you guys investigate?
Network team: Oh yeah that's normal, we don't investigate anything below 30s, that's our SLA 😎
😵5 -
FUUCCKKKK!! I need to hit smth. Or rant..
So that flaky ec2 issue.. These ec2s act as a shared environment for multiple apps. Our app is one of them. I have no access to those ec2s at all.
What I have access to is my app and some monitoring. Now the app randomly starts lagging while nearly idling. At the same random times monitoring stops completely and doesn't come back up. This happens to random app instances at random times.
Reached out to infra support, managed to get attention from the big boys [mgmt]. Today we got the fix deployed. I test it out -- problem persists.
I find this behaviour somewhat familiar. Managed to get some server stats from infra folks. Apparently cpu% is high as well as load avg [cpu queue]. Bingo! I know how to fix it!
So I write a long comment w/ all the commands and all the 'if that, do this'. Send it to one of the infra technitians
and I get a reply: 'we will apply cpu usage limitations to fix the issue'
wait... Cpu% limitations will do nothing but highlight the underlying problem...
'no, instances have high cpu utilisation which is causing those lags. We will limit cpu resources and it will be fixed'
oh ffs... Cpu utilization and cpu queue are VERY different things.. I tried explaining that to them like 7-9 times. And all I get is:
'yes, cpu utilization is the problem. We will limit it and solve the problem'
I would surely escalate all of this through higher channels if only I could get my hands on those ec2s and have a proof. But that is not happening and I'm forced to sit back and watch them break things even worse until they are out of options and mark my query as 'wont fix'....
Fuck that's frustrating....
*thinking to myself* so I've read about that new vulnerability 2 days ago that allows one to escape from docker container to the host... What if <...>4 -
Migration in progress (long one, lasting over a month)
PM: Let's schedule a meeting to discuss migration progress
PM: Let's schedule a meeting to agree on what should we test
PM: Let's schedule a meeting to get specs of the new infra
Seriously, PMs. One 1 hour long meeting costs at least 4 hours of productive time (1 hour for travel, 1 hour for the meeting, 1+ hour for preparing for the meeting, 1+ hour for post-meeting discussions). And more often than not all meetings end with "We will come back to you later in regards to <some question not answered during meeting>" and it always means "we'll continue this chat via emails"
Why can't you first ask "do we need a meeting or can we sort this out via email?" ??? Or are you intentionally wasting everyone's time?4 -
How surprising is it when a person designs code in a very clear and impressive structure and just when you think about asking them for guidance, they reveal themselves to be complete turds?
I've been working with this person's "infra" code, at work. I've rewritten some classes to use their infra. I had a vague idea of how the classes work. I had no idea of how their code works. Expectedly, there were some issues but now only minor ones remain.
I asked them for a description of what I'm supposed to do for the few bugs I'm facing. They replied in such a condescending tone, it made me want to punch them through the screen.
Almost a month later, we're still going back and forth with emails. I've been swallowing it and responding calmly. I never got direct answers. Always deflections to irrelevant things or veiled insults. I took it because they did correct one silly error of mine that actually my code reviewer should've caught. (What's worse is that it got introduced by me just before my review and commit.)
But does that give them the right to insult me in front of the whole team including my project manager? I got a reply today from them with everyone of note in cc implying very clearly that I have not done any work. They highlighted a line from my code with some todo tag (that was not meant for them) to make their invalid point. A line that's unrelated to the bug I asked them about. This is after I proved them wrong when they insisted that I had done something wrong about a feature related to the bug.
If you don't understand what I asked for fucking ask me to ask again. But do not fucking try establish yourself on higher ground by pointing out irrelevant things in my code.
I was shocked and enraged that they'd do such a thing. I double checked everything like a mad man. Despite knowing that the fix has to come from them, I was instantly transported to the noob stage, grasping at straws. I wanted to send a really scathing reply right away but my manager asked me to wait.
My mind is now a see saw shifting between a panicked noob questioning every fucking thing I ever did in my nada life and a hungry enraged monster looking to maul that fucking shithead for burning me like that.1 -
This is irritating. Fuck you stitchfix. If I were convicted of a felony and did time, my odds of finding a job are basically zero. But for some reason (I can only surmise weaponized wokeness, or has an executives sex tape) they want to keep this fuck on who maliciously deletes half of Cisco's AWS service infra, pleads guilty and is looking at 5 years and $250k in fines.
https://theregister.com/2020/08/...
This isn't even the first time their sourcing of resources has become a problem. Deloitte nailed them just last year with an audit that said their outsourcing had led to effectively no way for them to control their financials or secure customer data. And their response is apparently, double down.
https://wsj.com/amp/articles/...
Fucking MBA fucks. -
Speaking of.. What in your opinion would be an appropriate way to warn someone about security problems, like db passwords in git?
I once came across dozens of extremely sensitive services' infra accesses: alibaba/aliexpress, natuonal observatories, gov institutions, telecomms, etc. I had dozens [if not hundreds] routers' and firewalls' credentials along with addresses. I tried one to confirm validity - it worked. I wanted to warn them but did not want to get in trouble.
If it were servers, I'd set a motd or append some warning messages in .profile. But not sure how to do it for non-server devices
what would you do? How would you warn them?
P.S. Deleting that record was a smart move, buddy ;)
p.P.S. Sorry, wrong category... Can't edit now :(6 -
I explained last week in great detail to a new team member of a dev team (yeah hire or fire part 2) why it is an extremely bad idea to do proactive error handling somewhere down in the stack...
Example
Controller -> Business/Application Logic -> Infrastructure Layer
(shortened)
Now in the infrastructure layer we have a cache that caches an http rest call to another service.
One should not implement retry or some other proactive error handling down in the cache / infra stack, instead propagate the error to the upper layer(s) like application / business logic.
Let them decide what's the course of action, so ...
1) no error is swallowed
2) no unintended side effects like latency spikes / hickups due to retries or similar techniques happens
3) one can actually understand what the services do - behaviour should either be configured explicitly or passed down as a programmed choice from the upper layer... Not randomly implemented in some services.
The explanation was long and I thought ... Well let's call the recruit like the Gremlin he is... Gizmo got the message.
Today Gizmo presented a new solution.
The solution was to log and swallow all exceptions and just return null everywhere.
Yay... Gizmo. You won the Oscar for bad choices TM.
Thx for not asking whether that brain fart made any sense and wasting 5 days with implementing the worst of it all.6 -
I’ve just discover that some dudes in my previous work have puts whole web pages inside Redis.
mfw when Infra cost goes brrrrrrrrrr
4 -
FUCK EVERY PERSON IN THIS SHIT BANK!! FFS THE IDIOTS CREATED A NEW DATABASE USING SQL SERVER 2008! Yes, 2008 and its a new database if it was some type of legacy I could try to understand, but this shit is a completely new database. I have to use sequelize and guess what? It can't paginate results because shit server 2008 does not accept OFFSET FETCH syntax3
-
You can't break into what isn't turned on. We can now scale the admin interface down to zero nodes and spin it up on demand.2
-
Today during a follow-up meeting of the grand project I'm workng on...
TL: ... and I want to start working on the production environment and have it ready by next month.
Me: (interrupts) hold up! We are not ready, we have a huge backlog of technical tasks that need to be addressed and we are still not in possession of the very crucial business and functional requirements that you are supposed to provide. The acceptation environment is just set up on infra perspective but does not have anything running yet! The API we depend on is still not ready because you keep adding change tasks to it. We have a mountain of work to do to even get to a first release to integration yet and there is still the estimations on data loads and systems... your dream will not be possible until at least Q2 of 2024.
TL: stop being so negative @neatnerdprime and try to be more customer friendly. I want it by the end of the next month.
Me: remember what I said to you about moving prematurely. Remember I don't take any responsibility if things break because you rush the project. Please, reconsider!
TL: I just want it, please do it
FUCK YOU YOU SORRY EXCUSE OF A PEOPLE PERSON KNOWING JACK SHIT AND JUST LICKING THE MIDDLE MANAGEMENT ASSHOLE TO RECEIVE ATTABOY PETS ON YOUR UGLY ASS BALD HEAD AND CROOKED TEETH. YOU SHOULD FUCKING DIE IN A FURNACE AND LEAVE NO TRACE BEHIND.4 -
Why is planning so awful at so many tech companies (and in general I guess)?
I’m barely a month in and already having to drop current projects because someone made a promise to someone else and realized at the last minute “wait we need infrastructure for this” and came to me out of nowhere “hey we need this super complicated pipeline and complex architecture built”
I ask boss about how this should fall in priority, he basically says “figure it out”. Okay….thanks.
I ask for the business case from the requesters and when they need it and I’m told “we’re going live in late September”
Go back to boss, tell him what I “figured out” and asked if this was something he intended for me to take on. He skimmed through the ticket and gave a non-committal “I’m not sure what this is”. I’m still trying to figure out the infra here, still haven’t gotten access to half the things linked in the JIRA, but there’s been about 3 email chains and a Director recently DM’d me on slack asking for an update.
So I guess I’m flying in the dark on this one.
If you never hear from me again in this community I probably flew into the side of a mountain or something. This new job ain’t it.7 -
My college is full of assholic admins fucking around!
Increasing the fee, not giving WiFi, shitty mess, bad infra and what not!
Those fuckers are probably corrupted money eating leeches!
Oh god fuck them!
More on their website www.iiits.ac.in6 -
A member of infra team:
"Hey, we are migrating to a Microsoft office tools and we migrated your google drive data to One drive"
I go and check the new One Drive account and it's empty. So I point that out and the reply was:
"You should export your files to a zip and then import them to One Drive"
I didn't want to waste my time showing him that he is just contradicting himself in less than 5 mn and in two nearly consecutive messages.
I need more patience.1 -
Company wants to change to a different cloud provider because of credits :/
Thus rebuilding the infrastructure from ground up.
Worst part is Azure sucks.3 -
New twist on an old favorite.
Background:
- TeamA provides a service internal to the company.
- That service is made accessible to a cloud environment, also has a requirement to be made available to machines on the local network so you can develop against it.
- Company is too cheap/stupid to get a s2s vpn to their cloud provider.
- Company also only hosts production in the cloud, so all other dev is done locally, or on production non-similar infra, local dev is podman.
- They accomplish service connectivity by use of an inordinately complicated edge gateway/router/firewall/message translator/ouija board/julienne fry maker, also controlled by said service team.
Scenario:
Me: "Hey, we're cool with signing requests using an x509 cert. That said, doing so requires different code than connecting to an unsecured endpoint. Please make this service accessible to developer machines and lower environments on the internal network so we can, you know, develop."
TeamA: "The service should be accessible to [cloud ip range]"
Me: "Yes, that's a production range. We need to be able to test the signing code without testing in production"
TeamA: "Can you mock the data?"
Me: "The code we are testing is relating to auth, not business logic"
TeamA: "What are you trying to do?"
Me: "We are trying to test the code that uses the x509 you provide to connect to the service"
TeamA: "Can you deploy to the cloud"
Me: "Again, no, the cloud is only production per policy, all lower environments are in the local data center"
TeamA: "can you try connecting to the gateway?"
Me: "Yes, we have, it's not accessible, it only has public DNS, and only allows [cloud ip range]"
TeamA: "it work when we try it"
Me: "Can you please supply repro steps so we can adjust our process"
TeamA: "Yes, log into the gateway and try issuing the call from there"
Me: (╯°□°)╯︵ ┻━┻
tl;dr: Works on my server -
So the infra folks finally considers OpenStack after doing their own thing that is prone to breaking whenever they add new stuff on the IaaS
ABOUT FUCKING TIME -
The IT at my current work designed infra as such :
One repo for ALL the configs for every project and one config file per project that defines the version of the language (ex node 6) for all environment of a project. I don't even want to talk about deploying previous version or what happen when you update the version and AWS spawns new instances.
Jump into chatops hype approach and use one single script to deploy every app. Talk about a single point of failure but hey we use slack now it's great no?
Since I always think we are one character away from bad deployment and I'm into one click deployment then I've made a web app just to generate command and copy it or send it to slack.
I guess this is what happens when IT work for themselves only..2 -
Your "feature" just became my problem. Your "great idea" is now my migraine. What you did in 300 lines, another team was already doing in 5.
The next time you `brew install...` on your laptop, you should fucking think that the infra team has to install those dependencies, on every server, too.
In less time than it took you to create your code, I could have given you several functions to call. I could have saved all of us weeks of work. Fucking ask cross-team before you cowboy code your next big idea please.
Got a problem you need to be solved, somebody else probably solved it, just fucking ask.6 -
Never tell 60 users on a demo to "click on this button" at the same time.
Infra went "poof". No time to auto scale in this scenario.
Usual usage of app : 4 to 5 personns who do actions in the windows of 2 seconds.8 -
Allrighty, so we have a huge migration upcoming. The planning started early this spring. We've split the whole process into separate tasks and estimated each of them. Also marked all the tasks client should take care of itself so save funds and time. All-in-all the whole thing estimated like 4 months if we did it [single dev, tremendous amounts of communication with various parties, buy and prepare the infra, adapt app to the changes, testing, monitoring, etc.] and like a month if client did the tasks we shouldn't be doing. The funding for migration is time-bound and can only be used before December. Cool! We got notified that by the end of April we should be good to go! Plenty of time to do things right!
April comes. Silence. Mid-april we resch out to the client. Since there's plenty of time left migration is getting lower priority to other tasks. Well allright, sort of makes sense. We should migrate mid-July. Cool!
July comes. Client replies that everyone's on vacation now. Gotta wait for August - will do the quicker version of migration to make it on time. Well allright....
August comes. Everyone's vusy with whatever they've postponed during summer. Hopefully we'll start migration in September. Mhm...
September comes. We're invited to a meeting by project funders to explain tasks' breakdown, justify the time needed to make the migration. We're being blamed for surreal estimations and poor organization of tasks as nothing's happened yet... [they were the ones who always were postponing things....]. Moreover, they can only spare 20% of infra resources required for data alone anf they want us to make that enough for all environments, all components, all backups, all databases,... You get the pic.
The leader of the meeting semi silently mumbled to other participants 'Well then I'm afrsid we can't make a full migration in time.. Only partial. That's very unfortunate, very. That's why we should not have incopetent vendors [*glancing at us*]'
somehow we agreed we'll get the resources mid-November and we should be thankful for him bcz he'll have to pull some strings for... us..
I left the meeting with my fists squeezed so hard! But it's okay, we got smth useful: resources and start date. Although it leaves us with less than a month to do smth requiring a month for a sunny-day scenario. Nvm, still doable.
Last week we get an email that resources will be available at the beginning of December [after deadline] and we should start a full migration no sooner than Nov 12. Which leaves us with 50% of our estimated fucking optimistic scenario time and not enough resources to even move a single db.
Fuck I hate politics in dev... Is it wrong for me to want to tie them to a pole, set them on a veeery slow fire and take a piss on them while they're screaming their shitty lungs out? I'd enjoy the view and the scream. I know I would. And while enjoying I might be tempted to take a burning 20cm diameter wooden stick and shove it up their assholes. Repeatedly. Round-robin. Promissing them I'll take it out in 5 seconds and pulling it out after 2 minutes.
Can I?8 -
Am I Data Engineer or Software Development Engineer ?
I design the infrastructure for analytics data, and I build the infra entirely including an development. Except making reports out of the data.
What I'm supposed to be called ?
Data Engineer ?
Software Development Engineer ?
Definitely not an Data Scientist. Official designation given by company is Data Engineer II. But what I'm ?
Confused, someone help me please.5 -
As a dev, knowing some basic sh** about your infra comes in handy, esp when it's Saturday afternoon and you realise your Thursday release f'd up some obscure integration1
-
Sometimes it's a challenge to show how much work I've accomplished to a non-techie (<- any good nicknames for such people?).
I mean yes it looks like it's pretty simple but there were like ~5000 new lines of code and 2 weeks of work put into getting this thing working perfectly, looking sexy, and moving efficiently all while making sure it protects our infra from idiots like you!2 -
A Rant about my past self!
I thought I never did this shit myself! While restoring a old game server with it's various plugins and infra, I found this randomly.
It was for a Admin Panel that was never used in anything commercial. And I think no one ever saw this reponse anyway since I built the frontend that used it as well.
I'm still embarrassed that 16yo me did this exact thing, I think is just plain stupid today.
https://youtu.be/nSKp2StlS6s
11 -
I'm in a team of 3 in a small to medium sized company (over 50 engineers). We all work as full stack engineers.. but I think the definition of full stack here is getting super bloated. Let me give u an example. My team hold a few production apps, and we just launched a new one. The whole team (the 3 of us) are fully responsible on it from planning, design, database model, api, frontend (a react page spa), an extra client. Ok, so all this seems normal to a full stack dev.
Now, we also handle provisioning infra in aws using terraform, doing deployments, building a CI/CD pipeline using jenkins, monitoring, writing tests, building an analytics dashboard.
Recently our tech writer also left, so now we are also handling writing feature releases.
Few days ago, we also had a meeting where they sort of discussed that the maintenance of the engineering shared services, e.g. jenkins servers, (and about 2-3 other services) will now be split between teams in a shared board, previously this was handled only be team leads, but now they want to delegate it down.
And ofcourse not to mention supporting the app itself and updating bug tickets with findings.
I feel like my daily responsiblities are becoming the job responsibilities of at least 3 jobs.
Is this what full stack engineering looks like in your company? Do u handle everything from app design, building, cloud, ops, analytics etc..7 -
So I was rejected by the management today for promotion to Senior 2 although I have done several major feature developments + infra design and basically end to end ownership.
Reason for no promotion? That's the best fucking part, according to the feedback, the work I performed on the service I created is well-designed,
and the code quality is commendable. However, they pointed out a notable difference in code quality between the micro-service
I built and the rest of the project developed by others. This, apparently, suggests that I lack a strong sense of ownership over the broader product.
First of all, we have super tight deadlines (almost 996), and I burned midnight oil to make sure the service I am in-charge of is designed really well.
Also, how in the flying fuck the other how the inability of others to maintain good code quality elsewhere in the product is being used as evidence against my sense of ownership
and initiative in ensuring high engineering quality for the repository I wasn't even working on
What a delusional management, the entire feedback feels like just an excuse to fuck off, we are not promoting you...
May be instead of doing actual engineering work, I should have just do minimal work and write more design docs / technical artifacts
It is very demoralizing after I worked hard for so many months, product went out really well.. yet when performance review comes, rejected with a petty reason7 -
How to serve a static file when infra setup is something every hard to do?
<script src="/path/to/my/fav/lang.cxx?filename=mystaticfile.js" type="text/javascript" />
... Brilliant !5 -
fuck.. FUCK FUCK FUCK!!!
I'mma fakin EXPLODE!
It was supposed to be a week, maybe two weeks long gig MAX. Now I'm on my 3rd (or 4th) week and still got plenty on my plate. I'm freaking STRESSED. Yelling at people for no reason, just because they interrupt my train of thought, raise a hand, walk by, breathe, stay quiet or simply are.
FUCK!
Pressure from all the fronts, and no time to rest. Sleeping 3-5 hours, falling asleep with this nonsense and breaking the day with it too.
And now I'm fucking FINALLY CLOSE, I can see the light at the end of the tunne<<<<<TTTOOOOOOOOOOOOOTTTTT>>>>>>>
All that was left was to finish up configuring a firewall and set up alerting. I got storage sorted out, customized a CSI provider to make it work across the cluster, raised, idk, a gazillion issues in GH in various repositories I depend on, practically debugged their issues and reported them.
Today I'm on firewall. Liason with the client is pressured by the client bcz I'm already overdue. He propagates that pressure on to me. I have work. I have family, I have this side gig. I have people nagging me to rest. I have other commitments (you know.. eating (I practically finish my meal in under 3 minutes; incl. the 2min in the µ-wave), shitting (I plan it ahead so I could google issues on my phone while there), etc.)
A fucking firewall was left... I configured it as it should be, and... the cluster stopped...clustering. inter-node comms stopped. `lsof` shows that for some reason nodes are accessing LAN IPs through their WAN NIC (go figure!!!) -- that's why they don't work!!
Sooo.. my colleagues suggest me to make it faster/quicker and more secure -- disable public IPs and use a private LB. I spent this whole day trying to implement it. I set up bastion hosts, managed to hack private SSH key into them upon setup, FINALLY managed to make ssh work and the user_data script to trigger, only to find out that...
~]# ping 1.1.1.1
ping: connect: Network is unreachable
~]#
... there's no nat.
THERE"S NO FUCKING NAT!!!
HOW CAN THERE BE NO NAT!?!?!????? MY HOME LAPTOP HAS A NAT, MY PHONE HAS A NAT, EVEN MY CAT HAS A MOTHER HUGGING NAT, AND THIS FUCKING INFRA HAS NO FUCKING NAT???????????????????????
ALready under loads of pressure, and the whole day is wasted. And now I'll be spending time to fucking UNDO everything I did today. Not try something new. But UNDO. And hour or more for just that...
I don't usually drink, but recently that bottom shelf bottle of Captain Morgan that smells and tastes like a bottle of medical spirit starts to feel very tempting.
Soo.. how's your day2 -
Anyone using Huawei Cloud? How good is it? Does it have APIs to automate the infra creation and all?3
-
Started new contract recently, their main product is aiming to be some kind of automation holy grail for business, basically low code nonsense integrating with most of the industry standard tools like sap or confluence. Their entire infrastructure is setup manually, slowly transitioning from on premise to AWS. No infra as code, no playbooks, not even scripts, just "engineers" painstakingly clicking the UI. They don't seem to see the irony of being automation company that doesn't use automation, but I'm having a good laugh at least.
-
Does anyone even use AWS CDK?
I saw something about CDKv2 getting released and immediately made up my mind about it and just want to validate my opinion.
I'm having a hard time thinking of a case where I would need to use yet another layer of bullshit to deploy cloud infra.
It's bad enough with terraform(which I far prefer over cloud formation). But now you can use python or node? What's next, deploying with XSLT?
I'm partially ranting, because I know someone on my team is going to show this as the "new thing" and I'll be stuck maintaining my code...as code--and that really pisses me off.
I'm also legitimately curious on how many of you have run across this being used successfully and for what problem did it solve?10 -
Fun story
tl;dr; analog FTW!
so we've just had a nice game. A few teams internationally gathered together in the aws gameDay. We had aws accounts set up [one per team] and our goal was to maintain our t2.Micros to deal with incoming load. The higher the latency - the less points we get, the more 5xx - the more points we lose. The more infra we have, the more points we pay for it.
So we are quite new in aws, most of us know aws only in theory. And that's the best part!
So at first we had some steady, mild load incoming. But then bursts came up and we went offline. It's obvious we needed an lb w/ autoscaling. Lb was allright, we did set it up and got back online. We also created an autoscaling group and set it up.
Now what we couldn't figure out is how the f* do we make that group scale automatically, as a response to traffic! So we did what every sane person would do - we monitored LB's stats and changed autoscaling group's config manually 😁
needless to say we won the game w/ 23k points. 2nd place had 9k.
That was fun!3 -
A dedicated team has built an "infrastructure" for creating UI for c++ developers in the company. What looks like a poor attempt at recreating what Microsoft did with XAML at first glance, it actually is a horrible exercise in force feeding people the stinking pile of shit that their code is.
The idea is to make it easy to create UI for developers who aren't used to front end development. They should just need to declare the layout. Very noble.
But.
If you want to do anything more than show a checkbox or a radio button, if you dare to define relationships between the UI controls or worse, if you get ambitious with creating a simple UI that uses a lot of similar controls and similar relationships with dynamic content... be prepared to eat your own barf from eating too much of their shit.
Not only do you now need to write front end code (including JS among others), you need to do it with limited or poor support and you have to make sure that it sits well with the house of moist, crumbly cards the team proudly created. Or resort to some very stupid and performance costing "bypasses" that further cripple your application code. Usually you have to do both of these things.
To think that scores of other teams have welcomed this amazing enhancement with full support without any resistance. It's sickening.
I waste too much of energy (and good jokes!) with these people.4 -
I got an offer to be part of an asset management team in a position of "maintenance and reliability engineer" although I would be the IT/Infra/Network responsible. Basically chasing the integrator, and checking if they do their job according to expectations that the company currently doesn't have (🫣). I decided to let you guys decide.8
-
!dev
So my company says they are paying the market rate, and I should be grateful about it. When I went for interviews at "some big companies", I got offers from 2 different companies with the annual pay x2.5 of what I am getting here.
My current company mentality is so shitty that they only want to hire undergrads or people from oversea so they can pay less.
I do not even know what my fucking role is at this point, I do DevOps, I fucking do infra, I also do SRE shit, then I also write fucking code for servers. Yet my pay is slightly less than just "dev" working at good companies. Fuck my company, I am leaving soon.
I am not the only one in the shit show here, our PM also has to write code because most of our developers left. Most of the time, CTO is not even in the office, the company is basically run by some managements below CTO.6 -
Follow-up rant to my company. Today's day is fairly good, so let's talk about infra.
We're building upon an existing open-source project which is not intended to be extended (e.g. plugins).
Our backend-team somehow hacked symfony into the app, which made the actual work a little bit less annoying. But on the other side, there is absolutely no automation. Everything is setup by hand and I need to upload my sources to my dev-server and watch what files exactly are overwritten. Because if not, I accidentally overwrite core sources which will break the whole app, no matter what. If I forget what file I wrongly overwrote, I have no choice but to setup the core from scratch and apply our sources on-top, AGAIN.
The first time setup took me almost five days.
Oh yeah and the team shares one dev server, so whenever I feel like fucking with a mate, I can easily fuck up his system, since everyone has root-rights.
We're required to use windows, but our dev is linux and I am the only knowledgable linux guy. They need cheatsheets (to be fair, I need my powershell-cheatsheet).
We market the same app with some additional functionality, but we also have clients which require their own stuff. This case has never been thought-out, since for these specific clients, we also modify some core-parts. Which makes it a real hassle to add a basic new feature to that special customer.
At least our frontend is somewhat decent. Simple and without critical thinking, but it works and is decently understandable. I'll rant about that for another day, it's still tedious.
I know I won't stay there for long since I start my own stuff, but it's sad. Nothing is perfect and they _do_ want to make it better, but it's the usual "there is no time, client first" talk. On the other hand, they tell that we should be more efficient, but there is no way to be without looking back at the fundamental structure and what takes us so long.
I don't think I am able to change anything here and as I heard from co-workers, they already look for something new.
cheers -
This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...
Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place3 -
Well, I thought "hey, I need to urgently update my own infra for a change, been neglecting this way too long, but should not be longer then a day."
I spent the whole weekend dependency resolving, modulesyncing and ensure deps are met ... And every single goddamn time it's stdlib causing a whoupsie on another module...
Oh at least I am having fun. Sort of. -
Just found out our pre-production environment is the qa for another team.
Infra team has recommended to ignore the nomenclature. It's just a name they said. -
Just a reminder that Terraform is insecure by design and if you even THINK about using it to execute CI/CD deployments not built into the cloud (Jenkins, on-prem CI/CD, etc...), then you're a DOUBLE fool. God i hate my infra team sometimes...14
-
Airflow… airflow… I hate you so freaking much, you are a bloated piece of software that packages wayyy too much and increase the complexity of any solution we built on top of you. Plus unit-testing and integration tests are wayyy too difficult with you. But man… your recent UI changes are a massive welcome.
A year ago I was tasked with either upgrading airflow to version 2 or to migrate the code to another tool. Naively I thought “well might as well upgrade to avoid a rewrite”. Little did I know that the reason airflow didn’t scale out well for us, was due to people over the years not having a grasp on airflow primitives like “pools”, “workers” and “operators”. Ended up refactoring the entire codebase for both infra and DAGs anyhow AND upgrading the beast AND lower cost by a factor of 2 (from $100-$150 daily to $50-$70 daily).
But seriously feels like I could’ve solved the scheduling issue with literally any message queue+decent library (like celery or Faust) and I’d have half the headache.3 -
I have one question to everyone:
I am basically a full stack developer who works with cloud technologies for platform development. For past 5-6 months I am working on product which uses machine learning algorithms to generate metadata from video.
One of the algorithm uses tensorflow to predict locale from an image. It takes an image of size ~500 kb and takes around 15 sec to predict the 5 possible locale from a pre-trained model. Now when I send more than 100 r ki requests to the code concurrently, it stops working and tensor flow throws some error. I am using a 32 core vcpu with 120 GB ram. When I ask the decision scientists from my team they say that the processing is high. A lot of calculation is happening behind the scene. It requires GPU.
As far as I understand, GPU make sense while training but while prediction or testing I do not think we will need such heavy infra. Please help me understand if I am wrong.
PS : all the decision scientists in the team basically dumb fucks, and they always have one answer use GPU.8 -
What wrong with the MNC's. I am trying to create a new folder for email templates in DEV server. I have been busy for two days to take the approval from Manager, Infra team and architect. Are you serious two productive days to give approval to create a folder???
Fuck the system. Fuck you MNC. -
- load tests via web
- load tests via api
- figure out why the fuck hibernate started proxying Blob.class after migration rather than using jdbc implementation, like before
- fix ^^
- reconfigure tomcat to ditch random for urandom completely [still getting econnreset]
- continue conversation with sysadmin, tester, analyst, 2 PMs, infra architect, junior dev
- provide immediate support for analyst and tester as soon as they need it
- provide support to another dev on another project
and that's my today's todo list. I think I need more personalities [more threads] to keep going -
Today I finally found the bug that I was debugging the whole infra for last 3 days. Actual culprit was the first guess, which I ignored. But these low of long debugging makes me feel alive again.1
-
Should I infastructure as code or should I not? The god damn difficulty of approaching a cloud infra modernization project… so much to potentially do, don’t even know where to start and what is or isn’t relevant in the end… gaah11
-
We need to go live by this month 10.
Yep, infra is ready in production. You can push the events and see the results right away.
PM: Wow, great. Setup staging we need to test it.
Me: FML, staging machines are smaller, can't even start the program.1 -
My own server infra without configmgmt. Thanks to puppet i have been able to finally give my brain some more compute resources for other things in life, because managing them all by hand is almost the equivalent of a medieval monk copying the bible over and over again.
Now i can manage in the tens of thousands at relative ease. -
How did you get the people from Info Security and Compliance on board this continuous delivery thing ?
I am being asked to run antivirus scans on my own code and binaries as part of build.
Is this common practice? Am I missing something?
I am going to deploy stuff on Azure PaaS. I can understand having malware scan agent on azure VMs scanning the infra, but this?4 -
Colleague of mine deployed application using @reboot target in crom, LD_PRELOAD and screen. The only positive thing is I'm not responsible for infra :-)
-
How effective are visualizations for monitoring infra on AWS?
https://reddit.com/r/aws/...
Can visual infra monitoring be effective?
Pros and cons? -
untreated testicular cancer whore ass scrotum looking mouth wrapped in a cocksucker's infra-lingual malignant tumor vomiting a stupid stinking pussy fart of a retort can unequivocally swallow a vast jungle of hairy fucking balls4
-
Being stuck on bitch work like getting infra to decommissioning AWS stuff and figuring out test plans for emergency cutovers
-
Why in the fuck does everyone expose specific ports in Dockerfiles?
If I wanted to expose the port, I would fucking expose it.
Currently can't run my home infra platform because I'm running two separate instances of Maria DB on the same private internal network. These are two databases for two separate applications.
Why don't I run them on one? Because they're two separate fucking applications.
Why the fuck can I not do this when I used to be able to do it a week ago.
Stop exposing your fucking ports in your fucking Dockerfiles.
This shit is getting so bad, I'm just about to throw my towel in on all fucking containers and just install everything in multiple VM environments.
I am God damn appalled that after 8 years of using docker, core concepts like a port exposure is being leveraged as a way to somehow circumvent poor security practices.
You want a secure container environment? Expose your own goddamn ports.
Fuck you Maria DB, and fuck you docker.2
Top Tags
Weekly Rant
View