Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "servers"
"Hey, Root, someone screwed up and now all of our prod servers are running this useless query constantly. I know I already changed your priorities six times in the past three weeks, but: Go fix it! This is higher priority! We already took some guesses at how and supplied the necessary code changes in the ticket, so this shouldn't take you long. Remember, HIGH PRIORITY!"
1. I have no idea how to reproduce it.
2. They have no idea how to reproduce it.
3. The server log doesn't include queries.
4. The application log doesn't include queries.
5. The tooling intercepts and strips out some log entries the legendary devs considered useless. (Tangent: It also now requires a tool to read the logs because log entries are now long json blobs instead of plain text.)
6. The codebase uses different loggers like everywhere, uses a custom logger by default, and often overwrites that custom logger with the default logger some levels in. gg
7. The fixes shown in the ticket are pretty lame. (I've fixed these already, and added one they missed.)
8. I'm sick and tired and burned out and just can't bring myself to care. I'm only doing this so i don't get fired.
9. Why not have the person who screwed this up fix it? Did they quit? I mean, I wouldn't blame them.
Why must everything this company does be so infuriatingly complicated?11
I’m surrounded by idiots.
I’m continually reminded of that fact, but today I found something that really drives that point home.
Gather ‘round, everybody, it’s story time!
While working on a slow query ticket, I perused the code, finding several causes, and decided to run git blame on the files to see what dummy authored the mental diarrhea currently befouling my screen. As it turns out, the entire feature was written by mister legendary Apple golden boy “Finder’s Keeper” dev himself.
To give you the full scope of this mess, let me start at the frontend and work my way backward.
This function allows the user to better see the rows in the API Calls table, for which there is a also search feature — the very thing I’m tasked with fixing.
It’s worth noting that above the search feature are two inputs for a date range, with some helpful links like “last week” and “last month” … and “All”. It’s also worth noting that this table is for displaying search results of all the API requests and their responses for a given merchant… this table is enormous.
This search field for this table queries the backend on every character the user types. There’s no debouncing, no submit event, etc., so it triggers on every keystroke. The actual request runs through a layer of abstraction to parse out and log the user-entered date range, figure out where the request came from, and to map out some column names or add additional ones. It also does some hard to follow (and amazingly not injectable) orm condition building. It’s a mess of functional ugly.
The important columns in the table this query ultimately searches are not indexed, despite it only looking for “create_order” records — the largest of twenty-some types in the table. It also uses partial text matching (again: on. every. single. keystroke.) across two varchar(255)s that only ever hold <16 chars — and of which users only ever care about one at a time. After all of this, it filters the results based on some uncommented regexes, and worst of all: instead of fetching only one page’s worth of results like you’d expect, it fetches all of them at once and then discards what isn’t included by the paginator. So not only is this a guaranteed full table scan with partial text matching for every query (over millions to hundreds of millions of records), it’s that same full table scan for every single keystroke while the user types, and all but 25 records (user-selectable) get discarded — and then requeried when the user looks at the next page of results.
What the bloody fucking hell? I’d swear this idiot is an intern, but his code does (amazingly) actually work.
No wonder this search field nearly crashed one of the servers when someone actually tried using it.
Me talking to a recruiter (even though I am not looking for a job)
Me: If I walk into an interview, and they ask me to reverse a binary tree for a frontend Reac or Vue position or something along those lines, I will end the call and/or walk away from it.
Him: I get similar feelings from other programmers, I don't quite understand why the notion is as common
Me: Because it is fucking useless, it servers no purpose to a dev to know about that when building frontends with react, I link my github profile, for which they can find advanced backend-frontend related projects, compiler and interpreter projects, plus the title I currently have at my workplace and a bunch of other shit, I am not interviewing for a teaching position at an institute, but an actual place of work, for which if they want to know about DS and A they can review my profile which has a repo of DS and A in about 5 different languages including plain C++. I do not need to be offended by such notions since they server no purpose on the frontend, and neither do other devs. If anything it should be a casual conversation during the interview, not a basis for employment.
Recruiter: .........thank you for explaining this to me, I am sure I can bring it up to the agencies doing the reviews and interviews. Are you still interested?
Me: Are they going to give me a coding assignment for a project or a bs question like what I mentioned?
Him: I don't know
Me: then I am not interested12
We passed a milestone: 250,000 phpunit testcases.
If it weren't for a heavily parallelized build pipeline which splits it out over 20 servers, it would take about 7.5 hours to complete.
Not hating on PHP, and without tests it would truly be hell...
But still, fucking hell, we outgrew PHP.
Not having a solid type system just means you either accept more bugs, or write thousands of unit tests to guard all the foundational cracks in the system.
On the bright side, I get a coffee break after every commit 😄22
Darn it, I was having such a good day. Just sitting over here in sysadmin land watching the Java devs tear their hair out over the Log4j vulnerability, when someone just had to ask me about the Jenkins servers my team maintains.
Jenkins doesn't use Log4j! What a relief!
Jenkins does, however, have third-party plugins, some of which use Log4j. And thus my relief was short-lived and now I'm also tearing out my hair trying to patch this shit.18
What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!
I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.
Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.
So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.
Lunch time already. But the fun isn't over yet!
I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.
After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."
So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".
While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.
Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.5
Software is such an awe inspiring concept if you really think about it. We literally create our own reality from scratch. Binary for the wrong architecture? Don't worry we can emulate it. Network? Fully software defined. Heck, the servers don't even know if it's real or all in software. You know what? Fuck it! The machine isn't real either! All virtualized or software emulated
I'm really bad at putting things into words but the idea of software truly amazes me6
DEAR CTOs, PLEASE ASK THE DEVELOPER OF THE SOFTWARE WHICH YOU ARE PLANNING TO BUY IN WHAT LANGUAGE AND WHAT VERSION THEY ARE WRITTEN IN.
Background: I worked a LONG time for a software company which developed a BIG crm software suite for a very niche sector. The softwary company was quite successfull and got many customers, even big companies bought our software. The thing is: The software is written in Ruby 1.8.7 and Rails 2. Even some customer servers are running debian squeeze... Yes, this setup is still in production use in 2022. (Rails 7 is the current version). I really don't get it why no one asked for the specific setup, they just bought it. We always told our boss, that we need time to upgrade. But he told every time, no one pays for an tech upgrade... So there it is, many TBs of customer data are in systems which are totally old, not updated and with possibly security issues.9
Is this learning job cpu intensive or memory intensive?
I don't know and I don't give a flying fuck, because it's 6:20pm and I have not found any of my favorite servers free to rerun this shit the whole fucking week, so this server (which I have actually killed before, btw) can suck a dick and do its fucking job.
Following on from yesterday's catastrophe...
It happened again overnight!
And THEN another server failed when backing up the DBs...turns out 2 websites have activated logging .. each of around 20GB in the database!
So this morning both of our servers are down for different but similar reasons.
It's absolutely fine everybody.
God bless being a student. I just moved a massive calculation to uni's jupyter servers. Saved me from a shitton of effort and burning my laptop down. 🙏
Got drunk during work time because fuck openvpn, fuck anyconnect, and fuck badly designed vpn servers in general.
A fucking pptp connection, man, a fucking pptp.8
just saw a tweet praising a company because of their choice to use swiss servers and they had a pompous sentance in parenthesis like (upside to banking secret culture)
like, dude, at the end of the day, guaranteed their 'server' is just a linux box somewhere, just like anywhere else in the world just STFU
god i HATE ignorance, hype, and stupid tropes that managers just automatically subscribe to with their 2 brain cell NPC brain4
The most C++ I know is from 5 weeks of 'learning' it in college. Now I've been handed a legacy C++ project from sometime before 2004 and am expected to figure out how it works, update it to either a newer C++ version, and compile it to NOT a 16 bit dll (like the current version is) to replace the one on our servers.
Ummmm... wish me luck2
So, yet another "senior" web developer employed by my contractor who utterly fails to understand CORS.
I mean, easy enough to config their servers to provide the headers. A good and quick buck.
But I swear the level of idiocy I find in so called "seniors" infuriates me. I swear, he didn't even figure out that
A) you can't make the browser omit the Origin header.
(But it works on curl 😭😭😭)
B) it's the *server* who must include access-control-allow-origin in the response, not you in the request. Like, what use would that be? I don't even...
I guess if I ever need to hire web devs again my only question during the interview will be "explain CORS to me".8
New developers. Tip: There is no silver bullet.
If you like Python, please understand GIL's behavior before making a system that handles thousands of requests.
If you like Java, know that "Write once, run anywhere" is a fallacy. Even application servers don't like the same WAR.
If you like PHP, understand the life cycle of a request before connecting to the database from all corners.
If you like C#, don't make it a small command-line application that will be used on FreeBSD.
If you like C, meet valgrind.
If you like C++, templates are cool, but don't overdo it. And take the opportunity to meet valgrind.
Never use the same tool to do everything. Elect the language and framework for the given need with rationality.
Every time I see a "Java Man", a "C++ Chad" or anything like that, it comes to mind that if he were a carpenter, he would be tightening screws with hammers.
Every lock-in is bad.11
them: welcome new project members, this is our CI/CD pipeline which is completely different from the rest of the company, there won't be any great knowledge transfer, we just expect you to be able to know and use everything. but also, we expect you to work on your tasks and don't waste any time.
me: okay, so my tasks aren't going as fast as expected, because I need to invest some learning so i can set up my project correctly.
later: some help would be nice, i'm stuck right now
coworker: *helps me to fix my problems, which were partly due to misconfigured build servers* i know it's a lot, and unfortunately, for this topic sources on the web aren't so good. i can really recommend this book, this will give a deeper understanding of the topic.
me: okay, yeah i mean, tbh, i'll read the book if the project invests some time for me so i can learn everything that's required, but this won't happen. also, some initial workshop on the topic or anything would have been nice.
coworker: well, i mean, i am a software developer. for me, it is normal that i learn all that stuff in my free time. and i think that's what the PM expects from us.
me: okay, that's fine for you, i mean, if i'm interested in a topic, i will invest my private time. but in this case, PM would just expect me to do unpaid labor, to gain knowledge and skills that i can use in this specific project. i'm not willing to do that.
it's not that i don't want to learn. the thing is that there isn't any energy left by the end of the day. i'm actually trying to find some work life balance, because i don't feel balanced right now, haven't felt since i started this job.
also, this is only one of several projects i'm working on. it's like they expect me this project has top priority in my life. if it wasn't so annoying on different levels, maybe i'd have a more positive attitude towards it.
also, at the moment i find it fucking annoying that i have to invest so much time in this dev ops bullshit and this keeps me from doing my actual work.
if they are unhappy with my skills, either they can invest in my learning or kick me out. at this point, either is fine for me..12
I often wonder if our clients seriously think we have an all-knowing crystal ball of wisdom when they send tickets like "Cannot send emails - please check" while they have like 10 servers and email delivery is a complex matter on its own already.
If I didn't care what our clients think, I'd reply with equally informative email of "Maybe, who knows"...1
PM/PO/SM/dev(new!): so i've continued working on feature X, i did this and that (shows screenshot of UI in Jira) and for today i'm planning to do XYZ. anyquestionsnothennextoneplease.
me: sorry, i have a question. did you persist your changes in our repo?
PM: no, not yet.
me: okay, please do this, you've spent several days of work on this. i mean, it's fine if build servers don't build it yet, that's what our feature branches are for.
PM: you're absolutely right, and i will definitely do it at some point.
me: at some point?
PM: yes. dev x, your update please.9
My manager has explained in detail to me how to fix the problem I'm having. I know how to fix the problem. I just don't have permission on our servers to change the settings I need to.
Guess who does?2
FUCKING MICROSOFT IIS SHIT.
I'm a .NET dev since 13 years and EVERY FUCKING TIME STUPID IIS MOTHERFUCKER AND STUPID WINDOWS SERVER have a different problem setting up because of some permission.
You can't never get a site up in IIS without loosing time and patience having weird 400/500.x errors because every fucking machine have to set up some tweaky and hidden permissions.
I have 2 identical fucking win servers and deploying a .NET core applications and on one works (test server) and obviously, on the production server it gives troubles.
FUCK YOU MICROSOFT FUCK YOU I would take the IIS devs personally here and whip them to death until they don't resolve the fucking thing4
Today I learned that docker makes all ports publicly available by default on Ubuntu servers using UFW.
Why? Because for some reason docker bypasses the UFW and has done so since 2014.
Thinking about this, I'm a bit irritated to say the least. Infuriated about such reckless behavior would be another reaction.
Anyhow, in case you have docker running on some forgotten Ubuntu server without a dedicated FW/VPN see https://github.com/chaifeng/... for more details.12
A little update after yesterday's catastrophe:
No catastrophe today (so far). Managed to clear some space on the servers, and the backup ran correctly overnight.
Also...and I'm still checking this... but I think I've just received a pay rise.
wait....is today...a good day?2
There's one thing I hate about the new year. Everyone assumes it's the best time to do some cleanup, some maintenance. As the year starts fresh - everyone should start fresh, right?
No, Hetzner, you don't need to restart my servers on Dec 26th.
No, local VPS providers, you don't need to clean-up all my /tmp/ files for me on Dec 29th, leaving me in the dark and unaware I have to restart my apps.
That's just a dick move. Your intentions might be alright, but the consequences... Damn you!3
Wondering how many old smart TVs have log4j on them and are running web servers that log local requests...8
IT department of client still doesn't get its shit together. Previously, I've ranted that they insist I access their GitLab through a fucking RDP.
Me: requests an account to their Confluence space
Them: give me a Confluence account. Naturally, Confluence requests that I confirm my email. That needs to be confirmed in the inbox of firstname.lastname@example.org. Mail servers hosted by Azure, using Outlook.
Me: ok, let's configure my Outlook, 2FA as they configured to demand it from me... install MS's authenticator app, ok so far so good... Now I'm ready to login and find that email from Confluence and... ERROR 500 INVALID LICENSE
Fucking hell. You just love your siloes so much you actually make it impossible to access it and feel good about my own good will.
I made my passion my job, programming servers & web dev. Although it has been productive economically it has sucked the fun out of programming servers for me...so as a way to rediscover my passion I'm giving game dev a try. After a couple of weekends playing with a game engine this is what I've got, a monkey dev with a suit that jumps from project cabinet to project cabinet avoiding hazards, drinking coffee and trying to make some money (someone told me I should express myself and I took that personally).
I'm pretty much done but the hazard placeholders (a box and an arrow) don't convince me so I wanted to see if my fellow disenfranchised developers had some ideas of what my developer should be avoiding/being hit by, preferably something I could draw easily since as you can see I'm not much of an artist although I've also though of just words falling like "deadline" or something.
Anyway any feedback is welcome, take it ez I've never drawn anything more than a stickman and this is my first attempt at something playable. Small Rant plus question. Happy Monday.14
Brazilian health ministry got a ransomware attack this night.
Why? Not because every city is demanding you to show you're vaccinated in order to go somewhere. Because you have to show it using a 20+ year old system.
Don't get me wrong the UI is nice.
But the servers...
Well, at least I have a document where my shots are registered.
And good luck to us living in this country, where we're known for gorgeous cities (people too) but also for a government that earns 200k+ while working 2 days a week and can employee 40+ people for sitting there and do no fucking shit.
No wonder if you get bad news from here every now and then, it's all true.
The ministries are dumb.
The president is dumb.
And worst. People too.
People don't care. Because they don't know they are part of 94% of more than 200 MILLION that earns minimum wage and strive to live bc the country BUYS things that we ALREADY PRODUCE and have to put a tax to every product to compensate them paying 5x times more to buy in dollars.
At least I'm not depending on this sucker of government, never cared about it.
You guys deserve to collapse and become poor again6
So Yay just asked me to replace pulseaudio with pipewire. I was hesitant because I have meetings to attend and I don't want to have to fuck around with my audio config once again.
But it simply works.
PipeWire can replace pulseaudio simply by uninstalling pulse and installing pipewire-pulse.
Next I'll see if it can replace JACK as easily. If so, if I no longer have to juggle two fundamentally incompatible audio servers to do audio processing, then FOSS has just solved one of the greatest obstacles in its path to reach feature parity and performance superiority to Windows.7
What is your home network infrastructure like?
Ethernet or is everything WiFi? Is there a dedicated firewall? Servers running on a raspberry? Do you have VLANs configured?30
What the hell is WRONG with Windows 10. Why does it need so much storage space? I get to only use 219+38.6+13.8 GiB and Windows gets to use 564 GiB of data to piggyback on data and storage space to push nonsense updates to user who do not want them. Use your own fucking servers, MS. I wish this fucking OS burns in hell.10
One of our servers had a disk fail this week. Luckily it's 1 of 3 in a RAID5 array. And, luckily, it was our mostly-dev box and didn't have any production stuff on it, except for some support things. We scheduled a disk replacement with the hosting company, took everything down, waited. Somebody at the hosting company apparently didn't know we'd scheduled the replacement, saw the machine was down, and brought it up again. Sigh. Finally they did the replacement, got it back up, but now we're seeing an ethernet port flapping, suggested they have someone go in and make sure all the jacks are fully seated, maybe one got loose when they were doing the disk switch. Bureacracy reared up again and we got the boilerplate "if there's a hardware issue suspected please boot into rescue mode and run the tests"... sigh...8
I might be new to webdev , but wtf is wrong with imports in js ?
html seems to get the only decent way of dealing with js: all the files mentioned in subsequent <script> tags can access the functions of previous file
but when it comes to those generated html content(aka react projects) and servers, nobody seems to come to an agreement : react guys uses import while server people uses require. and both of these can't be used in the same file : import works in mjs files (or usual files too if type is defined as module) while require works in cjs file (or usual js files if type is NOT defined as module)
so i kind of like imports for its elegance and resembelence to java imports. and i might have got into some errors in unrelated areas , so my package.json has type=module . i want to use some cjs package (jsonwebtoken) and that shit for the love of god won't work with import, so i gotta use it with cjs file and then the whole project can't use that crappy cjs file.
WTAF ? has web world not got matured enough to not have this shitty import export situation?should i write caveman code and convert everything to require(..) ?
Manager encounters problem, hacks a load of stuff together to "get around" the problem. Asks me to fix the issue. I spend a week tidying up what he'd hacked together, then change it so I encounter the problem.
A morning of probing later, the problem?
One of our servers is running 2 firewalls, so we were having network connection issues on just that server.2
Our parents keep reminding us about the time change that happens every half a year. To remember that the clocks need to be adjusted. And we, the children, keep reminding them, that most devices are already connected to the internet and use the time servers for reference. Which surprises our parents every time. 🙃22
Small chaotic startup that never grew up (15 years atm).
Hosts/maintains a number of apps/sites for various customers.
At some point, someone decides that a CMS would be usefull to maintain the content across all products. Forgoing all sense, reason and the very notion of "additional maintenance and dev" it is decided that one should be built in-house.
Fast forward a number of years.
Ops performs routine maintenance on prod-servers. A java-patch accidently knocks out one of the pillars a 3rd party lib the CMS uses for storing images. CMS basically burst in to flames causing a.... significant incident.
Enter yours truly to fix the mess.
Spend a few days replacing the affected 3rd party lib. Run tests on CMS in test and staging environments. Apply java-patch. All seems fine.
When speaking to frontenders and app-devs, a significant hurdle present itself:
All test/staging instances of all websites/apps/etc ALL USE PRODUCTION CMS. Hardcoded. No way around.
There is -no- way to properly test and verify the functionality of any changes made to the home-brewed CMS.
My patch did indeed work in the end.
But did the company learn anything? Did they listen to my reasoning, pleading or even anguished screams for sanity?
Dev: Sends weekly update email on Friday, as drudgingly required by management, outlining that stress level is high recently in the past two weeks and constant polling for feedback and updates is stressing him out, needs to slow down the pace
Management: (ON A FUCKING SATURDAY): "Received your email @fullStackChris"
WOW! THANKS FOR THAT INSIGHTFUL UPDATE! I BET YOU DID! I HAVE NO DOUBT GOOGLE'S SMTP SERVERS ARE RUNNING PROPERLY. AND AFTER READING IT, YOU HAVE THE AUDACITY TO @ ME IN A WHATSAPP MESSAGE. ON A FUCKING SATURDAY. I DON'T WANT TO HEAR FROM YOU, I DON'T WANT TO THINK ABOUT THE COMPANY OR ITS TASKS UNTIL MONDAY. PLEASE STOP.
I mean they must be fucking with me at this point, right guys? Maybe I should start writing stuff like "I need more tasks and more messages throughout the day" then I would probably receive less, I mean wtf is actually going on.7
Someone's guts will be torn out tomorrow and put up on a nice clean razor barbed wire ...
I was wondering what the fucking fuck messed up my brain - till I realized that some dev mixed up the timezone on one of our servers. Dunno how the dev managed it - but the end result was not funny.
Due to the difference in time strings the newer backup had an older timestamp - and vice versa.
Which - when you want to do mass clean up and migration - is a very fucked up thing.
I had to manually check dozens of backups to make sure I got the right ones...
-.- knife goes in, gut goes out. Thx Bart Simpson.8
Others here hates how DevOps pushed parts of operations workload on Devs? Just this afternoon I have to fix a CI issue and then find a way to connect a microservice I built to production MongoDB; I'd be okay with that (I love to thinker with servers) if not for the fact which I have to do it trough leaky and badly documented abstractions put up by the customer. I was having a nice productivity streak but when I have to do this kind of shit the motivation quickly plummets.6
I hate servers that only support EFI boot with a passion. Yes, legacy / BIOS boot is old, but it was so simple. I've been spending hours trying to get EFI boot working on servers with swraid-ed disks and *nothing* works without ugly hackish patches all over!
Anyone successfully got an EFI partition (/boot/efi) on an MDRaid device? D':4
I hate power cuts..
I have an UPS though, so, I don't hate them so much..
Except for ones that last longer than the battery life of the UPS..
Then it means I have to turn off servers..
And guess what, one of them won't turn on again !
I do have a backup server, but it isn't as powerful, and will only cover the basic's..
So, lesson there, always have at least one running spare server as powerful as you need, ready to take over when another server decides its not going to play nice..
This is why I don't like powering things down, ever !7
IT admins of devRant, explain my dumbass the following:
Why would an IT department put servers in a VPN without TLS.
They presume they don't need because muh-VPN.
And then they don't want to hand out VPN connections to anyone and force me to use Citrix RDP 🤡
I know there are security reasons, but is there not a better way? Like goteleport.com ?
Asking for a friend (or several)7
Me at 9:00 AM: wide awake 🔥
Me at 10:00 AM: after spending 1 hour on trying to find out why storage space is low on some of our production servers: ready to fall asleep any second 😴
More and more, I am getting frustrated/depressed from the attitude of our customers who complain, moan and get angry about issues in their infrastructure, while at the same time, refusing to pay more so the issues could be mitigated.
Like, a client's angry with us today for having one of their non-production-critical databases inaccessible for... Hmm... About 8 hours now (So a whole workday).
Like... I get it, some of your employees couldn't work with it offline, but like... What the hell do we do? You keep data from as far back as several years ago in there, without partitioning, without exports, in a mix of innodb and myisam, so when the DB crashes, and its replication has to be reset from zero, reimporting all the data takes hours upon hours, and importing .sql files just takes time.
Or another client who got angry when their app fell out of the internet, cuz one of their myisam-based log tables crashed, and had to be repaired, with data spanning several years back, meaning it took hours to fix...
The more I work with these "basic" and "simple" infrastructure designs that is *not* redundant, or HA, the more I wonder -- How do the big names out there do it? How do you design systems with fault tolerance so a single DB table crash doesn't lead to the whole app getting inaccessible?
We have... One, exactly one, client, who uses MariaDB with Gallera, and that cluster is *amazing*, it just keeps chugging along, without a care in the world. But it cost them quite a lot, as they had to buy 3 DB servers, instead of 1...3
I've read the docs but my tired brain overrided an important detail.
"By default, HAProxy Enterprise will serve these pages only if it initiated the error itself. For example, it will return the page for a 503 Service Unavailable error if it can't reach any backend servers."
I had _the_ return part for interception of the error page from the backend added, not the default override for the error page of HAPRoxy itself.
Took me 4 hours, crying, madness and screaming to realize it.
This week is really wringing the last bits of the gooey slime what should be my brain out...
Another fun part is that I mistakenly thought the delimiter for multiple strings to an ACL comparison is a comma... It's a whitespace.
acl is_evil hdr(host) -i one,two is wrong.
acl is_evil hdr(host) -i one two is right.
I used to write HAPRoxy configurations blindly, today it was more like writing two lines of codes 100000000 times and still doing it wrong TM.
I need new brain.
Anyone got an offer?3
I'm convinced that the PRODUCTION servers can smell your long-awaited salary appraisals from a mile away.
So my company says they are paying the market rate, and I should be grateful about it. When I went for interviews at "some big companies", I got offers from 2 different companies with the annual pay x2.5 of what I am getting here.
My current company mentality is so shitty that they only want to hire undergrads or people from oversea so they can pay less.
I do not even know what my fucking role is at this point, I do DevOps, I fucking do infra, I also do SRE shit, then I also write fucking code for servers. Yet my pay is slightly less than just "dev" working at good companies. Fuck my company, I am leaving soon.
I am not the only one in the shit show here, our PM also has to write code because most of our developers left. Most of the time, CTO is not even in the office, the company is basically run by some managements below CTO.6
What the absolute fuck were you thinking Microsoft?
You're doing everything you can to ensure that those who continue to use Github are flogged and castrated?
What the fuck happened to the SSH clone link that was so easy to keep in all you had to do was *checks notes* fucking NOTHING.
It makes me question choices I have made over the last two years. Like, why don't I just host my own git server at this point? I have a couple servers running and it would cost me next to nothing.
Before anyone says anything about GitLab , I looked. I would be spending three times what I am now if I used them.
At this point it seems like a futile attempt to stay with you. I'm going to start calling you ShitHub now because it's a place where I can't get shit done without some kind of new shitty "improvement".
2022 is lining up to be a spectacular year!
Fuck you Microsoft.8
How do you guys monitor programs on your servers?
For example, I have a raspberry pi zero w running raspbian (headless). On this pi, I have a bunch of discord bots and web scrapers running at the same time. My solution was to run them all from a bash file:
Python3 discordbot1.py &
Python3 discordbot2.py &
Python3 webscraper1.py &
Node webscraper2.js & etc.
Is there a better way I could be running these services? How is stuff like this usually done?9
What's the best Linux distribution for servers? I don't want an UI and that fancy stuff. I just want a 1337 hax0r console.10
How fucked is the internet/web/servers/satellites/cyberworld if there is ever a worldwide zombie apocalypse?6
Can't believe I'm about to say this, but:
Systemd-container is a rather cool SysD extension.
It allows me (Root on most servers) to switch to a customer account in a completely new session, setting all the .profile and .bashrc stuff up, so I can do stuff like control their rootless docker, and no longer have to add my SSH key to their authorized_keys file then re-login under their user.
Tonight I learned that none of our automatically installed systemd-based servers had the /etc/machine-id created, and that that file used to be pretty central in the systemd world.
So that was the warning at the beginning of the boot log about a missing /etc/machine-id! Though until today, everything still worked fine. Only today, the machinectl utility was unable to find the local machine with the machine id missing.
Oops? At least I'll have stuff to fix tomorrow lol.6
In summary, you would use AWS Lambda when you want to run code in response to events, without having to manage servers, and you would use EC2 when you want more control over the infrastructure or when you need specific operating systems or runtime environments for your applications.1
I had the funniest thing today... So our company has some servers off somewhere in a VPN, as well as one server in our own office.
So, for simplicity, S1 is my own laptop, S2 is our office server, S3 is one VPN server, and S4 another.
I want to get a file from S2 to S4. S1 can SSH into S2 and S3, S2 can't ssh into any server, S3 can ssh into S2 and S3, and S4 can't ssh into any server.
So to get a file from S2 to S4, I took the path
S1 pull from S2 -> S1 push to S3 -> S3 push to S4
Part of it was preexisting keys meaning it was easier to send S1 to S4 via S3 than get my pubkey from S1 onto S4, but also S2 not being on the VPN meant I couldn't go straight from S2 to S3 or S4, so I had to route through S1, which I could add to the VPN (I'd sshed into S2 from home and thus couldn't put it on the VPN not to mention permissions, whereas I could put S1 easily onto it)
Twas certainly a fun time :P
Plus, port forwarding from a Docker container on S2 to S2's port to S1's port via ssh was fun to get set up.
Time to document this process :)2
I've searched for this a few times now but can't really find a good solution for my "problem".
I want to update multiple Debian servers / machines at ones or from one machine.
Puppet doesn't seem to support newer versions of Debian and I don't really want to force the updates via cron.
Are there any other ways to do it?1
Somebody: (whinwy) we need something to log into nonprivileged technical accounts without our rootssh proxy. We want this pammodule pam_X.so
me: this stuff is old (-2013) and i can't find any source for it. How about using SSSD with libsss_sudo? Its an modern solution which would allow this with an advantage of using the existing infrastructure.
somebody: NO I WANT THIS MODULE.
me: ok i have it packaged under this name. Could you please test it by manipulating the pam config?
Somebody: WHAT WHY DO I NEED TO MANIPULATE THE PAMCONFIG?
me: because another package on our servers already manipulates the config and i don't want to create trouble by manipulate it.
Somebody: why are we discussing this. I said clearly what we need and we need it NOW.
we have an package that changes the pam config to our needs, we are starting to roll out the config via ansible, but we still use configuration packages on many servers
For authentication as root we use cyberark for logging the ssh sessions.
The older solution allowed additionally the login into non-rootaccounts, but it is shut down in the next few weeks after over half an year of both systems active and over half an year with the information that the login into non-privileged accounts will be no more.7
Cisco Anyconnect can blow me.
I go through the process of connecting to the vpn, username, password, token.
Then it has its pop up "respond to the banner to connect" and I click accept . . . and it does nothing.
So I go through the process again. And this time it says connected
But now I still can't connect to any of my companies sharepoint, SQL servers, Azure Devops, JIRA, etc
And the only solution to that is a reboot.
And this happens swear to god at least every other day.
Like good lord, if I put in my credentials and they pass authentication/authorization, let me do my goddamn work.4
Ok, it's the second day and still the same problem: creating JS apps takes like forever. I think it's maybe with the servers since my Internet is completely fine.
To copy a file from a Windows Phone ( Lumia series. ) to a Windows 10 desktop PC, what are my options ?
Ideally I'd like simple solutions that don't involve lots of complications, or sending files several times around the world to get to my PC, something simple and local would be peachy.
Since my previous phone about 20 years old, involved a USB cable, some software that doesn't like to run on Windows 10..
Surely we have come along a bit since then, right..
Is there some magic app/etc. that gives me a folder on my Windows Phone that I can just copy stuff too and it magically appears on my PC..
Yes I have Wifi, no I don't really have any servers under the dining room table anymore, but there is a Vista laptop running the doorbell..
So, my options ?9
i want to deploy websockets on multiple servers with horizontal scaling. i don't know what to use. redis pub/sub? haproxy? i wanna know your opinions. ❤️6
What getting an AWS solutions architect certification got me:
Acceptance into the 4 top high schools in the area( one of which being in the top 5 public schools in the United States)
1 opportunity at the local college for some research experience
Vasts amounts of knowledge about servers and back end technologies I have never known about
And of course, the most important one, getting all the aunties attention at parties
People like to say that certifications don’t help but they get your foot into the door, it’s up to you to do the rest of the work3
A few years ago we had a fail-over which was successful until we started failing everything back to primary servers. The applications could not start at all.
4 hours into troubleshooting, only to find out some java security files were misbehaving. Update from another server and it worked.
Up to date i haven't understood how it failed
I wish sophos had some sort of out of the box mode for development which avoided slowing down IDEs and build servers1
Despite already having a few years of professional experience dealing with Linux servers, I still, to this day, confuse, which environment file gets sourced and when...
There's /etc/profile, /etc/bashrc, ~/.bash_profile, ~/.profile, ~/.bashrc
I think it's... Bashrc for interactive shells, profile for login shells.
But then I have examples like "ssh user@server 'echo $var'" that... Don't source any of the files!
You can enable user environment files for SSH that get sourced whenever a user logs on through SSH (~/.ssh/environment / environment specified for a key in ~/.ssh/authorized_keys)
Is there some sort of master environment file that gets sourced *every* time, no matter what kind of shell starts?1
So, as per usual, I am not sure what I am doing.
I want to make multiplayer games on the web using web sockets.
But of course I want there to be multiple game servers for horizontal scaling (I'm gonna hit it big)
Specifically for either Node.JS or ASP.Net (or both) how could I manage such a thing where there are 2 servers and 2 users. User A is assigned to server A by load balancer, and user B is assigned to server B. But they play in the same game?
Best I know of so far is to connect both game servers to a redis backing. But this seems like a convoluted way to communicate. I would rather have them both route to the game server (Whichever server the game starts on)17