Sometimes I think back to all the funny shit that happened and how simple stuff fucks everyone

- tired Database engineer deleting (not dropping, literally rm -rf) the database files on the wrong server

- Microsoft delivering viruses through updates

- Pissed and stubborn dev deleting his one line library repo which does something like removing a char left side of string fucking an unmeasurable amount of other projects

- Adobe getting hacked and exposed for storing passwords in plain texts

- a doubled line causing a bug called heartbleed in a fuckton of webservers

- a Tutorial Company getting kicked from github because their repo got so big github staff had to maintain the repo manually

- and an old one: bad code crashed a space shuttle

Boss wants to scale our webservers because it seems they're having performance/capacity issues....

I'VE BEEN TELLING HIM FOR WEEKS IT'S NOT THE SERVERS!!! IT'S THE FACT THAT EVERY SINGLE QUERY HITS A SINGLE MONGODB... AND NO CACHE EITHER... AND THE DB CANT BE ENTIRELY LOADED INTO MEMORY AS ITS TOO BIG FOR RAM ON A SINGLE SERVER...

HOW THE FUCK CAN YOU SCALE IF EVERYTHING HAS A DEPENDENCY ON 1 NON-DISTRIBUTED DATABASE?

That's actually something that happened fairly recently.. just that I didn't have the energy left at the time to write it down. That, or I got my ass too drunk to properly write anything.. not sure actually.

So on paper I'm unemployed, but I do spend some time still on pretty much voluntary work for HackingVision, along with a handful of other people.

At the time, we were just doing the usual chit-chat in the admin channel, me still sick in my bed (actually that means that I wasn't drunk but really tired for once.. amazing!) and catching up to what happened, but unable to do any useful work in this sick state. So, tablet, typing on glass, right. I didn't have any keyboard attached at the time.

One of the staff members (a wanketeer from India) apparently had an assignment in a few hours for which he needed to write a server application in Java. Now, performance issues aside, I figured.. well I've got quite a bit of experience with servers, as well as some with client-server protocols. So I got thinking.. mail servers, way too overengineered. Web servers.. well that could work, I've done some basic netcat webservers that just sent an HTTP 200 OK and the file, those worked fine.. although super basic of course. And then there's IRC, which I've actually talked to an InspIRCd server through telnet before (which by the way is pretty much the only thing that telnet is still useful for, something that was never its purpose, lol) and realized that that protocol is actually quite easy to develop around. That's why I like it so much over modern chat protocols like XMPP, MQTT and whatnot. So I recommended that he'd write a little IRC server in Java. Or even just a chatbot like I attempted to at the time, considering that that's - with a stretch of course - a sort-of server too.

His fucking response however, so goddamn fucking infuriating. "If the protocol is so easy, then please write me down how to implement it in Java."

Essentially do his fucking work for him. I don't know Java, but as a fucking HackingVision admin, YOU SHOULD FUCKING KNOW THAT HACKERS CAN'T STAND LAZY CUNTS THAT CAN'T EVEN BE ASSED TO GOOGLE SHIT!!! If I wanted to deal with cunts like that, I'd have opened the page inbox with all its Fb h4xx0ring questions, not the fucking admin chat!

And type it on a goddamn fucking piece of glass, while fucking sick?! Get your ass fucked by a bobs and vegana horny fuck from the untouchable caste, because that's where you fucking belong for expecting THAT from me, you fucking bhenchod.

But at least I didn't get my ass enraged like that to say that to him in the admin chat. Although that probably wouldn't have been a bad thing, to get his feet right back on the ground again.

Is it just me or is IIS fucking trash?

I’m fairly new to maintaining my own webservers. For the past week the servers (two of them) kept crashing constantly.

After some investigation I figured it was due to someone running a script trying to get ssh access.

I learned about fail2ban, DOS and DDOS attacks and had quite a fight configuring it all since I had 20 seconds on average between the server shutdowns and had to use those 20 second windows to configure fail2ban bit by bit.

Finally after a few hours it was up and running on both servers and recognized 380 individual IPs spamming random e-mail / password combos.

I fet relieved seeing that it all stopped right after fail2ban installation and thought I was safe now and went to sleep.

I wake up this morning to another e-mail stating that pinging my server failed once again.

I go back to the logs, worried that the attack became more sophisticated or whatever only to see that the 06:25 cronjob is causing another fucking crash. I can’t figure out why.

Fuck this shit. I’m setting another cronjob to restart this son of a bitch at 06:30.

I’m done.

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

So I asked this question yesterday to linuxxx and he had some great tips. But I thought there might be some Linux fanatics here who also have some experience working with vps's that you might have somethings to add something 😁

Recently I got asked if I want to help maintain our webservers (they all run Ubuntu 18.04 with standard webserver stuff, nginx, MySQL, php, ssh)

Does anyone know of some tips or share some helpful knowledge regarding maintaining a VPS? / Keeping it healty?

I can't decide on a linux distro because all I've tried are great. Seriously.

I'd call myself a novice-to-intermediate linux user (heavy on the novice part) and since I work as a web developer it's been a great learning experience to use the same OS on my workstation as the webservers my projects run on. (Ie I started out with Ubuntu and a LAMP setup).

The thing is I distrohop ad infinitum... Feels like I've tried out every desktop environment known to mankind (I just can't stop myself when I see a new one or a new take on an old one) and I've dipped my toes in Arch territory to. Loved Antergos when that still was a thing. Found EndeavourOS this weekend, kernel panic ensued. I'm a noob with sudo and that's never a good thing. 😆 (Try out in a virtual machine first you say? Bah. Where's the fun in that?!)

So now I'm on Linux Mint w Cinnamon because why not. (Because it's sluggish and boring, that's why...) I had to just get something up and running quickly so I could get back to work. 😬

But one day in and I'm realising I actually miss GNOME. And Ubuntu feels like home. I would feel much cooler using Arch but honestly I don't think I can be trusted with it. I love tinkering with settings, look and feel and whatnot but I can honestly do that just as well in an Ubuntu/GNOME environment.

Maybe Pop!_OS... could be something for me. 😏

I'd like to ask: What's trending at the moment instead....

Either I'm old and senile and missing something, or there is not really sth new.

Okay, JS might be crapping out new frameworks in their common "Not invented here" diarrhea....

But otherwise? What's really new?

I don't really know. I'm not only thinking about languages and stuff, but even in hardware there ain't really a big thing going on in my opinion.

Hab ich wat verpennt?

(Have I overslept?)

We had an interesting and frightening discussion regarding NGINX, as it is russian software today and that a new trend of a true, actively developed webserver is severely lacking... Apache looks semi dead and most other niche webservers, too.

That's all I've seen as a "trend" discussion in the latest time

I remember someday from a few years ago, because i just got off the phone with a customer calling me way too early! (meaning i still was in my pyjamas)

C:"Hey NNP, why si that software not available (He refers to fail2ban on his server)
Me: "It's there" (shows him terminal output)
C: " But i cannot invoke it, there is no fail2ban command! you're lieing"
Me: "well, try that sudoers command i gave you (basically it just tails all the possible log files in /var/log ) , do you see that last part with fail2ban on it?
C: "Yeah, but there is only a file descriptor! nothing is showing! It doesnt do anything.
Me: "That's actually good, it means that fail2ban does not detect any anomalies so it does not need to log it"
C:" How can you be sure!?"
Me: "Shut up and trust me, i am ROOT"

(Fail2ban is a software service that checks log files like your webserver or SSH to detect floods or brute force attempts, you set it up by defining some "jails" that monitor the things you wish to watch out for. A sane SSH jail is to listen to incoming connection attempts and after 5 or 10 attempts you block that user's IP address on firewall level. It uses IPtables. Can be used for several other web services like webservers to detect and act upon flooding attempts. It uses the logfiles of those services to analyze them and to take the appropriate action. One those jails are defined and the service is up, you should see as little log as possible for fail2ban.)

Json host files of a whole server networks root server passwords under the webservers configs directory open to the public.

just got a "privilege update" now i'm responsible for not only my software but for the webservers too...thanks admin guys...now i have more bullshit to deal with