Happened a few weeks ago but still awesome.

Me and a good friend have a website together but we don't monitor it too much.
He studied with me in the same class but went towards frontend/apps where I chose backend/servers/security. He knows how to do basic Linux stuff but that's about it.

We were at a party when he noticed that our site was offline. Walked over to me (because I manage the server) to notify me so I could look into it said I'd look into it (phone):
*visits site: nothing*
*online dig tool: got the server ip*
*remembered this one didn't have pubkey authentication - after three passwords attempts I'm in*

"service apache2 status"
*service doesn't exist*
*right, migrated this one from Apache to nginx....*
"history"
*ah, an nginx restart probably suffices...*
"service nginx restart"

BAM, site is reachable again.

*god damnit, lets encrypt cert expired...*
"history"
*sees command with certbot and our domain both in one*
"!892"
*20 seconds later: success message*
*service nginx reload*

BAM, site works securely again.

"Yo mate, check the site again"

Mate: 😶 w-w-what? *checks site and his watch* you started less than two minutes ago...?
Me: yeah..?
Mate: 😶 now this is why YOU manage our server and I don't 😐

His face was fucking gold. It wasn't that difficult for me (I do this daily) but to him, I was a God at that moment.

Awesome moment 😊

😐😓 oh shit!

Totally forgot 😳

Happy birthday Lets Encrypt 😁🤓😆
(just about a week late... 😅)

Lets encrypt!

No biggie, just forgot to renew my lets encrypt certs T_T

Am I the only one who's getting more and more aggrevated about how the large youtube channels misinform and make out VPN providers (I am looking at you, Nord VPN, mostly) as the messiahs of the internet? How they protect our data that would otherwise be in incredible "danger" otherwise?

I understand they need clients, and I know most of the YT channels probably do not know better, but... This is misinformation at best, and downright false advertising at the worst...

"But HTTP-only websites still exist!" - yes, but unlike the era before Lets Encrypt, they are a minority. Most of the important webpages are encrypted.

"Someone could MITM their connection and present a fake certificate!" - And have a huge, red warning about the connection being dangerous. If at that point, the user ignores it, I say its their fault.

Seriously... I don't know if Nord gives their partners a script or not... But... I am getting super sick of them. And is the main reason why I made my own VPN at home...

Guys, lets encrypt.

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

Docker with nginx-proxy and nginx-proxy-le (Lets Encrypt) is fucking awesome!

I only have to specify environment variables with email and host name when starting new containers with web servers, and the proxy containers will automatically make a proxy to the new container, and generate lets encrypt ssl certificates. I don’t have to lift a fucking finger, it is so ducking genius

Maaaan, we all knew it was coming, we were warned, again and again, yet still, when Lets Encrypt's old root CA expired today, we found out a tool we were using to get new certs (Not cerbot, custom wrapper around acme-tiny) included the old root in the chain.

So... A few hours ago, some of our servers started having connection issues.

Great final 3 hours of today. Better luck next time I guess? Still, despite the little hickup, Lets Encrypt still remains as one of the biggest revolutions in the adoption of SSL, they're the good guys.

Does anybody uses Lets Encrypt ? Is it reliable ?

Am I missing something here? Lets Encrypt auto renews SSL every 90 days....BUT it will fail if you have .htaccess re-direct set up to https. So you would have to switch off the https redirect, manually renew, then switch it back on again. Thats fucking crazy. I can’t find a way round this. The hosting co set this up but are encouraging people to buy one of theirs when the renewal fails. a cunning plot to get more of their own SSLs. Any ideas?

Do you guys think that lets encrypt is fine or should I buy a Comodo cert for my personal website?