So here I am, skrewing around with the Google Authenticator app and the dodgiest base32 code generator I've ever built and generating a 56 char unique ID, and a 8 digit time based code.

WTF, all these products, services and logins that use 6 digit codes... and this fucking thing can handle 8 without breaking 😑

Now... to hook it into a QR code class... and spit out an image I can actually scan, without calling google charts api.

I can't say I've written one of those before 🙃

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

I got a PDF job description from a recruiter which has an invisible paragraph

> How to Apply
> Please send CV to opportunities@acme.com referring to the Integration Developer role.

acme being stand-in for the employer of course (_not_ the recruiter)

What does this even mean?

I found out that apache had built-in support ( via a module - mod_md ) for automatic TLS certificate management with Let's Encrypt since October 2017.

Bloody Hell! Why didn't I hear of this sooner?

So, I ran off into my cloud to set up this so-called ManagedDomain ( mod_md ).

Found the module in the package repositories, installed it and started testing it out.
I started writing IfModule conditions under mod_ssl so that I wouldn't have to overwrite my existing TLS configurations ( which was already issued by Let's Encrypt via certbot, by the way ).

After a whole night of twisting and turning with the configurations, it turns out that the module in the package repositories were built for ACMEv1 and that API has been dead for as long as the module has been around.

I had noticed that the module was 'experimental', but I still hoped that they had the packaged the module.

Finally, I cozied back up with certbot. At least, until this so-called mod_md becomes stable and mainstream.
I hope certbot doesn't make a fuss. I'm sure, it got offended that I was trying to cheat it with mod_md.

Maaaan, we all knew it was coming, we were warned, again and again, yet still, when Lets Encrypt's old root CA expired today, we found out a tool we were using to get new certs (Not cerbot, custom wrapper around acme-tiny) included the old root in the chain.

So... A few hours ago, some of our servers started having connection issues.

Great final 3 hours of today. Better luck next time I guess? Still, despite the little hickup, Lets Encrypt still remains as one of the biggest revolutions in the adoption of SSL, they're the good guys.

Let me run something by all of you. Let's say you once started freelancing as a "Plan B" in case your full-time gig dropped you. Over 12 years you've managed to build a long-standing personal brand around that occasional freelancing. You have several clients who adore you and the work you do and they tell you they would be lost without your talent and have nowhere else to go and nobody else they trust. You know, because in the past you tried to send them elsewhere (for various reasons) and they just kept coming back.

You get laid off from the full-time gig and ACME Company calls and interviews you as a top candidate they're really interested in for that same type of work for a full-time job they're offering.

Here's the catch...if hired, you have two months to basically erase your personal brand and agree never to do any freelancing work as before, even on your own time on evenings and weekends. ACME wants your full focus and attention. Additionally, you find out that the person you'd be replacing is being let go because they weren't sufficiently tech-skilled for the job. And, with a little digging, you find out that person _also_ had several freelancing gigs going on the side. Probably for the same "Plan B" reason. Which is probably why ACME is demanding exclusivity.

Your client base is small. ACME says "we don't care". The work you do is 90% automated and easily achievable in just minutes a day on a weekend or evening. ACME says "doesn't matter". You already had full-time work to begin with so you weren't doing a ton on the side. ACME couldn't be less interested in this "excuse". And you're not keen on the idea of burning down your brand, especially with no guarantees of any kind in the present IT industry hiring/firing/layoffs climate. ACME says this issue is make or break for them.

If you get to the offer stage do you:
a) Flip the bird to your brand and clients you've built up for over a decade and memory-hole it?
b) Negotiate a non-compete clause with ACME, agreeing not to take on any new clients while working full time for them?
c) Flip the bird to ACME and look for something else?

Asking for a friend. ;)

So I finally got a rant to tell, about me myself and I. Were working on my web host (personal gladly) and was trying to get ACME working without root access. I messed up and somehow got a folder named “~”, in a directory. I thought “well that folder is unnecessary” and I ran “rm -rf ~”. The moment I pressed “enter”, I realized what could’ve happened, and it did happen. My whole web-root gone. No backups. Just a big facepalm on my forehead ...
Take lesson, fix backups...

So I'm building this environmental monitoring system for one of the Labs to monitor Temperature and Humidity. the "software" that comes as part of the package with these sensors is really just a website you host yourself if you don't choose the cloud option. No big deal really, (see my previous rant about getting windows server through SSC) I setup IIS and get the "software" registered get a couple sensors running looks good. However I don't like the error messages that popup because it's unsecured. do some reading and I find out that most browsers will give you a warning if your not using HTTPS even if it's for internal use only. OK we'll how hard can it be in implement encryption, turns out it's not that hard and you can do it for free how with letsencrypt and other places. I like free, now i have to use SSH to get into the server and run an ACME client. Hey open SSH is part of windows now cool, download an ACME client SSH into the server and nope doesn't work. Oh right I'm behind a corporate firewall and a bunch of other shit I can't control. Why is so damn arduous to setup this god dam internal website and the problems aren't even the site. Now I'm playing with AWS spinning up an instance to be able to try and get an SSL certificate just so i don't have to tell people it's OK to trust this site ignore the big angry warning.
Best part is other similar internal sites don;t use SSL and all have big messages about someone stealing your soul if you go there and these are commercial systems that run all the HVAC for all the campuses across Canada.
I need more Tylenol.