One step through the door my wife whips around, a look so disgusted she barely seems human. "What's that smell?" she cries. "It's you! You smell like...like bad code!"

Indeed, I am covered with the scent of the forbidden love child of a man who read half a chapter on if-then statements and then pushed out into the world, earthworm-like, a mangled misshapened gelatinous mass that my employer gave the title of line-of-business application purely out of pity.

For more days than I'd like to count I have been porting a ColdFusion 5 application to .NET. Initially written in 2000 and last touched in 2006, it has a data architecture comparable to Dresden after the second world war. It features a table solely comprised of seven columns of IDs so that joins can be made between other tables lacking a common key. Columns that should be contained within a single table spread out among multiple tables. Single columns containing data that should be multiple columns (with handy flags to separate the subsets). A view with 14 joins that playfully displays unintended results. And so much more spread out over almost 200 stored procedures, views, triggers, and tables on the SQL server, and dozens of additional ADO-like SQL statements within the ColdFusion itself. Fortunately, the application overcomes these issues by having absolutely no data validation while allowing nulls pretty much everywhere.

When I am done this will be a very nice ASP.NET MVC app with at least 150 less stored procs, views, and tables. Auto-generated duplicate entries will be a thing of the past. Pop-up windows that inexplicably refresh the underlying screen to display a different part of the program than the one the user wants will be eliminated. And a UI based on the colors of a Rubik's Cube with usability that Mr. Rubik would find challenging will disappear with only the trauma of using it left behind.

Sadly, this is not my worse legacy code experience. Just the most recent. Just the most recent stench added to a lifetime of bathing in code rot.

Me: JavaScript, you are crazy...

JavaScript: honey, you can not live without me... If you survive without me, leave now!!!

(That was three years ago)
and I am still writing #JavaScript applications

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

Don't you love it when banks decide to change their "security" policies without any warning?

Logging in via private wifi? Forbidden with a nebulous message. You better download their banking app or connect to your router via cable. As if the app they keep pushing wasn't a complete dumpster fire.

Need to transfer money? Okay, just register the account and the limitations will be removed after 24 hours... Does it say 24? I'm sorry we updated our policy to 72 hours and haven't bothered updating the message.

I swear one of my new year resolutions will be to switch banks asap