In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

  • 6
    PrivateTmp is insane.

    I'd similar issues....
  • 4
    As its a breaking change it should come with a big warning and unless being actively exploited be default off.
  • 3
    @Aldar one lil thing: the _upstream_ maintainers (Apache) do provide via a module a system unit.


    The _debian_ (or downstream) maintainers integrate (or create) this file with modifications.

    The trouble stems from SystemD: Downstream doesn't want to deviate from default behaviour, since it is a maintainer burden (they'd need to d
    decide and support every deviation, and there _will_ always be a negative backslash, since some will not like the decision made).

    Upstream could make a decision, but it would require that SystemD becomes a dependency with a minimum version requirement - the version in which PrivateTmp was introduced.

    I think SystemD is at fault here. While additional features might be nice, they shit on version compatibility / semantic versioning.

    Such a breaking change by default might be security wise great. But it fucks downstream and upstream in the worst kind possible.
  • 2
    So you disabled that SystemD feature, right?
  • 2
    @IntrusionCM if I understand you correctly, then it means private tmp is... On by default? And neither upstream (Apache Foundation), nor Downstream (Package maintainers) would dare to change it for reasons you put forth? It... Does make sense, but I didn't know this would be on, by default...

    @Fast-Nop in a way, not by modifying the package's unit file, but by adding an override into /etc/systemd/system/apache2.service.d/no-private-tmp.conf

    This way, the package's unit file may change yet this modification would remain.
  • 1
    @Aldar Bingo.

    I think it was version 212, but not 100 % sure...

    And your override is the correct way.

    Never overwrite an service by placing your own service file under /etc/systemd...
  • 1
    Having a web server depend on a particular init system is sheer madness.
  • 0
    "friday afternoon [...] have just migrated"

    oh boiii, here we go again =D
Add Comment