Update (December 14 ,2021): We’ve updated this article with information about the new Log4j version release, along with new exploit vectors, and risks related to all Java versions.
While you were watching the F1 title decider between Max Verstappen and Lewis Hamilton or excited for the Succession finale, companies running the internet were scared shitless.
You might not have noticed it because services like Twitter, Facebook, Gmail, and smaller ones all stayed up. But a bug in an open-source tech called Log4j was (and still is) causing panic amongst the infosec community across the world.
While the bug has affected billions of devices, and companies are scrambling to apply fixes, the open-source community has a raging debate going on about funding volunteers that maintain projects like Log4j.
Before we dive into all that, here’s a brief background about the technology and the issue.
What is Log4j?
Log4j 2 is a Java-based open-sourced logging framework that comes under the Apache Foundation services, so anyone can use it for free. The logging software is used by companies to track activity on their servers (or even client-side apps).
For instance, when you visit a website, your IP address, your browser, and the pages you visit are registered by the logger. This data related to activity can help companies solve any problem with their service.
Since the library is Java-based, there’s a chance that billions of devices supported by the framework might be at risk.
What is the Log4j bug?
The bug, listed under CVE-2021-44228 last week, allows attackers to remotely execute code through a specially crafted string. Since Log4j is so common, cybercriminals can easily manipulate log strings and control the server or the client.
One of the main reasons for this bug’s existence is that some versions of Log4j are capable of executing arbitrary text through a directory lookup protocol (LDAP protocol).
If you want to read a detailed explanation of the issue in simple language, read this Twitter thread.
Explaining #log4j for non technical people, because the internet is burning down and y’all might want to know what’s happening and why there’s all this “${jndi:ldap” stuff out there#Log4Shell#log4jRCE
⬇️
— Emy | eq 🌈 (@entropyqueen_) December 11, 2021
A new report from Bloomberg suggests the bug was first reported to the maintainers of the Log4j project on November 24 by a security researcher working at the Alibaba team.
Who’s affected?
To be honest, I should’ve named this section “Who’s not affected?” Log4j’s open-source nature and widespread compatibility means it’s a great choice and tons of companies use it — including Apple, Microsoft, Steam, Twitter, Baidu, and Cloudflare.
As soon as the details about the vulnerability were out, several people started scanning servers to check if they were open to exploitation.
Mass scanning activity detected from multiple hosts checking for servers using Apache Log4j (Java logging library) vulnerable to remote code execution (https://t.co/GgksMUlf94).
Query our API for “tags=CVE-2021-44228” for source IP addresses and other IOCs. #threatintel
— Bad Packets (@bad_packets) December 10, 2021
Some programmers tested various sites and services to check the potential reach of the attack; you can check out the full list here.
Importantly, as Ars Technica noted, many of these companies have several lines of defense against cyber attacks. So even if some of the servers are open to the Log4j attack, other security systems can thwart the attack.
The issue came to the fore through Minecraft sites, when researchers found out that a specific string passed to the server or even in-game chat could give attackers control of the system. Web infrastructure firm Cloudflare’s CEO, Matthew Prince, tweeted the company will roll out some protection to all customers — including the ones using the free tier.
We’ve made the determination that #Log4J is so bad we’re going to try and roll out at least some protection for all @Cloudflare customers by default, even free customers who do not have our WAF. Working on how to do that safely now.
— Matthew Prince 🌥 (@eastdakota) December 10, 2021
Notably, there are several appliances that run Java server components, and Log4j could be a part of that. So your toaster could be a hacking target too (?).
Data security platform LunaSec has a lengthy technical explanation about the impact of the bug on its blog.
If you were really thinking about how widespread Log4j’s usage is, this tweet will sum it up for you.
Did you know that Ingenuity, the Mars 2020 Helicopter mission, is powered by Apache Log4j? https://t.co/gV0uyE1ylk#Apache#OpenSource#innovation#community#logging#servicespic.twitter.com/aFX9JdquP1
— Apache – The ASF (@TheASF) June 4, 2021
What’s the mitigation?
Hours after the bug was discovered, the security team at the Apache Foundation released a new version (v15.0) of Log4j with patches that secure systems from attacks.
Minecraft’s owner, Mojang Studios, also published a guide for people running the official game client and game servers to fix the bug. It warned that people running third-party game clients might have to wait for the provider to patch the exploit. Plus, the Swiss government’s computer response team (CERT) has tweeted an easy-to-understand prevention guide in a picture format.
Heads up: We see a lot of scanning against the #log4j vulnerability and decided to publish a blog post with some guidance:https://t.co/dtEXfY1G16
Please patch now! pic.twitter.com/HSedlSed0V
— GovCERT.ch (@GovCERT_CH) December 12, 2021
Companies like Red Hat and VMWare released their own fixes to plug the issue. For server admins, severalsecurity blogs have detailed guides for mitigations strategies. A GitHub user has made a super useful list of patches issued by companies, including Netflix, Citrix, Docker, and Oracle.
Apache has released version 2.16.0 of the Log4j framework. This update completely removes message lookups to reduce the threat. Plus, it has also disabled automatically resolving lookups contained in a message or its parameters.
It’s not over yet
While service providers and the open-source community are working round the clock to fix the issue and secure as many systems as possible. The danger is looming over our heads.
As Bleeping Computer noted in its report last night, attackers are installing cryptominers, malware, and deploying botnets for DDoS (Distributed Denial of Service) attacks. Microsoft’s threat intelligence team has noted that many attackers are using the Colbat Strike penetration detection tool for credential theft.
Sean Gallagher, a senior threat researcher at Sophos, also said that the security company has detected many attempts of remote code execution:
Since Dec.9, Sophos has detected hundreds of thousands of attempts to remotely execute code using the Log4Shell vulnerability. Initially, these were Proof-of-Concept(PoC) exploit tests by security researchers and potential attackers, among others, as well as many online scans for the vulnerability. This was quickly followed by attempts to install coin miners, including the Kinsing miner botnet.
The most recent intelligence suggests attackers are trying to exploit the vulnerability to expose the keys used by Amazon Web Service accounts.
Security researcher Greg Linares tweeted that threat actors might be building a worm to attack many servers through the Log4j bug and cause damage or demand money.
#Log4J based on what I’ve seen, there is evidence that a worm will be developed for this in the next 24 to 48 hours.
Self propagating with the ability to stand up a self hosted server on compromised endpoints.
In addition to spraying traffic, dropping files, it will have c2c
— Greg Linares (@Laughing_Mantis) December 12, 2021
The Cybersecurity and Infrastructure Security Agency (CISA) has also issued a warning stating that the bug “is being widely exploited by a growing set of threat actors, presents an urgent challenge to network defenders given its broad use.” The agency suggested organizations to disable external-facing devices running Log4j and install additional firewall security with rules related to the vulnerability.
Security researchers found out last night that the Log4J bug can affect all java versions. So even if you’re running an updated Java version, you need to patch your Log4j service.
Just added support to LDAP Serialized Payloads in the JNDI-Exploit-Kit. This attack path works in *ANY* java version as long the classes used in the Serialized payload are in the application classpath. Do not rely on your java version being up-to-date and update your log4j ASAP! pic.twitter.com/z3B2UolisR
— Márcio Almeida (@marcioalm) December 13, 2021
A report from The Verge noted that researchers are getting pings from Apple and Tesla servers if they change the device name to specially crafted exploit string.
The open-source community enraged by low payment for contributors
This Log4j bug has once again established that large corporations around the globe depend on free open-source software heavily. That’s all fine and dandy, but the open-source community is now outraged by the fact about contributors not getting enough pay.
Filippo Valsorda, a cryptographer at Google, noted on his personal blog that the maintainer who fixed the Log4j bug contributed to the project part-time, and had just three GitHub Sponsors (a way for people to pay project volunteers).
He noted that contributors only get a small amount through GitHub or Patreon, while their projects might be responsible for millions of dollars of business. Plus, a bug in these important systems can break the internet, and it’s unfair to depend on people who work on these projects in their spare time without any assured rewards.
This is the maintainer who fixed the vulnerability that’s causing millions(++?) of dollars of damage.
“I work on Log4j in my spare time”
“always dreamed of working on open source full time”
“3 sponsors are funding @rgoers‘s work: Michael, Glenn, Matt”People, what are we doing. pic.twitter.com/2hAxUWCjuC
— Filippo ${jndi:ldap://filippo.io/x} Valsorda (@FiloSottile) December 10, 2021
Matthew Green, a professor of cryptography at Johns Hopkins University, argued that the industry needs a list of open-source libraries and technologies with widespread reach so more people can work on projects that are important to run the internet smoothly.
Ok, so what I’m saying is that if you want to solve this problem, to me the missing resource is not money. It’s *visibility*. We all know the landmines are out there — but we can’t see them. 8/
— Matthew Green (@matthew_d_green) December 11, 2021
Security firm Chainguard’s founder, Dan Lorenc, said that while corporations are willing to fund open-source projects, it’s hard for them to find out critical projects that need paid maintainers. He also noted that ideally, the industry should fund teams that maintain three to four projects.
This is going to sound blunt, but it’s a distribution problem not a funding problem. $ is easy.
Corporations have budget and are willing to spend, but it takes too much time. Finding projects that need help and maintainers willing to help in exchange for money is hard. pic.twitter.com/mFkOoOVYXn
— Dan Lorenc (@lorenc_dan) December 12, 2021
Contributors cited that their pay through contributions is super low, it’s really hard to maintain expenses by just doing that. That’s a sad state of affairs.
Hopefully, this issue will finally wake up the industry to pay attention towards coming together and contribute towards open-source projects that keep the internet afloat. But perhaps they might go back to sleep if the bug doesn’t cause much damage.
I have to end this with an XKCD comic summing up the situation.