Log4shell: Initial Thoughts
The first report I saw regarding the recent Log4j 0-day suggested that the scope was limited to Minecraft servers. By the time I had reviewed the CVE, there were already proof-of-concept exploits being showcased and it didn’t take long before widespread active exploitation was reported. It was soon apparent that the impact would be far reaching and severe with RCE being easily achieved. The potential attack surface is stunning. The Log4j library is seemingly ubiquitous in Java applications.
The Initial Response #
Once it was clear that immediate action was necessary, I broke the response into three steps:
- Enumerate all Java applications in our environment
- Identify if the current application version is vulnerable
- Remediate as necessary
Our estate is small so we have a solid inventory and working knowledge of our deployed apps. This made step 1 easy. Fortunately, many vendors have expeditiously posted vulnerability status, as well as patching or workaround recommendations, so we were quickly able to begin the remediation process. The vulnerability can be stopped by upgrading Log4J to 2.15.0. If the library cannot be updated, setting the -Dlog4j2.formatMsgNoLookups=true flag is an insufficient remediation that can be avoided by certain attack vectors. Vendor updates will be deployed as soon as they are available.
Key vendors in our environment have all published bulletins confirming that their are not vulnerable. They either do not use Java or do not use a vulnerable version of the Log4j library. Some vendors have lagged behind in their response but by 12/14 we have received confirmations from all. At the network level, appropriate signatures were very quickly developed by the firewall vendors based on current IOCs and have now been enabled.
Looking Forward #
We’ll be dealing with this vulnerability for quite some time. While I am confident I’ve dealt with the known Log4j instances in the environment, I can’t help but worry about the unknown unknowns. There are likely some “black box” services that I haven’t accounted for that will need to be investigated. Scanning machines for log4j*.jar files doesn’t turn up any surprises but they could easily be hidden in fat bundles. Much like NPM module hijacks, or other supply chain vulnerabilities, there are simply too many upstream dependencies for an app owner or ops team to vet and audit before deployment. It will be important to monitor vendor releases and make patching decisions once new information is available.
Defense In Depth #
This latest doomsday vulnerability has emphasized the importance of “defense in depth” strategies. An organization should have a robust patching schedule to address vulnerable code quickly. If vulnerable code cannot be patched, endpoint protection should be intelligently configured and signatures should be current. If it is missed by endpoint monitoring, host-based firewalls should be configured to only allow essential traffic. Networks should be segmented at appropriate trust boundaries. Firewalls should be inspecting egress traffic for known IOCs and need to be kept up to date with signatures and known malware IPs/domains. Most difficult of all, organizations need to have the policies and procedures in place to respond to emerging threats in a timely manner.
I’m thankful that my organization gives me the space to focus on security events like this. I feel for the SMB admins out there.