9 Modern-Day Best Practices for Log Management

Figure 1: (Image: 123tin via Adobe Stock)

Logs are central to forensic investigations, but only if they're collected, stored long enough, contain everything investigators need, and the bad guys don't get to them first.

That's a big "if."

"What can businesses [do] to mitigate the possibility that lots of attackers are trying to hide their tracks and even destroy log files? Obvious: Use a log management tool to centralize logs – the same advice as in 2021, 2011, 2001, and perhaps even 1991," says Dr. Anton Chuvakin, head of security solution strategy at Google Cloud and author of several books.

However, all security professionals know log management circa 1991 is nowhere near as vast and complex as it is today. Logs grow as needed to record data events – and make no mistake, modern-day businesses have tons of data.

That makes managing large and unwieldy numbers of logs is a daily challenge. Staying in compliance with a growing number of laws is raising the level of complexity, too. For example, the Health Insurance Portability and Accountability Act (HIPAA) requires logs to be held for up to six years, while the Sarbanes-Oxley Act (SOX) requires seven years and The Basel II Accord requires three to seven years.

So it's crucial that log management be done smartly, correctly, and concisely – not too much, not too little, but just right – and in a manner thorough enough to be of aid to forensics investigators, even when criminals hide their misdeeds.

Experts shared their tips and best practices to give Dark Reading readers the home advantage.

Figure 4:

Detach Logs From Their Device and System Origins

Criminals often target specific systems and devices, removing their own logs to cover their tracks. Using a tool capable of ingesting logs from devices and systems and storing them together in a separate, secure place ensures the good guys can still see what the bad guys did.

"Consider removing logs from the systems and devices that are creating them," says Nathan Salminen, senior associate at global law firm Hogan Lovells. "A security information and event management tool, or even a simple log aggregation tool, ingests logs from across the enterprise and keeps them together in one place where they can be retained and preserved, even if a threat actor successfully destroys or edits logs on the targeted system

Salminen, who is also a certified Offensive Security Certified Professional (OSCP), warns that while many organizations already have such tools, "some have not yet configured it to ingest logs from all of their critical systems and devices."

As an aside, Salminen says the most prevalent problem he sees is "organizations that do not log events at all or do not retain logs for a sufficient period of time to enable them to determine the extent of the threat actor's compromise of their systems."

(Image: adam121 via Adobe Stock)

Figure 5:

Log at Different Points

According to Kevin Madura, SVP at global consulting firm AlixPartners, businesses should turn on "more verbose logging" everywhere possible – within computational and financial restraints of course – including for applications, application servers, Web servers, load balancers, and network appliances such as firewalls, switches, routers, and endpoints.

"It's critical to log at these different points within the network, which can be useful for investigators to understand how the attacker got in and how they pivoted within the network, as well as where they went after initial intrusion," he says. "This also helps determine which systems and data may have been compromised during the attack to determine if any other systems should be examined forensically."  

(Image: Anikakodydkova via Adobe Stock)

Figure 6:

Take Cover in the Cloud

Moving logs to the cloud can be helpful because it adds extra layers of complexity to the criminal’s work.

"As the first mitigation to protect logging systems, consider using a cloud-based logging solution," says Elad Menahem, director of security at Cato Networks. "Moving the logging server to the cloud forces attackers to compromise not one but two networks. And often cloud providers will invest far more in protecting their networks than the typical enterprise."

But whether you keep your logging systems on your own servers or someone else's a la the cloud, backing up logs is a good idea. That's because sometimes the bad guys don't destroy logs. Instead, they modify or pollute log content through simple measures, such as an overload of activity, or by more complex efforts, such as to "surgically modify logs, such as changing a specific log row indicating suspicious activity," Menahem adds.

(Image: Shutter2U via Adobe Stock)

Figure 7:

Add Images of the Storage Media to the Forensic Data Mix

It's true that a picture can be worth 1,000 words – especially if the image captures information outside of the log's domain but is relevant to the scheme of things.

"Many forensic investigations are successfully resolved by reviewing forensic images of the storage media rather than logs. Taking and retaining periodic snapshots of virtual machines can yield very valuable intelligence about a threat actor's activities," says Peter Marta, a partner at global law firm Hogan Lovells who formerly served as global head of cybersecurity law at JPMorgan Chase and in the U.S. intelligence community. "Historical snapshots can be more useful than an image of the post-incident system, which may be encrypted or otherwise obfuscated by the threat actor."

However, backups are not necessarily a full cure for an attack that vexes you, he says, because they "often store only the data partitions and very rarely store unallocated space, which can be critical for identifying deleted files." 

(Image: chathuporn via Adobe Stock)

Figure 8:

Don't Shut Down Compromised Machines So Fast

Almost everyone rushes to shut down their machines once an attack is detected.

"While understandable, doing so can result in the loss of the full picture of what was going on in that machine, especially as malware has continued to evolve to variants that never touch storage media," says Hogan Lovells' Marta. "Taking an image of memory before shutting the machine down, once a company sees that it has been attacked, can be critical."

Indeed, memory is often the central point of attack, making it essential to create records of activity there.

"Make sure security teams have memory analysis capability, as most malware does not write to disk and runs strictly in memory, making it incredibly hard to analyze without the proper skillset and practice," says Steven Baker, a Deloitte Risk & Financial Advisory specialist master in the cyber and strategic risk practice. "IT should have a standing process in place that grabs memory from a suspect system before an investigation begins. If they triage and find no issues, then delete the memory and move on. If they find something, they have memory to look at."

(Image: Monster Ztudio via Adobe Stock)

Figure 9:

Know Which Info in Log Files Is Useful

While the answer varies depending on the type of incident, some general data points are almost always useful. According to Hogan Lovells' Salminen, they are:

(Image: Giovanni Cancemi via Adobe Stock)

Figure 10:

Test the Usefulness of Your Logs

Not all logs are created equal, so it's a good idea to check and see how useful your logs actually are – before you need to depend on them.

"Wargaming can help you get an idea of how useful an organization's log files are and where logging gaps occur," says Deloitte's Baker. "A simulation of a common or headline-driving cyberattack can help determine what data your organization needs to be able to fully scope that type of incident and conduct root-cause analysis."

It also can be helpful to know which logs other organizations find useful. Google Cloud's Chuvakin says his favorite log types are server/endpoint, VPN/remote access, various security tools (IDS/IPS, antivirus), and cloud (SaaS via CAS authentication protocol, IaaS directly) logs.

(Image: 3000ad via Adobe Stock)

Figure 11:

Purge Responsibly

The sheer volume of logs to be managed shows no signs of slowing. But don't dump logs too soon to make room for more.

"Even with a solid SIEM in place, it can still be overwhelming," says Keatron Evans, an infosec skills author at Infosec Institute. "Also, these logs are not kept indefinitely in most cases. The high volume means they have to generally be purged often to make room for new logs. Storage is a major challenge for enterprise logging."

While only one part of a strong security architecture and plan, logs are often critical to forensic investigations.

"Developing effective response strategies before a cyberattack occurs is key, as all the logging in the world won't detect an event if the logs weren't monitored to begin with," says Baker.

(Image: lassedesignen via Adobe Stock) 

Figure 12:

Don't Overdo It

Preparedness often saves the day, and that means managing logs well and in advance. But be forewarned: Overpreparation can cost you dearly, too.

"It's easy to just log everything, but without being deliberate in your approach, you may end up increasing risk instead of mitigating it," AlixPartners' Madura says.

Unencrypted PII, user data, passwords, or other sensitive information can be accidentally collected in efforts to log all things loggable, but that could "land you in hot [regulatory] water," he adds.

Even so, less isn't best either, adds Matt Ruddell, MEDEX trainer and lecturer at the Global Forensic and Justice Center at Florida International University.

"While the sheer amount of this data may seem daunting, good forensic examiners should have software and hardware tools beefy enough to comb through these logs – which can certainly be voluminous if you turn them all on," he says. "Admittedly, this can frequently be an issue as digital forensic examiners are often battling for budget dollars to keep their equipment and skills up to date."

(Image: minicel73 via Adobe Stock)