The Value of Documentation

If someone asked me to name the biggest shortcoming in DFIR, and in cybersecurity, in general, I'd have to say documentation.

We miss a lot of value because activity and analysis goes undocumented, and it is lost. Without documentation clearly delineating what was done, and what was observed, we can't identify and address issues that need to be fixed. 

Without that documentation, capabilities cannot be updated, and new capabilities cannot be added. We end up not being able to operationalize what was observed or learned.

Without that documentation, every analyst needs to learn the same lessons, from their own efforts, and cannot benefit from the efforts of others. This is inefficient, and leads to a great deal of corporate knowledge simply be unrealized and lost.

I learned lessons about the value of documentation a long time ago. In 1990, during the first Desert Shield, my unit deployed to South Korea for an exercise named "Valiant Blitz". After the exercise, we had an after action review, and one of the radar operators stood up and just flat out said, "Comm sucked." That's it. Being the Communications Officer, and having a senior officer from my own unit stand up and say that, hurt. Particularly because I could not remember a single complaint during the entire exercise. 

In January 1991, a unit on Okinawa was decommissioned, and an LDO 1stLt was assigned to our unit, and became my boss. Shortly thereafter, we deployed to South Korea again, this time for the "Team Spirit" exercise. As before, we had an after action meeting following our return to Okinawa, and as before, one of the senior officers got up and said, "Comm sucked." 

However, my boss had set up a rudimentary ticketing system, maintained in a log book that Marines are used to having around. We logged all trouble calls that came into the Communications/Electronics section, to whom they were assigned, and their resolution. When the Captain made his statement, my boss slide the log book across the table and said, "show me." There was no more discussion of the issue. 

About 8 yrs later, I was off of active duty, and in my third or fourth consulting role in the private sector. I'd been hired into a firm that provided networking services, but as a security engineer. My boss was hired after me, and after about 6 months, he came to my desk late one afternoon and asked to speak to me. He'd had some speed bumps since he was brought in, and had given notice that morning. Immediately upon giving notice, the president of the company had flown down from NY, walked right into his office, and asked what he needed to stay. My boss said, "I want a security lab.", and the president responded with, "Give me a purchase order." My boss then pulled out a folder containing 6 months of purchase requests for a variety of lab configurations, ranging from multiple desktops and a couple of blade servers, all the way down to just 2 additional laptops. All were denied. 

In 2014, I was in a consulting role, and our team was going through an operational debrief, where an analyst walked through an engagement, and shared what they had seen and learned. The call literally started with the analyst stating that they'd seen something, didn't know what it was, and that they never went back to figure out what it was, before simply moving on with the rest of their debrief. Two of us in particular were taken aback by this statement, so we reached out to get some specifics about what the "thing" was, but the analyst hadn't documented the "thing", and didn't remember. So, we dug around a bit and found out what it was...and found that it was an extremely high fidelity indicator of this threat actor's TTPs. We found that this indicator only existed because of the specific actions that the threat actor took, and we hadn't seen other threat actors, nor admins or users, take similar actions. This was almost lost because the analyst hadn't documented it; fortunately, someone investigated it, and was able to operationalize the finding so that, going forward, all analysts would be alerted to the indicator, without having to do a full cradle-to-grave investigation themselves.

Not having documentation means:
1. We cannot operationalize, or even share, what we've seen and learned.
2. We do not have the necessary information, data, or numbers to encourage decision makers to decide in our favor.
3. We cannot justify the things we feel we need, or really understand how much of that is "feel" versus really, truly "need". 

Investigations and Analysis

Something I've seen consistently, from the very beginning when I first brushed up against this business of "digital forensics", is the default focus on, "...this is  how you analyze this artifact...", and rarely extending beyond that. 

I'm guilty of this, too. My first 8 books took this approach, listing data sources and artifacts, describing what they were in a general sense, and then maybe showing how to parse them. Each of those efforts basically left it up to the reading to stitch together their investigation on their own, and I justified this with the adage that each investigation is different. 

But then, I realized that what makes each investigation different isn't the incident itself, or the data sources. No, it's the investigator. Everyone of us approaches an incident investigation supported by our own knowledge and experience, and that's really the only thing that makes each investigation different. So, yes, while it's important to understand the value and nuances of the various data sources available on a Windows endpoint, something I can to realize is that this must be done in the context of an overall investigation.

I recently ran across a blog post titled, How Do You Investigate Windows Event Logs?, with the subtitle, Windows Eventlog Analysis 101. While I am very supportive of analysts developing a thorough understanding of various data sources, so that they can leverage and truly exploit those data sources in their investigations, I also very firmly believe that this must be part of an overall analysis process. 

This is why I wrote Investigating Windows Systems, to show how all of this information is, or can be, tied together by an analysis process. Once you have a process down, adding or removing data sources becomes thoughtful, intentional, and justified by data, not something that's done on a whim or because we forgot.

There's a good bit of really good advice in the article, but what stood out to me most was one of the things I absolutely do NOT want to see in any discussion of investigative or analysis processes, and that's the recommendation to "search by event ID". 

I've said it before, and I'll say it again...event IDs are not unique. There's no requirement for that. In fact, MS itself doesn't use unique event IDs. For example, event IDs 4624 and 4625 mean something entirely different if the source is "Microsoft-Windows-Security-Auditing", or if it is "EventSystem". 

That's right...not even Microsoft uses unique event IDs. When you start digging into the Application Event Log, and even add other applications that either write their events to the Application Event Log, or to their own log, then things really start to take a turn. 

The point here is that any investigation or analysis advice that relies on a single data source, or suggests opening a single Windows Event Log in a viewer and then "search for event IDs" should be viewed as insufficient. Such advice is being shared by someone who doesn't have a defined analysis process. 

What we should be looking to is using several data sources (WEVTX, file system, EDR telemetry, etc.), normalizing them into a common format, enriching them with data from either our own prior investigations or from external sources (or both), and then displaying them in a timeline format, so that we can see the progression and timing of the activity. 

Another example of the narrow-aperture way of conducting analysis can be seen clearly in this LinkedIn post, and there are many more just like it.

The shortfall of all of this is that this is what AI models are being trained on, and because they're capable of doing it at scale, today's up-and-coming analysts will not be able to keep up. They won't be able to see the gaps, and they won't know when the model is hallucinating a finding. They simply won't be able to discern valid findings.