The Value of Documentation

If someone asked me to name the biggest shortcoming in DFIR, and in cybersecurity, in general, I'd have to say documentation.

We miss a lot of value because activity and analysis goes undocumented, and it is lost. Without documentation clearly delineating what was done, and what was observed, we can't identify and address issues that need to be fixed. 

Without that documentation, capabilities cannot be updated, and new capabilities cannot be added. We end up not being able to operationalize what was observed or learned.

Without that documentation, every analyst needs to learn the same lessons, from their own efforts, and cannot benefit from the efforts of others. This is inefficient, and leads to a great deal of corporate knowledge simply be unrealized and lost.

I learned lessons about the value of documentation a long time ago. In 1990, during the first Desert Shield, my unit deployed to South Korea for an exercise named "Valiant Blitz". After the exercise, we had an after action review, and one of the radar operators stood up and just flat out said, "Comm sucked." That's it. Being the Communications Officer, and having a senior officer from my own unit stand up and say that, hurt. Particularly because I could not remember a single complaint during the entire exercise. 

In January 1991, a unit on Okinawa was decommissioned, and an LDO 1stLt was assigned to our unit, and became my boss. Shortly thereafter, we deployed to South Korea again, this time for the "Team Spirit" exercise. As before, we had an after action meeting following our return to Okinawa, and as before, one of the senior officers got up and said, "Comm sucked." 

However, my boss had set up a rudimentary ticketing system, maintained in a log book that Marines are used to having around. We logged all trouble calls that came into the Communications/Electronics section, to whom they were assigned, and their resolution. When the Captain made his statement, my boss slide the log book across the table and said, "show me." There was no more discussion of the issue. 

About 8 yrs later, I was off of active duty, and in my third or fourth consulting role in the private sector. I'd been hired into a firm that provided networking services, but as a security engineer. My boss was hired after me, and after about 6 months, he came to my desk late one afternoon and asked to speak to me. He'd had some speed bumps since he was brought in, and had given notice that morning. Immediately upon giving notice, the president of the company had flown down from NY, walked right into his office, and asked what he needed to stay. My boss said, "I want a security lab.", and the president responded with, "Give me a purchase order." My boss then pulled out a folder containing 6 months of purchase requests for a variety of lab configurations, ranging from multiple desktops and a couple of blade servers, all the way down to just 2 additional laptops. All were denied. 

In 2014, I was in a consulting role, and our team was going through an operational debrief, where an analyst walked through an engagement, and shared what they had seen and learned. The call literally started with the analyst stating that they'd seen something, didn't know what it was, and that they never went back to figure out what it was, before simply moving on with the rest of their debrief. Two of us in particular were taken aback by this statement, so we reached out to get some specifics about what the "thing" was, but the analyst hadn't documented the "thing", and didn't remember. So, we dug around a bit and found out what it was...and found that it was an extremely high fidelity indicator of this threat actor's TTPs. We found that this indicator only existed because of the specific actions that the threat actor took, and we hadn't seen other threat actors, nor admins or users, take similar actions. This was almost lost because the analyst hadn't documented it; fortunately, someone investigated it, and was able to operationalize the finding so that, going forward, all analysts would be alerted to the indicator, without having to do a full cradle-to-grave investigation themselves.

Not having documentation means:
1. We cannot operationalize, or even share, what we've seen and learned.
2. We do not have the necessary information, data, or numbers to encourage decision makers to decide in our favor.
3. We cannot justify the things we feel we need, or really understand how much of that is "feel" versus really, truly "need".