Thoughts on Tool Verification

A recent thread over on ForensicFocus got me to thinking about the issue of tool verification. In that thread, there is discussion of verifying the results produced by RegRipper. Chris picked up on this and posted to his blog, as well.

First off, I'm a proponent of verification, but I also think that we, as a community and a profession, shouldn't be blindly running on autopilot when it comes to tool verification. Specifically, what is "tool verification"?

One way to look at this is that you'd like to verify that a tool does what it's supposed to do correctly. For example, if one tool (ie, AccessData's FTK Imager) generates an MD5 hash of an acquired image, you might want to verify that the algorithm used to generate that hash was done implemented correctly, so you'd use another tool (one that you trust as being correct...perhaps Jesse Kornblum's md5deep) to confirm that the MD5 hash is, indeed, correctly implemented. The implication here is that if both tools produce the same output, then the implementation employed by FTK Imager is correct and verified.

A similar way to look at this is with respect to a tool such as RegRipper, but how would you verify the results? For one, what other tools do what RegRipper does...besides RegExtract, which is, according to the author, based on RegRipper?

Well, similar to Nessus, RegRipper implements its actions based on plugins that are text-based. So if you want to determine what a plugin does (ie, what keys, values or data it extracts, and how it extracts or parses that data), all you need to do is open the plugin in Notepad or an editor. I know that a lot of folks will look at the Perl code and slip in to a coma, but for the most part, the data extraction and parsing logic is right there...in some cases, it's easy to see: read the binary data into a variable/Perl scalar, go to this offset within the data, read this many bytes, and translate it based on this table of values. Easy.

If you're not sure that what a plugin does is correct, I try to make sure that I include either descriptions in the comments of the plugin (lines that start with "#") or references to credible sources about the structure or use of the key, value, or data being parsed. In most cases, much like WFA 2/e, these references consist of MS KB articles. If there's some question about the data and how it is or should be interpreted...well, that's what I provide the references for, and as many of you know, I've tried to answer more than a few questions along those lines.

So, how do you verify what a tool like RegRipper does? One way is to use another tool that does what RegRipper does...but outside of RegExtract, there are no other tools that do what RegRipper does. You can open the hive file in a Registry viewer...some of the plugins present information in a manner similar to the way a viewer would, and you can always write your own plugins to do this. But what about parsing/interpreting binary data? Or "decrypting" ROT-13 value names? At that point, the only way to verify what RegRipper does is to sit down with the hive file open in a hex editor and do it all by hand.

Again, I'm not saying that tool verification isn't important...because it is. But so is understanding what a tool does. Blindly saying that I need a tool verify my findings against another tool is really pointless unless I understand what both tools do, and how they do it. I mean, the same holds true in other areas, such as live response. If I run two tools that both use to the same API calls to show me the active process list or the network connections, have I verified anything? No, I haven't. However, if I run two tools that use disparate APIs (and don't converge until somewhere very low in the API stack) then I have verified the tools and the findings...but only to the point where they ultimately converge somewhere down the API stack.

So, before discussing tool verification, we need to have an understand of the purpose and use of the tool, as well as an understanding of how the tool functions.