A recent thread over on ForensicFocus got me to thinking about the issue of tool verification. In that thread, there is discussion of verifying the results produced by RegRipper. Chris picked up on this and posted to his blog, as well.
First off, I'm a proponent of verification, but I also think that we, as a community and a profession, shouldn't be blindly running on autopilot when it comes to tool verification. Specifically, what is "tool verification"?
One way to look at this is that you'd like to verify that a tool does what it's supposed to do correctly. For example, if one tool (ie, AccessData's FTK Imager) generates an MD5 hash of an acquired image, you might want to verify that the algorithm used to generate that hash was done implemented correctly, so you'd use another tool (one that you trust as being correct...perhaps Jesse Kornblum's md5deep) to confirm that the MD5 hash is, indeed, correctly implemented. The implication here is that if both tools produce the same output, then the implementation employed by FTK Imager is correct and verified.
A similar way to look at this is with respect to a tool such as RegRipper, but how would you verify the results? For one, what other tools do what RegRipper does...besides RegExtract, which is, according to the author, based on RegRipper?
Well, similar to Nessus, RegRipper implements its actions based on plugins that are text-based. So if you want to determine what a plugin does (ie, what keys, values or data it extracts, and how it extracts or parses that data), all you need to do is open the plugin in Notepad or an editor. I know that a lot of folks will look at the Perl code and slip in to a coma, but for the most part, the data extraction and parsing logic is right there...in some cases, it's easy to see: read the binary data into a variable/Perl scalar, go to this offset within the data, read this many bytes, and translate it based on this table of values. Easy.
If you're not sure that what a plugin does is correct, I try to make sure that I include either descriptions in the comments of the plugin (lines that start with "#") or references to credible sources about the structure or use of the key, value, or data being parsed. In most cases, much like WFA 2/e, these references consist of MS KB articles. If there's some question about the data and how it is or should be interpreted...well, that's what I provide the references for, and as many of you know, I've tried to answer more than a few questions along those lines.
So, how do you verify what a tool like RegRipper does? One way is to use another tool that does what RegRipper does...but outside of RegExtract, there are no other tools that do what RegRipper does. You can open the hive file in a Registry viewer...some of the plugins present information in a manner similar to the way a viewer would, and you can always write your own plugins to do this. But what about parsing/interpreting binary data? Or "decrypting" ROT-13 value names? At that point, the only way to verify what RegRipper does is to sit down with the hive file open in a hex editor and do it all by hand.
Again, I'm not saying that tool verification isn't important...because it is. But so is understanding what a tool does. Blindly saying that I need a tool verify my findings against another tool is really pointless unless I understand what both tools do, and how they do it. I mean, the same holds true in other areas, such as live response. If I run two tools that both use to the same API calls to show me the active process list or the network connections, have I verified anything? No, I haven't. However, if I run two tools that use disparate APIs (and don't converge until somewhere very low in the API stack) then I have verified the tools and the findings...but only to the point where they ultimately converge somewhere down the API stack.
So, before discussing tool verification, we need to have an understand of the purpose and use of the tool, as well as an understanding of how the tool functions.
Harlan, this is an excellent point you posted which I fully concur with you. I am a proponent for tools verification too.
ReplyDeleteFelix
I am in full support of tool verification. Nevertheless, in order to verify a tool, you need to know what the correct output should be. In addition, that cannot be done by comparing TOOL A to TOOL B. You need to understand the fundamentals and do be able to do it by hand.
ReplyDeleteFor example, when using RegRipper, I noticed that it does not process VISTA or WIN7 OpenSaveMRU keys correctly. It was not that it was programmed incorrectly, that it had not been implemented yet. I gave Harlan a heads up and he added it to the plugin.
I wish more people actually did tool verification. Secretly, I wish more people did original research as well. We need more hands on deck. There are artifacts yet to discover.
Verify what someone writes in a book, blog, or teaches in a class such as SANS. "Trust, but verify." (Ronald Reagan taught us that with the USSR.)
Not only does it make you better. It makes us all better as well. No one has it all figured out.
Though I try to double check personally everything we teach, mistakes are made. How often has FTK or ENCASE been patched/updated? Do you really believe they had it all figured out at version 1.0? To truly verify, you have to see it yourself.
Trust, Verify, Research.
Rob Lee
SANS Institute
Guys, thanks for the comments...
ReplyDelete...I noticed that it does not process VISTA or WIN7 OpenSaveMRU keys correctly.
See, that's the cool thing about RegRipper...Perl's free, and the plugins are open source, so if something doesn't work the way you want it to, you can always modify it.
I wish more people actually did tool verification.
Agreed, but IMHO, there are a number of issues at play here. First, I don't think that there's a clear, solid understanding across the community of what comprises "verification". Second, not every can...or should have to...do tool verification. There needs to be clear documentation available for the process and results to tool verification available to everyone. Otherwise, we all do this verification and no one is left to actually do the work that needs to be done!
We do need more hands on deck, I agree, but we also need the folks that are doing the work now to collaborate and share. When you've got a couple of disparate groups working toward the same goal, you're likely to get better results faster if they collaborate and share information.
Finally, not everyone in the community is going to be able to do or take part in research, but pretty much everyone will want to be consumers of that information. As such, there should be community-wide support for the research. Right now, there isn't...it's really much more of a "gimme" attitude. Support the research and it'll be better, more complete and arrive faster.
I think that everyone does have to do tool verification to some extent. That doesn't mean reverse engineering FTK, but it may mean running companion tools side by side and going into an image with a hex editor to verify the results. For example, I just found some errors in a new tool in regard to the offsets at which it reported Internet data. That's not only important for obvious reasons, but it helps the publisher, too.
ReplyDeleteXWF puts out a variety of registry reports and is configurable. It can be used in conjunction with RegRipper. In regard to the registry, as well as topics discussed recently on the list (USB, link files), I think that validation also needs to go to the meaning or interpretation of the data, in addition to a determination that the corrrect data were found. For instance, there's been some misinterpretation of whether data in the SAM hive means that a password is required of a given user.
...I think that validation also needs to go to the meaning or interpretation of the data, in addition to a determination that the corrrect data were found.
ReplyDeleteI agree 100%. However, I think that there is always going to be an issue with this. Even with training or documentation (and references) for this, someone will walk away from the training and not touch an item for weeks. Then, after having not used it, they'll be in a position where they need the information and don't remember it correctly. This happens all the time with the USB stuff Cory and I did a while back, and I still find the same thing to be true, even with people who have WFA 2/e.
For instance, there's been some misinterpretation of whether data in the SAM hive means that a password is required of a given user.
Right, and I received some info from MS in that regard and posted it to my blog.
Harlan, Thanks for the recent holler-back. I invest the time in tracking down and sharing these tools and tips because they bring benefit to me, because I want to "pay it forward" for all the work other bloggers and developers have done to get me where I am in my knowledge base, and because it is just plain therapeutic in a relaxing way. So many great apps and info, just beneath the surface clutter of the Net. Just takes a bit of digging to uncover then share.
ReplyDeleteThank you in particular for the link to the woanware site in this post. It's an amazing collection of targeted utilities. Right up my alley. I'm going to be sorting through that site for a while!
FWIW, I'm currently going through an incident-response analysis on a production system and RegRipper came in dead-useful with my system admin hat on. It allowed me to parse out the areas I needed to focus in on. Then I grabbed my other toolchest and set to work. It and the other forensic tips from you in particular have really helped me pick apart the system and come up with a solid timeline of activity and events. I'm grateful. I'm not sure it will make daylight on my blog due to the nature, but I hope to post some particular lessons learned in the near future related to one or two particular area of focus.
Cheers!
--Claus V.
Claus,
ReplyDeleteThanks! I hope you do get a chance to share your lessons learned...one of the drawbacks of writing this stuff from the batcave is that I don't get to see how folks are actually using the tools too often...