Latest Testing: The Windows Registry, pt I

I've always been interested in and fascinated with the Windows Registry, in large part due to the fact that early in my DFIR career few really seemed to have a good grasp on, or interest in, just how central it is to the Windows operating system.  By extension, the grasp, for the community at large, as to the analysis value of the Registry was underestimated.  We've seen changes to this attitude over the years, and as the migration to incorporating the Windows Registry into analysis has progressed, so has the development of techniques for deriving even more analysis value from the Registry.

Interest in the Registry as a source of data for digital analysis and incident response has waxed and waned over the years. In the late '90s while I was performing vulnerability assessments, there was considerable utility in querying the Registry from live systems in order to determine the state and attack surface of those systems.  We were looking for (and finding) active modems in systems, as well as extracting information about software versions, etc.  To a large extent, this remains true today; there is a great deal of data available in the Registry to assist in developing an understanding of not only the state of the system at a given time, but also activity that occurred on the system.

In 2008, Jolanta Thomassen shared her dissertation regarding parsing Windows Registry hive file unallocated space; that is, retrieving deleted keys and values from unallocated space within the hive file itself.  She also released regslack.exe, a command line tool for extracting deleted keys and values from hive files.  Other tools have followed suit, providing similar functionality. Eventually, RegRipper included this functionality, as well.

In the past couple of years, attention has turned to Registry transaction logs, and incorporating data from those logs directly into hive files in order to achieve a more comprehensive and perhaps 'correct' view of the hive contents.  As such, I wanted to determine what tools were available to merge transaction logs into a hive file, and from there, begin looking at the effect of the merge operation.

I installed Python 3.7 on a Windows 10 system.  I opted for Python 3.7, as it is required for a tool I'm planning to test, Regipy.  I installed yarp on that system, as well.  This is the system I'm using as the platform for merging hives with their transaction logs.

I began testing by using yarp + registryFlush.py (thanks to Maxim) to merge the transaction logs (hive.log1, hive.log2) into a hive file, using hives from the LoneWolf scenario due in part to their availability.  Once I had successfully merged the transaction logs into the System hive, creating a 'new' System hive file, I used registry-diff.py and regdiff.pl to document the differences between the original and 'new' hives.  As an example of differences between the two hives, while they were reported by the file system to be the same size, regdiff.pl found 2561 differences between the original and 'new' hives, many of which pertained to USB devices.

I should note here that this is a very interesting finding.  Several years ago, I ran into a situation where I was comparing the System hive from a system to it's 'companion' in the RegBack folder (clearly, this goes back to when the hives in the config folder were being actively backed up).  In that case, I found 9 USB devices listed in the System hive, and 13 devices listed in the backed up hive.  I did a text search across both folders for one of the device names (one that was unique), and found an entry in one of the .log files, but at the time, there were no tools available for merging the transaction logs into the hive.

I then moved on to looking at differences in the hives via RegRipper.  Again, for this initial set of testing, I was focusing on the System hive.  As the AppCompatCache (or ShimCache) data is written to the System hive at shutdown, I did not expect to see significant differences between the two hives, and that expectation was borne out by running the appropriate plugin against both hives.  However, the BAM and Mounted Devices plugins proved to be a bit more interesting.  For example, the output of the mountdev.pl plugin run against the 'new' hive contained a drive signature for an F:\ volume that did not exist in the original hive.  The bam.pl plugin showed two additional entries for the user with RID 1001 when run against the 'new' hive.  Also, there was an entry for the Local Service account in the 'new' hive (again, via the bam.pl plugin) that was not in the original hive.

The del.pl plugin produced some interesting results.  When I ran it against the original System hive, it pulled out a number of deleted keys and values; however, running it against the 'new' (i.e., merged) hive, I got nothing.  Not one deleted key, not one deleted value.

So, what does all this mean?

I started this out wanting to simply see what tools and processes could be used to merge transaction logs into their hive file.  I posted a tweet to that effect, and Maxim was the first (and only) person to respond (to that query).  I do have additional testing to do, not just with this script, but with other tools that are available, as well.  Again, what I'm looking at here is simply how to go about merging the transaction logs into a hive, and if successful, looking to see differences between the two in order to demonstrate the value of taking these steps.

Something to keep in mind is that Windows systems no longer appear to be populating (by default) the backup copies of the Registry hives within the config/RegBack folder.  As such, merging transaction logs into the hive files prior to any parsing and analysis processes would seem to be a pretty valuable approach, if not simply provide more complete information. From that point, it's a matter of conducting analysis. 

As far as a process goes, you'd likely want to either extract the appropriate files, or mount the image with the appropriate tool such that you could access the necessary directories.  From there, you'd want to keep the original hive, and create a new version of the hive by merging the transaction logs.  The new hive would ideally give you a more complete view of the hive, but the original hive would provide you with historical information, particularly regarding deleted content.

So, more to come...

Resources
I recently became aware of RegRippy, a Python-based alternative to RegRipper.