Wednesday, May 29, 2019

Troubleshooting and Deep Knowledge

What do we do when a tool that we're using doesn't return what we expect to see?

How often does this happen?  How often do we run a tool in the course of our DFIR investigation process and for some reason, we don't see something in the output of the tool that we thought we would see?  How often do we look for something, only to find that it's not there?  More importantly, what do we do about it?

I can't speak for anyone else, but I tend to want to figure out why, particularly if the data I'm trying to parse is important to my overall analysis, based on my goals.  If it's just a curiosity, I'll leave it until later, but if the data itself is paramount to the analysis, I generally tend to want to know why I'm seeing (or not, as the case may be) what I'm seeing. 

Several years ago, while I was QSA certified and working on PCI cases (sssshhhh...don't let that little secret out...), we ran across a case where we knew that two specific brands of credit cards were being used at the site, but for some reason, our scans were coming up empty for those specific brands.  We were getting lots of search results for the major card brands, but nothing for these two brands that we expected to see.  We started peeling back the layers of the process, and go to the point where we were running scans of just straight text files that contained test data (several of the brands provided a small number of "valid" but unassigned credit card numbers), but we were still coming up empty on the brands.  As we investigated further, we narrowed the issue down to a built-in API function, and determined that it was not, in fact, returning "valid" credit cards numbers.  Rather, it was, just not all of them; it did not consider these two brands as 'valid' for some reason.  We got some programming assistance and built our own replacement function, one that was a bit slower but more complete.

The moral of the story is that we looked at the components involved...the data, the tool, and the examiner...and worked methodically to figure out where the issue lay.  Was it an assumption on our part, that these two brands were actually being used at the site?  How did we know?  Was it an assumption, or was it information that came from a specific, identifiable source?  What about the data?  Was there an issue with the data?  Or was it the tool?

As it turned out, it was an issue of deep knowledge.  We knew what the data should look like, and we had an understanding of where the data should reside.  As a bit of a follow-on to my post on Deep Knowledge in DFIR work,  I wanted to take a few moments to discuss an issue I ran into with the DefCon 2018 CTF File Server image, and what I did to address it.

I was reviewing some of the data, and I wanted to check and see if I could see a reference to the counter-forensics tool in the AppCompatCache data; as such, I ran the RegRipper plugin, and saw the following:

Launching appcompatcache v.20190112
appcompatcache v.20190112
(System) Parse files from System hive AppCompatCache

Signature: 0x0

That was it...nothing else.  No listing of file paths and names, no time stamps...nada.

Okay, what?  Do I throw up my hands and assume that the tool doesn't work?

No, not at all.  I start troubleshooting the issue.

As  I mentioned, there are three components to this process...the data, the tool, and the operator.  Any one of the three could have an "issue".  It's entirely possible that the tool does not function properly.  For example, when looking at a Registry hive via a viewer, I like to use the MiTeC Windows Registry Recovery (WRR) tool, even though I know that the tool has a deficiency.  Specifically, for value data over a certain size, a different data type is used to store that data (similar to non-resident files within the MFT using a run list), and WRR does not handle those data types. The AppCompatCache data is exactly that data type.  However, I could navigate through the Registry structure to the key in question, even though I could not directly view the raw data value via WRR.

First step...check the tool.  Is the plugin working correctly?  It works correctly against the System hive from the HR server, but that's a different version of Windows.  I don't have a System hive available that matches the Windows version from the file server image, but for the most part, the plugins seems to be working correctly.

RegRipper plugins are Perl scripts, which means that they're essentially text files.  Opening the plugin in Notepad++, we see that lines 96 and 97 are commented out (that is, the lines start with "#"):

# ::rptMsg("Length of data: ".length($app_data));
# probe($app_data);

This is essentially some troubleshooting code; when it's not commented out, it will tell me the length of the data, and then print a hex dump of the data (via the probe() function).

Assuming that the issue may have been with the plugin, I removed those "#" symbols, saved the file, and re-ran the plugin in hopes that I'd get some more information.  For example, if the plugin wasn't parsing the data correctly, I'd see a line telling me the length of the data, and then a hex dump of the data itself.  However, when I ran the plugin, I saw simply "Length of data:" and nothing else; no length value, no hex dump.

Next step...check the data itself.  I opened the hive in WRR, and went to the Control\Session Manager key to check the AppCompatibility subkey (the path is listed in the RegRipper plugin), and...nothing.  I couldn't find the AppCompatibility subkey.  It didn't seem to exist.

I ran the RegRipper plugin against the hive file, redirecting the output to a file: -r f:\defcon2\files\system -p del > f:\defcon2\files\system_del.txt

From the output:

------------- Deleted Data ------------
  598220                          10 00 00 00 6f 6d 70 61          ....ompa
  598230  74 43 61 63 68 65 00 00                          tCache..

Okay, so let's try another tool.  Using, I ran the following command line: --deleted e:\defcon2\files\system > e:\defcon2\files\system_del_yarp.txt

Opening the output file, there were quite a few recovered keys and values. I ran a search for "ompat", as well as one for "cache", and found nothing related to the AppCompatCache value.

Okay, so now we're pretty sure that the reason the RegRipper plugin didn't return what we expected is because the key and value question don't exist within the hive file.  This is pretty unusual, so the next question would be, "why?"

I checked the LastWrite time of the ControlSet001\Control\Session Manager key via WRR, and then pivoted into the timeline of system activity.  Events near that time are shown below:

Wed Aug  8 03:51:12 2018 Z
  FILE                       - M... [37092] C:\ProgramData\Microsoft\Windows Defender\Support\MPLog-02112017-053110.log

Wed Aug  8 03:51:03 2018 Z
  REG                        - M... HKLM/Software/Microsoft/MpSigStub 
  FILE                       - M... [5732] C:\Windows\Temp\MpSigStub.log
  FILE                       - MA.. [4096] C:\ProgramData\Microsoft\Windows Defender\Definition Updates\{8DCEFA0C-BEA3-48DC-B17D-E363C91F2F5D}\$I30
  REG                        - M... HKLM/Software/Microsoft/Windows/CurrentVersion/WindowsUpdate/Auto Update/Results/Install 
  FILE                       - MA.. [48] C:\Windows\Temp\2A62F43A-0724-4D0C-8B60-BC284D249D64-Sigs\
  FILE                       - MA.. [48] C:\ProgramData\Microsoft\Windows Defender\Definition Updates\{8DCEFA0C-BEA3-48DC-B17D-E363C91F2F5D}\

Wed Aug  8 03:51:02 2018 Z
  REG                        - M... HKLM/Software/Microsoft/Windows Defender/Signature Updates 
  FILE                       - M... [5699701] C:\ProgramData\Microsoft\Windows Defender\Scans\mpcache-008087D650ED729E08CB8F27E1DE2E1889585057.bin
  FILE                       - MA.B [1999848] C:\ProgramData\Microsoft\Windows Defender\Scans\mpcache-008087D650ED729E08CB8F27E1DE2E1889585057.bin.83
  REG                        - M... HKLM/System/ControlSet001/Control/Session Manager 

Interesting.  It looks as if Windows Defender was being updated or a scan was being run at about that time.  Checking the support log from 03:51:12, we see the following at the end of the file:

[Tue Aug 07 2018 20:51:03] Process scan started.
[Tue Aug 07 2018 20:51:04] Process scan completed.

Checking the timezone settings for the system, we see that the ActiveTimeBias is 7 hrs, and that a scan was started and completed shortly after the Registry key in question was last modified.

What about the other ControlSet?  Pivoting within the timeline to the point where the Control\Session Manager key within ControlSet002 was modified, we see the following:

Tue Aug  7 20:25:33 2018 Z
  FILE                       - MA.. [655360] C:\Windows\System32\$I30
  FILE                       - MA.. [56] C:\Program Files (x86)\
  FILE                       - MA.. [56] C:\Program Files\
  FILE                       - MA.. [56] C:\Program Files (x86)\$TXF_DATA
  FILE                       - MA.. [56] C:\Program Files\$TXF_DATA
  FILE                       - .A.. [20] C:\Windows\AppCompat\Programs\RecentFileCache.bcf
  EVTX     WIN-M5327EF98B9   - NetBT/4321;,WIN-M5327EF98B9:0,,
  REG                        - M... HKLM/Software/Wow6432Node/Microsoft/Windows/CurrentVersion/explorer 
  FILE                       - MA.. [520] C:\Users\mpowers\AppData\Local\Microsoft\Windows Mail\Backup\new\
  FILE                       - MA.. [56] C:\Windows\System32\
  FILE                       - MA.. [4096] C:\Windows\SysWOW64\config\systemprofile\AppData\LocalLow\Microsoft\CryptnetUrlCache\Content\$I30
  FILE                       - MA.. [4096] C:\Program Files\$I30
  FILE                       - MA.. [256] C:\Windows\System32\sysprep\Panther\IE\
  REG                        - M... HKLM/System/ControlSet002/Control/Session Manager

Looking nearby within the timeline, we see that there are a number of modifications to both the file system and Registry hives going on leading up to, as well as following that time.

Remember my previous blog post regarding the deletion of forensic artifacts? It appeared that the PrivaZer application was launched at approximately 20:21:51 Z on 7 Aug, and completed its work approximately 10 minutes later; the LastWrite time for the Session Manager key seen above is right in the middle of that activity. As such, it would appear that the reason why there is no AppCompatCache data available is due to the deletion of forensic artifacts by the PrivaZer application.  The later LastWrite time for the key found within ControlSet001 may be due to other factors, such as activity that occurred after the key of interest was deleted.

When engaged in analysis, there are number of components at play, in particular the data, the tool being used, and the analyst.  Any one of these, or any combination thereof, could result in an "issue".  For example, in this case, an examiner could have easily come to the determination that the tool (i.e., the RegRipper plugin) was the issue. I can't remember ever seeing a System hive file that did not have an AppCompatCache value.  As such, why wouldn't it be an issue with the tool? 

It turns out that all of the components for running this issue down were right there.  We had an open source tool, one which we could literally open in Notepad and read what it does, that would start us down the troubleshooting road.  We had other tools available; without WRR, I could have just as easily imported the hive into RegEdit, and from there, seen that the AppCompatibility subkey didn't exist. 

No comments: