Monday, March 02, 2015

How do you "do" analysis?

Everybody remembers "The Matrix", right?  So, you're probably wondering what the image to the right has to do with this article, particularly given the title.  Well, that's easy...this post is about employing various data sources and analysis techniques, and pivoting in order to add context and achieve a greater level of detail in your analysis.  Sticking with just one analysis technique or process, much like simply trying to walk straight through the building lobby to rescue Morpheus, would not have worked.  In order to succeed, Neo and Trinity had to pivot and mutually support each other in order to achieve their collective goal.  So...quite the metaphor for a blog post that involves pivoting, eh?

Timeline Analysis
Timeline analysis is a great technique for answering a wide range of questions.  For malware infections and compromises, timeline analysis can provide the necessary context to illustrate things like the initial infection (or compromise) vector, the window of compromise (i.e., based on when the system was really infected or compromised, if anti-forensics techniques were used), what actions may have been taken following the infection/compromise, the hours during which the intruder tends to operate, and other systems an intruder may have reached to (in the case of a compromise).

Let's say that I have an image of a system thought to be infected with malware.  All I know at this point is that a NIDS alert identified the system as being infected with a particular malware variant based on C2 communications that were detected on the wire, so I can assume that the system must have been infected on or before the date and time that the alert was generated.  Let's also say that based on the NIDS alert, we know that the malware (at least, some variants of it) persists via a Windows service.  Given this little bit of information, here's an analysis process that I might follow, including pivot points:
  1. Load the timeline into Notepad++, scroll all the way to the bottom, and do a search (going up from the bottom) to look for "Service Control Manager/7045" records.
  2. Locate the file referenced by the event record by searching for it in the timeline.  PIVOT to the MFT: parse the MFT, extract the parsed record contents for the file in question in order to determine if there was any time stomping involved.
  3. PIVOT within the timeline; start by looking "near" when the malware file was first created on the system to determine what other activity occurred prior to that event (i.e., what user was logged in, were there indications of web browsing activity, was the user checking their email, etc.)
  4. PIVOT to the file itself: parse the PE headers to get things like compile time, section names, section sizes, strings embedded in the file, etc.  These can all provide greater insight into the file itself.  Extract the malware file and any supporting files (DLLs, etc.) for analysis.
  5. If the malware makes use of DLL side loading, note the persistent application name, in relation to applications used on the system, as well as within the rest of the infrastructure.  
  6. If your timeline doesn't include AV log entries, and there are AV logs on the system, PIVOT to those in order to potentially get some additional detail or context.  Were there any previous attempts to install malware with the same or a similar name or location?  McAfee AV will flag on behaviors...was the malware installed from a Temp directory, or some other location?  
  7. If the system has a hibernation file that was created or modified after the system became infected, PIVOT to that file to conduct analysis regarding the malicious process.
  8. If the malware is known to utilize the WinInet API for off-system/C2 communications, see if the Local Service or Network Service profiles have a populated IE web history (location depends upon the version of Windows being examined). 
  9. If the system you're analyzing has Prefetch files available, were there any specific to the malware?  If so, PIVOT to those, parsing the modules and looking for anything unusual.  
Again, this is simply a notional analysis, meant to illustrate some steps that you could take during analysis.  Of course, it will all depend on the data that you have available, and the goals of your analysis.

Web Shell Analysis
Web shells are a lot of fun.  Most of us are familiar with web shells, at least to some extent, and recognize that there are a lot of different ways that a web shell can be crafted, based on the web server that's running (Apache, IIS, etc.), other applications and content management systems that are installed, etc.  Rather than going into detail regarding different types of web shells, I'll focus just on what an analyst might be looking for (or find) on a Windows server running the IIS web server.  CrowdStrike has a very good blog post that illustrates some web shell artifacts that you might find if an .aspx web shell is created on such a system.

In this example, let's say that you have received an image of a Windows system, running the IIS web server.  You've created a timeline and found artifacts similar to what's described in the CrowdStrike blog post, and now you're read to start pivoting in your analysis.

  1. You find indications of a web shell via timeline analysis; you now have a file name.
  2. PIVOT to the web server logs (if they're available), searching for requests for that page.  As a result of your search, you will know have (a) IP address(es) from where the requests originated, and (b) request contents illustrating the commands that the intruder ran via the web shell.
  3. Using the IP address(es) you found in step 2, PIVOT within the web server logs, this time using the class C or class B range for the IP address(es), to cast the net a bit wider.  This can give you additional information regarding the intruder's early attempts to fingerprint and compromise the web server, as you may find indications of web server vulnerability scans originating from the IP address range.  You may also find indications of additional activity originating from the IP address range(s).
  4. PIVOT back into your timeline, using the date/time stamps of the requests that you're seeing in the web server logs as pivot points, in order to see what events occurred on the systems as a result of requests that were sent via the web shell.  Of course, where the artifacts can be found may depend a great deal upon the type of web shell and the contents of the request.
  5. If tools were uploaded to the system and run, PIVOT to any available Prefetch files, and parse out the embedded strings that point to module loaded by the application, in order to see if there are any additional files that you should be looking to.
Once again, this is simply a notional example of how you might create and use pivot points in your analysis.  This sort of process works not just for web shells, but it's also very similar to the process I used on the IBM ISS ERS team when Chris and I were analyzing SQL injection attacks via IIS web servers; conceptually, there is a lot of overlap between the two types of attacks.

Additional Resources (Web Shells)
Security Disclosures blog post
Yara rules - 1aN0rmus, Loki

Memory Analysis
This blog post from Contextis provides a very good example of pivoting during analysis; in this case, the primary data source for analysis was system memory in the form of a hibernation file.  The case stated with disk forensics, and a hit for a particular item was found in a crash dump file, and then the analyst pivoted to the hibernation file.

Adam did a great job with the analysis, and in writing up the post.  Given that this post started with disk forensics, some additional pivot points for the analysis are available:

  1. Pivoting within the memory dump, the analyst could have identified any mutex utilized by the malware.  
  2. Pivoting into a timeline, the analyst may have been able to identify when the service itself was first installed (i.e., "Service Control Manager" record with event ID 7045).
  3. Determining when the malicious service was installed can lead the analyst to the initial infection vector (IIV), and will be extremely valuable if the bad guys used anti-forensic techniques such as time stomping the malware files to try to obfuscate the creation date.
  4. Pivot to the MFT and extract records for the malicious DLL files, as well as the keystroke log file.  Many of us have seen malware that includes a keylogger component that will continually time stomp the keystroke log file as new key strokes are added to it.  

"Doing" Analysis
I received an interesting question a while back, asking for tips on how I "do analysis".  I got to thinking about it, and it made sense to add my thoughts to this blog post.

Most times, when I receive an image, I have some sort of artifact or indicator to work with..a file name or path, a date/time, perhaps a notice from AV that something was detected.  That is the reason why I'm looking at the image in the first place.  And as a result, producing a timeline is obviated by the questions I need to answer; that is to say, I do not create a timeline simply because I received an image.  Instead, I create a timeline because that's often the best way to address the goals of my exam.

When I do create a timeline, I most often have something to look for, to use as an initial starting or pivot point for my analysis.  Let's say that I have a file that I'm interested in; the client received a notification or alert, and that led them to determine that the system was infected.  As such, they want to know what the malware is, how it got on the system, and what may have occurred after the malware infected the system.  After creating the timeline, I can start by searching the timeline for the file listing.  I will usually look for other events "around" the times where I find the file listed...Windows Event Log records, Registry keys being created/modified, etc.

Knowing that most tools (TSK fls.exe, FTK Imager "Export Directory Listing..." functionality) used to populate a timeline will only retrieve the $STANDARD_INFORMATION attributes for the file, I will often extract and parse the $MFT, and then check to see if there are indications of the file being time stomped.  If it does appear that the file was time stomped, I will go into the timeline and look "near" the $FILE_NAME attribute time stamps for further indications of activity.

One of the things I use to help me with my analysis is that I will apply things I learned from previous engagements to my current analysis.  One of the ways I do this is to use the wevtx.bat tool to parse the Windows Event Logs that I've extracted from the image.  This batch file will first run MS's LogParser tool against the *.evtx files I'm interested in, and then parse the output into the appropriate timeline format, while incorporating header tags from the eventmap.txt event mapping file.  If you open the eventmap.txt file in Notepad (or any other editor) you'll see that it includes not only the mappings, but also URLs that are references for the tags.  So, if I have a timeline from a case where malware is suspected, I'll search for the "[MalDetect]" tag.  I do this even though most of the malware I see on a regular basis isn't detected by AV, because often times, AV will have detected previous malware infection attempts, or it will detect malicious software downloaded after the initial infection (credential dumping tools, etc.).

Note: This approach of extracting Windows Event Logs from an acquired image is necessitated by two factors.  First, I most often do not want all of the records from all of the logs.  On my Windows 7 Ultimate system, there are 141 *.evtx files.  Now, not all of them are populated, but most of them do not contain records that would do much more than fill up my timeline.  To avoid that, there are a list of less than a dozen *.evtx files that I will extract from an image and incorporate into a timeline.

Second, I often work without the benefit of a full image.  When assisting other analysts or clients, it's often too cumbersome to have a copy of the image produced and shipped, when it will take just a few minutes for them to send me an archive containing the *.evtx files of interest, and for me to return my findings.  This is not a "speed over accuracy" issue; instead, it's a Sniper Forensics approach that lets me get to the answers I need much quicker.

Another thing I do during timeline analysis is that I keep the image (if available) open in FTK Imager for easy pivoting, so that I can refer to file contents quickly.  Sometimes it's not so much that a file was modified, as much as it is what content was added to the file.  Other times, contents of batch files can lead to additional pivot points that need to be explored.

Several folks have asked me about doing timeline analysis when asked to "find bad stuff".  Like many of you reading this blog post, I do get those types of requests.  I have to remember that sometimes, "bad stuff" leaves a wake.  For example, there is malware that will create Registry keys (or values) that are not associated with persistence; while they do not lead directly to the malware itself (the persistence mechanism will usually point directly to the malware files), they do help in other ways.  One way is that the presence of the key (or value, as the case may be) lets us know that the malware is (or was) installed on the system.  This can be helpful with timeline analysis in general, but also during instances when the bad guy uses the malware to gain access to the system, dump credentials, and then comes back and removes the malware files and persistence mechanism (yeah, I've seen that happen more than a few times).

Another is that the LastWrite time of the key will tell us when the malware was installed.  Files can be time stomped, copied and moved around the file system, etc., all of which will have an effect on the time stamps recorded in the $MFT.  Depending on the $MFT record metadata alone can be misleading, but having additional artifacts (spurious Registry keys created/modified, Windows services installed and started, etc.) can do a great deal to increase our level of confidence in the file system metadata.

So, I like to collect all of those little telltale IOCs, so that when I do get a case of "find the bad stuff", I can check for those indicators quickly.  Do you know where I get the vast majority of the IOCs I use for my current analysis?  From all of my prior analysis.  Like I said earlier in this post, I take what I've learned from previous analysis and apply it to my current analysis, as appropriate.

Sometimes I get indicators from others.  For example, Jamie/@gleeda from Volatility shared with me (it's also in the book) that when the gsecdump credential theft tool is run to extract LSA secrets, the HKLM/Security/Policy/Secrets key LastWrite time is updated.  So I wrote a RegRipper plugin to extract the information and include it in a timeline (without including all of the LastWrite times from all of the keys in the Security hive, which just adds unnecessary volume to my timeline), and since then, I've used it often enough that I'm comfortable with the fidelity of the data.  This indicator serves as a great pivot point in a timeline.

A couple of things I generally don't do during analysis:
I don't include EVERYTHING into the timeline.  Some times, I don't have everything...I don't have access to the entire image.  Someone may send me a few files ($MFT, Registry hives, Windows Event Logs, etc.) because it's faster to do that than ship the image.  However, when I do have an image, I very often don't want everything, as getting everything can lead to a great deal of information being put into the timeline that simply adds noise.  For example, if I'm interested in remote access to a system, I generally do not include Windows Event Logs that focus on hardware monitoring events in my timeline.

I have a script that will parse the $MFT and display the $STANDARD_INFORMATION and $FILE_TIME metadata in a timeline...but I don't use it very often.  In fact, I can honestly say that after creating it, I haven't once used it during my own analysis.  If I'm concerned with time stomping, it's most often only for a handful of files, and I don't see that as a reason for doubling the size of my timeline and making it harder to analyze.  Instead, I will run a script that will display various metadata from each record, and then search the output for just the files that I'm interested in.

I don't color code my timeline.  I have been specifically asked about this...for me, with the analysis process I use, color coding doesn't add any value.  That doesn't mean that if it works for you, you shouldn't do it...not at all.  All I'm saying is that it doesn't add any significant value for me, nor does it facilitate my analysis.  What I do instead is start off with my text-based timeline (see ch. 7 of Windows Forensic Analysis) and I'll create an additional file for that system called "notes"; I'll copy-and-paste relevant extracts from the full timeline into the notes file, annotating various things along the way, such as adding links to relevant web sites, making notes of specific findings, etc.  All of this makes it much easier for me to write my final report, share findings with other team members, and consolidate my findings.


Bellerophon said...

Excellent article! I hear this type of stuff all the time and when I tell people to "pivot" on data I get funny looks. I was once told that "pivot" is too technical of a word. Really? No one ever played basketball?

Harlan Carvey said...


Thanks for the you have any examples of pivoting during analysis that you can share?

Jonathan Rajewski said...

In your memory section under point "4" - are you saying the examiner should be looking at the MFT from disk or the memory image? Based on your experience does one offer better artifacts? I've never compared that before.

Harlan Carvey said...


What would you recommend?

Bellerophon said...

Well, at a very high level I have been working with level 1 SOC at my organization on how to use the SIEM and pivot on data from different log sources. For instance using static malware URL strings to pivot and find new domain host names or IPs. Or using a user agent string to do something similar.

P.S. - This is what I love about our small DFIR community: One of my Champlain professors commenting on a blog that apparently we all read. As well as seeing Harlan's comments on another associate's blog/linkedin post today. Gotta love the sharing/contributions/discussions made to the community.

Harlan Carvey said...


You should see the sharing that goes on in other home brewing, for example. So many more people in communities like that are willing to share, many times without prompting. It's absolutely amazing...

Anonymous said...

THIS was a GREAT article. I haven't really heard this methodology being referred to as 'pivoting' but it works. Here of late, I've been working more and more with Win8 and 'pivoting' as you call it, is really the only way to be thorough. Win8 has data spread out all over the place so that if an examiner sees something in one place.. it absolutely must be verified somewhere else...thus 'pivoting.'