Tuesday, June 23, 2015

The Blue Team Myth

The 2015 M-Trends Report states that the median number of days that threat groups were present in a victim's network before detection was 205 days, which is down 24 days from 2013.  But that's the median number of days...the longest persistence was 2,982 days.  That's over 8 years.  The report also states that 69% of organizations were informed of the breach by an external entity.

The 2015 TrustWave Global Security Report indicates that 81% of companies did not detect breaches on their own, and that the median length of time it took to detect a breach was 86 days.

What do these numbers tell us?   They tell us that the organizations in question did not detect the breaches, but when they called in consultants to assist, those consultants found evidence of the breaches that allowed them to identify for how long (to a point, of course) that the intruders had persisted within the infrastructure.  I'm going to go out on a limb here and just say that I recognize that the consultants may not have been able to trace the breach information back to the initial infection vector (IIV) in every case, so the numbers in the report are likely just a little fuzzy.  

What the numbers tell us is that the consultants, based on their combined experience, knowledge, and methodologies, were able to find indicators and artifacts that others did not.  This means that the local IT staff either wasn't looking, or didn't have a complete understanding of what to look for.

The numbers in the reports are supported by my own experiences, as well as those of a myriad of other responders.  A while back, I worked on an engagement during which we were able to trace the adversary's activities back to the original phishing email and attached weaponized PDF.  The emails had been received 21 months before we had been called.  I've done analysis on breaches where indicators went back two years or more, and we were never able to track down the IIV.

The numbers also tell us that the adage that attackers need to be right 1 out of 100 times and defenders need to be right 100 out of 100 times doesn't hold, and that it's a myth.  What artifacts did the consultants who provided input to the reports find, and could administrators of the compromised infrastructure have found those artifacts, as well, if they'd been looking?  Based on my own personal experiences, I would say that the answer is a definite yes.

By now, we are all well aware that prevention doesn't work, that regardless of how many technical measures we put in place, that as long as people are involved in some way, at some point someone will make their way into an infrastructure.  So what we need to do is include detection and response; put our protection mechanisms in place, monitor them, look/hunt for issues throughout our infrastructure, and be ready to take some sort of action when issues are found.

In my previous blog post, I shared some thoughts on hunting, in short, describing, in general terms, what folks can look for within their environments.  One person commented (via Twitter) that I should spend less time talking about it, and more time listing those artifacts that hunters should look for.

So what should you look for? Check out this video of Lee's presentation...he shares a lot of great information.  Much like the information here, or in any of the HowTo blog posts from July, 2013.  That's a start.  Eventmap.txt has some interesting tips, as do many RegRipper plugins.  Want more?  Take a hard look at your threat intelligence...if you're getting it from a vendor, does it give you what you need?  How about the threat intel you're developing internally?  Similarly, does it meet your needs?  MD5 hashes and IP addresses aren't going to get you as far as TTPs, if you have the processes and methodologies to find those TTPs.

Monday, June 22, 2015

Hunting, and Knowing What To Hunt For

Like many others of my generation, when I was a kid I'd go and play outside for hours and hours.  Sometimes, while running through the woods, I'd see trash...but seeing it often, I wouldn't think much of it.  Sometimes it was a tire in the creek, and other times it might be bottles or cigarette butts in a small cluster.

When I went through my initial military training, we spent a lot of time in the outdoors, but during the first 6 months, there was no real "this is what to look for training".  It wasn't until a couple of years later, when I returned to be an instructor that the course was teaching new officers the difference between moving through the woods during the day and at night.

Much later, after my military service ended, I got engaged in horseback riding, particularly trail riding.  Anyone who's ever done this knows that you become more aware of your surroundings, for a variety of reasons, but most importantly, the safety of your horse, you, and those you may be riding with.  You develop an awareness of your surroundings; what's moving near you, and what's further away.  Are there runners or walks with dogs (or children in strollers) down the trail?  Sometimes you can be warned of the impending approach of others not so much visually, as by what you hear or smell (some riders like to smoke while they're riding).  You can tell what is or might be in the area by visual observation (scat, droppings, etc.), as well as listening, and even smell.  Why is this important?   There's a lot that can spook a horse...horses will smell a carcass well before a human will, and as you ride up, a gaggle of vultures might suddenly take flight.  Depending on your horse's temperament, a flock of turkey hens running through a field might spook them or just catch their attention.  

If it's recently rained in the area where I'm riding, I'll visually sweep the ground looking for footprints and signs of animals and people.  If I see what appear to be fresh impressions of running shoes with dog paw prints near by, I might be looking for a walker or runner with a dog.  Most runners have a sense of courtesy to slow down to a walk and may be even say something when approaching horses from the rear.  Most...not all.  And not everyone who walks a dog really thinks about whether their dog is habituated to horses, or even how the dog will reach when they see horses.  I'll also look for signs of deer, because they usually (albeit not always) move in groups.  More than once in the springtime, I've come across a fawn coiled up in the tall grass...deer teach their fawns to remain very still, regardless of the circumstances, while they go off to forage.  More than once I've seen the fawn well before the fawn has lost its nerve and suddenly bolted from its hiding place.

Now, because of this level of awareness, I had an interesting experience several years ago while hiking with friends at the base of Mount Rainier.  At the park entrance, there was a small lake with signs prohibiting fishing, and a sign telling hikers what to do if they spotted a bear.  We hiked about 3 1/2 miles to a small meadow with wild blueberries growing.  The entire way out and back, there were NO signs of animal life...no insects, no sounds of birds or squirrels.  No scat or markings of any kind.  The absence of any and all fauna was very extremely evident, and more than a bit odd.

So what?
Okay...so what's my point?  If you're familiar with the environment, and aware of your surroundings while performing DFIR work, and know what should be there, you know what to look for, as well as what data sources to go to if you're looking for suspicious activity.  It's all about knowing what to hunt for when you're hunting.  This is particularly true if you're performing hunting operations in your own environment; however, it can also be applied in cases where you're a consultant (like me), engaging with an unfamiliar infrastructure.

In a recent presentation  (BrightTalk, may require registration) at InfoSecurity Europe 2015, Lee Lawson discussed some of the indicators that are very likely available to you right now, particularly if you've done nothing to modify the default audit configuration on Windows systems.

Not long ago, the folks at Rapid7 shared this video, describing four indicators you could use to detect lateral movement within your infrastructure.  Unfortunately, two of them are not logged as a result of the default audit configuration of Windows systems, and the presenter doesn't mention alternate Windows Event Log source/ID pairs you can look for instead.

There are a lot of ways to hunt for indications of activity, such as lateral movement, but what needs to happen is that admins need to start looking, even if it means building their list of indicators (and hopefully automating as much of it as makes sense to do...) over time.  If you don't know what to look for, ask someone.

If someone were to ask me (this is my opinion, and should not be misconstrued as a statement of my employer's opinion or business model) what was the one thing they could do to increase their odds of detecting malicious behavior, I'd say, "install Sysmon", but that's predicated on someone actually looking at the logs.

Addendum: @Cyborg_DFIR over on Twitter read the original version of this post (prior to the addendum) and felt that I talked more about explaining what you meant instead of talking about it. Fair enough.  I don't dispute that.  But what I will say is that I, and others, have talked about what to look for.  A lot. Does that mean it should stop?  No, of course not.  Just because something's been talked about before doesn't mean that it doesn't bear repeating or being brought up again.

During July 2013, I wrote 12 articles whose titles started with "HowTo:" and went on to describe what to look for, if you were looking for specific things.  More recently, I addressed the topic of detecting lateral movement within the last month.

So, to @Cyborg_DFIR and others who have similar thoughts, questions or comments...I'm more than willing to share what I know, but it's so much easier if you can narrow it down a bit.  Is there something specific you're interested in hearing about...or better yet, discussing?

Thursday, June 11, 2015

RegRipper plugin update

I just pushed out an update to the appcompatcache.pl plugin, and committed it to the Github archive.  The update was based on a request I'd received to make the output of the tool a bit more manageable, and that was right along the lines of something I'd been thinking about doing for some time, anyway.

In short, the update simply puts the output for each entry on a single line, with the app path and name first, then the date, then other data (that's specific to 32-bit XP, actually), and finally, the executed flag.  Each segment is separated by two spaces.

So, what does this mean?  Well, the format puts everything on one line, so if you redirect the output to a file and any searches you run will give you the entire line, not just the first line (as with the old format).

Another way to use this new format is the way I like to use it, to determine if there's anything in the ShimCache data that requires my attention:

rip.pl -r d:\cases\local\system -p appcompatcache | find "programdata" /i

Done.  That's it.  Using DOS commands, it's quick and easy to run through a data sample quickly to see if there's anything of interest.

What techniques do you use during Registry analysis?

Sunday, June 07, 2015


If you haven't heard, the new SANS DFIR "Evidence of..." poster is available.  I can see looking at the second page of the poster that they've added some items of interest that have been talked about recently.  For example, under "Browser Usage", I see mention of Mari's Google Analytics Cookies, and under "Program Execution", I see a reference to the AmCache.hve and the RecentFileCache.bcf files.

What's New in Windows 10
Speaking of "evidence of...", one question I see quite often is, "..what's new in Windows {insert newest version number}?"  Some folks at Champlain College have written up a Windows 10 Forensics guide.

The Problem with RegRipper 
I was reading herrcore's blog post in which malware that persists via a user's shell extensions, and came across the following statement in the post:

The problem with the two tools I mentioned; RegRipper (shellext.pl plugin) and Autoruns is that they rely on the Shell Extension to be registered using the standard method with HKEY_CLASSES_ROOT.

This quote is taken from the first sentence under a header that says, "A Blind Spot in our Incident Response Tools".

Since I first released RegRipper, my intention was that it would be community-based. That is, that by providing a framework such as RegRipper, the strength of the tool would be not simply that it would be downloaded and blindly pointed at Registry hives, but rather that analysts would either write and contribute their own plugins, or request (providing sample data, as well) plugins be developed.  That's right.  I totally get that not everyone programs in Perl...which is why I provide Windows executable files for the framework.  This is also why if someone doesn't program in Perl but strongly believes that they have a good idea for a plugin, all they need to do is write up a concise description of what it is they'd like to see, and email it to me along with some sample data.  When this has happened, I've been able to turn around a functioning plugin pretty quickly...usually within 4 hrs, and often within 1 hr.

This sort of thing has happened before.  At the SANS DFIR Summit in 2012, I sat in a presentation, during which the speaker stated, "...RegRipper does not have a plugin for {insert functionality}...".  That speaker also stated that RegRipper "does not scale to the enterprise" (which is was NEVER designed to do); however, that speaker had never reached to me to say, "...here's a plugin I wrote...", nor did they ever ask me for such functionality in a plugin.  I did find out after the presentation that the presentation had been written several months prior, and that a plugin had been written...the speaker had simply not updated their materials.

I get that there are some roadblocks to having a community-based tool...I completely understand that not everyone programs (nor wants to learn), and that of those who do, not all program in Perl.  That's okay.  I also understand that a major roadblock for a lot of DFIR analysts is simply...communicating.  There are a lot of reasons for this, but the fact is that most DFIR analysts simply do not want to communicate with others within the community.

I would suggest that the real "blindspot" or "problem" isn't whether or not RegRipper (or any other tool) contains specific functionality, but rather, what are we, as analysts, are willing to do to communicate with each other.

That being said, I contacted herrcore, and as a result of an email exchange where he sent me some sample data.  I'm researching his findings a little bit more to see if I can modify a current plugin (inprocserver.pl, shellext.pl), or would it make sense to write a new one?

In the meantime, here are some links where I've discussed RegRipper in this blog:
Mar, 2011 - Using RegRipper
Jul, 2012 - Thoughts on RegRipper Support
Aug, 2012 - RegRipper Updates

Addendum, 8 June: This morning, I updated a plugin, and created two others, and committed them to the plugin repository.

inprocserver.pl - I updated this plugin by adding "programdata" to the @alerts list in the alertCheckPath() function, and ran it against the test data that herrcore provided.  The result was:

ALERT: inprocserver: programdata found in path: c:\programdata\{9a88e103-a20a-4e

cached.pl - new plugin; run it against the NTUSER.DAT hive to get the values from the \Shell Extensions\Cached key.  Running it via rip.exe displays the time value, and the two available CLSID values; I added a simple lookup for the second one.  As such, the output appears as follows:

Tue May 26 05:31:59 2015  First Load: {BDFA381D-D8C6-441A-BA17-95EB7FEBEA81} (IDriveFolderExt)
Tue May 26 05:34:57 2015  First Load: {9C73F5E5-7AE7-4E32-A8E8-8D23B85255BF} (IShellFolder)
Tue May 26 05:34:57 2015  First Load: {7007ACC7-3202-11D1-AAD2-00805FC1270E} (IShellFolder)
Tue May 26 05:35:05 2015  First Load: {F6BF8414-962C-40FE-90F1-B80A7E72DB9A} (IDriveFolderExt)

Simple enough.  However, if you use your command line fu, you can do something like this:

rip.pl -r d:\cases\herrcore\ntusertest2.dat -p cached | find "F6BF"
Launching cached v.20150608
Tue May 26 05:35:05 2015  First Load: {F6BF8414-962C-40FE-90F1-B80A7E72DB9A} (IDriveFolderExt)

cached_tln.pl - new plugin; similar to cached.pl, except that this plugin's output can be added directly to a timeline.  Also, this plugin does not have the lookup that cached.pl has...below is an example of the output:

rip.pl -r d:\cases\herrcore\ntusertest2.dat -p cached_tln -u keydet89 | find "F6BF"
Launching cached_tln v.20150608
1432618505|REG||keydet89|Cached Shell Ext First Load: {F6BF8414-962C-40FE-90F1-B80A7E72DB9A}

So, there you have it.  Typical caveats apply...such as "..these are based on extremely limited test data..", etc.

Also, something I wanted to mention as a result of looking through the hives herrcore sent me was that there were other indicators added to the USRCLASS.DAT hive, as seen in the image to the left.  You can see that in addition to the CLSID\{GUID}\InprocServer32 key, another key was added with the path Drive\ShellEx\FolderExtensions\{GUID}.

Thursday, May 28, 2015

Detecting Lateral Movement

Almost two years ago, I posted this article that addressed how to track lateral movement within an infrastructure.  At the time, I'd been using this information successfully during engagements, and I still use it today.

This morning, I saw this video from Rapid7, and I thought that Mike did a great job with the presentation.  Mike made some very good points during his presentation.  For example, "SMB" is native to a Windows infrastructure, and with the right credentials, an adversary can go just about anywhere they please.

There were some things missing in the presentation, some caveats that need to be mentioned; I do understand that they were likely left out for the sake of time.  However, they are important.  For example:

Security-Auditing/4698 events - Scheduled Task creation; under Advanced Security Audit Policy settings, for Object Access, you need to have Audit Other Object Access Events enabled for this event to appear in your Windows Event Logs.

Security-Auditing/4697 events - Service installation; similar to the previous events, systems are not configured to audit for system creation via the Security Event Log by default.

So, the take-away here is that in order for these (and other) events to be useful, what admins need to do is properly configure auditing on systems, as well as employ a SEIM with some sort of filtering capability.  Increasing auditing alone will not be useful...I've seen that time and time again when an incident is identified; auditing is ramped up suddenly, and the Security Event Logs start filling up and rolling over in a matter of a few hours, causing valuable information to be lost.  The best thing to do is to enable auditing that makes sense within your infrastructure ahead of time, employing the appropriate settings (what to audit, increasing the default size of the Windows Event Log files, etc.) before an incident occurs.

Also, consider the use of MS's Sysmon, sending the collected data to a SEIM (Splunk??).  Monitoring process creation (including the command line) is extremely valuable, and not just in incident response.  For IR, having the process creation information available (along with a means to monitor it in a timely manner) reduces IR engagements from days or weeks to hours or even minutes.  If setting up Sysmon, Splunk, and filters is too daunting a task, consider employing something like Carbon Black.

Thanks to Rapid7 for sharing the video...it's some great information.

Description of Security Events in Windows 7/Windows Server 2008 R2

Tuesday, May 26, 2015

Links and Stuff

Registry Goodness
I recently wrote a RegRipper plugin, based on this KB article; on 26 May, I committed it to the plugin repository.  I had tweeted to ask the DFIR community if this information was relevant to their investigations, and there was not a great deal of response on the topic...although there was apparently some confusion.  I hope that folks take the time to try it, and I hope it's of some use to the DFIR community.  I don't often (scratch that...in 15+ years of doing DFIR work, I've never...) need to determine the history of GPOs assigned to a system.

Speaking of RegRipper plugins, Dan posted recently about how he completed the SANS CEIC 2015 Challenge.  While he completed the challenge using Eric Zimmerman's Registry Explorer tool, he did state toward the end of the post that he could've used RegRipper to complete the challenge, as well.

From the recent CEIC Conference, you can see David Dym's slides for his Improving Windows External Device Investigations presentation.  I know that no matter how many times this subject is addressed and discussed, there will always be confusion as to what resources are available on Windows systems if you are conducting one of these investigations.  I think it's great that we've got others talking about this topic, particularly because there seems to be so much confusion in this area.  Cory Altheide and I published the initial research into this topic in 2005 (there's a link here), and as new versions of Windows have come out, more information has become available regarding not only which devices were connected to systems, but also which user may have accessed the device.

Speaking of the Registry, Eric Zimmerman recently released a command line tool for interacting with (including searching) a Registry hive file for specific items.  Be sure to get version 0.6 of the tool.  Eric's been doing a lot of work in creating freeware tools for accessing the Registry, so be sure to check out his other offerings.

Not related to Registry analysis, but TrendMicro recently had a blog post about what they seem to be presenting as a variation in autostarting malware.  More than anything else, the post left me more than a little confused...it says that the intruders found an application that was set to run when the system started, and then modified the application's import table by adding a reference to a malicious DLL.  It was the next sentence that left me confused:

It is almost impossible to find differences between the original version and the modified ones, as even their file sizes are almost identical.

The post then goes on to say that 4 of the 5 infected applications were discovered as the modified versions weren't signed.  However, there still seems to be more going on here, because adding a DLL to the import table of a .exe file, and then referencing the malicious function should make something of an impact on the size of the application, as well as other aspects of the system itself (MFT, USN change journal, etc.).

Speaking of stuff starting, Corey posted recently regarding some testing he'd done with an MSWord document that would launch an executable.

Something I really like about Corey's post is that there's enough detail in the way he presented the material to not only replicate what he did (if you can or want to get a copy of the file he used...).  Also, there's enough information in the post to create things like searches for the pattern after running LogParser against the Sysmon Event Log file, as well as to write Carbon Black watchlist queries.

Tuesday, May 05, 2015


Plugin Updates
Eric Zimmerman reached to me a little while ago and let me know that he'd taken a look at the AppCompatCache data from a Windows 10 system, and found that...wait for it...the format of the data was different from previous versions of Windows.  O.  M.  G.

Thanks to the heads up from Eric, I've updated the RegRipper plugins for parsing this data.  However, my testing was extremely limited; I had only one System hive file (from a Windows 10 TP VM that I'd set up) on which to test the parsing code.  A dearth of testing data has been an issue since I started writing tools, and it seems that even TaoSecurity has recognized the need for test data.

Interestingly enough, I happened across a System hive file from a Windows 2012 system, and the updates to the parser seemed to work just fine.  Again, I said "a" hive...so testing has been extremely limited.

When working with the AppCompatCache or "ShimCache" data, analysts need to remember the context of the information, and in particular, the time stamps in the data. In most cases, the time stamp is the last modification time for the file in question, from the file system metadata, specifically, the $STANDARD_INFORMATION attribute. It is NOT the date and time that the application was executed.

Dumping Passwords
Speaking of the Registry, I ran across this little gem that describes how to dump passwords in plain text from Windows 8.1/2012 systems.  I'm a big proponent for finding out what things look like on systems, and it's pretty clear reading through the blog post what an analyst would look for in an acquired image, to determine if something similar to what was described in the post had occurred on the system.

However, I'll put this out there...rather than hoping to find these indicators on a system, why not make IR scoping easier on yourself through the use of process creation monitoring at the endpoints?

Malware Persistence
Here's a good blog post on malware persistence.  There's some focus on MS's AutoRuns tool, so it's likely to be familiar to a lot of folks.  What I thought was interesting is the number of times we see the Run key being the point of persistence for malware; while many will suggest that this location is 'well known', there are also those of us who see it used time and time again, even when the incident is 'detected' through external, third-party notification.  Some may think that the Run key is passe, but hey, it still works, and works well...so why not use it, right?

Every now and then, I still get an opportunity to analyze Windows XP and 2003 systems...most often, I tend to find myself working with Windows 7 and Windows 2008 R2 systems.  Working with older versions of Windows can sometimes necessitate the use of different tools in order to conduct analysis; specifically, when working with the Event Logs, they're in a different location, as well as in a different binary format.

When working with Windows XP and 2003 (and yes, Windows 2000) Event Logs (*.evt files), I'll use evtrpt and evtparse.  These tools were written specifically for the binary format of the *.evt files found on Windows 2000, XP, and 2003 systems, and are not intended for use against the *.evtx files found on Vista+ systems.

This is a tool I wrote a while back to provide me with information about the contents of an EVT file; how many records exist, how many source/ID pairs exist, and what date range do the events cover.  What's cool about this tool is that it doesn't use the MS API...which means that it can be run on Linux, and it doesn't rely on the header information of the EVT file to tell it how many records exist.

Evrtp is a command line tool, and takes just one argument...the path to the file you're interested in parsing.  If you just type the name of the tool at the command prompt, you'll get a message that says "You must enter a filename".  That seems to be pretty straightforward, and adding the full path to the .evt file of interest is all I need to do.

So, I run the tool against an Event Log file that I've extracted from an image, and one of the things I see in the output is the date range for the event records in that file:

Fri Sep 27 17:32:26 2013 to Tue Apr 28 17:37:58 2015

Cool, it covers the time that I'm interested in.  Above the date range in the output, I get a list of event sources and IDs, as well as a count of each.  For login attempts, the tool also breaks down the login/logoff type.  For example, I see the following entry:

Security                                      538,3    18938

What this tells me is that for the source "Security", there are 18938 events with ID 538 and type 3.

I ran the tool against the other Event Log files from the system, as well.  For the Application Event Log, the date range was:

Thu Jan 10 01:52:02 2013 to Tue Apr 28 15:36:04 2015

What I found most interesting from the output of the tool was this entry:

Symantec AntiVirus                               51        2

Remember the tool I use for Windows Event Logs, wevtx.bat?  The one that uses the eventmap.txt file?  Well, in that file, Symantec AntiVirus/51 event records indicate that the product detected malware.  Understanding the context of these event source/ID pairs can be extremely valuable, and the evtrpt tool can give you an idea of what pairs are available.  For McAfee, I'd look for McLogEvent/257 pairs.

Something of interest from the output of the System Event Log was:

Application Popup                                26       11
Application Popup                                44        1

So, hopefully by now you can see how useful this tool can be for a quick look at the Event Log files.

This tool operates similar to the evtrpt tool, but the output allows you to see more about the Event Log records.  Typing just the name of the tool at the prompt will show you the syntax information, along with example command lines you can use to run the tool.  I use this tool to parse through the *.evt file (or files) and add the entries to a timeline of system activity.  The command line to launch this tool does not take many arguments, because most of the data that you may want to include in each timeline entry (system name, user name, etc.) is included within each event record.

Why Do I Need The Event Source and the event ID?
Something I see a great deal of within the DFIR community is that Windows Event Log records are referred to only by their ID number.  Ok, you're probably wondering...so what?  Who cares?  Well, it can be important...if someone says that they have mulitple event ID 4100 records, I would have to ask, which source?  There are a number of event IDs that have multiple different event sources, each of which can provide completely different context to the situation.

So What?
Why does any of this matter? Context is extremely important when conducting analysis, and in particular when communicating findings to other analysts, as well as to clients.  One example was mentioned earlier in this post...the time stamps in the AppCompatCache/ShimCache data is the last modification time of the file, from the file system metadata (the Mandiant white paper is only 5 pages long and is an easy read...).

Another example comes from the output of another RegRipper plugin...the LastWrite time for one of the subkeys from the USBStor key is just that...the key LastWrite time, which is analogous to a file's last modification time.  It is NOT the last time that USB device was written to.

So, when communicating findings from the Event Log (or, Windows Event Log on Vista+ systems), providing the event source AND ID can be extremely important for context.  If someone just says, "...event ID 4100...", then the very next question should be, "...which event source?"  When documenting findings in your case notes, include both the source and ID, as well as a reference, for (possible) inclusion in the eventmap.txt file.