Tuesday, May 08, 2018

EDR Obviates Compliance

Okay, you'll have to excuse me for the title of this blog post...I needed something to get your attention, and get you to read this, because it's important. I know that stating that "EDR obviates compliance" is going to cause a swirl of emotions in anyone's mind, and my intention was to get you to read on and hear me out; 280 characters simply is not enough to share this line of reasoning.

GDPR is just around the corner, and article 4 of GDPR defines a "personal data breach as"...

...‘personal data breach’ means a breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data transmitted, stored or otherwise processed...

Something to be aware of is that this definition includes ransomware, as it includes "access to" and "alteration of". 

As security professionals have been saying for quite some time now, breaches of your perimeter are going to happen, it's simply inevitable.  However, per GDPR, it's not a reportable breach until something happens with respect to 'personal data'.  The simple fact is that without adequate/suitable instrumentation and visibility, you simply won't know if personal data has been accessed during a breach.  EDR provides that visibility.

Where Are We Now?
Right now, we're "right of breach".  Most organizations are simply unprepared for breaches to occur, and breach identification comes in the form of third-party external notification.  This notification comes weeks or even months after the initial intrusion occurred, during with time the dearth of instrumentation and "evidence decay" makes it nearly impossible to definitively determine what the adversary may have done, with any degree of certainty. 

The 2018 Nuix Black Report states that 98% of those surveyed stated that they could breach the perimeter of a target organization and exfil data in 15 hrs or less.  Compare that to the 2017 Ponemon Institute report that indicated that the 'dwell time' (the time between a breach actually occurring and it being detected) was 191 days, and it's still in the triple digits.  I hardly seems far, but that's reality in today's terms.

Where we need to be is "left of breach".  We know breaches are going to occur, and we know that clean up is going to be messy, so what we need to do is get ahead of the game and be prepared.  It's not unlike boxing...if you step into the ring with gloves on, you know you're going to get hit, so it behooves you to get your gloves up and move around a bit, lessening the impact and effect of your opponent's punches. In cybersecurity, we move "left of breach" by employing monitored EDR (MDR) so that breaches can be detected and responded to early in the adversary's attack cycle, allowing us to contain and eradicate the adversary before they're able to access "critical value data", of CVD.  CVD can consist of "personal data", or PII; credit card data, or PCI; healthcare data, or PHI; or it can be intellectual property, or IP.  Regardless of what your "CVD" is, the point is that you want to detect, respond to and stop the bad guy before they're able to access it, and the only way you can do that is to instrument your infrastructure for visibility.

Chris Pogue wrote this blog post, addressing the "cost of breach" topic, almost three years ago.  With the help of our wonderful marketing department, I recently gave this webinar on reducing costs associated with breaches.  Things haven't changed...breaches are becoming more expensive; more expensive to clean up, more expensive in terms of down-time and lapses in productivity, and more expensive in terms of legal fees and fines.

While researching information for the webinar, one of the things I ran across was the fines that would have been associated with GDPR, had GDPR been in effect at the time that the Equifax breach.  In 2016, Equifax reportedly earned $3.145B USD.  High end GDPR fines can be "20M euros or 4% of world-wide operating revenue, whichever is greater", bringing the fine to $125.8M USD.  That's just the GDPR fine; that does not include direct costs associated with the IR response and investigation, subsequent notification, nor any indirect costs.

On 1 May, Alabama became the 50th US state to enact a data breach notification law, SB 318.  As you read over the stipulations regarding reporting detailed in the law, keep in mind that for the 50 US states, everyone is different.  If you store or process the personal data for US citizens, the cost associated with simply determining to which states you actually have to report is enormous!  And that's before you even report...it doesn't include the actual cost of notification, it's just the cost of determining to whom (which states) you need to report!

So where does endpoint detection and response (EDR) come in?

I've been doing DFIR work for about two decades now, and most of the work I've done has been after an incident has occurred.  Early on, the work involved a system or two thought to be infected or compromised in some manner, but it was always well after the supposed infection or compromise occurred.  In most cases, questions involving things like, "...what data was taken?" simply could not be answered.  Either the information to answer the question didn't exist due to evidence decay, or it didn't exist because that information was was never actually recorded or logged.

EDR/MDR solutions allow you to detect a breach early in the adversary's attack cycle, as one of the things monitored by most, if not all, EDR tools is the processes executed on systems.  Yes, you get a lot of 'normal' user stuff (allowing you to detect fraud and insider threats), but you also get a detailed look at the adversary's activities.

To see some really good examples of process trees showing what various attacks look like, go to Phil Burdette's Twitter feedThis tweet includes an example of a phish, with the user clicking "Enable Content" on a malicious document, leading to the implant being downloaded via bitsadminThis tweet illustrates a great example of credential theft.  I could go on and on about this, but I think that the point has been made...EDR gives you a level of visibility that you don't get through any other instrumentation.

Where I've used this information to great effect was detecting an adversary's activities within a client's infrastructure several years ago; the tool being used at the time alerted on the use of an archive utility known as "Rar".  The adversary had renamed the executable so that it was no longer 'rar.exe', but adding a password to protect the archive is not something the adversary could change, and it's what was detected in this case.  So, we had the fact that data was marshaled and archived for exfiltration (along with the password), and we further was that after exfiltrating the archives, the adversary deleted the archived data.  However, once we got an image of the system we were able to carve unallocated space and recover a number of the archives, and using the adversary's password, we were able to open the recovered archives and see exactly what data was taken. 

Today, the way to address this behavior is that rather than alerting on the condition, instead, block the process itself.  And the same thing is true with other adversary behaviors; intelligence gives us excellent insight into adversary behaviors, and using these behaviors, we can implement our EDR solution to alert on some things, but if other specific conditions are met, prevent those actions from completing.  We can further isolate the adversary by isolating the system on the network.

In March of this year, the City of Atlanta was hit with Samsam ransomware.  Interestingly enough, Secureworks, a security MSSP located within the corporate limits of the city, has a good bit of documentation as to the methodology used by the adversary that deploys the ransomware; had the city had an EDR/MDR solution in place prior to the incident, they would've detected and been able to respond to the initial breach, before the adversary was able to deploy the ransomware. 

Given all this, the benefits of employing an EDR/MDR solution are:

1. The solution becomes part of your budget.  Employing an EDR/MDR solution is something you plan for, and include in your annual budget.  Breaches are rarely planned for.

2.  You now have definitive data illustrating the actions taken by an adversary prior to your containment and eradication procedures. You have a record of the commands they ran, the actions they took, and hence their behaviors.  You know exactly where they are within your infrastructure, and you can track back to how they got there, meaning that you can quickly contain and eradicate them.

3. Due to early detection and response, you now have definite data to support not having to report the breach.  Let's say that you detect someone on your network running the "whoami" command; this is a behavior that's been observed in many adversaries.  Alerting on this behavior will allow you to respond to the adversary during their initial recon phase, before they're able to perform credential theft, privilege escalation, and lateral movement.  All of their activity and behavior is recorded, and it's a short list because you've detected and responded to them very early in their attack cycle.  As 'personal data' was not accessed, you've obviated your need to report, and obviated costs associated with fines, notification, clean-up, etc.

Monday, May 07, 2018

Tools, Books, Lessons Learned

As part of my Sunday morning reading a bit ago, I was fascinated by an evolving tweet thread...Eric recently tweeted some random threat hunting advice involving shimcache data.  In response, Nick tweeted regarding an analytic approach to using shimcache/appcompatcache data, at scale, along with AmCache data.  Nick also provided a link to the FireEye blog post that describes their AppCompatProcessor tool.

I really like this approach, and I think that it's a fantastic step forward, not only in single system analytics, analyzing one system through the traditional, "dead box" analysis, but also with respect to an enterprise-wide approach to threat hunting and response.  In many ways, threat hunting and response is the next logical step in DFIR work, isn't it?  Taking what you learned from one system and expanding it to many systems just kinda makes sense, and it looks as though what the FireEye folks did was take what they learned from many single system examinations and developed a process, with an accompanying tool, from that information.

From even a single system analysis approach, there are still a number of analysts within the community who either aren't parsing the AppCompatCache data as a matter of course, or they're misinterpreting the time stamp data all together. Back in Dec, 2016, Matt Bromiley wrote an article describing some of the important aspects of the AppCompatCache data ("important" with respect to interpretation of the data).  Also, be sure to check out read thoroughly the section named "AppCompatSecrets" here, as well as Luis's article here.  A correct understanding of the data and it's context is important going forward.

From an enterprise perspective, there are tools that do nothing more than collect/parse this data from systems, but do not provide any analytics to the data itself, in short, simply dumping this massive trove of data on the analyst.  The analytic approach described by the FireEye folks is similar in nature to the Hamming distance, something I learned about while taking Dr. Hamming's seminar at NPS; in short, the closer the temporal execution of two entries (smaller Hamming distance), the stronger the tcorr value.  This can be applied to great effect as a threat hunting technique across the enterprise.  In the case of AppCompatCache data, the time stamp does not indicate temporal execution; rather, it's a combination of a flag value and the position of the entry, in the order that it's stored within the Registry value.  The "temporal" aspect, in this case, refers to the location of entries in the data with respect to each other.

This can also be applied on a single system dead box analysis, when you look at it as a system, and include not just the System hive, but the System hive in the RegBack folder, as well as any that exist in VSCs, as well as within a memory dump.

I'd recently blogged about using several Registry and Registry-like artifacts in a much smaller approach to "analytics", looking at the use and correlation of multiple sources of data on a single system, with my base assumption being that of single system dead box analysis.  I figured, hey, it's a place to start.

Getting a Book Published
I recently had an opportunity to read Scar's blog post entitled, Finding a Publisher For Your Book.  I found it fascinating because what she wrote about is not too dissimilar from my own experiences.

I have not read her book, Windows Forensic Cookbook.  In fact, I wasn't aware of it until recently.  Once I found out about it, I checked the Amazon site and didn't see any reviews of the book; I'm not surprised...after all, this is the DFIR "community"...but I did hope to get a sense of the material within the book.  So, this isn't a book review, but rather commentary in solidarity with her experiences.

Like Scar, I've put a lot of work into writing a manuscript, only to get the proofs back for review and thinking...what?  In one instance, every single instance of "plugin" had been changed to "plug-in" in a chapter. Uh...no.  My response back to the copy editor was simply to state at the being of the chapter, "...change it back...all of it."

Scar's comments about expectations caught my attention, as well, particularly when the publisher wanted a chapter review turned around in 48 hrs.  What?!?  This is one of those things you have to be prepared for when working with publishers...they simply do not understand their suppliers, at all.  A publisher or editor works on getting multiple books out, that's what they do all day.  However, those of us writing books and providing material do other things all day, and write books when we have time.  That's right...we do the stuff during the day that makes us qualified to write books, and then actually get to write the books in our copious "free time", after putting in a full day or week of work.  These crazy deadlines are something I've pushed against, more so since I've developed greater credibility with the publisher by writing more books.  When my tech editor is working full days just like everyone else and gets a 35+ page chapter with a demand to have the review returned within 48 hrs, that's simply an unrealistic expectation that needs to be addressed up front.

I've been writing books for a while now...it didn't occur to me just how long I've been doing it until last week.  I have a book in the process of being published right now...I'm waiting on the proofs to come back for review...and I submitted a prospectus recently for another book, and the reviews of the proposal I sent in have started coming back.  One reviewer referred to WFA 1/e, published in 2005; this means that I actually started working on it late in 2003, or really in 2004.  And that wasn't my first book. All of this is to say that when I see someone write a blog post where they share their experiences in writing a book and getting it published, what I find most interesting about it is that nothing seems to have changed.  As such, a lot of what Scar wrote in her blog post rings very true, even to this day.

Finally, I've mentioned this before but I'll say it again...over the years, I've heard stories about issues folks have had working with publishers; I've had some of my own, but I like to think that I've learned from them.  Some folks don't get beyond the proposal stage for their book, and some folks have gotten to a signed contract, but drop out due to apparently arbitrary changes made by the editorial staff, after the contract was signed by both parties.  What I had proposed to the publisher I have worked with is to create a liaison position, one where I would work directly with authors (singles, groups) to help them navigate the apparent labyrinth of going from a blank sheet of paper to a published book, using what I've learned over the years.  This never really went anywhere, in part due to the turnover with the publisher...once I had found a champion for the idea, that person left the company.  The fact that after attempting to do this three times and not succeeding, and that the publisher hasn't come back to me to pursue it, tells me that they (the publisher) are happy with the status quo.

If you're interested in writing a book, you don't have to be.  Read over Scar's blog post, ask questions, etc., before you make a decision to commit to writing a book.  It isn't easy, but it can be done.  If your main fear against writing a book is that someone else is going to read it and be critical, keep in mind that no one will be as passionate as you about what you write.

Lessons Learned
I was engaged in an exchange with a trusted and respected colleague recently, and he said something to me that really struck a chord...he said that if I wanted to progress in the direction we were discussing, I've got to stop "giving stuff away for free".  He was referring to my blog (I think), and his point was well taken.  If you're like me, and really (REALLY) enjoy the work...the discovery, the learning, solving a problem in what may be a unique manner, etc...then what does something like writing books and blog posts get you?  I'm not sure what it gets others, but it doesn't lead to being able to conduct analysis, that's for sure.  I mean, why should it, right...if I put it in writing what I did or would do, any someone else can replicate it (like a recipe for tollhouse cookies) then why reach out and say, "Hey, Harlan...I could really use your help on this...", or "...can you analyze these images for malicious activity..."?

Monday, April 09, 2018

Mommy, where do plugins come from?

This is one of those questions kids have been asking their parents throughout history, and in more recent times, those parents may have resorted to a book.  Just sayin'.

Well, they come from three general sources, really, none of which involves a stork or a cabbage leaf.

Recently, there were a couple of requests for functionality to be added to RegRipper.  One was for the ability to automatically update the default profiles in RegRipper.  I was speaking with someone recently, and demonstrating the RegRipper extension that had been added to Nuix's Workbench product.  As part of the discussion, I explained that I do not update the default profiles when I create new plugins (something I've mentioned a number of times in this blog), as I don't want to overwrite any customized profiles folks have made to their installations.  This person then asked, "...can you add the ability to update the default profiles automatically?"  I thought for a minute and realized that rip already has about 2/3 of the code I would need to do exactly that.  So, I opened an editor, used that code to populate a hash of arrays, and then wrote the lists of plugin names, each to their own file.  Boom.  Done.

The other "feature" was for a new plugin to be created.  Someone I know reached out to me to say that they'd found value in a particular Registry key/value during an investigation, and that it might make a good plugin to retrieve the value in question.  This person didn't initially provide any test data, and when they did, it was an exported .reg file; I know it sounds easy enough to handle, but this adds several additional steps (i.e., open a VM, transfer the .reg file to it, import the .reg file into a hive, then shut down the VM, open the .vmdk file in FTK Imager, and extract the hive...), as well as a level of uncertainty (are there variations based on the version of Windows, etc.), to the testing process.

Not having data to test on makes it difficult to write a plugin, as well as test the plugin before releasing it. 

Intel from IR engagements
So, where do other plugins come from?  Similar to the request for the new plugin, sometimes I'll find something during an examination that might make a good plugin.  For example, during an examination, we found a Registry value of interest, and as such, I added the key LastWrite times from the user's NTUSER.DAT hive to a timeline, using regtime.exe.  For context, I did the same for the Software hive from that system, and as an aside, found some interesting Registry keys/values associated with the installed AV product.  In this instance, the keys and values were related to the responder's activities, but writing a plugin (or two) to extract data from the Registry keys/values would facilitate research activities, and as such, make it easier to determine their nature and context.

Interestingly enough, this is how RegRipper started, and was the source of most of the plugins I've written in the past decade.

Online research
Another source of plugins is OSINT, and online research.

For example, FireEye recently released their 2018 M-Trends report, and page 23 includes a Registry key that an attacker modified to hide their activity, by adding a folder to the AV exclusion list.  If I had data on which to test a plugin, I'd write one; online research indicates that there's a key within the path that may vary, and as such, I'd need a bit better understanding of the path in order to write a useful plugin. 

Oddly enough, I don't think I have ever received a request for a plugin based on something published online, via a blog post, or an annual report.  I don't see (or know) everything, and it's likely that I may simply have not seen that post or report. 

Another analysis aspect that RegRipper can be used for is the check or verify system configuration.  For example, see this Microsoft documentation regarding making remote calls to the local SAM database; including a plugin to extract these values may help an analyst narrow down the original attack vector, or at least identify possibilities.

Friday, April 06, 2018


Based on some feedback I received recently, I updated RegRipper's rip.pl (and the corresponding .exe, of course) to include the "-uP" switch.

What this switch does is run through all of the plugins, determine to which hives they apply, and the automatically update the default profiles with those plugins.  As I've stated in the past, when I create a new plugin, I do not update that appropriate profile...I just add the plugin to the repository.  If you want to run all of the plugins available for the NTUSER.DAT hive against all NTUSER.DAT hives you get, run rip.exe with the "-uP" switch, and the profile named "ntuser" will include all of those plugins, EXCEPT those with "_tln" in their name.

This switch will create or overwrite (if they already exist) profiles named for the hives (lower case, without ".dat" at the end).  This does NOT affect any custom profiles you've created, unless they use the same names.

I recently received a request from someone to create a new plugin to retrieve the IE Search Scopes.  The 'searchscopes.pl' plugin was added to the repository today, along with the updated rip.pl/.exe mentioned above.

Something I wanted to point out about both of these updates...they started with someone asking.  That's it.  I don't use RegRipper's profiles for the analysis work I do, but I know that others do.  If you use the GUI, then that's pretty much what you use...you use profiles, rather than individual plugins.  The profiles are also valuable when using the RegRipper extension added to the Nuix Workbench product (fact sheet); the extension relies on a mapping of the hive type to the RegRipper profile.  You can edit/update the profiles themselves, or you can create your own custom profiles and edit the mapping file (JSON, just open it in Notepad...).  I showed the extension to someone, and they asked, "hey, can you create a tool that automatically updates the default profiles?"

The same is true with the searchscopes.pl plugin...someone said, hey, there's this thing that I found useful during an investigation, it might make a good plugin.  Boom.  Done. 

If you've been thinking about something along these lines, or trying to find a way to do it manually, maybe there's a way to do it in an automated fashion.  Sometimes, the smallest interaction can lead to a big result.  Don't isolate yourself on your own island.

Saturday, March 31, 2018

DFIR Analysis and EDR/MDR Solutions

There is only so much the DFIR analysis can do.

There it is, I said it.  And it's especially true when the DFIR analysis is the result of external third party incident notification, which we ultimately determine to come months after the incident originally occurred.

Some artifacts exist forever.  Until they don't.  Some artifacts are recorded and exist for an unspecified and indeterminate period of time...nanoseconds, microseconds, weeks, or months.  Processes execute, finish, and the memory they used is freed for use by another process.  Text files exist until they are deleted, and the last modification times on the files remain the same until the next time they're modified.  Windows Event Logs record events, but some event logs "roll over" more quickly than others; events in some may exist for only a few days, while events in others may exist for weeks or even months.  As time passes, artifact clusters corrode to the point where, by the time DFIR analysts get the data for analysis, their ability to definitively answer questions is severely hampered.

The 2017 Ponemon Institute Data Breach Report indicates an average "dwell time" (the time between initial breach and discovery of the breach) of 191 days.  The Nuix 2018 Black Report findings indicate that professional red teamers/pen testers report that they can target, compromise, and exfil data within 15 hrs.  Hardly seems fair, particularly when you consider that if legitimate, scheduled pen tests go undetected, what chance do we have of detecting an unscheduled, uninvited intruder?

Some artifacts are created, are extremely transient (although they do exist), but are never recorded.  An example of this is process command lines; if an adversary runs the "whoami" command as part of their initial attempts to orient themselves, that process exists for a very short time, and then the memory used gets freed for later use.  By default, this is not recorded, so it ceases to exist very quickly, and the ceases to be available a short time later.   The same is true when an intruder runs the command "net user /add" to create a user account on the system; the command runs, and the command line no longer exists.  Yes, the user account is created, so the results of the command persist...but the command line itself, which likely included the password for the account, is no longer available.  Finally, when the adversary stages files or data for exfiltration, many times they'll use rar.exe (often renamed) to archive the collected data with a password...the command line for the process will include the password, but once the command has completed, the plain text password issued at the command line is no longer available.

Several years ago, I was working a targeted threat response engagement, and we'd observed the adversary staging data for exfiltration.  We alerted on the command line used...it was rar.exe, although the executable had been renamed.  The full command line included the password that the adversary used, and was recorded by the EDR/MDR solution.  We acquired an image of the system, and through analysis determined that the archive files were no longer visible within the "active" file system.  As such, we carved unallocated space for RAR archives and were able to open the twelve archives we retrieved, using the password recorded in the command line used to archive the files.  The web server logs definitively illustrated that the files had been exfiltrated (the IIS logs included the requested file name, number of bytes transferred, and the success code of "200"), and we had several of the archives themselves; other deleted archives have enough sectors overwritten that we were not able to successful recover the entire files.

That's a great example of how an EDR/MDR solution can be so powerful in today's world.  Also, consider this recent Tanium blog post regarding the Samsam ransomware, the same ransomware family that recently hit the City of Atlanta...using the EDR/MDR solution to detect malicious actor activity prior to them deploying the ransomware, such as during their initial orientation and recon phase, means that you have a better chance of inhibiting, hampering, or completely obviating their end goal.

Finally, EDR/MDR solutions become budget items; I'm pretty sure that Equifax never budgeted for their breach, or budgeted enough.  So, not only do you detect breaches early in the attack cycle, but with an IR plan, you can also respond in such a way as to prevent the adversary from accessing whatever it is they're after, obviating compliance and regulatory issues, notification, and keeping costs down.

Monday, March 26, 2018

Even More DFIR Brain Droppings...

Something I've seen over the years that I've been interested in addressing is the question of "how do I analyze X?"  Specifically, I'll see or receive questions regarding analysis of a particular artifact on Windows systems, such as "...how do I analyze this Windows Event Log?" and I think that after all this time, this is a good opportunity to move beyond the blogosphere to a venue or medium that's more widely accessible.

However, addressing the issue here, the simple fact is that artifacts viewed in isolation are without context.  A single artifact, by itself, can have many potential meanings.  For example, Jonathon Poling correctly pointed out that the RemoteConnectionManager/1149 event ID does not specifically indicate a successful login via Terminal Services; rather, it indicates a network connection.  By itself, in isolation, any "definitive" statement made about the event, beyond the fact that a network connection occurred, amounts to speculation.  However, if we know what occurred "near" that event, with respect to time, we can get enough information to provide come much needed context.  Sure, we can add Security Event Log records, but what if (and this happens a lot) the Security Event Log only goes back a day or two, and the event you're interested in occurred a couple of months ago?  File system metadata might provide some insight, as would UserAssist data from user accounts.  As you can see, as we start adding specific data sources...not willy-nilly, because some data sources are more valuable than others under the circumstances...we begin to develop context around the event of interest.

The same can be said for other events, such as a Registry key LastWrite time...this could indicate that a key was modified by having a value added, or deleted, or even that the key had been created on that date/time.  In isolation, we don't know...we need more context.  I generally tend to look to the RegBack folder, and then any available VSCs for that additional context.  Using this approach, I've been able to determine when a Registry key was most likely modified, versus the key being created and first appearing in the Registry hive.

As such, going back to the original questions, I strongly recommend against looking at any single artifact in isolation.  In fact, for any artifact with a time stamp, I strongly recommend developing a timeline in order to see that event in context with other events. 

LNK Shell Items...what's old is new again
It's great to see LNK Shell Items being discussed on the Port139 blog, as a lot of great stuff is being shared via the blog.  It's good to see stuff that's been discussed previously being raised and discussed again, as over time, artifacts that we don't see a lot of get forgotten and it's good to revisit them.  In this case, being able to parse LNK files is a good thing, as adversaries are using LNK files for more than just simple persistence on systems.  For example, they've have been observed sending LNK files to their intended victims, and as was described by JPCERT/CC about 18 months ago, those files can provide clues to the adversary's development environment. LNK files have been sent as attachments, as well as embedded in OLE objects; both can be parsed to provide insight into not just the adversary's development environment, but also potentially to track a single actor/platform used across multiple campaigns.

Another use for parsing LNK files is that adversaries can also use them for maintaining the ability to return to a compromised environment, by modifying the icon filename variable to point to a remote system.  The adversary records and decrypts authentication attempt, and gets passwords that may have changed.

Something else that hasn't been discussed in some time is the fact that shell items can be used to point to devices attached to a system.  Sure, we know about USB devices, but what about digital cameras, and being able to determine the difference between possession of images, and production?

EDR Solutions
Something I've encountered fairly regularly throughout my DFIR experience is Locard's Exchange Principle. I've also blogged and presented on the topic, as well.  Applied to DFIR, this means that when an adversary connects to/engages with a system on a compromised infrastructure, digital material is exchanged between the two systems.  Now, this "digital material" may be extremely transient and persist for only a few micro-seconds, but the fact is that it's there.  As most commercial operating systems are not created with digital forensics and incident response in mind, most (if not all) of these artifacts are not recorded in any way (meaningful or otherwise).  This is where EDR solutions come in.

For the sake of transparency, I used to work for a company that created endpoint technology that was incorporated into its MSSP offering.  My current employer includes a powerful EDR product amongst the other offerings within their product suite.

For something to happen on a system, something has to be executed.  Nothing happens, malicious or otherwise, without instructions being executed by the CPU.  Let's say that an adversary is able to get a remote access Trojan (RAT) installed on a system, and then accesses that system.  For this to occur, something needed to have happened, something that may have been extremely transient and fleeting.  From that point, commands that the adversary runs to, say, perform host and network reconnaissance,  will also be extremely transient.

For example, one command I've seen adversaries execute is "whoami".  This is a native Windows command, and not often run by normal users.  While the use of the tool is not exclusive to adversaries, it's not a bad idea to consider it a good indicator.  When the command is executed, the vast majority of the time involved isn't in executing the command, but rather in the part of the code that sends the results to the console.  Even so, once the command is executed, the process block in memory is freed for use by other processes, meaning that even after a few minutes, without any sort of logging, there's no indication that this command was ever executed; any indication that the adversary ran the command is gone.

Now, extend this to things like copy commands (i.e., bad guy collects files from the local system or remote shares), archival commands (compressing the collected files into a single archive, for staging), exfiltration, and deletion of the archive.  All of these commands are fleeting, and more importantly, not recorded.  Once the clean-up is done, there're few, if any, artifacts to indicate what occurred, and this is something that, as many DFIR practitioners are aware, is significantly impacted by the passage of time.

This is where an EDR solution comes in.

The dearth of this instrumentation and visibility is what leads to speculation (most often incorrect and over exaggerated) about the "sophistication" of an adversary.  When we don't see the whole picture, because we simply do not accept the fact that we do not have the necessary instrumentation and visibility, we tend to fill the gaps in with assumption and speculation.  We've all been in meetings where someone would say, "...if I were the attacker, this is what I would do...", simply because there's no available data to illustrate otherwise.  Also, it's incredibly easy under such circumstances for us to say that the attacker was "sophisticated", when all they really did was modify the hosts file, and then create, run, and then delete an FTP script file.

Why does any of this matter?

Well, for one, current and upcoming legislation (i.e., GDPR) levies 'cratering' fines for breaches; that is, fines that have what can be a hugely significant impact on the financial status of a company.  If we continue the way we're going now...receiving external notification of an intrusion weeks or months after the attack actually occurred...we're going to see significant losses, beyond what we're seeing now.  Beyond paying for a consulting firm (or multiple firms) to investigate the breach, along with loss of productivity, reporting/notification, law suits, impact to brand, drop in stock price, etc...now there are these huge fines. 

Oh, and the definition of a breach includes ransomware, so yeah...there's that.

And all of these costs, both direct and indirect, are included in the annual budget for companies...right?  We sit down at a table each year and look at our budget, and take a swag...we're gonna have five small breaches and one epic "Equifax-level" breach next year, so let's set aside this amount of money in anticipation...that actually happens, right?

Why not employ an EDR solution?  It's something you can plan for, include in your budget...costs are known ahead of time.  The end result is that you detect breaches early in the attack cycle, obviating the need to report.  In addition, you know have the actual data to definitively demonstrate that 'sensitive data' was NOT accessed, so why would you need to notify?  If client data is not accessed, and you can demonstrate that, why would you need to notify? 

Recently, following a ransomware attack, an official with a municipality in the US stated that there was "no evidence" that sensitive data was accessed.  What should have been said was that there was simply no evidence...the version of ransomware that impacted that municipality was one that is not email-borne; delivery of the ransomware required that someone access the infrastructure remotely, locate the servers, and deploy the ransomware.  If all of this occurred and no one noticed until the files are no longer accessible and the ransom note was displayed, how can you then state definitively that sensitive data was not accessed?

You can't.  All you can say is that there is "no evidence".

It's almost mid-year 2018...if you don't already have a purchase of an EDR product planned, rest assured that you'll be part of the victim pool in the coming year.

Sunday, March 25, 2018

More DFIR Brain Droppings

Ransomware (TL;DR)
This past week, the City of Atlanta was hit with ransomware. This is not unusually...I've been seeing a lot of municipalities in the US...cities, counties, etc...getting hit, since last fall.  Some of them aren't all that obvious to anyone who isn't trying to obtain government services, of course, and the media articles themselves may not be all that easy to find.  Here are just a few examples:

Mecklenburg County update
Spring Hill, TN update
Murfreesboro, TN update

Just a brief read of the above articles, and maybe doing a quick Google search for other articles specific to the same venues will quickly make it clear how services in these jurisdictions are affected by these attacks.

The ArsTechnica article that covered the Atlanta attack mentioned that the ransomware used in the attack was Samsam.  I've got some experience with this variant, and Kevin Strickland wrote a great blog post about the evolution of the ransomware itself, as well.  (Note: both of those blog posts are almost 2 yrs old at this point).  The intel team at SWRX followed that up with some really good information about the Samas ransomware campaigns recently. The big take-away from this is that the Samsam (or Samas) ransomware has not historically been email-borne. Instead, it's delivered (in most cases, rather thoughtfully) after someone has gained access to the organization, escalated privileges (if necessary), located the specific systems that they want to target, and then manually deployed the ransomware.  All of these actions could have been detected early in the attack cycle. Think of it this way...instead of being delivered via mail (like a letter), it's more like Amazon delivery folks dropping the package...for something that you never ordered...off in your house...only you didn't have one of those special lock-and-camera combinations that they talked about, there was just a door or window left open.  Yeah, kind of like that.

On the topic of costs, the CNN article includes:

The Federal Bureau of Investigation and Department of Homeland Security are investigating the cyberattack...


The city engaged Microsoft and a team from Cisco's Incident Response Services in the investigation...

Okay, so we're getting a lot of help, that's good.  But they're getting a lot of help, and that's going to be expensive.

Do you, the reader, think that when the staff and leadership for the City of Atlanta sat down for their budget meetings last year, that they planned for this sort of thing?  When something like this occurs, the direct costs include not only the analyst's time, but food, lodging, etc.  Not only does it add up over time, but it's multiplied...$X per analyst per day, times Y analysts, times Z days.

From the USAToday article on the topic:

Such attacks are increasingly common. A report last year from the Ponemon Institute found that half of organizations surveyed had had one or more ransomware incidents in 2017, and 40% had experienced multiple attacks.   

An IBM study found that 70% of businesses have been hit with ransomware. Over half of those paid more than $10,000 to regain their data and 20% paid more than $40,000.

In January, an Indiana hospital system paid a $50,000 ransom to hackers who hijacked patient data. The ransomware attack accessed the computers of Hancock Health in Greenfield through an outside vendor's account Thursday. It quickly infected the system by locking out data and changing the names of more than 1,400 files to "I'm sorry."

Something else that's not addressed in that Ponemon report, or at least not in the quote from the USAToday article, is that even when the ransom is paid, there's no guarantee that you'll get your files back.  Just over a year ago, I did some DFIR analysis work for a client that paid the ransom and didn't get all of their files back, and the paid us to come in and try to provide some answers.  So the costs are stacking up.

What about other direct and/or indirect costs associated with ransomware attacks?  There are hits to productivity and the ability to provide services, costs of downtime, costs to bring consultants in to assist in discovery and recovery, etc.  While we don't have the actual numbers, we can see these stacking up.  But what about other costs?  Not too long ago, another municipality got hit with ransomware, and a quote from one of the articles was along the lines of, "...we have no evidence that sensitive information was accessed."

Yes, of course you don't.  There was no instrumentation, nor visibility, to detect the early stages of the attack, and as a consequence, no evidence of anything about the attack was recorded.  On the surface that sounds like a statement meant to comfort those whose personal data was held by that organization, but focusing just 1nm beyond that statement reveals an entirely different picture; there is "no evidence" because no one was watching.

So what can we expect to see in the future?  We're clearly going to see more of these attacks, because they pay; there is a monetary incentive to conducting these attacks, and an economic benefit (to the attacker) for doing so.  As such, there is no doubt in my mind that two things are going to happen; one is that with the advent of GDPR and other similar legislation, this is going to open up a whole new avenue for extortion.  Someone's going to gain access to an infrastructure, collect enough information to have solid proof that they've done so, and use that as their extortion.  Why is this going to work?  Because going public with that information is going to put the victim organization in the legislative spotlight in a way that they will not be able to avoid.

The other thing that's going to happen is that when statements regarding access to 'sensitive data' are made, there are going to be suits demanding proof.  I don't think that most individuals (regardless of country) have completely given up on "privacy" yet.  Someone...or a lot of someones...is/are going to go to court to demand definitive answers.  A hand-waving platitude isn't going to be enough, and in fact, it's going to be the catalyst for more questions.

Something else I thought was pretty interesting from the CNN article:

When asked if the city was aware of vulnerabilities and failed to take action, Rackley said the city had implemented measures in the past that might have lessened the scope of the breach. She cited a "cloud strategy" to migrate critical systems to secure infrastructure.

This is a very interesting question (and response), in part because I think we're going to see more of questions just like this.  We will never know if this question was a direct result of the Equifax testimony (by the former CEO, before Congress), but I do think that it's plausible enough to assume that that's the case.

And the inevitable has occurred...the curious have started engaging in their own research and posted their findings publicly.  Now, this doesn't explicitly mean that this is the avenue used by the adversary, but it does speak to the question from the CNN article (i.e., "...were you aware of vulnerabilities...").

Attacker Sophistication
Here's a fascinating GovTech article that posits that some data breaches may be the result of professional IT pride.  As I read through the article, I kept thinking of the Equifax breach, which reportedly occurred for want of a single patch, and then my mind pivoted over to what was found recently via online research conducted against the City of Atlanta.

From the article, "...it’s sometimes an easy out to say: “the bad guys are just too good.” "  Yes, we see that a lot...statements are made without qualification about the "sophisiticated" attacker, but for those of us who've been in the trenches, analyzed the host and log data, and determined the initial avenue that the attacker used to get into the infrastructure...how "sophisticated" does one have to be to guess that password for an Internet-accessible Terminal Services/RDP account when that password is on every password list?  Or to use a publicly available available exploit against an unpatched, unmanaged system?  "Hey, here's an exploit against JBoss servers up through and including version 6, and I just found a whole bunch of JBoss servers running version 4...hold my beer."

In my experience, the attacker only needs to be as sophisticated as he needs to be.  I worked an engagement once where the adversary got in, collected and archived data, and exfiltrated the archive out of the infrastructure.  Batch files were left in place, and the archive was copied (not moved) from system to system, and not deleted from the previous location.  The archive itself didn't have a password on it.  Someone said that the adversary was sloppy and not sophisticated.  However, the client had been informed of the breach via third party notification, weeks after the data was taken.

Tool Testing
Daniel Bohannon has a great article up about testing your tools.

I'd like to add to his comments about tool testing...specifically, if you're going to effectively test your tools, you need to understand what the tools do (or at least what they're supposed to do...) and how they work.  Perhaps not at a bit-level (you don't have to be the tool developer), but there are some really simple troubleshooting steps you can follow if you have an issue with a tool and you feel that you want to either ask a question, or report the issue.

For one...and Jamie Levy can back me up on this..."don't work" don't work.  What I mean is, if you're going to go to a tool developer (or you're going to bypass the developer and just go public), it's helpful to understand how the tool works so that you can describe how it appears to not be working.  A long time ago, I learned that Volatility does NOT work if the 8Gb memory dump you received is just 8Gb of zeroes.  Surprising, I know.

Oddly enough, the same is true for RegRipper; if you extract a hive file from an image, and for some reason it's just a big file full of zeroes, the RegRipper will through errors.  This is also true it you use reg.exe to export hive files, or portions thereof.  RegRipper is intended to be run against the raw hive file, NOT text files with a .reg extension.  Instead of using "reg export", use "reg save".

The overall point is that it's important to test the tools you're using, but it's equally important to understand what the tools are supposed/designed to do.

Speaking of tools, not long ago I ran across a reference to Rattler, which is described as, "...a tool that automates the identification of DLL's which can be used for DLL preloading attacks."  Reading through the blog post that describes the issue that Rattler addresses leads me to believe that this is the DLL search order hijacking issue, in that if you have malicious DLL in the same folder as the executable, and the executable calls a DLL with the same name that's found in another folder (i.e., C:\Windows\system32), then your malicious DLL will be loaded before the legit DLL.