DFIR Open Mic Night

So, here's a thought...

At a well-attended DFIR conference, there should be a DFIR open mic comedy night.  In the evening, after the event is done for the day, use the venue for something a little light-hearted. For example, OSDFCon has had a mixer at the end of one of the days, where there've been finger foods and adult beverages, and there's a bar right there in the lobby.  Since the venue already has chairs and a mic, why not use them? 

So, Chatham House rules apply, as well as:

• No sales pitches
• No shaming anyone, any company or organization, product, etc.
• Use no names, unless you're sending praise/shout-outs
• Be cool - a little light profanity isn't an issue, but don't be vulgar or disgusting

I think a lot of folks would enjoy something like this...it's a great way to engage, and provide some folks who maybe didn't get their talk accepted a chance to get up on stage and see how they do with public speaking.

After all, there are a LOT of weird or funny things that go on during an IR engagement.  Some may not be funny at the time, but with a little embellishment ("don't let the facts get in the way of a good story...") some time later, they're freakin' hilarious!  So why not share these with everyone else?

For example, I did an IR years ago with a global organization, one that had multiple tools available.  We'd seen Mimikatz being run in the environment; in fact, the SOC (which was located in another country) had alerted the headquarters organization to this finding.  A manager in charge of one the other tools was running searches for "mimikatz" across the infrastructure; unfortunately, one of the detections being used was, "any command line that includes "mimikatz"".  This was intended to catch things like "Invoke-Mimikatz", but it caught everything else.  The first couple of times this happened, the SOC would send an alert, and the local SOC manager (a guy about 6' 7", 275 lbs, bodybuilder type) would go nuts, telling (well, not "telling", per se...) us that the bad guy was back, while the veins in his head and neck were popping out.

So, not funny at the time, but something we can laugh about now...

Thoughts?  Go?  No go?  Is this something you'd participate in, or just want to watch?  Or watch until maybe you got your nerve up (liquid courage) and then got up on stage? 

Program Execution...Or Not

Over the years, different means have been used to discuss the DFIR analysis process, and one of those has been artifact categories.  This is where categories are created and artifacts placed in the various columns, as they relate to those categories.  One such example is the SANS IR poster, which provides a great visual reminder for folks looking to employ this approach.  Honestly, it is a good way to approach analysis...looking at even a single system image, artifacts have grown over the years as the versions of Windows have progressed from XP to Win7, through to Win10, and as such, it benefits a large portion of the community to have a repeatable approach to analysis.

However, when using approaches such as this, we need to keep in mind the context of the category titles.  Yes, the artifacts of "program execution" provide an indication of applications that were launched on a system, but what does that mean?

At this point, I can guess what you're thinking...wait, what?? And believe me, I get it.  "Program execution" means exactly that...that a program was executed.  End of story.  But wait...is that what it really means?

Consider this for a second...someone unfamiliar with a program or application might "open" it on their first try, to see what it does.  Command line tools, for example, often contain information about their usage, which we can see by either typing the name of the program at the command prompt (no arguments), or by adding "/?" or "-h" ("--help" for Linuxphiles).  This causes the program to run, but not in the sense that the functionality of the program is actually used.

As an example, I opened a command prompt, and changed directories to the C:\Windows\Prefetch folder.  Most analysts who've been in the industry for some time are familiar with the application prefetch files, often referred to as simply "Prefetch". Specifically, these are files are widely known as an artifact of "program execution".

I first typed "dir ftp*.pf" to see if there were any Prefetch files that appeared to point to the use of ftp.exe, and got the expected result: File Not Found.  Next, I typed "ftp /?" at the prompt, which displayed the usage syntax of the application.

I then retyped (actually, I hit the up arrow twice...) the 'dir' command, and this time, I found that there was a file named FTP.EXE-7BA637EA.pf, which was 2,685 bytes in size.

So, what happened?  I ran the program, but only to the point where I could read the usage syntax. I didn't actually use the program to transfer files or exfil data in any way.  However, the artifacts of program execution were populated.

Now, the same thing applies to GUI applications, maybe even more so.  You can launch a GUI application, look around at the interface, maybe click a few of the options to see what functionality is available, and then close the UI without ever having employed the functionality provided by the application.

Case in point...consider this analysis of the DefCon 2018 CTF file server image.  Other publicly available write-ups addressed the question of interest (which application was used to delete forensic artifacts?) with various findings.  One was the result of the itempos.pl RegRipper plugin; not an artifact normally associated with program execution, but rather that the application was resident on the desktop.  The two other write-ups went with the UserAssist artifacts, widely associated with program execution; however, there was no verification that the application was actually used to, as stated in the CTF question, delete forensic artifacts.  As such, the GUI application could have been launched, closed, and then something else could have been used to take the specified actions. In fact, the actions in question were never verified.

As such, something to consider going forward is, when artifacts of program execution are found, what do they really mean?

Finally, a question...there is a way to make use of the FTP protocol on Windows workstations (XP, 7, 8, 10) that does not leave the 'normal' artifacts of program execution (i.e., Prefetch file, UserAssist entry) that does not involve disabling any default functionality.  What is it, and how would you determine/verify it?

Addendum, 18 Aug: So far, there's only been one attempt to answer the final question.  I know that there's more out there...check the comments to see the answer, but there's at least one more, and maybe even more than one!

Chasing the DFIR Cure, pt II

Following my first post on this topic, an interesting comment was shared that I thought would really benefit the discussion, as well as benefit from a further look.

To paraphrase, the comment was along the lines of,"...how do you justify the additional cost of a second (or third) look when the results are coming out the same?"

In my experience, that hasn't been an issue.

First, when someone has decided that additional eyes on the data is a justified step to take, the value is already understood, and the cost-benefit analysis has already been done.  Take the pro bono case I mentioned in my previous post; in that case, the value was understood prior to the attorney contacting me, and the decision was made after contact and initial discussions/scoping to do the work pro bono.

Second, while I have been aware of and party to "second looks", in my experience, the results are rarely the same as the "first look". The comment above assumes that the results would be the same, but I simply haven't found that to be the case. 

I did some pro bono work (different case from the one previously mentioned) involving someone leaving a firm, and "salting the earth" on their way out.  What I mean by this is that someone submitted their letter of resignation, and after they left the building, it was found that their system was infected with ransomware.  In this case, the system they were assigned was "communal", in that there was a critical application installed on the system, with just one license, so the CEO needed to have access to it at all times.  My analysis was actually the third look, and I was able to demonstrate that the evening prior to the "incident", someone had logged in at 9pm and browsed the web for about 6 minutes.  During that time, the system became infected, and a persistence mechanism was established, which led to the ransomware launching when the user (not the CEO) logged in the next morning and wrote out their letter of resignation.  The case was thrown out, and a counter suit for libel went forward, based on my report.

This is just one example, but the findings were different enough from the law enforcement officer's report and another consultant's report that the outcome was significantly impacted, and the direction of the case altered. 

Chasing the DFIR Cure

I've wondered for some time now, how do "customers" (recipients of DFIR services) determine the quality and accuracy of the rendered product?  Throughout my time in the industry, I've known some customers to "seek a second opinion", but is this something that's really pervasive?

I was watching a new medical mystery show recently called "Chasing the Cure".  This show is all about folks with severe, debilitating illnesses that have gone un- or mis-diagnosed for extended periods of time. This subject is near and dear to me because about 25 years ago, I went through a similar situation, although the duration of my issue was a few months, rather than years.  In one of the show segments, a woman was suffering from a debilitating ailment that had gone misdiagnosed for several years, and she was able to receive a confident diagnosis on the show (i.e., PCOS).  What I found interesting was that the doctors who made the diagnosis (on the show) were looking at the results of tests that had been ordered by previous doctors, meaning that several medical professionals had the same data in front of them, and in the case, the woman had been told by previous doctors that did not have the condition for which she was finally diagnosed.  So, this was not just a matter of two (or more) professionals having the same data and arriving at different conclusions, as much as it was about a  professional in the field specifically discounting a diagnosis.  The result was that the woman and her family suffered for several more years, where she could have received treatment much earlier.  However, not being satisfied with the answer she was given led her to continue seeking a diagnosis.

That got me to thinking...how do those who contract for DFIR services know if the analysis and findings they're receiving are correct and accurate?  Sure, if the findings don't go their way, they can seek a second (or third) opinion, but how do to they know that what they receive is correct? 

Several years ago, I was contacted by an attorney who had a case that involved a person being in a specific location based on computer evidence.  In short, the case had to do with someone claiming that they were at a convenience store, and this attorney's case would be held up if that person had been behind the computer keyboard.  The attorney had first 'contracted' with a part-time IT sysadmin who serviced their office to conduct analysis of the computer data, and the sysadmin had reportedly found evidence that the person had, in fact, been behind the keyboard.  The attorney asked me to confirm the finding, which I was able to do.  However,

The question, "...are these findings correct?" ever asked?  Does it matter?  I believe it does, and I also believe that there are a number of circumstances where it may behoove a customer/recipient to seek a second opinion:

PCI Investigations - the "window of compromise" is a variable in the "potentially how many credit card numbers were compromised" equation.  This then leads to corrective or punitive actions, such as fines, and as such, is significantly impacted by the findings from the investigation.  For example, misinterpreting the time stamp associated with the AppCompatCache data can extend the "window of compromise" from weeks to years, and severely impact the merchant, who receives a much greater fine.

Compliance - this goes back to things like PCI investigations, as during the course of analysis, the analyst may find that the merchant was not compliant with the PCI DSS (or standards set by another regulatory body) at the time of the breach.

Cyber Insurance - the results of an investigation can significantly impact the results of a claim; issues with data collection and interpretation may lead to findings that indicate considerable gaps in "due diligence", and claims may not be paid.

HR - findings as a result of a DFIR investigation in support of HR can significantly impact the employee, or the company.  Misinterpretations of the data may lead to an employee being unjustly accused or dismissed; I worked a pro bono case to this effect several years ago.

Ransomware - something we see reported quite often in the media is that "...there was no evidence of data exfiltration found...", and that's a good thing which fits the desired narrative.  But is it correct?  Were the DFIR analysts aware of the artifact locations within Windows systems that might provide a different view of that answer?  After all, the actors behind Samas and Ryuk ransomware deployments have been observed spending months (yes, I did spell that correctly...) within an infrastructure before deploying the ransomware, so...yeah...

I'm not suggesting that the industry is rampant with errors in data collection and interpretation, not at all.  There are a lot of great analysts out there doing a lot of great work, and providing accurate results and findings to their customers.  However, like any industry, these things do happen, and like other situations, when they happen they can have a serious impact. We also have to look at the fact that operating systems are getting more sophisticated all the time, and applications are flourishing and getting more numerous.  This is all to say that things are much more complex than they were 20 years ago, and with the number of people coming into the DFIR field, how do we keep up on keeping everyone at a common level of knowledge?

Another aspect of the industry that I've seen change over time is the use of collection, parsing, and pre-processing frameworks.  When I started out in the industry, even if a DFIR analyst collected a dozen or more images, they analyzed those images themselves.  Over time, as there's been a move to cover and address the enterprise, there's been a subsequent increase in the amount of available data.  As such, in a move establish a level of consistency, a lot of DFIR teams have developed means for the enterprise-level collection and pre-processing of data.  All of this can add an additional layer of abstraction between the data and the analyst.

I'm also fully aware that over time, we learn things.  I was talking to Brett Shavers recently, and he brought up the scenario of going back and looking at previous cases.  Like Brett, when I've done this, I've marveled at how far I've come since that case; what are some of the things I've learned since then that I could apply to the case if I were to address the issue today?

I would think, then, that without some compelling reason, most who purchase DFIR services accept the findings they receive as correct and accurate.  In this age of legal and regulatory requirements that both impact and depend on the results of DFIR analysis, the correct and accurate collection and interpretation of digital data is paramount, and there are a number of cases where the "customer" may benefit from a second, or even a third opinion.  After all, we do this with medical issues, don't we?

To that point, that 'compelling reason' would likely consist of findings that are markedly contrary and contradictory to the desired narrative.  There is likely a 'threshold' that some may accept; for example, consider the PCI example above...there are likely merchants who receive information about those findings and are able to absorb whatever judgement is levied by the bank or the PCI Council.  However, there are also those merchants for whom the judgement is what I've referred to as a "cratering fine"; that is, once the fine is levied, the business (a small mom-and-pop restaurant, for example) ceases to exist.  I've seen this happen.  In such cases, given what's at stake, it may behoove the merchant to seek a second opinion.