I had the distinct honor of speaking at @BSidesCincy this past weekend, and I greatly appreciate the opportunity that Justin, Josh, and the entire crew provided for me to speak.In my time, I've been to a number of conferences. I started with Usenix back in '99, and that experience was a bit different for me, in that the vast majority of my public speaking to that point had been in the military (during training, and while providing training). Over the years, I've attended (I prefer to speak at conferences in an attempt to keep costs to my employer down...) several conferences that left me wondering what the point was. Sometimes, what the speaker actually spoke about had little or nothing to do with the advertised title of their talk, and in a few cases, with the conference theme itself. With BSidesCincy, it was clear that folks from the same community, with similar experiences and concerns, were coming together to share and discuss those experiences and concerns., and IMHO, that's the real way to make forward progress in this community.
Unfortunately due to flights (or rather, the lack of direct flights), I had to depart the venue after scarfing down lunch. I did get to see John Davidson's presentation, and I've got to say, it was pretty fascinating. Even though I'm more of a DFIR guy, my day job does include hunting (albeit largely from a host-based perspective), so a lot of what John talked about made complete sense, as at one point or another, I had some similar thoughts. John took those thoughts and then took them further. From a 50,000 ft view, John's presentation was about "here's the issue I ran into and here's how I addressed it...", which resulted in a really good presentation. Further, it addressed something that I've heard over the years throughout the community...that folks don't want to hear, "...this is what you need to do...", as much as they want to hear, "...this is the issue I faced, and here's what I did to address it...", illustrating their methodology and thought processes throughout.
Thanks to Adrian, here's a link to the video of my presentation.
Again, to Justin, Josh, Adrian, and everyone on the BSidesCincy crew...thanks so much for having me out and for giving me the opportunity to share a bit with everyone. I had a really great time engaging with everyone I got to speak with and meet. I hope that for you, it was worth it and that some folks came away with something that they could use.
Addendum, 30 July: I was asked recently via Twitter if I was going to post the slides somewhere, and to be honest, I really don't see the point. First, I don't read off of my slides...most of what I talk about during a presentation isn't in the slides (on the screen or in the notes). Second, if you're looking to use the slides as a reference, the information that is posted in the slides is already available in my blog, or someplace else. Many of the event IDs mentioned in this presentation were taken from eventmap.txt.
I didn't get many comments on this thought at the time...am I on target, or am I way off and clearly making things up? Does it make sense? Does it generate any additional or follow-on thoughts?
So, what would be the point of releasing the slides? I talk to this slide in the video, and so far, haven't heard any comments or feedback on the point, so why release the slides, just to get...well...still NOT get comments or thoughts? So far, there have been a good number of RTs and "Favorites" for the original version of this post, but as of yet, no comments regarding the content of the video.
Harlan, first time blogger and fairly new to the this scene... But 30 years in IT, development, project mgmt roles seems to apply... I enjoyed meeting you at the conference and thanks for your presentation and time to chat.
ReplyDeleteYour presentation was also thought provoking and really tried to pull the audience in with questions that applied to your presentation. You provided us with a difference of opinion and came to the core of what is on my mind while doing IR. I try to confine myself to the who, what, when, where and how when doing my work. I agree Johns' presentation rocked. There are always ways to make improvements and hearing how you and John go about doing these is inspirational. As a long time perl writer, I'm inspired to take some of what I've done over the years and automate it.
My last comment to you was I need to spend more time in the lab.... Your comment back was to leverage what others have done, which I'll do as it will aid in the speed of things....
Thank again!
Here are my feelings on your slide. I think it can be summed up in four words. Time, freedom, capacity and skills.
ReplyDeleteTime: Detecting the truly damaging intrusions is time consuming. There is time that is needed to be spent researching, building and testing detection, analyzing alert data and hunting. I would guess that most orgs spend way too little time with this either because they aren't aware that they should be doing these things, they don't have the manpower or they don't have buy in from their management that these activities are required.
Understanding: I feel there is a general lack of understanding regarding what it "looks like". I've worked with many new people and it takes a while for them to be able to look at something and say "oh yeah, that's bad" and feel confident in doing so. People generally don't want to be wrong and may tend to discount subtle indicators.
Capacity; I was at the SANS SOC summit earlier this year and I was shocked at some of the duties these teams are required to do. From device policy management to trouble shooting network issues. In most cases I would doubt the majority of teams don't have the manpower to perform the duties they currently have as well as what I described under Time.
Skills: Finding the people who are really good at finding this activity is hard and if you do find them they are expensive and difficult to retain for any amount of time.
Jack,
ReplyDeleteThat's an interesting perspective...thanks for sharing it.
I have to say that thinking back to some of the organizations I've worked with over the past 15 or so years, this is definitely the case. In others, all of these factors have been in place, in some form or another, but the issue was simply lack of focus.
One thing keeps popping into my head...a line from "Remember the Titans"; "attitude reflect(s) leadership." In some instances, I've seen well-staffed, capable teams that are completely disorganized, and then I'd see senior management the same way.
Thanks for the comment.
Slides - I think presentation slides are useful from the standpoint that I can keep them locally with other info I collect. Saves me some typing, basically. Yes, you covered everything and then some in the video which is what I would expect. Having said that, your blog and github repo are really all that's needed. It's all in there.
ReplyDeleteI echo Jack's comments about the extra duties given to DFIR folks, it's insane what folks in some corp environments have to do along with DFIR. Just one more reason they leave (or won't accept job offers once they find this out...)
Throne,
ReplyDeleteI've added a folder to my GitHub repository...you'll find the PDF of the presentation there.
Thanks.
Hi Harlan,
ReplyDeleteI agree with Jack and throne's points. I also think there is an emphasis on network indicators and what some of these indicators look like when you get to the exfil stage by some orgs. This in turn leaves them blind to endpoint activity and what it looks like (clusters of activity).
As you mentioned in your bsides presentation there are sysadmins and even those tasked with infosec duties that don't know about the at command or what valuable data is in the default windows event logs or any of the other sources of info like prefetch, shimcache, shellbags, reg keys to name a few that IR consultants are familiar with. In my opinion i think it comes down to these possibilities as to why orgs aren't catching these earlier....
-Lack of training
-Lack of attitude/mentality
-Lack of visibility on endpoints/network
-Lack of incident handling processes
-emphasis on easy indicators- hashes, filenames, network sigs
Lastly to go back to my initial point on emphasis of "what it looks like" on the network, take FireEye's report on Hammertoss. It's a great report on how the malware communicates and obfuscation techniques it leverages and it would seem difficult to detect on the network but what about how it looks like on the endpoint? there were a couple of nuggets of info like executing powershell with uncommon switches that one could look for using process monitoring but just that tidbit and not much else. granted this wouldn't be a definitive indicator of that malware or actor but its start on monitoring for suspicious behavior assuming an org doesn't have some of the issues i mentioned.
Daniel,
ReplyDeleteI agree with you, and in particular with your reference to HammerToss. I took another look at the FireEye report this morning, and found the command you referred to...I see the reference to the use of Powershell, but in 14 pages, there's very little in the way of host-based artifacts. There is one instance of the word "peristent", and none of "persistence"; there appears to be no mention of the persistence mechanism used by this malware.
I also found the discussion of steganography on pg 11 to be very interesting, particularly given that the authors define steganography as "concealing a message, image or file within another message...", and then state that HammerToss appends data to an image, after the end-of-file marker. I would say that I'm really unclear as to why they spend time defining and using the term steganography, but it's clear that it's marketing.
I've seen similar reports from other TI providers before; where a report will give a description of encrypted network traffic, and maybe even provide detection IF you have some means of decrypting the traffic. However, rarely do you see that the endpoint then becomes the focus for detection.
Hi Harlan,
ReplyDeletethanks very much for sharing the slides and linking to your presentation. You are one of the few that provide valuable comments instead of just reciting from slides.
You shared many valuable points for me to consider when I do IR. Some I knew of before, like browser session restore capability. However, the sticky key attack you discussed around minute 37 and the leaving of the RAT in the local admin profile waiting for the login for it to be executed was new to me. Awesome, thank you for mentioning them.
You summarized the next IR imperative "Process creation logging" and I hope everyone is listening.
That needs to be combined with something you have said for many years... our lack of specificity of language.
Br, mitch
Mitch,
ReplyDeleteThanks for your comment. I did follow up with Justin and ask if anyone else had shared thoughts/comments/experiences from the conference, and he wasn't aware of any (at the time).
With respect to process creation logging in IR, a great example of how incredibly valuable that is, I did a response engagement a bit ago, where the total time taken for the response was under 4 hrs. Process creation monitoring was used in detection, and additional steps were taken to track the process(es) back to the origin. As with most breaches, had they not had the process creation monitoring in place, they would not have even noticed the breach until the adversary was well-entrenched.
Hi Harlan,
ReplyDeleteAbsolutely, process logging seem to be extremely useful.I have installed Sysmon on a few Windows based vm's and this is really providing a lot of extra visibility. Im currently testing some common tasks / events on my home network to see the results. I am curious to see what happens when this extra logging information is added to timeline examinations, it can only be a good thing.
Given the current trends of malware using of Powershell, RDP, net use, etc perhaps Microsoft will consider adding some extra immutable process logging options. Or perhaps solutions will start showing up from various 3rd party commercial vendors ?
In any event, there is no fear of IR being made obsolete just yet :)
Br, mitch
Mitch,
ReplyDeleteLike you, I have installed Sysmon in testing VMs, and found a great deal of extremely valuable information when testing various credential theft tools, for example.
... perhaps solutions will start showing up from various 3rd party commercial vendors ?
They already have; and actually, some of them have been out for quite a while now. However, they will be of little use if the adversary has access via RDP.