Friday, February 09, 2007

Why I like Perl for Forensic Analysis

Many times when I post to a list or give a presentation and talk about using Perl, I almost immediately hear comments back such as, "I can't do that because I don't know Perl".

Well, I have to say, it's not about knowing Perl, or how to program, really. Most of the stuff I'm doing with Perl is translating binary stuff into something readable (timestamps, checking flag values), and just doing repetitive stuff. For example, in Windows Memory Analysis, I just use Perl to implement various processes that are repetitive and boring, and therefore I would be prone to making mistakes. The same is true with tools like SAMParse and SECParse that I use to parse the contents of raw Registry files. The information is there for anyone to see, extract, and interpret. I simply use Perl to do it automatically. In some cases, I write the scripts I do because there is commercial application that provides anywhere close to the information I want or need.

I have written tools to parse RAM dumps, parse Event Log .evt files (without using the Windows API), parse raw Registry files, etc. I wrote these tools because I needed them, and as it was very likely that I would need to do those things again, I automated the process in a Perl script.

I also hear things like, "I don't want to have to keep Perl up-to-date". I think in a lot of cases, comments like these come from folks who don't know a great deal about Perl...and you know...that's fine. It's okay, people. I'm not being a Perl snob or forensics snob or whatever snob, I simply use Perl to automate the repetitive tasks that make up my work day. Besides, I also use Perl2Exe, so all the tools that will be provided on the DVD that comes with my next book (Windows Forensic Analysis, from Syngress/Elsevier) will be provided with Perl "source" code, as well as a standalone EXE file that will allow you to run the tools on a Windows system without having to install Perl.

Perl is an interpretted language, so I don't have to "compile" my scripts to run them, and I can easily make modifications and changes to the scripts, updating or correcting them quickly. The scripts are somewhat self-documenting, although I do add my own comments to give myself a better view of what's going on, so I can figure out was I was doing a year or two later, rather than just look at it and think, "*I* wrote this??". Finally, Perl gives me a freedom and flexibility that is not available in commercial tools. For example, there are commercial tools that will allow you to view certain Registry data in a nice GUI, but you can only view that data, and only one entry at a time. With Perl, I can output multiple entries, as well as correlate data from multiple keys, such as when I want to show USB removable storage devices, any drive letters they were mapped to, and the last time each device was connected to the system (I should call this one "USBParse"). Don't like ASCII output in block format? In a few moments, I can have CSV style output. Ta-da!

So...some folks don't like to use Perl or open-source tools for forensic analysis, and that's okay. I use Perl because I know that if I have to perform some kind of analysis or correlation once, I'll probably have to do it again, and in order to avoid (as much as possible) forgetting a step or making a mistake, I'll automate the process.

Besides, I give most of this stuff away for free, as in beer.


Anonymous said...

I completely agree with you on the power and extensibility of Perl. I use Perl all the time and to tell the truth, had gone through a "perl withdrawal syndrom" when I was coding in other interpreted languages - CPAN saves a lot of effort. It also makes me a lazy programmer at times. It's just most of the building blocks are available :).

Portability is another issue.
Perl2exe is great, mostly for smaller projects. In my experience porting from my primary development platform ( *nix ) to windows may require actual perl interpreter.

For some reason I found many people in windows world are more reluctant to install and use Perl as an "add-on" engine, plus the modules.
I think CPAN vs. activestate module availability may be the issue. To many, having Cygwin as a yet another dependency is a whole new game.

H. Carvey said...


I think CPAN vs. activestate module availability may be the issue. To many, having Cygwin as a yet another dependency is a whole new game.

I'm not sure I follow...if you're using ActiveState (which is free, as in beer), you don't need Cygwin. Modules can be easily searched for and installed using PPM...and I've provided very straightforward installation instructions for important modules that are on CPAN, but haven't crossed over to ActiveState's repository as of yet.

Part of what I like about Perl is that, like I said, it's partially self-documenting. If you are parsing a PE file for example, and you're not sure where the various values are actually located in the file, look at the source -> "Oh, this DWORD at offset 0x00-whatever is the address of the entry point!"

Anonymous said...

Yes, short and modular programs dealing with common win API are not a problem.
I found the limitation getting into more of a framework-based code. Graphical presentation of the results, for example.

In my world, due to general absense of a pre-installed compiler on WinOS, it may be very hard to bring useful modules like GD and HTTP POE objects to the windows side. Too many dependencies...

Yes, you can scour the web and get the binary distribution somewhere. But..
Activestate does not build all the modules available on CPAN to provide alternative for windows folks. This is why I mentioned Cygwin.

Maintaining PPM and Cygwin seems to be an overhead. Maybe it's just me.

H. Carvey said...


So how do you use the GD and HTTP POE modules for forensic analysis?


Anonymous said...

I guess, at some point I have extended the scope of the discussion to include functionality supporting forensic data mining itself :)

These we just examples. But since you asked...
For example, in one of the tools I am working on charting/graphing is an important piece as it may provide clues to the distribution and occurrence of the potentially useful forensic events. One of the choices was Perl w/GD image library. Same story with the HTTP interface. Do I want to hook it up to IIS/Apache or do I provide Perlish HTTP component independent of the yet another external dependency - a web server. I am not even going into the possibility of coding the front-end in Perl/Tk.
... At the end of the day Perl was not the primary choice for this.

My point was really on portability of a perl application to Windows platform in it's entirety and the dependencies one needs to meet when exploring out of range of the available modules/tools.

Again, I agree with the Perl usage,
I just think that it has limitation in acceptance and therefore appropriate baseline support on Windows platform to code freely.

H. Carvey said...

...I have extended the scope of the discussion...

Thanks for your comments, as they are greatly appreciated, but they do extend the discussion well beyond the scope of my blog post.

I was referring more to:
-parsing Event Log files on a binary level, bypassing the API
-parsing PE files
-parsing RAM dumps
-parsing raw Registry files

None of these require additional modules beyond the base install, although I do use the platform independant Parse::Win32Registry to parse Registry files...

Anonymous said...
This comment has been removed by a blog administrator.