Reversing software compressions: Tale of dragons and men who slay them

In that spirit numerous presentations, during the last few years, have been held involving the great abundance of ways one can make a single generic solution that unpacks it all. This presentation is its exact opposite as it will focus on reverse engineering specifics for numerous commonly used software compressions.

When building a system for automated file analysis our goal is to make an optimal system that accurately identifies files and unpacks them in the blink of an eye. Such system must be able to be deployed in any environment without the risk of anything going even remotely wrong. That kind of requirements eliminate most generic unpacking solutions making us focus on what is without a doubt hardest unpacking scenario; static unpacking. Writing static unpackers is a hard task which is why it is more than often avoided by reverse engineers. However it is necessary as their performance far overtakes the difficulty of implementation.

We will focus on reverse engineering of all known and possible implementations of various transformations performed by the compression solution in an aim to show that the best way to observe the software compression is as subset of its parts. Detailed descriptions of reverse engineering procedures needed to analyze internal data structures along with ways to restore them to original PECOFF format will be provided. These techniques will be applied to both custom and traditional compression & encryption algorithms with examples that shows how to reverse engineer vital functions from assembly back to source code. In addition to this first step in reversing we will tackle the problems of data layout and import, resource, relocation and TLS table transformation and analysis. Differences between x86, x64 and .net packers and the ways to unpack them will also be covered. Solution to all of these problems will be presented from a standpoint of writing a high load static unpacker that operates in a multi-threaded environment. As an implementation platform upcoming TitanEngine3 unique design will be presented along with approach it uses to solve the problems that come with writing static unpackers.

More information here.