Thursday, January 26, 2012

Catch Debuggers With The Step Over Trap

Everyone who spent a short while playing with debuggers knows very well what Step into and Step over mean.

Step into is used to execute one instruction at a time, giving the chance to check registers, stack, etc, it is something called Single Stepping. Step over does the same job except when it comes to call or rep instructions, as Step over does not single step them, it simply jumps over but still executes them. How is that accomplished?

The answer is definitely software breakpoints. When you Step over a call instruction, OllyDbg or any other debugger places an int3 software breakpoint just after the call instruction. In other words, Step over is exactly the same as manually pressing F2 just after the call instruction and then pressing F9.

After we have seen how debuggers implement "Step over", how can we use this for our advantage?
We can easily use this fact as an anti-debug trick by inserting a small macro in the prologue of each function in our application. The macro should extract the return address from the stack and check the first byte pointed to by the return address. If it is 0xCC, it means that someone is trying to step over the protected function and it should escape execution as such.

Walied, Stop talking. Show me some damn code!
If we compile the code in the image above and load the resulting executable in OllyDbg, we will get something like this.

In OllyDbg, if we step over func1, func2, or func3, we will get this error message.

You can imagine how ridiculous it could be for a novice reverse engineer to debug your application as he or she is less likely to step into every single procedure in the application.
The demo mentioned above can be found here. You can also find its source code here.

An example demonstrating how Step over rep movsb can be used as anti-debug trick can be found here.

The interesting thing about rep instructions e.g. rep movsb is that reverse engineers usually step over them since rep movsb instructions often involve copying long strings or large chunks of data.

Everything mentioned in this post has been tested on OllyDbg v1.10, but as far as i know, it should work anywhere else.

You can follow me on Twitter @waleedassar 


  1. nice tut bro...

    I may need a lot to learn about OllyDbg from you bro... :D

    visit to my blog bro...

  2. Won't work in general since you can't know how the compiler sets up the stack or which registers are live at function entry (when, e.g., link time code optimization are enabled). If you assume VC++ you can use the compiler intrinsic "_ReturnAddress" to obtain the return address of the current function.

    1. It is up to the developer to choose the proper macro depending on the chosen compiler options.

      _ReturnAddress is not used in the source code above, since i am using VC++ 6.0, which seems to lack this compiler intrinsic.

      When using the "Step over" trap with the "rep movsb" instructions, you don't have to care that much about compiler options.

  3. Well, my point is that your macro might work or might not work without using the intrinsic. Regardless of the compiler options, you have no guarantee that ebp+4 is *always* the return address. For instance, imagine the compiler inlines your function, then ebp+4 might point to something entirely different.

    1. First, thanks for leaving me comments. I completely understand your point. Anyway, the example above is only for demonstration. It is up to the developer to make his/her own custom macro and add it to proper functions e.g. ebp-based ones. Again, i just wrote the above post only for demonstration.

  4. Very nice tut!! Thanks a lot!!