Everyone who spent a short while playing with debuggers knows very well what Step into and Step over mean.
Step into is used to execute one instruction at a time, giving the chance to check registers, stack, etc, it is something called
Single Stepping. Step over does the same job except when it comes to
call or
rep instructions, as Step over does not single step them, it simply jumps over but still executes them. How is that accomplished?
The answer is definitely software breakpoints. When you Step over a
call instruction, OllyDbg or any other debugger places an
int3 software breakpoint just after the
call instruction. In other words, Step over is exactly the same as manually pressing F2 just after the
call instruction and then pressing F9.
After we have seen how debuggers implement "Step over", how can we use this for our advantage?
We can easily use this fact as an anti-debug trick by inserting a small macro in the prologue of each function in our application. The macro should extract the return address from the stack and check the first byte pointed to by the return address. If it is 0xCC, it means that someone is trying to step over the protected function and it should escape execution as such.
Walied, Stop talking. Show me some damn code!
If we compile the code in the image above and load the resulting executable in OllyDbg, we will get something like this.
In OllyDbg, if we step over func1, func2, or func3, we will get this error message.
You can imagine how ridiculous it could be for a novice reverse engineer to debug your application as he or she is less likely to step into every single procedure in the application.
The demo mentioned above can be found
here. You can also find its source code
here.
An example demonstrating how Step over rep movsb can be used as anti-debug trick can be found
here.
The interesting thing about
rep instructions e.g.
rep movsb is that reverse engineers usually step over them since
rep movsb instructions often involve copying long strings or large chunks of data.
Everything mentioned in this post has been tested on OllyDbg v1.10, but as far as i know, it should work anywhere else.
You can follow me on Twitter
@waleedassar