Why does this code enable me to detect a debugger?
assembly, debugging, reverse-engineering, windows, x86
Solution
When program is debugged (i.e. single step) prefetch queue is flushed at each step (when interrupt occurs). However, when executed normally that will not happen to `rep stosb`. Older processors didn't flushed it even when there was memory write to the cached area, in order to support self-modifying code that was changed except `rep movs` and `rep stosb`. (IIRC it was eventually fixed in i7 processors.)
That's why if there is a debugger (single step) code will execute correctly and when `rep stosb` is replaced by `ret` `l2` will be executed. When there is no debugger `rep stosb` will continue, since `ecx` is the biggest possible it will eventually write somewhere it is not supposed to write and an exception will occur.
This anti-debugging technique is described in this paper.
Problem
Why the following assembly code is an anti-debugging tool? ``` l1: call l3 l2: ;some code l3: mov al, 0c3h mov edi, offset l3 or ecx, -1 rep stosb ``` I know that C3h is `RETN` and I know that `stobs` writes the value in `al` as opcode according to the offset in `edi` and it is done for `ecx` times because of `rep`. I am also aware the fact that `stobs` and `stosw` will run if they were pre-fetched on intel architecture as their original format. If we run the program in debugged mode the pre-fetch is irrelevant and the l2 label will run (because it is single-step) otherwise if there is no debugger it will be ping-pong between l1 and l3 am I right?