Using x86's ENTER instruction with a non-zero nesting level?

Question

Consider the x86 instruction `ENTER`. From Intel's instruction set reference. Creates a stack frame for a procedure. The first operand (size operand) specifies the size of the stack frame (that is, the number of bytes of dynamic storage allocated on the stack for the procedure). The second operand (nesting level operand) gives the lexical nesting level (0 to 31) of the procedure. The nesting level determines the number of stack frame pointers that are copied into the “display area” of the new stack frame from the preceding frame. Both of these operands are immediate values. I was wondering how the `ENTER` instruction works when a nonzero nesting level is passed as a second operand. In this case, according to Intel's manual, the processor should push additional frame pointers on the stack. Sounds simple, but I can't figure why it doesn't work as expected on sample programs. I compiled the following example with FASM and debugged with OllyDbg. ``` format PE section '.text' code readable…

Accepted Answer

The nested form of the `enter` instruction assumes you already have a valid frame pointer in `ebp`, which is not the case on linux at the start of the process. Presumably that is why you get a fault.

Try this instead:

```
_start:
    enter $0, $0        # set up initial stack frame
    push $1
    push $2             # simulate two previous frame pointers
    enter $16, $3       # nested proc level 3, esp should be 32 less
    movl (%ebp), %eax   # should be previous ebp
    movl -4(%ebp), %eax # should be 1
    movl -8(%ebp), %eax # should be 2
    movl $1, %eax
    movl $0, %ebx
    int $0x80
```

See also section 6.5.1 ENTER Instruction in the Intel® 64 and IA-32 Architectures Software Developer's Manual Volume 1: Basic Architecture

Using x86's ENTER instruction with a non-zero nesting level?

Solution

Problem