GCC Segmentation fault with -O1 and inline assembler
debugging, gcc, inline-assembly, linux, ubuntu
Solution
No - gcc is not capable of detecting this. It doesn't perform any analysis of the instructions that appear in the `asm` block. It is your responsibility to inform the compiler of any side effects. Can you explain what test you are performing?
Also, you should consider using a single `asm` block for this code; `volatile` may prevent reordering of the `asm` blocks, but you cannot assume this yields consecutive instructions.
Problem
I have detected a strange segmentation fault in my code and I would like to hear your opinion if that could be a GCC bug or is just my fault! The function looks like that: ``` void testMMX( ... ) { unsigned long a = ...; unsigned char const* b = ...; unsigned long c = ...; __asm__ volatile ( "pusha;" ); __asm__ volatile ( "mov %0, %%eax;" : : "m"( a ) : "%eax" ); // with "r"( a ) it just works fine! __asm__ volatile ( "add %0, %%eax;" : : "m"( b ) : "%eax" ); __asm__ volatile ( "mov %0, %%esi;" : : "m"( c ) : "%eax", "%esi" ); __asm__ volatile ( "sub %eax, %esi;" "dec %esi;" "movd (%esi), %mm0;" "popa;" ); } ``` If I compile this with -O0 it just works fine. But it SegFaults with -O1 and -O2. It took me a long time to figure out that this segfault was caused by frame pointer omission. The pusha instruction increases the stack size by 4*8=32 byte (x86_32) and therefore ESP should be increases as well. But gcc does not recognize this. If I add the ESP fix manually ``` __asm__("add $32, %esp") ``` or use the "-fno-omit-frame-pointer" flag in gcc I can compile and run it with -O1 and -O2 without any errors! So my question now is: why does gcc not adjust the ESP with any push/pop inline assembler operations if frame-pointer-omission is enabled? Is this a gcc bug? Is gcc even capable of detecting this? Am I missing something? It would be very interesting to solve this. Thanks in advance!