close() x86_64 system call weird return value
c, kernel, system-calls, x86-64
Solution
As expected, a rollback to a previous kernel version solved the problem. I'm not really a kernel specialist but as far as I understand, the answer given by @R.. makes sense:
This is a 64-bit machine, so 1<<32-9 should never appear. The problem is that the kernel is internally using unsigned instead of int for the return value of some of these functions, then returning -EBADF which gets reduced modulo 2^32 rather than modulo 2^64
The problem is that the generic code in the libc syscall wrappers that handles syscall error returns has to treat the return value as a long (since it could be a pointer or long for some syscalls) when making the comparison to see if it's a small negative value that would indicate an error. But the kernel returned (long)(unsigned)-9 which is very different from (long)-9. or (unsigned long)-9 (either of which would have worked).
Problem
My xinetd daemon suddenly stopped working after a kernel upgrade (from 2.6.24 to 2.6.33). I've run an strace and found this: ``` [...] close(3) = 0 munmap(0x7f1a93b43000, 4096) = 0 getrlimit(RLIMIT_NOFILE, {rlim_cur=8*1024, rlim_max=16*1024}) = 0 setrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0 close(3) = 4294967287 exit_group(1) = ? ``` So basically, it looks like the close system call returned something different than 0 or -1 I did several tests and it appears that it happens only with 64bit executables: ``` $ file closetest32 closetest32: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped $ strace closetest32 execve("./closetest32", ["closetest32"], [/* 286 vars */]) = 0 [ Process PID=4731 runs in 32 bit mode. ] open("/proc/mounts", O_RDONLY) = 3 close(3) = 0 close(3) = -1 EBADF (Bad file descriptor) _exit(0) = ? $ file closetest64 closetest64: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), statically linked, not stripped $ strace closetest64 execve("./closetest64", ["closetest64"], [/* 286 vars */]) = 0 open("/proc/mounts", O_RDONLY) = 3 close(3) = 0 close(3) = 4294967287 _exit(0) = ? ``` I'm running the following kernel: ``` Linux foobar01 2.6.33.9-rt31.64.el5rt #1 SMP PREEMPT RT Wed May 4 10:34:12 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux ``` The worst part is that I cannot reproduce the bug on another machine with the same kernel. Any ideas ? EDIT: as requested: here's the code used for closetest32 and closetest64 closetest32.asm: ``` .section .data filename: .ascii "/proc/mounts" .section .text .globl _start _start: xorl %edi, %edi movl $5, %eax # open() i386 system call leal filename, %ebx # %ebx ---> filename movl $0, %esi # O_RDONLY flag into esi int $0x80 xorl %edi, %edi movl $6, %eax # close() i386 system call movl $3, %ebx # fd 3 int $0x80 xorl %edi, %edi movl $6, %eax # close() i386 system call movl $3, %ebx # fd 3 int $0x80 ## terminate program via _exit () system call movl $1, %eax # %eax = _exit() i386 system call xorl %ebx, %ebx # %ebx = 0 normal program return code int $0x80 ``` compiled as: ``` as test32.asm -o test32.o --32 ld -m elf_i386 test32.o -o closetest32 ``` closetest64.asm: ``` .section .data filename: .ascii "/proc/mounts" .section .text .globl _start _start: xorq %rdi, %rdi movq $2, %rax # open() system call leaq filename, %rdi # %rdi ---> filename movq $0, %rsi # O_RDONLY flag into rsi syscall xorq %rdi, %rdi movq $3, %rax # close() system call movq $3, %rdi # fd 3 syscall xorq %rdi, %rdi movq $3, %rax # close() system call movq $3, %rdi # fd 3 syscall ## terminate program via _exit () system call movq $60, %rax # %rax = _exit() system call xorq %rdi, %rdi # %rdi = 0 normal program return code syscall ``` compilation: ``` as test64.asm -o test64.o ld test64.o -o closetest64 ```