Disable AVX-optimized functions in glibc (LD_HWCAP_MASK, /etc/ld.so.nohwcap) for valgrind & gdb record

avx, gdb, glibc, linker, linux

Solution

There does not seem a straightforward runtime method to patch feature detection. This detection happens rather early in the dynamic linker (ld.so).

Binary patching the linker seems the easiest method at the moment. @osgx described one method where a jump is overwritten. Another approach is just to fake the cpuid result. Normally `cpuid(eax=0)` returns the highest supported function in `eax` while the manufacturer IDs are returned in registers ebx, ecx and edx. We have this snippet in glibc 2.25 `sysdeps/x86/cpu-features.c`:

__cpuid (0, cpu_features->max_cpuid, ebx, ecx, edx);

/* This spells out "GenuineIntel".  */
if (ebx == 0x756e6547 && ecx == 0x6c65746e && edx == 0x49656e69)
  {
      /* feature detection for various Intel CPUs */
  }
/* another case for AMD */
else
  {
    kind = arch_kind_other;
    get_common_indeces (cpu_features, NULL, NULL, NULL, NULL);
  }

The `__cpuid` line translates to these instructions in `/lib/ld-linux-x86-64.so.2` (`/lib/ld-2.25.so`):

172a8:       31 c0                   xor    eax,eax
172aa:       c7 44 24 38 00 00 00    mov    DWORD PTR [rsp+0x38],0x0
172b1:       00 
172b2:       c7 44 24 3c 00 00 00    mov    DWORD PTR [rsp+0x3c],0x0
172b9:       00 
172ba:       0f a2                   cpuid  

So rather than patching branches, we could as well change the `cpuid` into a `nop` instruction which would result in invocation of the last `else` branch (as the registers will not contain "GenuineIntel"). Since initially `eax=0`, `cpu_features->max_cpuid` will also be 0 and the `if (cpu_features->max_cpuid >= 7)` will also be bypassed.

Binary patching `cpuid(eax=0)` by `nop` this can be done with this utility (works for both x86 and x86-64):

#!/usr/bin/env python
import re
import sys

infile, outfile = sys.argv[1:]
d = open(infile, 'rb').read()
# Match CPUID(eax=0), "xor eax,eax" followed closely by "cpuid"
o = re.sub(b'(\x31\xc0.{0,32}?)\x0f\xa2', b'\\1\x66\x90', d)
assert d != o
open(outfile, 'wb').write(o)

An equivalent Perl variant, `-0777` ensures that the file is read at once instead of separating records at line feeds:

perl -0777 -pe 's/\x31\xc0.{0,32}?\K\x0f\xa2/\x66\x90/' < /lib/ld-linux-x86-64.so.2 > ld-linux-x86-64-patched.so.2
# Verify result, should display "Success"
cmp -s /lib/ld-linux-x86-64.so.2 ld-linux-x86-64-patched.so.2 && echo 'Not patched' || echo Success

That was the easy part. Now, I did not want to replace the system-wide dynamic linker, but execute only one particular program with this linker. Sure, that can be done with `./ld-linux-x86-64-patched.so.2 ./a`, but the naive gdb invocations failed to set breakpoints:

$ gdb -q -ex "set exec-wrapper ./ld-linux-x86-64-patched.so.2" -ex start ./a
Reading symbols from ./a...done.
Temporary breakpoint 1 at 0x400502: file a.c, line 5.
Starting program: /tmp/a 
During startup program exited normally.
(gdb) quit
$ gdb -q -ex start --args ./ld-linux-x86-64-patched.so.2 ./a
Reading symbols from ./ld-linux-x86-64-patched.so.2...(no debugging symbols found)...done.
Function "main" not defined.
Temporary breakpoint 1 (main) pending.
Starting program: /tmp/ld-linux-x86-64-patched.so.2 ./a
[Inferior 1 (process 27418) exited normally]
(gdb) quit                                                                                                                                                                         

A manual workaround is described in How to debug program with custom elf interpreter? It works, but it is unfortunately a manual action using `add-symbol-file`. It should be possible to automate it a bit using GDB Catchpoints though.

An alternative approach that does not binary linking is `LD_PRELOAD`ing a library that defines custom routines for `memcpy`, `memove`, etc. This will then take precedence over the glibc routines. The full list of functions is available in `sysdeps/x86_64/multiarch/ifunc-impl-list.c`. Current HEAD has more symbols compared to the glibc 2.25 release, in total (`grep -Po 'IFUNC_IMPL \(i, name, \K[^,]+' sysdeps/x86_64/multiarch/ifunc-impl-list.c`):

memchr, memcmp, __memmove_chk, memmove, memrchr, __memset_chk, memset, rawmemchr, strlen, strnlen, stpncpy, stpcpy, strcasecmp, strcasecmp_l, strcat, strchr, strchrnul, strrchr, strcmp, strcpy, strcspn, strncasecmp, strncasecmp_l, strncat, strncpy, strpbrk, strspn, strstr, wcschr, wcsrchr, wcscpy, wcslen, wcsnlen, wmemchr, wmemcmp, wmemset, __memcpy_chk, memcpy, __mempcpy_chk, mempcpy, strncmp, __wmemset_chk,

Problem

Original source

Related problems

• How to debug program with custom elf interpreter?