How can I create a spectre gadget in practice?

Question

I'm developing (NASM + GCC targetting ELF64) a PoC that uses a spectre gadget that measures the time to access a set of cache lines (FLUSH+RELOAD). How can I make a reliable spectre gadget? I believe I understand the theory behind the FLUSH+RELOAD technique, however in practice, despiste some noise, I'm unable to produce a working PoC. Since I'm using the Timestamp counter and the loads are very regular I use this script to disable the prefetchers, the turbo boost and to fix/stabilize the CPU frequency: ``` #!/bin/bash sudo modprobe msr #Disable turbo sudo wrmsr -a 0x1a0 0x4000850089 #Disable prefetchers sudo wrmsr -a 0x1a4 0xf #Set performance governor sudo cpupower frequency-set -g performance #Minimum freq sudo cpupower frequency-set -d 2.2GHz #Maximum freq sudo cpupower frequency-set -u 2.2GHz ``` I have a continuous buffer, aligned on 4KiB, large enough to span 256 cache lines separated by an integral number GAP of lines. ``` SECTION .bss ALIGN=4096 buffer: resb 256 * (1 + GAP) *…

Accepted Answer

The buffer is allocated from the `bss` section and so when the program is loaded, the OS will map all of the `buffer` cache lines to the same CoW physical page. After flushing all of the lines, only the accesses to the first 64 lines in the virtual address space miss in all cache levels1 because all2 later accesses are to the same 4K page. That's why the latencies of the first 64 accesses fall in the range of the main memory latency and the latencies of all later accesses are equal to the L1 hit latency3 when `GAP` is zero.

When `GAP` is 1, every other line of the same physical page is accessed and so the number of main memory accesses (L3 misses) is 32 (half of 64). That is, the first 32 latencies will be in the range of the main memory latency and all later latencies will be L1 hits. Similarly, when `GAP` is 63, all accesses are to the same line. Therefore, only the first access will miss all caches.

The solution is to change `mov eax, [rdi]` in `flush_all` to `mov dword [rdi], 0` to ensure that the buffer is allocated in unique physical pages. (The `lfence` instructions in `flush_all` can be removed because the Intel manual states that `clflush` cannot be reordered with writes4.) This guarantees that, after initializing and flushing all lines, all accesses will miss all cache levels (but not the TLB, see: Does clflush also remove TLB entries?).

You can refer to Why are the user-mode L1 store miss events only counted when there is a store initialization loop? for another example where CoW pages can be deceiving.

I suggested in the previous version of this answer to remove the call to `flush_all` and use a `GAP` value of 63. With these changes, all of the access latencies appeared to be very high and I have incorrectly concluded that all of the accesses are missing all cache levels. Like I said above, with a `GAP` value of 63, all of the accesses become to the same cache line, which is actually resident in the L1 cache. However, the reason that all of the latencies were high is because every access was to a different virtual page and the TLB didn't have any of mappings for each of these virtual pages (to the same physical page) because by removing the call to `flush_all`, none of the virtual pages were touched before. So the measured latencies represent the TLB miss latency, even though the line being accessed is in the L1 cache.

I also incorrectly claimed in the previous version of this answer that there is an L3 prefetching logic that cannot be disabled through MSR 0x1A4. If a particular prefetcher is turned off by setting its flag in MSR 0x1A4, then it does fully get switched off. Also there are no data prefetchers other than the ones documented by Intel.

Footnotes:

(1) If you don't disable the DCU IP prefetcher, it will actually prefetch back all the lines into the L1 after flushing them, so all accesses will still hit in the L1.

(2) In rare cases, the execution of interrupt handlers or scheduling other threads on the same core may cause some of the lines to be evicted from the L1 and potentially other levels of the cache hierarchy.

(3) Remember that you need to subtract the overhead of the `rdtscp` instructions. Note that the measurement method you used actually doesn't enable you to reliably distinguish between an L1 hit and an L2 hit. See: Memory latency measurement with time stamp counter.

(4) The Intel manual doesn't seem to specify whether `clflush` is ordered with reads, but it appears to me that it is.

How can I create a spectre gadget in practice?

Solution

Problem

Related problems