Skip to content

x86/retbleed: Call depth tracking mitigation

Peter Zijlstra kvm lkml 2 versions – Applied

Status from this message: The following commit has been merged into the x86/paravirt branch of tip:

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest59 patches 19 repliesApplied
  2. v259 patches 21 repliesSuperseded

Patches in v3 Applied

#Subject and filesSizeRepliesApplied
1/59x86/paravirt: Ensure proper alignmentarch/x86/include/asm/paravirt.h arch/x86/include/asm/paravirt_types.h+2 -00 replies
2/59x86/cpu: Remove segment load from switch_to_new_gdt()arch/x86/include/asm/processor.h arch/x86/kernel/cpu/common.c+31 -170 replies
3/59x86/cpu: Get rid of redundant switch_to_new_gdt() invocationsarch/x86/include/asm/processor.h arch/x86/kernel/cpu/common.c arch/x86/kernel/setup_percpu.c +2 more+14 -150 replies
4/59x86/cpu: Re-enable stackprotectorarch/x86/kernel/cpu/Makefile arch/x86/kernel/cpu/common.c+3 -30 replies
5/59x86/modules: Set VM_FLUSH_RESET_PERMS in module_alloc()arch/x86/kernel/ftrace.c arch/x86/kernel/kprobes/core.c arch/x86/kernel/module.c+5 -70 replies
6/59x86/vdso: Ensure all kernel code is seen by objtoolarch/x86/entry/vdso/Makefile+6 -50 replies
7/59x86: Sanitize linker scriptarch/x86/kernel/vmlinux.lds.S+7 -61 reply
8/59arch: Introduce CONFIG_FUNCTION_ALIGNMENTMakefile arch/Kconfig arch/ia64/Kconfig +6 more+36 -100 replies
9/59x86/asm: Differentiate between code and function alignmentarch/x86/include/asm/linkage.h+9 -50 replies
10/59x86/error_inject: Align function properlyarch/x86/lib/error-inject.c+1 -00 replies
11/59x86/paravirt: Properly align PV functionsarch/x86/include/asm/paravirt.h arch/x86/include/asm/qspinlock_paravirt.h arch/x86/kernel/kvm.c +1 more+5 -11 reply
12/59x86/entry: Align SYM_CODE_START() variantsarch/x86/entry/entry_64.S arch/x86/entry/thunk_64.S+10 -60 replies
13/59crypto: x86/camellia: Remove redundant alignmentsarch/x86/crypto/camellia-aesni-avx-asm_64.S arch/x86/crypto/camellia-aesni-avx2-asm_64.S+0 -60 replies
14/59crypto: x86/cast5: Remove redundant alignmentsarch/x86/crypto/cast5-avx-x86_64-asm_64.S+0 -20 replies
15/59crypto: x86/crct10dif-pcl: Remove redundant alignmentsarch/x86/crypto/crct10dif-pcl-asm_64.S+0 -10 replies
16/59crypto: x86/serpent: Remove redundant alignmentsarch/x86/crypto/serpent-avx-x86_64-asm_64.S arch/x86/crypto/serpent-avx2-asm_64.S+0 -40 replies
17/59crypto: x86/sha1: Remove custom alignmentsarch/x86/crypto/sha1_ni_asm.S+0 -10 replies
18/59crypto: x86/sha256: Remove custom alignmentsarch/x86/crypto/sha256-avx-asm.S arch/x86/crypto/sha256-avx2-asm.S arch/x86/crypto/sha256-ssse3-asm.S +1 more+0 -40 replies
19/59crypto: x86/sm[34]: Remove redundant alignmentsarch/x86/crypto/sm3-avx-asm_64.S arch/x86/crypto/sm4-aesni-avx-asm_64.S arch/x86/crypto/sm4-aesni-avx2-asm_64.S+0 -140 replies
20/59crypto: twofish: Remove redundant alignmentsarch/x86/crypto/twofish-avx-x86_64-asm_64.S+0 -20 replies
21/59crypto: x86/poly1305: Remove custom function alignmentarch/x86/crypto/poly1305-x86_64-cryptogams.pl+0 -10 replies
22/59x86: Put hot per CPU variables into a structarch/x86/include/asm/current.h arch/x86/kernel/cpu/common.c arch/x86/kernel/process_32.c +2 more+24 -150 replies
23/59x86/percpu: Move preempt_count next to current_taskarch/x86/include/asm/current.h arch/x86/include/asm/preempt.h arch/x86/kernel/cpu/common.c+16 -200 replies
24/59x86/percpu: Move cpu_number next to current_taskarch/x86/include/asm/current.h arch/x86/include/asm/smp.h arch/x86/kernel/setup_percpu.c+7 -110 replies
25/59x86/percpu: Move current_top_of_stack next to current_taskarch/x86/entry/entry_32.S arch/x86/entry/entry_64.S arch/x86/entry/entry_64_compat.S +8 more+19 -280 replies
26/59x86/percpu: Move irq_stack variables next to current_taskarch/x86/include/asm/current.h arch/x86/include/asm/irq_stack.h arch/x86/include/asm/processor.h +6 more+24 -280 replies
27/59x86/softirq: Move softirq pending next to current taskarch/x86/include/asm/current.h arch/x86/include/asm/hardirq.h+3 -10 replies
28/59objtool: Allow !PC relative relocationstools/objtool/arch/x86/decode.c tools/objtool/check.c tools/objtool/include/objtool/arch.h+27 -10 replies
29/59objtool: Track init sectiontools/objtool/check.c tools/objtool/include/objtool/elf.h+11 -80 replies
30/59objtool: Add .call_sites sectionarch/x86/kernel/vmlinux.lds.S tools/objtool/check.c tools/objtool/include/objtool/objtool.h +1 more+60 -00 replies
31/59objtool: Add --hacks=skylakescripts/Makefile.lib tools/objtool/builtin-check.c tools/objtool/check.c +1 more+15 -60 replies
32/59objtool: Allow STT_NOTYPE -> STT_FUNC+0 tail-callstools/objtool/check.c+20 -96 replies
33/59objtool: Fix find_{symbol,func}_containing()tools/include/linux/interval_tree_generic.h tools/objtool/elf.c tools/objtool/include/objtool/elf.h+229 -540 replies
34/59objtool: Allow symbol range comparisons for IBT/ENDBRarch/x86/entry/entry_64_compat.S tools/objtool/check.c+28 -10 replies
35/59x86/entry: Make sync_regs() invocation a tail callarch/x86/entry/entry_64.S+2 -50 replies
36/59ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLEinclude/asm-generic/vmlinux.lds.h kernel/trace/Kconfig tools/objtool/check.c+18 -20 replies
37/59x86/putuser: Provide room for paddingarch/x86/lib/putuser.S+49 -130 replies
38/59x86/Kconfig: Add CONFIG_CALL_THUNKSarch/x86/Kconfig+8 -00 replies
39/59x86/Kconfig: Introduce function paddingarch/x86/Kconfig arch/x86/Makefile arch/x86/entry/vdso/Makefile +1 more+40 -40 replies
40/59x86/retbleed: Add X86_FEATURE_CALL_DEPTHarch/x86/Kconfig arch/x86/include/asm/cpufeatures.h arch/x86/include/asm/disabled-features.h+28 -10 replies
41/59x86/alternatives: Provide text_poke_copy_locked()arch/x86/include/asm/text-patching.h arch/x86/kernel/alternative.c+23 -150 replies
42/59x86/entry: Make some entry symbols globalarch/x86/entry/entry_64.S+6 -30 replies
43/59x86/paravirt: Make struct paravirt_call_site unconditionally availablearch/x86/include/asm/paravirt.h arch/x86/include/asm/paravirt_types.h+14 -100 replies
44/59x86/callthunks: Add call patching for call depth trackingarch/x86/Kconfig arch/x86/include/asm/alternative.h arch/x86/kernel/Makefile +5 more+287 -100 replies
45/59x86/modules: Add call patchingarch/x86/include/asm/alternative.h arch/x86/kernel/callthunks.c arch/x86/kernel/module.c+43 -10 replies
46/59x86/returnthunk: Allow different return thunksarch/x86/include/asm/nospec-branch.h arch/x86/kernel/alternative.c arch/x86/kernel/ftrace.c +2 more+23 -80 replies
47/59x86/asm: Provide ALTERNATIVE_3arch/x86/include/asm/alternative.h+30 -30 replies
48/59x86/retbleed: Add SKL return thunkarch/x86/entry/entry_64.S arch/x86/include/asm/current.h arch/x86/include/asm/nospec-branch.h +3 more+156 -108 replies
49/59x86/retpoline: Add SKL retthunk retpolinesarch/x86/include/asm/nospec-branch.h arch/x86/kernel/alternative.c arch/x86/lib/retpoline.S +1 more+135 -120 replies
50/59x86/retbleed: Add SKL call thunkarch/x86/kernel/callthunks.c+20 -50 replies
51/59x86/calldepth: Add ret/call counting for debugarch/x86/include/asm/nospec-branch.h arch/x86/kernel/callthunks.c arch/x86/lib/retpoline.S+91 -50 replies
52/59static_call: Add call depth tracking supportarch/x86/include/asm/alternative.h arch/x86/kernel/callthunks.c arch/x86/kernel/static_call.c +2 more+44 -50 replies
53/59kallsyms: Take callthunks into accountkernel/kallsyms.c+40 -50 replies
54/59x86/orc: Make it callthunk awarearch/x86/include/asm/alternative.h arch/x86/kernel/callthunks.c arch/x86/kernel/unwind_orc.c+38 -10 replies
55/59x86/bpf: Emit call depth accounting if requiredarch/x86/include/asm/alternative.h arch/x86/kernel/callthunks.c arch/x86/net/bpf_jit_comp.c+48 -91 reply
56/59x86/ftrace: Remove ftrace_epilogue()arch/x86/kernel/ftrace_64.S+6 -151 reply
57/59x86/ftrace: Rebalance RSBarch/x86/kernel/ftrace_64.S+11 -00 replies
58/59x86/ftrace: Make it call depth tracking awarearch/x86/include/asm/nospec-branch.h arch/x86/kernel/callthunks.c arch/x86/kernel/ftrace.c +8 more+64 -80 replies
58/59x86/ftrace: Make it call depth tracking awarearch/x86/include/asm/nospec-branch.h arch/x86/kernel/callthunks.c arch/x86/kernel/ftrace.c +8 more+68 -81 reply
59/59x86/retbleed: Add call depth tracking mitigationarch/x86/kernel/cpu/bugs.c+30 -20 replies

Reviews trailers per patch

2 of 60 patches have a Reviewed-by or Acked-by.

PatchJuergen GrossLinus TorvaldsNathan Chancellor
1/59R-b
2/59
3/59
4/59
5/59
6/59
7/59
8/59Requested-by
9/59
10/59
11/59R-b
12/59
13/59
14/59
15/59
16/59
17/59
18/59
19/59
20/59
21/59
22/59
23/59
24/59
25/59
26/59
27/59
28/59
29/59
30/59
31/59
32/59
33/59
34/59
35/59
36/59
37/59
38/59
39/59
40/59
41/59
42/59
43/59
44/59
45/59
46/59
47/59
48/59T-b
49/59
50/59
51/59
52/59
53/59
54/59
55/59
56/59
57/59
58/59
58/59
59/59

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v3 thread

Cover letter [PATCH v3 00/59] x86/retbleed: Call depth tracking mitigation

Hi!

Previous postings:

v2: https://lkml.kernel.org/r/[email protected]
v1: https://lkml.kernel.org/r/[email protected]

Changes since v2 are minimal; I reworked the alignment thing per Linus' request (patch #8) and collected a few tags.

Barring great objections I'm hoping to merge this soon so we can all get on with other things.

--- text from v2 ---

This version is significantly different from the last in that it no longer makes use of external call thunks allocated from the module space. Instead every function gets aligned to 16 bytes and gets 16 bytes of (pre-symbol) padding. (This padding will also come in handy for other things, like the kCFI/FineIBT work.)

Prior to these patches function alignment is basically non-existent, as such any instruction fetch for the first instructions of a function will have (on average) half the fetch window filled with whatever comes before. By pushing the alignment up to 16 bytes this improves matters for chips that happen to have a 16 byte i-fetch window size (Intel) while not making matters worse for chips that have a larger 32 byte i-fetch window (AMD Zen). In fact, it improves the worst case for Zen from 31 bytes of garbage to 16 bytes of garbage.

As such the first many patches of the series fix up lots of alignment quirks.

The second big difference is the introduction of struct pcpu_hot. Because the compiler managed to place two adjacent (in code) DEFINE_PER_CPU() variables in random cachelines (it is absolutely free to do so) the introduction of the per-cpu x86_call_depth variable sometimes introduced significant additional cache pressure, while other times it would sit nicely in the same line with preempt_count and not show up at all.

In order to alleviate this problem; introduce struct pcpu_hot and collect a number of hot per-cpu variables in a way the compiler can't mess up.

Since these changes are 'unconditional', Mel was gracious enough to help test this on his test setup across all the relevant uarchs (very much including both Intel and AMD machines) and found that while these changes cause some very small wins and losses across the board it is mostly noise.

Aside from these changes; the core of the depth tracking is still the same.

- objtool creates a list of functions and a list of function call sites.

- for every function the padding is overwritten with the call accounting thunk; for every call site the call target is adjusted to point to this thunk.

- the retbleed return thunk mechanism is used for a custom return thunk that includes return accounting and does RSB stuffing when required.

This ensures no new compiler is required and avoids almost all overhead for non affected machines. This new option can still be selected using:

"retbleed=stuff"

on the kernel command line.

As a refresher; the theory behind call depth tracking is:

The Return-Stack-Buffer (RSB) is a 16 deep stack that is filled on every call. On the return path speculation will "pop" an entry and takes that as the return target. Once the RSB is empty, the CPU falls back to other predictors, e.g. the Branch History Buffer, which can be mistrained by user space and misguides the (return) speculation path to a disclosure gadget of your choice -- as described in the retbleed paper.

Call depth tracking is designed to break this speculation path by stuffing speculation trap calls into the RSB whenver the RSB is running low. This way the speculation stalls and never falls back to other predictors.

The assumption is that stuffing at the 12th return is sufficient to break the speculation before it hits the underflow and the fallback to the other predictors. Testing confirms that it works. Johannes, one of the retbleed researchers, tried to attack this approach and confirmed that it brings the signal to noise ratio down to the crystal ball level.

Excerpts from IBRS vs stuff from Mel's testing:

perfsyscall

		6.0.0-rc1		6.0.0-rc1
                tglx-mit-spectre-ibrs	tglx-mit-spectre-retpoline-retstuff
Duration User         136.16		   69.10
Duration System       100.50		   33.04
Duration Elapsed      237.20		  102.65

That's a massive improvement with a major reduction in system CPU usage.

Kernel compilation is variable. Skylake-X was modest with 2-18% gain depending on degree of parallelisation.

Git checkouts are roughly 14% faster on Skylake-X

Network test were localhost only so are limited but even so, the gain is large. Skylake-X again;

Netperf-TCP

                                  6.0.0-rc1              6.0.0-rc1
                      tglx-mit-spectre-ibrs  tglx-mit-spectre-retpoline-retstuff
Hmean     send-64         241.39 (   0.00%)      298.00 *  23.45%*
Hmean     send-128        489.55 (   0.00%)      610.46 *  24.70%*
Hmean     send-256        990.85 (   0.00%)     1201.73 *  21.28%*
Hmean     send-1024      4051.84 (   0.00%)     5006.19 *  23.55%*
Hmean     send-2048      7924.75 (   0.00%)     9777.14 *  23.37%*
Hmean     send-3312     12319.98 (   0.00%)    15210.07 *  23.46%*
Hmean     send-4096     14770.62 (   0.00%)    17941.32 *  21.47%*
Hmean     send-8192     26302.00 (   0.00%)    30170.04 *  14.71%*
Hmean     send-16384    42449.51 (   0.00%)    48036.45 *  13.16%*

While this is UDP_STREAM, TCP_STREAM is similarly impressive.

FIO measurements done by Tim Chen:

read (kIOPs) Mean stdev mitigations=off retbleed=off CPU util
================================================================================

mitigations=off         357.33  3.79    0.00%           6.14%           98.93%
retbleed=off            336.67  6.43    -5.78%          0.00%           99.01%
retbleed=ibrs           242.00  0.00    -32.28%         -28.12%         99.41%
retbleed=stuff (pad)    314.33  1.53    -12.03%         -6.63%          99.31%

read/write Baseline Baseline
70/30 (kIOPs) Mean stdev mitigations=off retbleed=off CPU util
================================================================================

mitigations=off         349.00  5.29    0.00%           9.06%           96.66%
retbleed=off            320.00  5.05    -8.31%          0.00%           95.54%
retbleed=ibrs           238.60  0.17    -31.63%         -25.44%         98.18%
retbleed=stuff (pad)    293.37  0.81    -15.94%         -8.32%          97.71%

Baseline Baseline
write (kIOPs) Mean stdev mitigations=off retbleed=off CPU util
================================================================================

mitigations=off         296.33  8.08    0.00%           6.21%           93.96%
retbleed=off            279.00  2.65    -5.85%          0.00%           93.63%
retbleed=ibrs           230.33  0.58    -22.27%         -17.44%         95.92%
retbleed=stuff (pad)    266.67  1.53    -10.01%         -4.42%          94.75%

---

The patches can also be found in git here:

git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git call-depth-tracking