x86/retbleed: Call depth tracking mitigation
Subsystems (24)
cover x86/retbleed: Call depth tracking mitigation
Folks!
Back in the good old spectre v2 days (2018) we decided to not use IBRS. In hindsight this might have been the wrong decision because it did not force people to come up with alternative approaches.
It was already discussed back then to try software based call depth accounting and RSB stuffing on underflow for Intel SKL[-X] systems to avoid the insane overhead of IBRS.
This has been tried in 2018 and was rejected due to the massive overhead and other shortcomings of the approach to put the accounting into each function prologue:
1) Text size increase which is inflicted on everyone. While CPUs are good in ignoring NOPs they still pollute the I-cache.
2) That results in tail call over-accounting which can be exploited.
Disabling tail calls is not an option either and adding a 10 byte padding in front of every direct call is even worse in terms of text size and I-cache impact. We also could patch calls past the accounting in the function prologue but that becomes a nightmare vs. ENDBR.
As IBRS is a performance horror show, Peter Zijstra and me revisited the call depth tracking approach and implemented it in a way which is hopefully more palatable and avoids the downsides of the original attempt.
We both unsurprisingly hate the result with a passion...
The way we approached this is:
1) objtool creates a list of function entry points and a list of direct call sites into new sections which can be discarded after init.
2) On affected machines, use the new sections, allocate module memory and create a call thunk per function (16 bytes without debug/statistics). Then patch all direct calls to invoke the thunk, which does the call accounting and then jumps to the original call site.
3) Utilize the retbleed return thunk mechanism by making the jump target run-time configurable. Add the accounting counterpart and stuff RSB on underflow in that alternate implementation.
This does not need a new compiler and avoids almost all overhead for non-affected machines. It can be selected via 'retbleed=stuff' on the kernel command line.
The memory consumption is impressive. On a affected server with a Debian config this results in about 1.8MB call thunk memory and 2MB btree memory to keep track of the thunks. The call thunk memory is overcommitted due to the way how objtool collects symbols. This probably could be cut in half, but we need to allocate a 2MB region anyway due to the following:
Our initial approach of just using module_alloc() turned out to create a massive ITLB miss rate, which is not surprising as each call goes out to a randomly distributed call thunk. Peter added a method to allocate with large 2MB TLBs which made that problem mostly go away and gave us another nice performance gain. It obviously does not reduce ICache overhead. It neither cures the problem that the ITLB cache has one slot permanently occupied by the call-thunk mapping.
The theory behind call depth tracking is:
RSB is a stack with depth 16 which is filled on every call. On the return path speculation "pops" entries to speculate down the call chain. Once the speculative RSB is empty it switches to other predictors, e.g. the Branch History Buffer, which can be mistrained by user space and misguides the speculation path to a disclosure gadget as described in the retbleed paper.
Call depth tracking is designed to break this speculation path by stuffing speculation trap calls into the RSB which are never getting a corresponding return executed. This stalls the prediction path until it gets resteered,
The assumption is that stuffing at the 12th return is sufficient to break the speculation before it hits the underflow and the fallback to the other predictors. Testing confirms that it works. Johannes, one of the retbleed researchers. tried to attack this approach and confirmed that it brings the signal to noise ratio down to the crystal ball level.
There is obviously no scientific proof that this will withstand future research progress, but all we can do right now is to speculate about that.
So much for the theory. Here are numbers for various tests I did myself and FIO results provided by Tim Chen.
Just to make everyone feel as bad as I feel, this comes with two overhead
values: one against mitigations=off and one against retbleed=off.
hackbench Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 24.41s 0.00% -15.26%
retbleed=off 28.80s +18.00% 0.00%
retbleed=ibrs 34.92s +43.07% +21.24%
retbleed=stuff 31.95s +30.91% +10.94%
----------------------------------------------------------------------------
sys_time loop Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 1.28s 0.00% -78.66%
retbleed=off 5.98s +368.50% 0.00%
retbleed=ibrs 9.73s +662.81% +62.82%
retbleed=stuff 6.15s +381.93% +2.87%
----------------------------------------------------------------------------
kernel build Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 197.55s 0.00% -3.64%
retbleed=off 205.02s +3.78% 0.00%
retbleed=ibrs 218.00s +10.35% +6.33%
retbleed=stuff 209.92s +6.26% +2.39%
----------------------------------------------------------------------------
microbench deep
callchain Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 1.72s 0.00% -39.88%
retbleed=off 2.86s +66.34% 0.00%
retbleed=ibrs 3.92s +128.23% +37.20%
retbleed=stuff 3.39s +97.05% +18.46%
----------------------------------------------------------------------------
fio rand-read Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 352.7 kIOPS 0.00% +3.42%
retbleed=off 341.0 kIOPS -3.31% 0.00%
retbleed=ibrs 242.0 kIOPS -31.38% -29.03%
retbleed=stuff 288.3 kIOPS -18.24% -15.44%
----------------------------------------------------------------------------
fio 70/30 Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 349.0 kIOPS 0.00% +10.49%
retbleed=off 315.9 kIOPS -9.49% 0.00%
retbleed=ibrs 241.5 kIOPS -30.80% -23.54%
retbleed=stuff 276.2 kIOPS -20.86% -12.56%
----------------------------------------------------------------------------
fio write Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 299.3 kIOPS 0.00% +10.49%
retbleed=off 275.6 kIOPS -9.49% 0.00%
retbleed=ibrs 232.7 kIOPS -22.27% -15.60%
retbleed=stuff 259.3 kIOPS -13.36% -5.93%
----------------------------------------------------------------------------
Sockperf 14 byte
localhost Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 4.495 MBps 0.00% +33.19%
retbleed=off 3.375 MBps -24.92% 0.00%
retbleed=ibrs 2.573 MBps -42.76% -23.76%
retbleed=stuff 2.725 MBps -39.38% -19.26%
----------------------------------------------------------------------------
Sockperf 1472 byte
localhost Baseline: mitigations=off retbleed=off
----------------------------------------------------------------------------
mitigations=off 425.494 MBps 0.00% +26.21%
retbleed=off 337.132 MBps -20.77% 0.00%
retbleed=ibrs 261.243 MBps -38.60% -22.51%
retbleed=stuff 275.111 MBps -35.34% -18.40%
----------------------------------------------------------------------------
The micro benchmark is just an artificial 30 calls deep call-chain through a syscall with empty functions to measure the overhead. But it's also interestingly close to the pathological FIO random read case. It just emphasizes the overhead slightly more. I wrote that because I was not able to take real FIO numbers as any variant was close to the max-out point of that not so silly fast SSD/NVME in my machine.
So the benefit varies depending on hardware and workload scenarios. At least it does not get worse than IBeeRS.
The whole lot is also available from git:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git depthtracking
The ftrace and BPF changes need some extra scrunity from the respective experts.
Peter and me went great length to analyze the overhead. Unsurprisingly the call thunks are contributing the largest amount after the actual stuffing in the return thunk. The call thunks are creating trouble in various ways:
1) An extra ITLB which has been very prominent in our first experiments where we had 4k PTEs instead of the 2M mapping which we use now. But this still causes ITLB misses in certain workloads
2) Icache footprint is obviously larger and the out of line call-thunks are not prefetcher friendly.
3) The extra jump adds overhead for predictors and branch history
A potential solution for this is to let the compiler add a 16 byte padding in front of every function. That allows to bring the call thunk into ITLB/Icache locality and avoids the extra jump.
Obviously this would only affect SKL variants and no other machine would go through that code (which is just padding in the compiled binary). Contrary to adding this to the prologue and NOPing it out, this avoids the ICache overhead for the non SKL case completely.
The function alignment option does not work for that because it just guarantees that the next function entry is aligned, but the padding size depends on the position of the last instruction of the previous function which might be anything between 0 and padsize-1 obviously, which is not a good starting point to put 10 bytes of accounting code into it reliably.
I hacked up GCC to emit such padding and from first experimentation it brings quite some performance back.
IBRS stuff stuff(pad)
sockperf 14 bytes: -23.76% -19.26% -14.31%
sockperf 1472 bytes: -22.51% -18.40% -12.25%
microbench: +37.20% +18.46% +15.47%
hackbench: +21.24% +10.94% +10.12%
For FIO I don't have numbers yet, but I expect FIO to get a significant gain too.
Quoted 1 line
From a quick survey it seems to have no impact for the case where the
thunks are not used. But that really needs some deep investigation and there is a potential conflict with the clang CFI efforts.
The kernel text size increases with a Debian config from 9.9M to 10.4M, so about 5%. If the thunk is not 16 byte aligned, the text size increase is about 3%, but it turned out that 16 byte aligned is slightly faster.
The 16 byte function alignment turned out to be beneficial in general even without the thunks. Not much of an improvement, but measurable. We should revisit this independent of these horrors.
The implementation falls back to the allocated thunks when padding is not available. I'll send out the GCC patch and the required kernel patch as a reply to this series after polishing it a bit.
We also evaluated another approach to solve this problem, which does not involve call thunks and solely relies on return thunks. The idea is to have a return thunk which is not reliably predictable. That option would be "retbleed=confuse". It looks like this:
retthunk:
test $somebit, something jz 1f ret int3 1: test $someotherbit, something jnz 2f ret int3 2: ....
There are two questions to this approach:
1) How does $something have enough entropy to be effective
Ideally we can use a register for that, because the frequent memory operation in the return thunk has obviously a significant performance penalty.
In combination with stack randomization the RSP looks like a feasible option, but for kernel centric workloads that does not create much entropy.
Though keep in mind that the attacks rely on syscalls and observing their side effects.
Which in turn means that entry stack randomization is a good entropy source. But it's easy enough to create a scenario where the attacking thread spins and relies on interrupts to cause the kernel entry.
Interrupt entry from user-space does not do kernel task stack randomization today, but that's easy enough to fix.
If we can't use RSP because it turns out to be too predictable, then we can still create entropy in a per CPU variable, but RSP is definitely faster than a memory access.
2) How "easy" is it to circumvent the "randomization"
It makes it impossible for the occasional exploit writer like myself and probably for the vast majority of script kiddies, but we really need input from the researchers who seem to have way more nasty ideas than we can imagine.
I asked some of those folks, but the answers were not conclusive at all. There are some legitimate concerns, but the question is whether they are real threats or purely academic problems (at the moment).
That retthunk "randomization" can easily be extended to have several return thunks and they can be assigned during patching randomly. As that's a boot time decision it's possible to figure it out, but in the worst case we can randomly reassign them once in a while.
But also this approach has quite some performance impact:
For 2 RET pathes randomized with randomize_kstack_offset=y and RSP bit 3:
IBRS stuff stuff(pad) confuse
microbench: +37.20% +18.46% +15.47% +6.76%
sockperf 14 bytes: -23.76% -19.26% -14.31% -9.04% sockperf 1472 bytes: -22.51% -18.40% -12.25% -7.51%
For 3 RET pathes randomized with randomize_kstack_offset=y and RSP bit 3, 6:
IBRS stuff stuff(pad) confuse
microbench: +37.20% +18.46% +15.47% +7.12%
sockperf 14 bytes: -23.76% -19.26% -14.31% -12.03% sockperf 1472 bytes: -22.51% -18.40% -12.25% -10.49%
For 4 RET pathes randomized with randomize_kstack_offset=y and RSP bit 3, 6, 5:
IBRS stuff stuff(pad) confuse
microbench: +37.20% +18.46% +15.47% +7.46%
sockperf 14 bytes: -23.76% -19.26% -14.31% -16.80% sockperf 1472 bytes: -22.51% -18.40% -12.25% -15.95%
So for the more randomized variant sockperf tanks and is already slower than stuffing with thunks in the compiler provided padding space.
I send out a patch in reply to this series which implements that variant, but there needs to be input from the security researchers how protective this is. If we could get away with 2 RET pathes (perhaps multiple instances with different bits), that would be amazing.
Thanks go to Andrew Cooper for helpful discussions around this, Johannes Wikner for spending the effort of trying to break the stuffing defense and to Tim Chen for getting the FIO numbers to us.
And of course many thanks to Peter for working on this with me. We went through quite some mispredicted branches and had to retire some "brilliant" ideas on the way.
Thanks,
tglx
--- arch/x86/Kconfig | 37 + arch/x86/entry/entry_64.S | 26 arch/x86/entry/vdso/Makefile | 11 arch/x86/include/asm/alternative.h | 69 ++ arch/x86/include/asm/cpufeatures.h | 1 arch/x86/include/asm/disabled-features.h | 9 arch/x86/include/asm/module.h | 2 arch/x86/include/asm/nospec-branch.h | 166 +++++ arch/x86/include/asm/paravirt.h | 5 arch/x86/include/asm/paravirt_types.h | 21 arch/x86/include/asm/processor.h | 1 arch/x86/include/asm/text-patching.h | 2 arch/x86/kernel/Makefile | 2 arch/x86/kernel/alternative.c | 114 +++ arch/x86/kernel/callthunks.c | 799 ++++++++++++++++++++++++++++ arch/x86/kernel/cpu/bugs.c | 32 + arch/x86/kernel/cpu/common.c | 17 arch/x86/kernel/ftrace.c | 20 arch/x86/kernel/ftrace_64.S | 31 + arch/x86/kernel/kprobes/core.c | 1 arch/x86/kernel/module.c | 86 --- arch/x86/kernel/static_call.c | 3 arch/x86/kernel/unwind_orc.c | 21 arch/x86/kernel/vmlinux.lds.S | 14 arch/x86/lib/retpoline.S | 106 +++ arch/x86/mm/Makefile | 2 arch/x86/mm/module_alloc.c | 68 ++ arch/x86/net/bpf_jit_comp.c | 56 + include/linux/btree.h | 6 include/linux/kallsyms.h | 24 include/linux/module.h | 72 +- include/linux/static_call.h | 2 include/linux/vmalloc.h | 3 init/main.c | 2 kernel/kallsyms.c | 23 kernel/kprobes.c | 52 + kernel/module/internal.h | 8 kernel/module/main.c | 6 kernel/module/tree_lookup.c | 17 kernel/static_call_inline.c | 23 kernel/trace/trace_selftest.c | 5 lib/btree.c | 8 mm/vmalloc.c | 33 - samples/ftrace/ftrace-direct-modify.c | 2 samples/ftrace/ftrace-direct-multi-modify.c | 2 samples/ftrace/ftrace-direct-multi.c | 1 samples/ftrace/ftrace-direct-too.c | 1 samples/ftrace/ftrace-direct.c | 1 scripts/Makefile.lib | 3 tools/objtool/arch/x86/decode.c | 26 tools/objtool/builtin-check.c | 7 tools/objtool/check.c | 160 ++++- tools/objtool/include/objtool/arch.h | 4 tools/objtool/include/objtool/builtin.h | 1 tools/objtool/include/objtool/check.h | 20 tools/objtool/include/objtool/elf.h | 2 tools/objtool/include/objtool/objtool.h | 1 tools/objtool/objtool.c | 1 58 files changed, 1970 insertions(+), 268 deletions(-)
PATCH 1/38 x86/paravirt: Ensure proper alignment
The entries in the .parainstr sections are 8 byte aligned and the corresponding C struct makes the array offset 16 bytes.
Though the pushed entries are only using 12 bytes. .parainstr_end is therefore 4 bytes short.
That works by chance because it's only used in a loop:
for (p = start; p < end; p++)
But this falls flat when calculating the number of elements:
n = end - start
That's obviously off by one.
Ensure that the gap is filled and the last entry is occupying 16 bytes.
--- arch/x86/include/asm/paravirt.h | 1 + arch/x86/include/asm/paravirt_types.h | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/paravirt.h +1 -0
@@ -743,6 +743,7 @@ extern void default_banner(void); word 771b; \ .byte ptype; \ .byte 772b-771b; \ + _ASM_ALIGN; \ .popsection
arch/x86/include/asm/paravirt_types.h +1 -0
@@ -294,6 +294,7 @@ extern struct paravirt_patch_template pv" .byte " type"\n" \" .byte 772b-771b\n" \" .short " clobber"\n" \ + _ASM_ALIGN"\n" \".popsection\n" /* Generate patchable code, with the default asm parameters. */
PATCH 2/38 x86/cpu: Use native_wrmsrl() in load_percpu_segment()
In reply to Thomas Gleixnerload_percpu_segment() is using wrmsr() which is paravirtualized. That's an issue because the code sequence is:
__loadsegment_simple(gs, 0); wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu));
So anything which uses a per CPU variable between setting GS to 0 and writing GSBASE is going to end up in a NULL pointer dereference. That's can be triggered with instrumentation and is guaranteed to be triggered with callthunks for call depth tracking.
Use native_wrmsrl() instead. XEN_PV will trap and emulate, but that's not a hot path.
Also make it static and mark it noinstr so neither kprobes, sanitizers or whatever can touch it.
--- arch/x86/include/asm/processor.h | 1 - arch/x86/kernel/cpu/common.c | 12 ++++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-)
arch/x86/include/asm/processor.h +0 -1
@@ -673,7 +673,6 @@ extern struct desc_ptr early_gdt_descr;extern void switch_to_new_gdt (int );extern void load_direct_gdt (int );extern void load_fixmap_gdt (int ); -extern void load_percpu_segment (int );extern void cpu_init (void );extern void cpu_init_secondary (void );extern void cpu_init_exception_handling (void );
arch/x86/kernel/cpu/common.c +10 -2
@@ -701,13 +701,21 @@ static const char *table_lookup_model(st__u32 cpu_caps_cleared[NCAPINTS +NBUGINTS ]__aligned(sizeof(unsigned long)) ;__u32 cpu_caps_set[NCAPINTS +NBUGINTS ]__aligned(sizeof(unsigned long)) ; -void load_percpu_segment (int cpu) +static noinstr void load_percpu_segment(int cpu) {#ifdef CONFIG_X86_32 loadsegment (fs, __KERNEL_PERCPU);#else __loadsegment_simple (gs,0 ); -wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu)); +/* +* Because of the __loadsegment_simple(gs, 0) above, any GS-prefixed +* instruction will explode right about here. As such, we must not have +* any CALL-thunks using per-cpu data. +* +* Therefore, use native_wrmsrl() and have XenPV take the fault and +* emulate. +*/ +native_wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu));#endif }
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 47 lines · Thomas Gleixner
load_percpu_segment()is usingwrmsr()which is paravirtualized. That's an issue because the code sequence is:__loadsegment_simple(gs, 0); wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu));So anything which uses a per CPU variable between setting GS to 0 and writing GSBASE is going to end up in a NULL pointer dereference. That's can be triggered with instrumentation and is guaranteed to be triggered with callthunks for call depth tracking.
Use
native_wrmsrl()instead.XEN_PVwill trap and emulate, but that's not a hot path.Also make it static and mark it noinstr so neither kprobes, sanitizers or whatever can touch it.
Signed-off-by Thomas Gleixner <[email protected]>--- arch/x86/include/asm/processor.h | 1 - arch/x86/kernel/cpu/common.c | 12 ++++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-)arch/x86/include/asm/processor.h +0
-1@@ -673,7 +673,6 @@ extern struct desc_ptr early_gdt_descr;extern void switch_to_new_gdt (int );extern void load_direct_gdt (int );extern void load_fixmap_gdt (int ); -extern void load_percpu_segment (int );extern void cpu_init (void );extern void cpu_init_secondary (void );extern void cpu_init_exception_handling (void );arch/x86/kernel/cpu/common.c +1
-2@@ -701,13 +701,21 @@ static const char *table_lookup_model(st__u32 cpu_caps_cleared[NCAPINTS +NBUGINTS ]__aligned(sizeof(unsigned long)) ;__u32 cpu_caps_set[NCAPINTS +NBUGINTS ]__aligned(sizeof(unsigned long)) ; -void load_percpu_segment (int cpu) +static noinstr void load_percpu_segment(int cpu) {#ifdef CONFIG_X86_32 loadsegment (fs, __KERNEL_PERCPU);#else __loadsegment_simple (gs,0 ); -wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu));
arch/x86/kernel/cpu/common.c static const char *table_lookup_model(st · line 719
Quoted 10 lines
arch/x86/kernel/cpu/common.c +9
-0+/* +* Because of the __loadsegment_simple(gs, 0) above, any GS-prefixed +* instruction will explode right about here. As such, we must not have +* any CALL-thunks using per-cpu data. +* +* Therefore, use native_wrmsrl() and have XenPV take the fault and +* emulate. +*/ +native_wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu));#endif
Lovely :-/
But I still don't see how that works, because __loadsegment_simple() is a memory clobber and cpu_kernelmode_gs_base() has a per-cpu lookup in it.
That said, this only has a sole caller, and in context, it's bogus for 64bit. Can't we fix all the problems by just doing this:
arch/x86/kernel/cpu/common.c +0 -0
@@ -701,16 +701,6 @@ static const char *table_lookup_model(struct
cpuinfo_x86 *c)
__u32 cpu_caps_cleared[NCAPINTS + NBUGINTS] __aligned(sizeof(unsigned long));
__u32 cpu_caps_set[NCAPINTS + NBUGINTS] __aligned(sizeof(unsigned long));
-void load_percpu_segment(int cpu)
-{
-#ifdef CONFIG_X86_32
- loadsegment(fs, __KERNEL_PERCPU);
-#else
- __loadsegment_simple(gs, 0);
- wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu)); -#endif
-}
-
#ifdef CONFIG_X86_32
/* The 32-bit entry code needs to find cpu_entry_area. */ DEFINE_PER_CPU(struct cpu_entry_area *, cpu_entry_area);
@@ -742,12 +732,15 @@ EXPORT_SYMBOL_GPL(load_fixmap_gdt);
* Current gdt points %fs at the "master" per-cpu area: after this,
* it's on the real one.
*/
-void switch_to_new_gdt(int cpu)
+void __noinstr switch_to_new_gdt(int cpu)
{
/* Load the original GDT */
load_direct_gdt(cpu);
+
+#ifdef CONFIG_X86_32
/* Reload the per-cpu base */
- load_percpu_segment(cpu);
+ loadsegment(fs, __KERNEL_PERCPU);
+#endif
}
static const struct cpu_dev *cpu_devs[X86_VENDOR_NUM] = {};
It's only 32bit where the percpu pointer is tied to the GDT. On 64bit, gsbase is good before this, and remains good after.
With this change,
# Make sure load_percpu_segment has no stackprotector CFLAGS_common.o := -fno-stack-protector
comes up for re-evaluation too.
~Andrew
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 3 lines · Andrew Cooper
It's only 32bit where the percpu pointer is tied to the GDT. On 64bit, gsbase is good before this, and remains good after.
That sounds sensible to me, but somebody should check that there's nothing that accidentally relied on the MSR_GS_BASE setting (or the segment selector clearing, for that matter).
Not that I can necessarily see how anything could work with it wrong, but..
Linus
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
In reply to Andrew CooperQuoted 21 lines · Andrew Cooper
Quoted 8 lines
(quoted diff) +1
-2-void load_percpu_segment(int cpu) +static noinstr void load_percpu_segment(int cpu) { #ifdef CONFIG_X86_32 loadsegment(fs, __KERNEL_PERCPU); #else __loadsegment_simple(gs, 0); - wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu));Review
Quoted 10 lines
(quoted diff) +9
-0+ /* + * Because of the __loadsegment_simple(gs, 0) above, any GS-prefixed + * instruction will explode right about here. As such, we must not have + * any CALL-thunks using per-cpu data. + * + * Therefore, use native_wrmsrl() and have XenPV take the fault and + * emulate. + */ + native_wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu)); #endifLovely :-/
Quoted 3 lines
But I still don't see how that works, because
__loadsegment_simple()is a memory clobber andcpu_kernelmode_gs_base()has a per-cpu lookup in it.
No. It uses an array lookup :)
Quoted 49 lines
That said, this only has a sole caller, and in context, it's bogus for 64bit. Can't we fix all the problems by just doing this:
arch/x86/kernel/cpu/common.c +0
-0@@ -701,16 +701,6 @@ static const char *table_lookup_model(struct cpuinfo_x86 *c)__u32 cpu_caps_cleared[NCAPINTS + NBUGINTS]
__aligned(sizeof(unsigned long));
__u32 cpu_caps_set[NCAPINTS + NBUGINTS]__aligned(sizeof(unsigned long));arch/x86/kernel/cpu/common.c +0
-10-void load_percpu_segment (int cpu) -{ -#ifdef CONFIG_X86_32 - loadsegment(fs, __KERNEL_PERCPU); -#else - __loadsegment_simple(gs, 0); - wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu)); -#endif -} -#ifdef
CONFIG_X86_32
/* The 32-bit entry code needs to findcpu_entry_area. */DEFINE_PER_CPU(struct cpu_entry_area *, cpu_entry_area);arch/x86/kernel/cpu/common.c +5
-2@@ -742,12 +732,15 @@ EXPORT_SYMBOL_GPL(load_fixmap_gdt);* Current gdt points %fs at the "master" per-cpu area: after this, * it's on the real one. */ -void switch_to_new_gdt (int cpu) +void __noinstr switch_to_new_gdt(int cpu) { /* Load the original GDT */ load_direct_gdt(cpu); + +#ifdef CONFIG_X86_32 /* Reload the per-cpu base */ - load_percpu_segment(cpu); + loadsegment(fs, __KERNEL_PERCPU); +#endif }
static const structcpu_dev*cpu_devs[X86_VENDOR_NUM] = {};It's only 32bit where the percpu pointer is tied to the GDT. On 64bit, gsbase is good before this, and remains good after.
Quoted 6 lines
With this change,
# Make sure
load_percpu_segmenthas no stackprotector CFLAGS_common.o := -fno-stack-protectorcomes up for re-evaluation too.
Good point. Let me stare at it some more.
Thanks,
tglx
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 27 lines · Thomas Gleixner
Quoted 26 lines · Andrew Cooper
#ifdef
CONFIG_X86_32
/* The 32-bit entry code needs to findcpu_entry_area. */DEFINE_PER_CPU(struct cpu_entry_area *, cpu_entry_area);(quoted diff) +5
-2@@ -742,12 +732,15 @@ EXPORT_SYMBOL_GPL(load_fixmap_gdt); * Current gdt points %fs at the "master" per-cpu area: after this, * it's on the real one. */ -void switch_to_new_gdt(int cpu) +void __noinstr switch_to_new_gdt(int cpu) { /* Load the original GDT */ load_direct_gdt(cpu); + +#ifdef CONFIG_X86_32 /* Reload the per-cpu base */ - load_percpu_segment(cpu); + loadsegment(fs, __KERNEL_PERCPU); +#endif }
static const structcpu_dev*cpu_devs[X86_VENDOR_NUM] = {};It's only 32bit where the percpu pointer is tied to the GDT. On 64bit, gsbase is good before this, and remains good after.
Quoted 8 lines
Quoted 6 lines
With this change,
# Make sure
load_percpu_segmenthas no stackprotector CFLAGS_common.o := -fno-stack-protectorcomes up for re-evaluation too.
Good point. Let me stare at it some more.
If it only would be that simple :)
loadsegment_simple() was a red herring. The gs segment is already zero.
So what explodes here is the early boot when switching from early per CPU to the real per CPU area.
start_kernel()
.....
setup_per_cpu_areas();
smp_prepare_boot_cpu()
switch_to_new_gdt()
load_direct_gdt(cpu);
load_percpu_segment(cpu)
GS: 0
GS_BASE: 0xffffffff829d0000 (early PERCPU)
wrmsrl()
GS_BASE: 0xffff888237c00000 (real PERCPU)So the explosion happens when accessing a per CPU variable after loading the GDT and before GS_BASE is fixed up.
That's the only case AFAICT where this matters. In all other invocations GS_BASE is already correct.
Let me fix this proper.
Thanks,
tglx
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 10 lines · Thomas Gleixner
On Sun, Jul 17 2022 at 21:08, Thomas Gleixner wrote:
loadsegment_simple()was a red herring. The gs segment is already zero.So what explodes here is the early boot when switching from early per CPU to the real per CPU area.
start_kernel()..... setup_per_cpu_areas(); smp_prepare_boot_cpu()
Bah. switch_to_new_gdt() is already invoked from setup_per_cpu_areas() and then again in smp_prepare_boot_cpu() and once more in cpu_init(),
What a mess.
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 12 lines · Thomas Gleixner
Quoted 10 lines · Thomas Gleixner
On Sun, Jul 17 2022 at 21:08, Thomas Gleixner wrote:
loadsegment_simple()was a red herring. The gs segment is already zero.So what explodes here is the early boot when switching from early per CPU to the real per CPU area.
start_kernel()..... setup_per_cpu_areas(); smp_prepare_boot_cpu()
Quoted 4 lines
Bah.
switch_to_new_gdt()is already invoked fromsetup_per_cpu_areas()and then again insmp_prepare_boot_cpu()and once more incpu_init(),What a mess.
So the below builds and boots at least on 64bit. I'll stare at it some more tomorrow. I have no idea whether native_load_gdt() works with XEN_PV. It should, but what do I know.
Thanks,
tglx
---
arch/x86/include/asm/desc.h +1 -1
@@ -205,7 +205,7 @@ static inline void native_set_ldt(const } } -static inline void native_load_gdt (const struct desc_ptr *dtr) +static __always_inline void native_load_gdt(const struct desc_ptr *dtr) { asmvolatile ("lgdt %0" ::"m" (*dtr)); }
arch/x86/include/asm/processor.h +1 -2
@@ -670,10 +670,9 @@ extern int sysenter_setup(void);/* Defined in head.S */ extern struct desc_ptr early_gdt_descr; -extern void switch_to_new_gdt (int ); +extern void switch_to_real_gdt (int );extern void load_direct_gdt (int );extern void load_fixmap_gdt (int ); -extern void load_percpu_segment (int );extern void cpu_init (void );extern void cpu_init_secondary (void );extern void cpu_init_exception_handling (void );
arch/x86/kernel/cpu/Makefile +7 -3
@@ -7,20 +7,24 @@ ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_common.o = -pg CFLAGS_REMOVE_perf_event.o = -pg +CFLAGS_REMOVE_switch_gdt.o = -pg endif # If these files are instrumented, boot hangs during the first second. KCOV_INSTRUMENT_common.o := n KCOV_INSTRUMENT_perf_event.o := n +KCOV_INSTRUMENT_switch_gdt.o := n # As above, instrumenting secondary CPU boot code causes boot hangs. KCSAN_SANITIZE_common.o := n +KCSAN_SANITIZE_switch_gdt.o := n -# Make sure load_percpu_segment has no stackprotector -CFLAGS_common.o := -fno-stack-protector +# Make sure that switching the GDT and the per CPU segment +# does not have stack protector enabled. +CFLAGS_switch_gdt.o := -fno-stack-protector obj-y := cacheinfo.o scattered.o topology.o -obj-y += common.o +obj-y += common.o switch_gdt.o obj-y += rdrand.o obj-y += match.o obj-y += bugs.o
arch/x86/kernel/cpu/common.c +0 -28
@@ -701,16 +701,6 @@ static const char *table_lookup_model(st__u32 cpu_caps_cleared[NCAPINTS +NBUGINTS ]__aligned(sizeof(unsigned long)) ;__u32 cpu_caps_set[NCAPINTS +NBUGINTS ]__aligned(sizeof(unsigned long)) ; -void load_percpu_segment (int cpu) -{ -#ifdef CONFIG_X86_32 -loadsegment (fs, __KERNEL_PERCPU); -#else -__loadsegment_simple (gs,0 ); -wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu)); -#endif -} -#ifdef CONFIG_X86_32 /* The 32-bit entry code needs to find cpu_entry_area. */ DEFINE_PER_CPU(struct cpu_entry_area *, cpu_entry_area);
@@ -738,18 +728,6 @@ void load_fixmap_gdt(int cpu) }EXPORT_SYMBOL_GPL (load_fixmap_gdt); -/* -* Current gdt points %fs at the "master" per-cpu area: after this, -* it's on the real one. -*/ -void switch_to_new_gdt (int cpu) -{ -/* Load the original GDT */ -load_direct_gdt (cpu); -/* Reload the per-cpu base */ -load_percpu_segment (cpu); -} -static const struct cpu_dev *cpu_devs[X86_VENDOR_NUM ] = {};static void get_model_name (struct cpuinfo_x86 *c)
@@ -2228,12 +2206,6 @@ void cpu_init(void) boot_cpu_has(X86_FEATURE_TSC) || boot_cpu_has(X86_FEATURE_DE))cr4_clear_bits (X86_CR4_VME |X86_CR4_PVI |X86_CR4_TSD |X86_CR4_DE ); -/* -* Initialize the per-CPU GDT with the boot GDT, -* and set up the GDT descriptor: -*/ -switch_to_new_gdt (cpu); -if (IS_ENABLED (CONFIG_X86_64 )) {loadsegment (fs,0 );memset (cur->thread.tls_array,0 ,GDT_ENTRY_TLS_ENTRIES *8 );
arch/x86/kernel/cpu/switch_gdt.c new file +31 -0
@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <asm/processor.h> +#include <asm/segment.h> +#include <asm/desc.h> + +/* +* Invoked during early boot to switch from early GDT and early per CPU +* (%fs on 32bit, GS_BASE on 64bit) to the real GDT and the runtime per CPU +* area. +* +* This has to be done atomic because after switching from early GDT to +* the real one any per cpu variable access is going to fault because +* %fs resp. GS_BASE is not yet pointing to the real per CPU data. +* +* As a consequence this uses the native variants of load_gdt() and +* wrmsrl(). So XEN_PV has to take the fault and emulate. +*/ +void __init switch_to_real_gdt (int cpu) +{ +struct desc_ptr gdt_descr; + + gdt_descr.address = (long )get_cpu_gdt_rw (cpu); + gdt_descr.size =GDT_SIZE -1 ; +native_load_gdt (&gdt_descr); + +#ifdef CONFIG_X86_32 +loadsegment (fs, __KERNEL_PERCPU); +#else +native_wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu)); +#endif +}
arch/x86/kernel/setup_percpu.c +1 -1
@@ -211,7 +211,7 @@ void __init setup_per_cpu_areas(void)* area. Reload any changed state for the boot CPU. */ if (!cpu) -switch_to_new_gdt (cpu); +switch_to_real_gdt (cpu); }/* indicate the early static arrays will soon be gone */
arch/x86/kernel/smpboot.c +5 -1
@@ -1457,7 +1457,11 @@ void arch_thaw_secondary_cpus_end(void)void __init native_smp_prepare_boot_cpu (void ) {int me =smp_processor_id (); -switch_to_new_gdt (me); + +/* SMP invokes this from setup_per_cpu_areas() */ +if (!IS_ENABLED (CONFIG_SMP )) +switch_to_real_gdt (me); +/* already set me in cpu_online_mask in boot_cpu_init() */ cpumask_set_cpu (me, cpu_callout_mask);cpu_set_state_online (me);
arch/x86/xen/enlighten_pv.c +1 -1
@@ -1164,7 +1164,7 @@ static void __init xen_setup_gdt(int cpu pv_ops.cpu.write_gdt_entry = xen_write_gdt_entry_boot; pv_ops.cpu.load_gdt = xen_load_gdt_boot; -switch_to_new_gdt (cpu); +switch_to_real_gdt (cpu); pv_ops.cpu.write_gdt_entry = xen_write_gdt_entry; pv_ops.cpu.load_gdt = xen_load_gdt;
Juergen Gross
maintainerRe: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 13 lines · Thomas Gleixner
Quoted 12 lines · Thomas Gleixner
Quoted 10 lines · Thomas Gleixner
On Sun, Jul 17 2022 at 21:08, Thomas Gleixner wrote:
loadsegment_simple()was a red herring. The gs segment is already zero.So what explodes here is the early boot when switching from early per CPU to the real per CPU area.
start_kernel()..... setup_per_cpu_areas(); smp_prepare_boot_cpu()
Quoted 8 lines
Quoted 4 lines
Bah.
switch_to_new_gdt()is already invoked fromsetup_per_cpu_areas()and then again insmp_prepare_boot_cpu()and once more incpu_init(),What a mess.
So the below builds and boots at least on 64bit. I'll stare at it some more tomorrow. I have no idea whether
native_load_gdt()works withXEN_PV. It should, but what do I know.
No, shouldn't work. But ...
Quoted 174 lines
Thanks,
tglx
---arch/x86/include/asm/desc.h +1
-1@@ -205,7 +205,7 @@ static inline void native_set_ldt(const } } -static inline void native_load_gdt (const struct desc_ptr *dtr) +static __always_inline void native_load_gdt(const struct desc_ptr *dtr) { asmvolatile ("lgdt %0" ::"m" (*dtr)); }arch/x86/include/asm/processor.h +1
-2@@ -670,10 +670,9 @@ extern int sysenter_setup(void);/* Defined in head.S */ extern struct desc_ptr early_gdt_descr; -extern void switch_to_new_gdt (int ); +extern void switch_to_real_gdt (int );extern void load_direct_gdt (int );extern void load_fixmap_gdt (int ); -extern void load_percpu_segment (int );extern void cpu_init (void );extern void cpu_init_secondary (void );extern void cpu_init_exception_handling (void );arch/x86/kernel/cpu/Makefile +7
-3@@ -7,20 +7,24 @@ ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_common.o = -pg CFLAGS_REMOVE_perf_event.o = -pg +CFLAGS_REMOVE_switch_gdt.o = -pg endif # If these files are instrumented, boot hangs during the first second. KCOV_INSTRUMENT_common.o := n KCOV_INSTRUMENT_perf_event.o := n +KCOV_INSTRUMENT_switch_gdt.o := n # As above, instrumenting secondary CPU boot code causes boot hangs. KCSAN_SANITIZE_common.o := n +KCSAN_SANITIZE_switch_gdt.o := n -# Make sure load_percpu_segment has no stackprotector -CFLAGS_common.o := -fno-stack-protector +# Make sure that switching the GDT and the per CPU segment +# does not have stack protector enabled. +CFLAGS_switch_gdt.o := -fno-stack-protector obj-y := cacheinfo.o scattered.o topology.o -obj-y += common.o +obj-y += common.o switch_gdt.o obj-y += rdrand.o obj-y += match.o obj-y += bugs.oarch/x86/kernel/cpu/common.c +0
-28@@ -701,16 +701,6 @@ static const char *table_lookup_model(st__u32 cpu_caps_cleared[NCAPINTS +NBUGINTS ]__aligned(sizeof(unsigned long)) ;__u32 cpu_caps_set[NCAPINTS +NBUGINTS ]__aligned(sizeof(unsigned long)) ; -void load_percpu_segment (int cpu) -{ -#ifdef CONFIG_X86_32 -loadsegment (fs, __KERNEL_PERCPU); -#else -__loadsegment_simple (gs,0 ); -wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu)); -#endif -} -#ifdef CONFIG_X86_32 /* The 32-bit entry code needs to find cpu_entry_area. */ DEFINE_PER_CPU(struct cpu_entry_area *, cpu_entry_area);@@ -738,18 +728,6 @@ void load_fixmap_gdt(int cpu) }EXPORT_SYMBOL_GPL (load_fixmap_gdt); -/* -* Current gdt points %fs at the "master" per-cpu area: after this, -* it's on the real one. -*/ -void switch_to_new_gdt (int cpu) -{ -/* Load the original GDT */ -load_direct_gdt (cpu); -/* Reload the per-cpu base */ -load_percpu_segment (cpu); -} -static const struct cpu_dev *cpu_devs[X86_VENDOR_NUM ] = {};static void get_model_name (struct cpuinfo_x86 *c)@@ -2228,12 +2206,6 @@ void cpu_init(void) boot_cpu_has(X86_FEATURE_TSC) || boot_cpu_has(X86_FEATURE_DE))cr4_clear_bits (X86_CR4_VME |X86_CR4_PVI |X86_CR4_TSD |X86_CR4_DE ); -/* -* Initialize the per-CPU GDT with the boot GDT, -* and set up the GDT descriptor: -*/ -switch_to_new_gdt (cpu); -if (IS_ENABLED (CONFIG_X86_64 )) {loadsegment (fs,0 );memset (cur->thread.tls_array,0 ,GDT_ENTRY_TLS_ENTRIES *8 );arch/x86/kernel/cpu/switch_gdt.c new file +31
-0@@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <asm/processor.h> +#include <asm/segment.h> +#include <asm/desc.h> + +/* +* Invoked during early boot to switch from early GDT and early per CPU +* (%fs on 32bit, GS_BASE on 64bit) to the real GDT and the runtime per CPU +* area. +* +* This has to be done atomic because after switching from early GDT to +* the real one any per cpu variable access is going to fault because +* %fs resp. GS_BASE is not yet pointing to the real per CPU data. +* +* As a consequence this uses the native variants of load_gdt() and +* wrmsrl(). So XEN_PV has to take the fault and emulate. +*/ +void __init switch_to_real_gdt (int cpu) +{ +struct desc_ptr gdt_descr; + + gdt_descr.address = (long )get_cpu_gdt_rw (cpu); + gdt_descr.size =GDT_SIZE -1 ; +native_load_gdt (&gdt_descr); + +#ifdef CONFIG_X86_32 +loadsegment (fs, __KERNEL_PERCPU); +#else +native_wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu)); +#endif +}arch/x86/kernel/setup_percpu.c +1
-1@@ -211,7 +211,7 @@ void __init setup_per_cpu_areas(void)* area. Reload any changed state for the boot CPU. */ if (!cpu) -switch_to_new_gdt (cpu); +switch_to_real_gdt (cpu); }/* indicate the early static arrays will soon be gone */ arch/x86/kernel/smpboot.c +5
-1@@ -1457,7 +1457,11 @@ void arch_thaw_secondary_cpus_end(void)void __init native_smp_prepare_boot_cpu (void ) {int me =smp_processor_id (); -switch_to_new_gdt (me); + +/* SMP invokes this from setup_per_cpu_areas() */ +if (!IS_ENABLED (CONFIG_SMP )) +switch_to_real_gdt (me); +/* already set me in cpu_online_mask in boot_cpu_init() */ cpumask_set_cpu (me, cpu_callout_mask);cpu_set_state_online (me);
arch/x86/xen/enlighten_pv.c static void __init xen_setup_gdt(int cpu · line 1167
Quoted 5 lines
arch/x86/xen/enlighten_pv.c +1
-1@@ -1164,7 +1164,7 @@ static void __init xen_setup_gdt(int cpu pv_ops.cpu.write_gdt_entry = xen_write_gdt_entry_boot; pv_ops.cpu.load_gdt = xen_load_gdt_boot; -switch_to_new_gdt (cpu); +switch_to_real_gdt (cpu);
... can't you use the paravirt variant of load_gdt in switch_to_real_gdt() ?
arch/x86/xen/enlighten_pv.c static void __init xen_setup_gdt(int cpu
Quoted 4 lines
arch/x86/xen/enlighten_pv.c +0
-0pv_ops.cpu.write_gdt_entry = xen_write_gdt_entry; pv_ops.cpu.load_gdt = xen_load_gdt;
Juergen
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 19 lines · Juergen Gross
Quoted 18 lines · Thomas Gleixner
void __init native_smp_prepare_boot_cpu(void) { int me = smp_processor_id();arch/x86/kernel/smpboot.c +5
-1-switch_to_new_gdt (me); + +/* SMP invokes this from setup_per_cpu_areas() */ +if (!IS_ENABLED (CONFIG_SMP )) +switch_to_real_gdt (me); +/* already set me in cpu_online_mask in boot_cpu_init() */ cpumask_set_cpu (me, cpu_callout_mask);cpu_set_state_online (me);arch/x86/xen/enlighten_pv.c +0
-0@@ -1164,7 +1164,7 @@ static void __init xen_setup_gdt(int cpu pv_ops.cpu.write_gdt_entry = xen_write_gdt_entry_boot; pv_ops.cpu.load_gdt = xen_load_gdt_boot;
Quoted 4 lines
arch/x86/xen/enlighten_pv.c static void __init xen_setup_gdt(int cpu · line 1167
Quoted 2 lines
arch/x86/xen/enlighten_pv.c +1
-1-switch_to_new_gdt (cpu); +switch_to_real_gdt (cpu);... can't you use the paravirt variant of
load_gdtinswitch_to_real_gdt()?
That does not solve the problem of having a disagreement between GDT and GS_BASE. Let me dig into this some more.
Thanks,
tglx
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 8 lines · Thomas Gleixner
Quoted 4 lines · Juergen Gross
Review
Quoted 2 lines
(quoted diff) +1
-1- switch_to_new_gdt(cpu); + switch_to_real_gdt(cpu);... can't you use the paravirt variant of
load_gdtinswitch_to_real_gdt()?That does not solve the problem of having a disagreement between GDT and
GS_BASE. Let me dig into this some more.
Bah. The real problem is __loadsegment_simple(gs, 0). After that GS_BASE is 0. So any per CPU access before setting MSR_GS_BASE back to working state is going into lala land.
So it's not the GDT. It's the mov 0, %gs which makes stuff go south, but as %gs is already 0, we can keep the paravirt load_gdt() and use native_write_msr() and everything should be happy.
Thanks,
tglx
Peter Zijlstra
maintainerRe: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 10 lines · Thomas Gleixner
Quoted 8 lines · Thomas Gleixner
Quoted 4 lines · Juergen Gross
Review
Quoted 2 lines
(quoted diff) +1
-1- switch_to_new_gdt(cpu); + switch_to_real_gdt(cpu);... can't you use the paravirt variant of
load_gdtinswitch_to_real_gdt()?That does not solve the problem of having a disagreement between GDT and
GS_BASE. Let me dig into this some more.
Quoted 7 lines
Bah. The real problem is
__loadsegment_simple(gs, 0). After thatGS_BASEis 0. So any per CPU access before settingMSR_GS_BASEback to working state is going into lala land.So it's not the GDT. It's the mov 0, %gs which makes stuff go south, but as %gs is already 0, we can keep the paravirt
load_gdt()and usenative_write_msr()and everything should be happy.
How is the ret from xen_load_gdt() not going to explode?
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Thomas Gleixner
Quoted 8 lines · Thomas Gleixner
Quoted 4 lines · Juergen Gross
Review
Quoted 2 lines
(quoted diff) +1
-1- switch_to_new_gdt(cpu); + switch_to_real_gdt(cpu);... can't you use the paravirt variant of
load_gdtinswitch_to_real_gdt()?That does not solve the problem of having a disagreement between GDT and
GS_BASE. Let me dig into this some more.Bah. The real problem is
__loadsegment_simple(gs, 0). After thatGS_BASEis 0. So any per CPU access before settingMSR_GS_BASEback to working state is going into lala land.
Quoted 5 lines
Quoted 3 lines
So it's not the GDT. It's the mov 0, %gs which makes stuff go south, but as %gs is already 0, we can keep the paravirt
load_gdt()and usenative_write_msr()and everything should be happy.How is the ret from
xen_load_gdt()not going to explode?
This is only for the early boot _before_ all the patching happens. So that goes through the default retthunk.
Secondary CPUs do not need that as they set up GDT and GS_BASE in the low level asm code before coming out to C.
I'm still trying to figure out how this works on XENPV and on 32bit.
Sigh...
Re: [patch 02/38] x86/cpu: Use native_wrmsrl() in load_percpu_segment()
Quoted 22 lines · Thomas Gleixner
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Thomas Gleixner
Quoted 8 lines · Thomas Gleixner
Quoted 4 lines · Juergen Gross
Review
Quoted 2 lines
(quoted diff) +1
-1- switch_to_new_gdt(cpu); + switch_to_real_gdt(cpu);... can't you use the paravirt variant of
load_gdtinswitch_to_real_gdt()?That does not solve the problem of having a disagreement between GDT and
GS_BASE. Let me dig into this some more.Bah. The real problem is
__loadsegment_simple(gs, 0). After thatGS_BASEis 0. So any per CPU access before settingMSR_GS_BASEback to working state is going into lala land.Quoted 5 lines
Quoted 3 lines
So it's not the GDT. It's the mov 0, %gs which makes stuff go south, but as %gs is already 0, we can keep the paravirt
load_gdt()and usenative_write_msr()and everything should be happy.How is the ret from
xen_load_gdt()not going to explode?
Quoted 7 lines
This is only for the early boot _before_ all the patching happens. So that goes through the default retthunk.
Secondary CPUs do not need that as they set up GDT and
GS_BASEin the low level asm code before coming out to C.I'm still trying to figure out how this works on XENPV and on 32bit.
On 32bit the CPU comes out with GDT and FS correctly set too.
For XEN_PV it looks like cpu_initialize_context() hands down GDT and GSBASE info to the hypercall which kicks the CPU so we should be good there as well. Emphasis on should. Jürgen?
Thanks,
tglx
cover x86/cpu: Sanitize switch_to_new_gdt()
In reply to Thomas GleixnerThis series is a replacement for patch 2/38 of the call depth stuffing series as a follow up to the review feedback.
Thanks,
tglx
PATCH 1/3 x86/cpu: Remove segment load from switch_to_new_gdt()
In reply to Thomas GleixnerOn 32bit FS and on 64bit GS segments are already set up correctly, but load_percpu_segment() still sets [FG]S after switching from the early GDT to the direct GDT.
For 32bit the segment load has no side effects, but on 64bit it causes GSBASE to become 0, which means that any per CPU access before GSBASE is set to the new value is going to fault. That's the reason why the whole file containing this code has stackprotector removed.
But that's a pointless exercise for both 32 and 64 bit as the relevant segment selector is already correct. Loading the new GDT does not change that.
Remove the segment loads and inline load_percpu_segment() into the only caller. Add comments while at it.
--- arch/x86/include/asm/processor.h | 1 arch/x86/kernel/cpu/common.c | 42 ++++++++++++++++++++++++--------------- 2 files changed, 26 insertions(+), 17 deletions(-)
arch/x86/include/asm/processor.h +0 -1
@@ -673,7 +673,6 @@ extern struct desc_ptr early_gdt_descr;extern void switch_to_new_gdt (int );extern void load_direct_gdt (int );extern void load_fixmap_gdt (int ); -extern void load_percpu_segment (int );extern void cpu_init (void );extern void cpu_init_secondary (void );extern void cpu_init_exception_handling (void );
arch/x86/kernel/cpu/common.c +26 -16
58 lines not on this page · raw message
Re: [patch 1/3] x86/cpu: Remove segment load from switch_to_new_gdt()
So I appreciate the added big comments in this code, but looking at this patch:
Quoted 25 lines · Thomas Gleixner
arch/x86/kernel/cpu/common.c +17
-3+/** +* switch_to_new_gdt - Switch form early GDT to the direct one +* @cpu: The CPU number for which this is invoked +* +* Invoked during early boot to switch from early GDT and early per CPU +* (%fs on 32bit, GS_BASE on 64bit) to the direct GDT and the runtime per +* CPU area. */ void switch_to_new_gdt (int cpu) { -/* Load the original GDT */ load_direct_gdt (cpu); -/* Reload the per-cpu base */ -load_percpu_segment (cpu); + +/* +* No need to load the %gs (%fs for 32bit) segment. It is already +* correct, %gs is 0 on 64bit and %fs is __KERNEL_PERCPU on 32 bit. +* +* Writing %gs on 64bit would zero GSBASE which would make any per +* CPU operation up to the point of the wrmsrl() fault. +* +* 64bit requires to point GSBASE to the new offset. Until the +* wrmsrl() happens the early mapping is still valid. That means
arch/x86/kernel/cpu/common.c void load_fixmap_gdt(int cpu) · line 761
Quoted 10 lines
arch/x86/kernel/cpu/common.c +9
-0+* the GSBASE update will lose any prior per CPU data which was +* not copied over in setup_per_cpu_areas(). +* +* For secondary CPUs this is not a problem because they start +* already with the direct GDT and the real GSBASE. This invocation +* is pointless and will be removed in a subsequent step. +*/ +if (IS_ENABLED (CONFIG_X86_64 )) +wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu)); }
... while those comments are nice and all, I do think this retains the basic insanity of having "switch_to_new_gdt()" do magical things on x86-64 that don't really match the name.
So honestly, I'd be happier of that whole
if (IS_ENABLED(CONFIG_X86_64))
wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu));was migrated to the callers instead. There aren't *that* many callers.
I expect that it is then quite possible that several of the call-sites would go "GS_BASE is already correct here, I can remove this".
But even if every single caller keeps that wrmsrl() around, at least it wouldn't be hidden behind a function call that has a name that implies something completely different is happening.
And no, I don't care *that* deeply, so this is just a suggestion.
But wouldn't it be nice if this function was actually named by what it does, rather than by what it used to do back in the i386 days when the GDT affected the segment bases?
Linus
Re: [patch 1/3] x86/cpu: Remove segment load from switch_to_new_gdt()
Quoted 16 lines · Linus Torvalds
So I appreciate the added big comments in this code, but looking at this patch:
arch/x86/kernel/cpu/common.c void load_fixmap_gdt(int cpu) · line 761
Quoted 7 lines · Thomas Gleixner
arch/x86/kernel/cpu/common.c +6
-0+* For secondary CPUs this is not a problem because they start +* already with the direct GDT and the real GSBASE. This invocation +* is pointless and will be removed in a subsequent step. +*/ +if (IS_ENABLED (CONFIG_X86_64 )) +wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu)); }... while those comments are nice and all, I do think this retains the basic insanity of having "
switch_to_new_gdt()" do magical things on x86-64 that don't really match the name.So honestly, I'd be happier of that whole
Quoted 7 lines
if (IS_ENABLED(CONFIG_X86_64)) wrmsrl(MSR_GS_BASE, cpu_kernelmode_gs_base(cpu));was migrated to the callers instead. There aren't *that* many callers.
I expect that it is then quite possible that several of the call-sites would go "
GS_BASEis already correct here, I can remove this".
With the next patch we have only two left. The SMP and the UP case. Let me look whether the UP needs it at all.
Quoted 9 lines
But even if every single caller keeps that
wrmsrl()around, at least it wouldn't be hidden behind a function call that has a name that implies something completely different is happening.And no, I don't care *that* deeply, so this is just a suggestion.
But wouldn't it be nice if this function was actually named by what it does, rather than by what it used to do back in the i386 days when the GDT affected the segment bases?
Yes. Let me come up with a sensible name.
Thanks,
tglx
PATCH 2/3 x86/cpu: Get rid of redundant switch_to_new_gdt() invocations
In reply to Thomas GleixnerThe only place where switch_to_new_gdt() is required is early boot to switch from the early GDT to the direct GDT. Any other invocation is completely redundant because it does not change anything.
Secondary CPUs come out of the ASM code with GDT and GSBASE correctly set up. The same is true for XEN_PV.
Remove all the voodoo invocations which are left overs from the ancient past, rename the function to switch_to_direct_gdt() and mark it init.
--- arch/x86/include/asm/processor.h | 2 +- arch/x86/kernel/cpu/common.c | 14 ++------------ arch/x86/kernel/setup_percpu.c | 2 +- arch/x86/kernel/smpboot.c | 6 +++++- arch/x86/xen/enlighten_pv.c | 2 +- 5 files changed, 10 insertions(+), 16 deletions(-)
arch/x86/include/asm/processor.h +1 -1
@@ -670,7 +670,7 @@ extern int sysenter_setup(void);/* Defined in head.S */ extern struct desc_ptr early_gdt_descr; -extern void switch_to_new_gdt (int ); +extern void switch_to_direct_gdt (int );extern void load_direct_gdt (int );extern void load_fixmap_gdt (int );extern void cpu_init (void );
arch/x86/kernel/cpu/common.c +2 -12
@@ -729,14 +729,14 @@ void load_fixmap_gdt(int cpu)EXPORT_SYMBOL_GPL (load_fixmap_gdt);/** -* switch_to_new_gdt - Switch form early GDT to the direct one +* switch_to_direct_gdt - Switch form early GDT to the direct one * @cpu: The CPU number for which this is invoked * * Invoked during early boot to switch from early GDT and early per CPU * (%fs on 32bit, GS_BASE on 64bit) to the direct GDT and the runtime per * CPU area. */ -void switch_to_new_gdt (int cpu) +void __init switch_to_direct_gdt (int cpu) {load_direct_gdt (cpu);
@@ -751,10 +751,6 @@ void switch_to_new_gdt(int cpu)* wrmsrl() happens the early mapping is still valid. That means * the GSBASE update will lose any prior per CPU data which was * not copied over in setup_per_cpu_areas(). -* -* For secondary CPUs this is not a problem because they start -* already with the direct GDT and the real GSBASE. This invocation -* is pointless and will be removed in a subsequent step. */ if (IS_ENABLED (CONFIG_X86_64 ))wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu));
@@ -2238,12 +2234,6 @@ void cpu_init(void) boot_cpu_has(X86_FEATURE_TSC) || boot_cpu_has(X86_FEATURE_DE))cr4_clear_bits (X86_CR4_VME |X86_CR4_PVI |X86_CR4_TSD |X86_CR4_DE ); -/* -* Initialize the per-CPU GDT with the boot GDT, -* and set up the GDT descriptor: -*/ -switch_to_new_gdt (cpu); -if (IS_ENABLED (CONFIG_X86_64 )) {loadsegment (fs,0 );memset (cur->thread.tls_array,0 ,GDT_ENTRY_TLS_ENTRIES *8 );
arch/x86/kernel/setup_percpu.c +1 -1
@@ -211,7 +211,7 @@ void __init setup_per_cpu_areas(void)* area. Reload any changed state for the boot CPU. */ if (!cpu) -switch_to_new_gdt (cpu); +switch_to_direct_gdt (cpu); }/* indicate the early static arrays will soon be gone */
arch/x86/kernel/smpboot.c +5 -1
@@ -1457,7 +1457,11 @@ void arch_thaw_secondary_cpus_end(void)void __init native_smp_prepare_boot_cpu (void ) {int me =smp_processor_id (); -switch_to_new_gdt (me); + +/* SMP invokes this from setup_per_cpu_areas() */ +if (!IS_ENABLED (CONFIG_SMP )) +switch_to_direct_gdt (me); +/* already set me in cpu_online_mask in boot_cpu_init() */ cpumask_set_cpu (me, cpu_callout_mask);cpu_set_state_online (me);
arch/x86/xen/enlighten_pv.c +1 -1
@@ -1164,7 +1164,7 @@ static void __init xen_setup_gdt(int cpu pv_ops.cpu.write_gdt_entry = xen_write_gdt_entry_boot; pv_ops.cpu.load_gdt = xen_load_gdt_boot; -switch_to_new_gdt (cpu); +switch_to_direct_gdt (cpu); pv_ops.cpu.write_gdt_entry = xen_write_gdt_entry; pv_ops.cpu.load_gdt = xen_load_gdt;
PATCH 3/3 x86/cpu: Re-enable stackprotector
In reply to Thomas GleixnerCommit 5416c2663517 ("x86: make sure load_percpu_segment has no stackprotector") disabled the stackprotector for cpu/common.c because of load_percpu_segment(). Back then the boot stack canary was initialized very early in start_kernel(). Switching the per CPU area by loading the GDT caused the stackprotector to fail with paravirt enabled kernels as the GSBASE was not updated yet. In hindsight a wrong change because it would have been sufficient to ensure that the canary is the same in both per CPU areas.
Commit d55535232c3d ("random: move rand_initialize() earlier") moved the stack canary initialization to a later point in the init sequence. As a consequence the per CPU stack canary is 0 when switching the per CPU areas, so there is no requirement anymore to exclude this file.
Add a comment to load_percpu_segment().
--- arch/x86/kernel/cpu/Makefile | 3 --- arch/x86/kernel/cpu/common.c | 3 +++ 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/Makefile +0 -3
@@ -16,9 +16,6 @@ KCOV_INSTRUMENT_perf_event.o := n # As above, instrumenting secondary CPU boot code causes boot hangs. KCSAN_SANITIZE_common.o := n -# Make sure load_percpu_segment has no stackprotector -CFLAGS_common.o := -fno-stack-protector - obj-y := cacheinfo.o scattered.o topology.o obj-y += common.o obj-y += rdrand.o
arch/x86/kernel/cpu/common.c +3 -0
@@ -751,6 +751,9 @@ void __init switch_to_direct_gdt(int cpu* wrmsrl() happens the early mapping is still valid. That means * the GSBASE update will lose any prior per CPU data which was * not copied over in setup_per_cpu_areas(). +* +* This works even with stackprotector enabled because the +* per CPU stack canary is 0 in both per CPU areas. */ if (IS_ENABLED (CONFIG_X86_64 ))wrmsrl (MSR_GS_BASE ,cpu_kernelmode_gs_base (cpu));
PATCH 3/38 x86/modules: Set VM_FLUSH_RESET_PERMS in module_alloc()
In reply to Thomas GleixnerInstead of resetting permissions all over the place when freeing module memory tell the vmalloc code to do so. Avoids the exercise for the next upcoming user.
--- arch/x86/kernel/ftrace.c | 2 -- arch/x86/kernel/kprobes/core.c | 1 - arch/x86/kernel/module.c | 9 +++++---- 3 files changed, 5 insertions(+), 7 deletions(-)
arch/x86/kernel/ftrace.c +0 -2
@@ -412,8 +412,6 @@ create_trampoline(struct ftrace_ops *ops/* ALLOC_TRAMP flags lets us know we created it */ ops->flags |=FTRACE_OPS_FL_ALLOC_TRAMP ; -set_vm_flush_reset_perms (trampoline); -if (likely (system_state !=SYSTEM_BOOTING ))set_memory_ro ((unsigned long )trampoline, npages);set_memory_x ((unsigned long )trampoline, npages);
arch/x86/kernel/kprobes/core.c +0 -1
@@ -416,7 +416,6 @@ void *alloc_insn_page(void)if (!page)return NULL ; -set_vm_flush_reset_perms (page);/* * First make the page read-only, and only then make it executable to * prevent it from being W+X in between.
arch/x86/kernel/module.c +5 -4
@@ -74,10 +74,11 @@ void *module_alloc(unsigned long size)return NULL ; p =__vmalloc_node_range (size,MODULE_ALIGN , -MODULES_VADDR +get_module_load_offset (), -MODULES_END , gfp_mask, -PAGE_KERNEL ,VM_DEFER_KMEMLEAK ,NUMA_NO_NODE , -__builtin_return_address (0 )); +MODULES_VADDR +get_module_load_offset (), +MODULES_END , gfp_mask,PAGE_KERNEL , +VM_FLUSH_RESET_PERMS |VM_DEFER_KMEMLEAK , +NUMA_NO_NODE ,__builtin_return_address (0 )); +if (p && (kasan_alloc_module_shadow (p, size, gfp_mask) <0 )) {vfree (p);return NULL ;
PATCH 4/38 x86/vdso: Ensure all kernel code is seen by objtool
In reply to Thomas Gleixnerextable.c is kernel code and not part of the VDSO
--- arch/x86/entry/vdso/Makefile | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
arch/x86/entry/vdso/Makefile +6 -5
@@ -30,11 +30,12 @@ vobjs32-y += vdso32/vclock_gettime.o vobjs-$(CONFIG_X86_SGX) += vsgx.o # files to link into kernel -obj-y += vma.o extable.o -KASAN_SANITIZE_vma.o := y -UBSAN_SANITIZE_vma.o := y -KCSAN_SANITIZE_vma.o := y -OBJECT_FILES_NON_STANDARD_vma.o := n +obj-y += vma.o extable.o +KASAN_SANITIZE_vma.o := y +UBSAN_SANITIZE_vma.o := y +KCSAN_SANITIZE_vma.o := y +OBJECT_FILES_NON_STANDARD_vma.o := n +OBJECT_FILES_NON_STANDARD_extable.o := n # vDSO images to build vdso_img-$(VDSO64-y) += 64
PATCH 5/38 btree: Initialize early when builtin
In reply to Thomas GleixnerAn upcoming user of btree needs it early on. Initialize it in start_kernel().
--- include/linux/btree.h | 6 ++++++ init/main.c | 2 ++ lib/btree.c | 8 +++++++- 3 files changed, 15 insertions(+), 1 deletion(-)
include/linux/btree.h +6 -0
@@ -5,6 +5,12 @@#include <linux/kernel.h> #include <linux/mempool.h> +#if IS_BUILTIN (CONFIG_BTREE ) +extern void btree_cache_init (void ); +#else +static inline void btree_cache_init (void ) {} +#endif +/** * DOC: B+Tree basics *
init/main.c +2 -0
@@ -75,6 +75,7 @@#include <linux/signal.h> #include <linux/idr.h> #include <linux/kgdb.h> +#include <linux/btree.h> #include <linux/ftrace.h> #include <linux/async.h> #include <linux/shmem_fs.h>
@@ -1125,6 +1126,7 @@ asmlinkage __visible void __init __no_sacgroup_init ();taskstats_init_early ();delayacct_init (); +btree_cache_init ();poking_init ();check_bugs ();
lib/btree.c +7 -1
@@ -787,15 +787,21 @@ static int __init btree_module_init(voidreturn 0 ; } +#if IS_MODULE (CONFIG_BTREE ) static void __exit btree_module_exit(void) {kmem_cache_destroy (btree_cachep); } -/* If core code starts using btree, initialization should happen even earlier */ module_init (btree_module_init);module_exit (btree_module_exit);MODULE_AUTHOR ("Joern Engel <[email protected]>" );MODULE_AUTHOR ("Johannes Berg <[email protected]>" );MODULE_LICENSE ("GPL" ); +#else +void __init btree_cache_init (void ) +{ +BUG_ON (btree_module_init ()); +} +#endif
PATCH 6/38 objtool: Allow GS relative relocs
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Objtool doesn't currently much like per-cpu usage in alternatives:
arch/x86/entry/entry_64.o: warning: objtool: .altinstr_replacement+0xf: unsupported relocation in alternatives section
f: 65 c7 04 25 00 00 00 00 00 00 00 80 movl $0x80000000,%gs:0x0 13: R_X86_64_32S __x86_call_depth
Allow this.
--- tools/objtool/arch/x86/decode.c | 26 +++++++++++++++++++++----- tools/objtool/check.c | 6 ++---- tools/objtool/include/objtool/arch.h | 4 +--- tools/objtool/include/objtool/check.h | 20 +++++++++++--------- 4 files changed, 35 insertions(+), 21 deletions(-)
tools/objtool/arch/x86/decode.c +21 -5
@@ -103,24 +103,37 @@ unsigned long arch_jump_destination(stru#define rm_is_mem (reg) (mod_is_mem() && !is_RIP() && rm_is(reg))#define rm_is_reg (reg) (mod_is_reg() && modrm_rm == (reg)) -static bool has_notrack_prefix (struct insn *insn) +static bool has_prefix (struct insn *insn,u8 prefix) {int i;for (i =0 ; i < insn->prefixes.nbytes; i++) { -if (insn->prefixes.bytes[i] ==0x3e ) +if (insn->prefixes.bytes[i] == prefix)return true; }return false; } +static bool has_notrack_prefix (struct insn *insn) +{ +return has_prefix (insn,0x3e ); +} + +static bool has_gs_prefix (struct insn *insn) +{ +return has_prefix (insn,0x65 ); +} +int arch_decode_instruction (struct objtool_file *file,const struct section *sec,unsigned long offset,unsigned int maxlen, -unsigned int *len,enum insn_type *type, -unsigned long *immediate, -struct list_head *ops_list) +struct instruction *instruction) { +struct list_head *ops_list = &instruction->stack_ops; +unsigned long *immediate = &instruction->immediate; +enum insn_type *type = &instruction->type; +unsigned int *len = &instruction->len; +const struct elf *elf = file->elf;struct insn insn;int x86_64, ret;
@@ -149,6 +162,9 @@ int arch_decode_instruction(struct objtoif (insn.vex_prefix.nbytes)return 0 ; +if (has_gs_prefix (&insn)) + instruction->alt_reloc_safe =1 ; + prefix = insn.prefixes.bytes[0 ]; op1 = insn.opcode.bytes[0 ];
tools/objtool/check.c +2 -4
@@ -396,9 +396,7 @@ static int decode_instructions(struct ob ret =arch_decode_instruction (file, sec, offset, sec->sh.sh_size - offset, - &insn->len, &insn->type, - &insn->immediate, - &insn->stack_ops); + insn);if (ret) goto err;
@@ -1620,7 +1618,7 @@ static int handle_group_alt(struct objto* accordingly. */ alt_reloc =insn_reloc (file, insn); -if (alt_reloc && +if (alt_reloc && !insn->alt_reloc_safe && !arch_support_alt_relocation (special_alt, insn, alt_reloc)) { WARN_FUNC("unsupported relocation in alternatives section",
tools/objtool/include/objtool/arch.h +1 -3
@@ -73,9 +73,7 @@ void arch_initial_func_cfi_state(structint arch_decode_instruction (struct objtool_file *file,const struct section *sec,unsigned long offset,unsigned int maxlen, -unsigned int *len,enum insn_type *type, -unsigned long *immediate, -struct list_head *ops_list); +struct instruction *insn);bool arch_callee_saved_reg (unsigned char reg);
tools/objtool/include/objtool/check.h +11 -9
@@ -47,15 +47,17 @@ struct instruction {unsigned long immediate; u16 dead_end : 1, - ignore : 1, - ignore_alts : 1, - hint : 1, - save : 1, - restore : 1, - retpoline_safe : 1, - noendbr : 1, - entry : 1; -/* 7 bit hole */ + ignore : 1, + ignore_alts : 1, + hint : 1, + save : 1, + restore : 1, + retpoline_safe : 1, + noendbr : 1, + entry : 1, + alt_reloc_safe : 1; + +/* 6 bit hole */ s8 instr;u8 visited;
PATCH 7/38 objtool: Track init section
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
For future usage of .init.text exclusion track the init section in the instruction decoder and use the result in retpoline validation.
--- tools/objtool/check.c | 17 ++++++++++------- tools/objtool/include/objtool/elf.h | 2 +- 2 files changed, 11 insertions(+), 8 deletions(-)
tools/objtool/check.c +10 -7
@@ -380,6 +380,15 @@ static int decode_instructions(struct ob !strncmp(sec->name, ".text.__x86.", 12)) sec->noinstr = true; +/* +* .init.text code is ran before userspace and thus doesn't +* strictly need retpolines, except for modules which are +* loaded late, they very much do need retpoline in their +* .init.text +*/ +if (!strcmp (sec->name,".init.text" ) && !opts.module) + sec->init = true; +for (offset =0 ; offset < sec->sh.sh_size; offset += insn->len) { insn =malloc (sizeof (*insn));if (!insn) {
@@ -3720,13 +3729,7 @@ static int validate_retpoline(struct objif (insn->retpoline_safe)continue ; -/* -* .init.text code is ran before userspace and thus doesn't -* strictly need retpolines, except for modules which are -* loaded late, they very much do need retpoline in their -* .init.text -*/ -if (!strcmp (insn->sec->name,".init.text" ) && !opts.module) +if (insn->sec->init)continue ;if (insn->type ==INSN_RETURN ) {
tools/objtool/include/objtool/elf.h +1 -1
@@ -38,7 +38,7 @@ struct section {Elf_Data *data;char *name;int idx; -bool changed, text, rodata, noinstr; +bool changed, text, rodata, noinstr, init; };struct symbol {
PATCH 8/38 objtool: Add .call_sites section
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
In preparation for call depth tracking provide a section which collects all direct calls.
--- arch/x86/kernel/vmlinux.lds.S | 7 ++++ tools/objtool/check.c | 51 ++++++++++++++++++++++++++++++++ tools/objtool/include/objtool/objtool.h | 1 tools/objtool/objtool.c | 1 4 files changed, 60 insertions(+)
arch/x86/kernel/vmlinux.lds.S +7 -0
@@ -290,6 +290,13 @@ SECTIONS *(.return_sites) __return_sites_end = .; } + + . = ALIGN(8); + .call_sites : AT(ADDR(.call_sites) - LOAD_OFFSET) { + __call_sites = .; + *(.call_sites) + __call_sites_end = .; + } #endif #ifdef CONFIG_X86_KERNEL_IBT
tools/objtool/check.c +51 -0
69 lines not on this page · raw message
tools/objtool/include/objtool/objtool.h +1 -0
@@ -28,6 +28,7 @@ struct objtool_file {struct list_head static_call_list;struct list_head mcount_loc_list;struct list_head endbr_list; +struct list_head call_list;bool ignore_unreachables, hints, rodata;unsigned int nr_endbr;
tools/objtool/objtool.c +1 -0
@@ -106,6 +106,7 @@ struct objtool_file *objtool_open_read(cINIT_LIST_HEAD (&file.static_call_list);INIT_LIST_HEAD (&file.mcount_loc_list);INIT_LIST_HEAD (&file.endbr_list); +INIT_LIST_HEAD (&file.call_list); file.ignore_unreachables = opts.no_unreachable; file.hints = false;
PATCH 9/38 objtool: Add .sym_sites section
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
In preparation for call depth tracking provide a section which collects all all !init symbols to generate thunks for.
--- arch/x86/kernel/vmlinux.lds.S | 7 +++++ tools/objtool/check.c | 55 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+)
arch/x86/kernel/vmlinux.lds.S +7 -0
@@ -297,6 +297,13 @@ SECTIONS *(.call_sites) __call_sites_end = .; } + + . = ALIGN(8); + .sym_sites : AT(ADDR(.sym_sites) - LOAD_OFFSET) { + __sym_sites = .; + *(.sym_sites) + __sym_sites_end = .; + } #endif #ifdef CONFIG_X86_KERNEL_IBT
tools/objtool/check.c +55 -0
67 lines not on this page · raw message
PATCH 10/38 objtool: Add --hacks=skylake
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Make the call/func sections selectable via the --hacks option.
--- scripts/Makefile.lib | 3 ++- tools/objtool/builtin-check.c | 7 ++++++- tools/objtool/check.c | 18 ++++++++++-------- tools/objtool/include/objtool/builtin.h | 1 + 4 files changed, 19 insertions(+), 10 deletions(-)
scripts/Makefile.lib +2 -1
@@ -231,7 +231,8 @@ objtool := $(objtree)/tools/objtool/objt objtool_args = \ $(if $(CONFIG_HAVE_JUMP_LABEL_HACK), --hacks=jump_label) \ - $(if $(CONFIG_HAVE_NOINSTR_HACK), --hacks=noinstr) \ + $(if $(CONFIG_HAVE_NOINSTR_HACK), --hacks=noinstr) \ + $(if $(CONFIG_CALL_DEPTH_TRACKING), --hacks=skylake) \ $(if $(CONFIG_X86_KERNEL_IBT), --ibt) \ $(if $(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL), --mcount) \ $(if $(CONFIG_UNWINDER_ORC), --orc) \
tools/objtool/builtin-check.c +6 -1
@@ -57,12 +57,17 @@ static int parse_hacks(const struct opti found = true; } +if (!str ||strstr (str,"skylake" )) { + opts.hack_skylake = true; + found = true; + } +return found ?0 :-1 ; }const struct option check_options[] = {OPT_GROUP ("Actions:" ), -OPT_CALLBACK_OPTARG ('h' ,"hacks" ,NULL ,NULL ,"jump_label,noinstr" ,"patch toolchain bugs/limitations" , parse_hacks), +OPT_CALLBACK_OPTARG ('h' ,"hacks" ,NULL ,NULL ,"jump_label,noinstr,skylake" ,"patch toolchain bugs/limitations" , parse_hacks),OPT_BOOLEAN ('i' ,"ibt" , &opts.ibt,"validate and annotate IBT" ),OPT_BOOLEAN ('m' ,"mcount" , &opts.mcount,"annotate mcount/fentry calls for ftrace" ),OPT_BOOLEAN ('n' ,"noinstr" , &opts.noinstr,"validate noinstr rules" ),
tools/objtool/check.c +10 -8
@@ -4372,15 +4372,17 @@ int check(struct objtool_file *file) goto out; warnings += ret; - ret =create_direct_call_sections (file); -if (ret <0 ) - goto out; - warnings += ret; +if (opts.hack_skylake) { + ret =create_direct_call_sections (file); +if (ret <0 ) + goto out; + warnings += ret; - ret =create_sym_thunk_sections (file); -if (ret <0 ) - goto out; - warnings += ret; + ret =create_sym_thunk_sections (file); +if (ret <0 ) + goto out; + warnings += ret; + } }if (opts.mcount) {
tools/objtool/include/objtool/builtin.h +1 -0
@@ -14,6 +14,7 @@ struct opts {bool dump_orc;bool hack_jump_label;bool hack_noinstr; +bool hack_skylake;bool ibt;bool mcount;bool noinstr;
PATCH 11/38 objtool: Allow STT_NOTYPE -> STT_FUNC+0 tail-calls
In reply to Thomas GleixnerAllow STT_NOTYPE to tail-call into STT_FUNC, per definition STT_NOTYPE is not a sub-function of the STT_FUNC.
--- tools/objtool/check.c | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-)
tools/objtool/check.c +20 -9
@@ -1420,6 +1420,16 @@ static void add_return_call(struct objtostatic bool same_function (struct instruction *insn1,struct instruction *insn2) { +if (!insn1->func && !insn2->func) +return true; + +/* Allow STT_NOTYPE -> STT_FUNC+0 tail-calls */ +if (!insn1->func && insn1->func != insn2->func) +return false; + +if (!insn2->func) +return true; +return insn1->func->pfunc == insn2->func->pfunc; }
@@ -1537,18 +1547,19 @@ static int add_jump_destinations(struct strstr(jump_dest->func->name, ".cold")) { insn->func->cfunc = jump_dest->func; jump_dest->func->pfunc = insn->func; - - } else if (!same_function(insn, jump_dest) && -is_first_func_insn (file ,jump_dest )) { -/* -* Internal sibling call without reloc or with -* STT_SECTION reloc. -*/ -add_call_dest (file, insn, jump_dest->func, true); -continue ; } } +if (!same_function (insn, jump_dest) && +is_first_func_insn (file, jump_dest)) { +/* +* Internal sibling call without reloc or with +* STT_SECTION reloc. +*/ +add_call_dest (file, insn, jump_dest->func, true); +continue ; + } + insn->jump_dest = jump_dest; }
PATCH 12/38 x86/entry: Make sync_regs() invocation a tail call
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
No point in having a call there. Spare the call/ret overhead.
--- arch/x86/entry/entry_64.S | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
arch/x86/entry/entry_64.S +2 -5
@@ -1060,11 +1060,8 @@ SYM_CODE_START_LOCAL(error_entry) UNTRAIN_RET leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ -.Lerror_entry_from_usermode_after_swapgs: - /* Put us onto the real thread stack. */ - call sync_regs - RET + jmp sync_regs /* * There are two places in the kernel that can potentially fault with
@@ -1122,7 +1119,7 @@ SYM_CODE_START_LOCAL(error_entry) leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ call fixup_bad_iret mov %rax, %rdi - jmp .Lerror_entry_from_usermode_after_swapgs + jmp sync_regs SYM_CODE_END(error_entry) SYM_CODE_START_LOCAL(error_return)
PATCH 13/38 x86/modules: Make module_alloc() generally available
In reply to Thomas Gleixnermodule_alloc() allocates from the module region which is also available when CONFIG_MODULES=n. Non module builds should be able to allocate from that region nevertheless e.g. for creating call thunks.
Split the code out and make it possible to select for !MODULES builds.
--- arch/x86/Kconfig | 3 ++ arch/x86/kernel/module.c | 58 ------------------------------------------ arch/x86/mm/Makefile | 2 + arch/x86/mm/module_alloc.c | 62 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 58 deletions(-)
arch/x86/Kconfig +3 -0
@@ -2236,6 +2236,9 @@ config RANDOMIZE_MEMORY_PHYSICAL_PADDING If unsure, leave at the default value. +config MODULE_ALLOC + def_bool MODULES + config HOTPLUG_CPU def_bool y depends on SMP
arch/x86/kernel/module.c +0 -58
78 lines not on this page · raw message
arch/x86/mm/Makefile +2 -0
@@ -59,3 +59,5 @@ obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_enc obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o + +obj-$(CONFIG_MODULE_ALLOC) += module_alloc.o
arch/x86/mm/module_alloc.c new file +62 -0
62 lines not on this page · raw message
PATCH 14/38 x86/Kconfig: Add CONFIG_CALL_THUNKS
In reply to Thomas GleixnerIn preparation for mitigating the Intel SKL RSB underflow issue in software, add a new configuration symbol which allows to build the required call thunk infrastructure conditionally.
--- arch/x86/Kconfig | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/Kconfig +8 -0
@@ -2439,6 +2439,14 @@ config CC_HAS_SLS config CC_HAS_RETURN_THUNK def_bool $(cc-option,-mfunction-return=thunk-extern) +config HAVE_CALL_THUNKS + def_bool y + depends on RETHUNK && OBJTOOL + +config CALL_THUNKS + def_bool n + select MODULE_ALLOC + menuconfig SPECULATION_MITIGATIONS bool "Mitigations for speculative execution vulnerabilities" default y
PATCH 15/38 x86/retbleed: Add X86_FEATURE_CALL_DEPTH
In reply to Thomas GleixnerIntel SKL CPUs fall back to other predictors when the RSB underflows. The only microcode mitigation is IBRS which is insanely expensive. It comes with performance drops of up to 30% depending on the workload.
A way less expensive, but nevertheless horrible mitigation is to track the call depth in software and overeagerly fill the RSB when returns underflow the software counter.
Provide a configuration symbol and a CPU misfeature bit.
--- arch/x86/Kconfig | 13 +++++++++++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 9 ++++++++- 3 files changed, 22 insertions(+), 1 deletion(-)
arch/x86/Kconfig +13 -0
@@ -2498,6 +2498,19 @@ config CPU_UNRET_ENTRY help Compile the kernel with support for the retbleed=unret mitigation. +config CALL_DEPTH_TRACKING + bool "Mitigate RSB underflow with call depth tracking" + depends on CPU_SUP_INTEL && HAVE_CALL_THUNKS + select CALL_THUNKS + default y + help + Compile the kernel with call depth tracking to mitigate the Intel + SKL Return-Speculation-Buffer (RSB) underflow issue. The mitigation + is off by default and needs to be enabled on the kernel command line + via the retbleed=stuff option. For non-affected systems the overhead + of this option is marginal as the call depth tracking is using + run-time generated call thunks and call patching. + config CPU_IBPB_ENTRY bool "Enable IBPB on kernel entry" depends on CPU_SUP_AMD
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -302,6 +302,7 @@#define X86_FEATURE_RETPOLINE_LFENCE (11*32+13)/* "" Use LFENCE for Spectre variant 2 */ #define X86_FEATURE_RETHUNK (11*32+14)/* "" Use REturn THUNK */ #define X86_FEATURE_UNRET (11*32+15)/* "" AMD BTB untrain return */ +#define X86_FEATURE_CALL_DEPTH (11*32+16)/* "" Call depth tracking for RSB stuffing */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */
arch/x86/include/asm/disabled-features.h +8 -1
@@ -69,6 +69,12 @@# define DISABLE_UNRET (1 << (X86_FEATURE_UNRET & 31))#endif +#ifdef CONFIG_CALL_DEPTH_TRACKING +# define DISABLE_CALL_DEPTH_TRACKING 0 +#else +# define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) +#endif +#ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0#else
@@ -101,7 +107,8 @@#define DISABLED_MASK8 (DISABLE_TDX_GUEST)#define DISABLED_MASK9 (DISABLE_SGX)#define DISABLED_MASK10 0 -#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET) +#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ + DISABLE_CALL_DEPTH_TRACKING)#define DISABLED_MASK12 0#define DISABLED_MASK13 0#define DISABLED_MASK14 0
PATCH 16/38 modules: Make struct module_layout unconditionally available
In reply to Thomas GleixnerTo simplify the upcoming call thunk code it's desired to expose struct module_layout even on !MDOULES builds. This spares conditionals and #ifdeffery.
--- include/linux/module.h | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-)
include/linux/module.h +22 -22
@@ -67,6 +67,28 @@ struct module_version_attribute {const char *version; }; +struct mod_tree_node { +struct module *mod; +struct latch_tree_node node; +}; + +struct module_layout { +/* The actual code + data. */ +void *base; +/* Total size. */ +unsigned int size; +/* The size of the executable code. */ +unsigned int text_size; +/* Size of RO section of the module (text+rodata) */ +unsigned int ro_size; +/* Size of RO after init section */ +unsigned int ro_after_init_size; + +#ifdef CONFIG_MODULES_TREE_LOOKUP +struct mod_tree_node mtn; +#endif +}; +extern ssize_t __modver_version_show (struct module_attribute *,struct module_kobject *,char *);
@@ -316,28 +338,6 @@ enum module_state { MODULE_STATE_UNFORMED, /* Still setting it up. */ }; -struct mod_tree_node { -struct module *mod; -struct latch_tree_node node; -}; - -struct module_layout { -/* The actual code + data. */ -void *base; -/* Total size. */ -unsigned int size; -/* The size of the executable code. */ -unsigned int text_size; -/* Size of RO section of the module (text+rodata) */ -unsigned int ro_size; -/* Size of RO after init section */ -unsigned int ro_after_init_size; - -#ifdef CONFIG_MODULES_TREE_LOOKUP -struct mod_tree_node mtn; -#endif -}; -#ifdef CONFIG_MODULES_TREE_LOOKUP /* Only touch one cacheline for common rbtree-for-core-layout case. */ #define __module_layout_align ____cacheline_aligned
PATCH 17/38 module: Add arch_data to module_layout
In reply to Thomas GleixnerFor the upcoming call depth tracking it's required to store extra information in the module layout. Add a pointer.
--- include/linux/module.h | 3 +++ 1 file changed, 3 insertions(+)
include/linux/module.h +3 -0
@@ -87,6 +87,9 @@ struct module_layout {#ifdef CONFIG_MODULES_TREE_LOOKUP struct mod_tree_node mtn;#endif +#ifdef CONFIG_CALL_THUNKS +void *arch_data; +#endif }; extern ssize_t __modver_version_show(struct module_attribute *,
PATCH 18/38 mm/vmalloc: Provide huge page mappings
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Provide VM_HUGE_VMAP, which unconditionally tries to use huge mappings. Unlike VM_ALLOW_HUGE_VMAP it doesn't care about the number of NUMA nodes or the size of the allocation.
If the page allocator fails to provide huge pages, it will silently fall back.
--- include/linux/vmalloc.h | 3 ++- mm/vmalloc.c | 33 +++++++++++++++++++-------------- 2 files changed, 21 insertions(+), 15 deletions(-)
include/linux/vmalloc.h +2 -1
@@ -27,10 +27,11 @@ struct notifier_block; /* in notifier.h#define VM_FLUSH_RESET_PERMS 0x00000100/* reset direct map and flush TLB on unmap, can't be freed in atomic context */ #define VM_MAP_PUT_PAGES 0x00000200/* put pages and free array in vfree */ #define VM_ALLOW_HUGE_VMAP 0x00000400/* Allow for huge pages on archs with HAVE_ARCH_HUGE_VMALLOC */ +#define VM_HUGE_VMAP 0x00000800/* Force for huge pages on archs with HAVE_ARCH_HUGE_VMALLOC */ #if (defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS )) && \ !defined(CONFIG_KASAN_VMALLOC ) -#define VM_DEFER_KMEMLEAK 0x00000800/* defer kmemleak object creation */ +#define VM_DEFER_KMEMLEAK 0x00001000/* defer kmemleak object creation */ #else #define VM_DEFER_KMEMLEAK 0#endif
mm/vmalloc.c +19 -14
@@ -3099,23 +3099,28 @@ void *__vmalloc_node_range(unsigned longreturn NULL ; } -if (vmap_allow_huge && (vm_flags &VM_ALLOW_HUGE_VMAP )) { -unsigned long size_per_node; +if (vmap_allow_huge && (vm_flags & (VM_HUGE_VMAP |VM_ALLOW_HUGE_VMAP ))) { -/* -* Try huge pages. Only try for PAGE_KERNEL allocations, -* others like modules don't yet expect huge pages in -* their allocations due to apply_to_page_range not -* supporting them. -*/ +if (vm_flags &VM_ALLOW_HUGE_VMAP ) { +unsigned long size_per_node; - size_per_node = size; -if (node ==NUMA_NO_NODE ) - size_per_node /=num_online_nodes (); -if (arch_vmap_pmd_supported (prot) && size_per_node >=PMD_SIZE ) +/* +* Try huge pages. Only try for PAGE_KERNEL allocations, +* others like modules don't yet expect huge pages in +* their allocations due to apply_to_page_range not +* supporting them. +*/ + + size_per_node = size; +if (node ==NUMA_NO_NODE ) + size_per_node /=num_online_nodes (); +if (arch_vmap_pmd_supported (prot) && size_per_node >=PMD_SIZE ) + shift =PMD_SHIFT ; +else + shift =arch_vmap_pte_supported_shift (size_per_node); + }else { shift =PMD_SHIFT ; -else - shift =arch_vmap_pte_supported_shift (size_per_node); + } align =max (real_align,1UL << shift); size =ALIGN (real_size,1UL << shift);
PATCH 19/38 x86/module: Provide __module_alloc()
In reply to Thomas GleixnerProvide a function to allocate from module space with large TLBs. This is required for callthunks as otherwise the ITLB pressure kills performance.
--- arch/x86/include/asm/module.h | 2 ++ arch/x86/mm/module_alloc.c | 10 ++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
arch/x86/include/asm/module.h +2 -0
@@ -13,4 +13,6 @@ struct mod_arch_specific { #endif }; +extern void *__module_alloc (unsigned long size,unsigned long vmflags); +#endif /* _ASM_X86_MODULE_H */
arch/x86/mm/module_alloc.c +8 -2
@@ -39,7 +39,7 @@ static unsigned long int get_module_load }#endif -void *module_alloc (unsigned long size) +void *__module_alloc (unsigned long size,unsigned long vmflags) {gfp_t gfp_mask =GFP_KERNEL ;void *p;
@@ -47,10 +47,11 @@ void *module_alloc(unsigned long size)if (PAGE_ALIGN (size) >MODULES_LEN )return NULL ; + vmflags |=VM_FLUSH_RESET_PERMS |VM_DEFER_KMEMLEAK ; p =__vmalloc_node_range (size,MODULE_ALIGN ,MODULES_VADDR +get_module_load_offset (),MODULES_END , gfp_mask,PAGE_KERNEL , -VM_FLUSH_RESET_PERMS |VM_DEFER_KMEMLEAK , + vmflags,NUMA_NO_NODE ,__builtin_return_address (0 ));if (p && (kasan_alloc_module_shadow (p, size, gfp_mask) <0 )) {
@@ -60,3 +61,8 @@ void *module_alloc(unsigned long size)return p; } + +void *module_alloc (unsigned long size) +{ +return __module_alloc (size,0 ); +}
PATCH 20/38 x86/alternatives: Provide text_poke_[copy|set]_locked()
In reply to Thomas GleixnerThe upcoming call thunk patching must hold text_mutex and needs access to text_poke_copy() and text_poke_set(), which take text_mutex.
Provide _locked postfixed variants to expose the inner workings.
--- arch/x86/include/asm/text-patching.h | 2 + arch/x86/kernel/alternative.c | 48 +++++++++++++++++++++-------------- 2 files changed, 32 insertions(+), 18 deletions(-)
arch/x86/include/asm/text-patching.h +2 -0
@@ -45,6 +45,8 @@ extern void *text_poke(void *addr, constextern void text_poke_sync (void );extern void *text_poke_kgdb (void *addr,const void *opcode,size_t len);extern void *text_poke_copy (void *addr,const void *opcode,size_t len); +extern void *text_poke_copy_locked (void *addr,const void *opcode,size_t len); +extern void *text_poke_set_locked (void *addr,int c,size_t len);extern void *text_poke_set (void *addr,int c,size_t len);extern int poke_int3_handler (struct pt_regs *regs);extern void text_poke_bp (void *addr,const void *opcode,size_t len,const void *emulate);
arch/x86/kernel/alternative.c +30 -18
81 lines not on this page · raw message
PATCH 21/38 x86/entry: Make some entry symbols global
In reply to Thomas Gleixnerparanoid_entry(), error_entry() and xen_error_entry() have to be exempted from call accounting by thunk patching because they are before UNTRAIN_RET.
Expose them so they are available in the alternative code.
--- arch/x86/entry/entry_64.S | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
arch/x86/entry/entry_64.S +6 -3
@@ -326,7 +326,8 @@ SYM_CODE_END(ret_from_fork) #endif .endm -SYM_CODE_START_LOCAL(xen_error_entry) +SYM_CODE_START(xen_error_entry) + ANNOTATE_NOENDBR UNWIND_HINT_FUNC PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8
@@ -904,7 +905,8 @@ SYM_CODE_END(xen_failsafe_callback) * R14 - old CR3 * R15 - old SPEC_CTRL */ -SYM_CODE_START_LOCAL(paranoid_entry) +SYM_CODE_START(paranoid_entry) + ANNOTATE_NOENDBR UNWIND_HINT_FUNC PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8
@@ -1039,7 +1041,8 @@ SYM_CODE_END(paranoid_exit) /* * Switch GS and CR3 if needed. */ -SYM_CODE_START_LOCAL(error_entry) +SYM_CODE_START(error_entry) + ANNOTATE_NOENDBR UNWIND_HINT_FUNC PUSH_AND_CLEAR_REGS save_ret=1
PATCH 22/38 x86/paravirt: Make struct paravirt_call_site unconditionally available
In reply to Thomas GleixnerFor the upcoming call thunk patching it's less ifdeffery when the data structure is unconditionally available. The code can then be trivially fenced off with IS_ENABLED().
--- arch/x86/include/asm/paravirt.h | 4 ++-- arch/x86/include/asm/paravirt_types.h | 20 ++++++++++++-------- 2 files changed, 14 insertions(+), 10 deletions(-)
arch/x86/include/asm/paravirt.h +2 -2
@@ -4,13 +4,13 @@/* Various instructions on x86 need to be replaced for * para-virtualization: those hooks are defined here. */ +#include <asm/paravirt_types.h> +#ifdef CONFIG_PARAVIRT #include <asm/pgtable_types.h> #include <asm/asm.h> #include <asm/nospec-branch.h> -#include <asm/paravirt_types.h> -#ifndef __ASSEMBLY__#include <linux/bug.h> #include <linux/types.h>
arch/x86/include/asm/paravirt_types.h +12 -8
@@ -2,6 +2,17 @@#ifndef _ASM_X86_PARAVIRT_TYPES_H#define _ASM_X86_PARAVIRT_TYPES_H +#ifndef __ASSEMBLY__ +/* These all sit in the .parainstructions section to tell us what to patch. */ +struct paravirt_patch_site { +u8 *instr;/* original instructions */ +u8 type;/* type of this instruction */ +u8 len;/* length of original instruction */ +}; +#endif + +#ifdef CONFIG_PARAVIRT +/* Bitmask of what can be clobbered: usually at least eax. */ #define CLBR_EAX (1 << 0)#define CLBR_ECX (1 << 1)
@@ -584,16 +595,9 @@ unsigned long paravirt_ret0(void);#define paravirt_nop ((void *)_paravirt_nop) -/* These all sit in the .parainstructions section to tell us what to patch. */ -struct paravirt_patch_site { -u8 *instr;/* original instructions */ -u8 type;/* type of this instruction */ -u8 len;/* length of original instruction */ -}; -extern struct paravirt_patch_site __parainstructions[], __parainstructions_end[];#endif /* __ASSEMBLY__ */ - +#endif /* CONFIG_PARAVIRT */ #endif /* _ASM_X86_PARAVIRT_TYPES_H */
PATCH 23/38 x86/callthunks: Add call patching for call depth tracking
In reply to Thomas GleixnerMitigating the Intel SKL RSB underflow issue in software requires to track the call depth. This could be done with help of the compiler by adding at least 7 bytes NOPs before every direct call, which amounts to a 15+ percent text size increase for a vmlinux built with a Debian kernel config. While CPUs are quite efficient in ignoring NOPs this still is a massive penalty in terms of I-cache for all CPUs which do not have this issue.
Inflict the pain only on SKL CPUs by creating call thunks for each function and patching the calls to invoke the thunks instead.
The thunks are created in module memory to stay within the 32bit displacement boundary. The thunk then does:
ACCOUNT_DEPTH
JMP function
The functions and call sites lists are generated by objtool. The memory requirement is 16 bytes per call thunk and btree memory for keeping track of them. For a Debian distro config this amounts to ~1.6MB thunk memory and 2MB btree storage. This is only required when the call depth tracking is enabled on the kernel command line. So the burden is solely on SKL[-X].
The thunks are all stored in one 2MB memory region which is mapped with a large TLB to prevent ITLB pressure.
The thunks are generated from a template and the btree is used to store them by destination address. The actual call patching retrieves the thunks from the btree and replaces the original function call by a call to the thunk.
Module handling and the actual thunk code for SKL will be added in subsequent steps.
--- arch/x86/Kconfig | 13 + arch/x86/include/asm/alternative.h | 13 + arch/x86/kernel/Makefile | 2 arch/x86/kernel/alternative.c | 6 arch/x86/kernel/callthunks.c | 459 +++++++++++++++++++++++++++++++++++++ 5 files changed, 493 insertions(+)
arch/x86/Kconfig +13 -0
@@ -125,6 +125,7 @@ config X86 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH + select BTREE if CALL_DEPTH_TRACKING select BUILDTIME_TABLE_SORT select CLKEVT_I8253 select CLOCKSOURCE_VALIDATE_LAST_CYCLE
@@ -2511,6 +2512,18 @@ config CALL_DEPTH_TRACKING of this option is marginal as the call depth tracking is using run-time generated call thunks and call patching. +config CALL_THUNKS_DEBUG + bool "Enable call thunks and call depth tracking debugging" + depends on CALL_DEPTH_TRACKING + default n + help + Enable call/ret counters for imbalance detection and build in + a noisy dmesg about callthunks generation and call patching for + trouble shooting. The debug prints need to be enabled on the + kernel command line with 'debug-callthunks'. + Only enable this, when you are debugging call thunks as this + creates a noticable runtime overhead. If unsure say N. + config CPU_IBPB_ENTRY bool "Enable IBPB on kernel entry" depends on CPU_SUP_AMD
arch/x86/include/asm/alternative.h +13 -0
@@ -80,6 +80,19 @@ extern void apply_returns(s32 *start, s3extern void apply_ibt_endbr (s32 *start,s32 *end);struct module ; +struct paravirt_patch_site ; + +struct callthunk_sites { +s32 *syms_start, *syms_end; +s32 *call_start, *call_end; +struct paravirt_patch_site *pv_start, *pv_end; +}; + +#ifdef CONFIG_CALL_THUNKS +extern void callthunks_patch_builtin_calls (void ); +#else +static __always_inline void callthunks_patch_builtin_calls(void) {} +#endif #ifdef CONFIG_SMP extern void alternatives_smp_module_add(struct module *mod, char *name,
arch/x86/kernel/Makefile +2 -0
@@ -141,6 +141,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_ obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev.o +obj-$(CONFIG_CALL_THUNKS) += callthunks.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/alternative.c +6 -0
@@ -936,6 +936,12 @@ void __init alternative_instructions(voi*/ apply_alternatives (__alt_instructions, __alt_instructions_end); +/* +* Now all calls are established. Apply the call thunks if +* required. +*/ +callthunks_patch_builtin_calls (); +apply_ibt_endbr (__ibt_endbr_seal, __ibt_endbr_seal_end);#ifdef CONFIG_SMP
arch/x86/kernel/callthunks.c new file +459 -0
459 lines not on this page · raw message
PATCH 24/38 module: Add layout for callthunks tracking
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Various things will need to be able to tell if a specific address is a callthunk or not (ORC, BPF, static_call). In order to answer this question in the face of modules it is necessary to (quickly) find the module associated with a specific (callthunk) address.
Extend the __module_address() infrastructure with knowledge of the (per module) callthunk range.
--- include/linux/module.h | 21 +++++++++++++++++++-- kernel/module/internal.h | 8 ++++++++ kernel/module/main.c | 6 ++++++ kernel/module/tree_lookup.c | 17 ++++++++++++++++- 4 files changed, 49 insertions(+), 3 deletions(-)
include/linux/module.h +19 -2
@@ -424,6 +424,9 @@ struct module {/* Core layout: rbtree is accessed frequently, so keep together. */ struct module_layout core_layout __module_layout_align;struct module_layout init_layout; +#ifdef CONFIG_CALL_THUNKS +struct module_layout thunk_layout; +#endif #ifdef CONFIG_ARCH_WANTS_MODULES_DATA_IN_VMALLOC struct module_layout data_layout;#endif
@@ -590,9 +593,23 @@ static inline bool within_module_init(un addr < (unsigned long )mod->init_layout.base + mod->init_layout.size; } -static inline bool within_module (unsigned long addr,const struct module *mod) +static inline bool within_module_thunk (unsigned long addr, +const struct module *mod) +{ +#ifdef CONFIG_CALL_THUNKS +return (unsigned long )mod->thunk_layout.base <= addr && + addr < (unsigned long )mod->thunk_layout.base + mod->thunk_layout.size; +#else +return false; +#endif +} + +static inline bool within_module (unsigned long addr, +const struct module *mod) { -return within_module_init (addr, mod) ||within_module_core (addr, mod); +return within_module_core (addr, mod) || +within_module_thunk (addr, mod) || +within_module_init (addr, mod); }/* Search for module by name: must be in a RCU-sched critical section. */
kernel/module/internal.h +8 -0
@@ -219,6 +219,14 @@ static inline struct module *mod_find(un }#endif /* CONFIG_MODULES_TREE_LOOKUP */ +#if defined(CONFIG_MODULES_TREE_LOOKUP ) && defined(CONFIG_CALL_THUNKS ) +void mod_tree_insert_thunk (struct module *mod); +void mod_tree_remove_thunk (struct module *mod); +#else +static inline void mod_tree_insert_thunk (struct module *mod) { } +static inline void mod_tree_remove_thunk (struct module *mod) { } +#endif +void module_enable_ro (const struct module *mod,bool after_init);void module_enable_nx (const struct module *mod);void module_enable_x (const struct module *mod);
kernel/module/main.c +6 -0
@@ -1154,6 +1154,7 @@ static void free_module(struct module *m*/ mutex_lock (&module_mutex); mod->state =MODULE_STATE_UNFORMED ; +mod_tree_remove_thunk (mod);mutex_unlock (&module_mutex);/* Remove dynamic debug info */
@@ -2770,6 +2771,10 @@ static int load_module(struct load_infoif (err <0 ) goto free_modinfo; +mutex_lock (&module_mutex); +mod_tree_insert_thunk (mod); +mutex_unlock (&module_mutex); +flush_module_icache (mod);/* Setup CFI for the module. */
@@ -2859,6 +2864,7 @@ static int load_module(struct load_infomutex_lock (&module_mutex);/* Unlink carefully: kallsyms could be walking list. */ list_del_rcu (&mod->list); +mod_tree_remove_thunk (mod);mod_tree_remove (mod);wake_up_all (&module_wq);/* Wait for RCU-sched synchronizing before releasing mod->list. */
kernel/module/tree_lookup.c +16 -1
@@ -66,11 +66,26 @@ static noinline void __mod_tree_insert(slatch_tree_insert (&node->node, &tree->root, &mod_tree_ops); } -static void __mod_tree_remove (struct mod_tree_node *node,struct mod_tree_root *tree) +static noinline void __mod_tree_remove(struct mod_tree_node *node, struct mod_tree_root *tree) {latch_tree_erase (&node->node, &tree->root, &mod_tree_ops); } +#ifdef CONFIG_CALL_THUNKS +void mod_tree_insert_thunk (struct module *mod) +{ + mod->thunk_layout.mtn.mod = mod; +if (mod->thunk_layout.size) +__mod_tree_insert (&mod->thunk_layout.mtn, &mod_tree); +} + +void mod_tree_remove_thunk (struct module *mod) +{ +if (mod->thunk_layout.size) +__mod_tree_remove (&mod->thunk_layout.mtn, &mod_tree); +} +#endif +/* * These modifications: insert, remove_init and remove; are serialized by the * module_mutex.
PATCH 25/38 x86/modules: Add call thunk patching
In reply to Thomas GleixnerAs for the builtins create call thunks and patch the call sites to call the thunk on Intel SKL CPUs for retbleed mitigation.
Note, that module init functions are ignored for sake of simplicity because loading modules is not something which is done in high frequent loops and the attacker has not really a handle on when this happens in order to launch a matching attack. The depth tracking will still work for calls into the builtins and because the call is not accounted it will underflow faster and overstuff, but that's mitigated by the saturating counter and the side effect is only temporary.
--- arch/x86/include/asm/alternative.h | 7 +++++ arch/x86/kernel/callthunks.c | 49 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/module.c | 29 +++++++++++++++++++++ include/linux/module.h | 4 +++ 4 files changed, 88 insertions(+), 1 deletion(-)
arch/x86/include/asm/alternative.h +7 -0
@@ -90,8 +90,15 @@ struct callthunk_sites {#ifdef CONFIG_CALL_THUNKS extern void callthunks_patch_builtin_calls (void ); +extern void callthunks_patch_module_calls (struct callthunk_sites *sites, +struct module *mod); +extern void callthunks_module_free (struct module *mod);#else static __always_inline void callthunks_patch_builtin_calls(void) {} +static __always_inline void +callthunks_patch_module_calls (struct callthunk_sites *sites, +struct module *mod) {} +static __always_inline void callthunks_module_free(struct module *mod) { }#endif #ifdef CONFIG_SMP
arch/x86/kernel/callthunks.c +49 -0
64 lines not on this page · raw message
arch/x86/kernel/module.c +28 -1
@@ -196,7 +196,8 @@ int module_finalize(const Elf_Ehdr *hdr, {const Elf_Shdr *s, *text =NULL , *alt =NULL , *locks =NULL , *para =NULL , *orc =NULL , *orc_ip =NULL , - *retpolines =NULL , *returns =NULL , *ibt_endbr =NULL ; + *retpolines =NULL , *returns =NULL , *ibt_endbr =NULL , + *syms =NULL , *calls =NULL ;char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
@@ -216,6 +217,10 @@ int module_finalize(const Elf_Ehdr *hdr, retpolines = s;if (!strcmp (".return_sites" , secstrings + s->sh_name)) returns = s; +if (!strcmp (".sym_sites" , secstrings + s->sh_name)) + syms = s; +if (!strcmp (".call_sites" , secstrings + s->sh_name)) + calls = s;if (!strcmp (".ibt_endbr_seal" , secstrings + s->sh_name)) ibt_endbr = s; }
@@ -241,10 +246,31 @@ int module_finalize(const Elf_Ehdr *hdr,void *aseg = (void *)alt->sh_addr;apply_alternatives (aseg, aseg + alt->sh_size); } +if (calls || syms || para) { +struct callthunk_sites cs = {}; + +if (syms) { + cs.syms_start = (void *)syms->sh_addr; + cs.syms_end = (void *)syms->sh_addr + syms->sh_size; + } + +if (calls) { + cs.call_start = (void *)calls->sh_addr; + cs.call_end = (void *)calls->sh_addr + calls->sh_size; + } + +if (para) { + cs.pv_start = (void *)para->sh_addr; + cs.pv_end = (void *)para->sh_addr + para->sh_size; + } + +callthunks_patch_module_calls (&cs, me); + }if (ibt_endbr) {void *iseg = (void *)ibt_endbr->sh_addr;apply_ibt_endbr (iseg, iseg + ibt_endbr->sh_size); } +if (locks && text) {void *lseg = (void *)locks->sh_addr;void *tseg = (void *)text->sh_addr;
@@ -266,4 +292,5 @@ int module_finalize(const Elf_Ehdr *hdr,void module_arch_cleanup (struct module *mod) {alternatives_smp_module_del (mod); +callthunks_module_free (mod); }
include/linux/module.h +4 -0
@@ -525,6 +525,10 @@ struct module {struct pi_entry **printk_index_start;#endif +#ifdef CONFIG_CALL_THUNKS +char *callthunk_name; +#endif +#ifdef CONFIG_MODULE_UNLOAD /* What modules depend on me? */ struct list_head source_list;
PATCH 26/38 x86/returnthunk: Allow different return thunks
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
In preparation for call depth tracking on Intel SKL CPUs, make it possible to patch in a SKL specific return thunk.
--- arch/x86/include/asm/nospec-branch.h | 6 ++++++ arch/x86/kernel/alternative.c | 19 ++++++++++++++----- arch/x86/kernel/ftrace.c | 2 +- arch/x86/kernel/static_call.c | 2 +- arch/x86/net/bpf_jit_comp.c | 2 +- 5 files changed, 23 insertions(+), 8 deletions(-)
arch/x86/include/asm/nospec-branch.h +6 -0
@@ -168,6 +168,12 @@ extern void __x86_return_thunk(void);extern void zen_untrain_ret (void );extern void entry_ibpb (void ); +#ifdef CONFIG_CALL_THUNKS +extern void (*x86_return_thunk)(void ); +#else +#define x86_return_thunk (&__x86_return_thunk) +#endif +#ifdef CONFIG_RETPOLINE #define GEN (reg) \
arch/x86/kernel/alternative.c +14 -5
@@ -509,6 +509,11 @@ void __init_or_module noinline apply_ret }#ifdef CONFIG_RETHUNK + +#ifdef CONFIG_CALL_THUNKS +void (*x86_return_thunk)(void )__ro_after_init = &__x86_return_thunk; +#endif +/* * Rewrite the compiler generated return thunk tail-calls. *
@@ -524,14 +529,18 @@ static int patch_return(void *addr, stru {int i =0 ; -if (cpu_feature_enabled (X86_FEATURE_RETHUNK )) -return -1 ; - - bytes[i++] =RET_INSN_OPCODE ; +if (cpu_feature_enabled (X86_FEATURE_RETHUNK )) { +if (x86_return_thunk == __x86_return_thunk) +return -1 ; + + i =JMP32_INSN_SIZE ; +__text_gen_insn (bytes,JMP32_INSN_OPCODE , addr, x86_return_thunk, i); + }else { + bytes[i++] =RET_INSN_OPCODE ; + }for (; i < insn->length;) bytes[i++] =INT3_INSN_OPCODE ; -return i; }
arch/x86/kernel/ftrace.c +1 -1
@@ -358,7 +358,7 @@ create_trampoline(struct ftrace_ops *ops ip = trampoline + size;if (cpu_feature_enabled (X86_FEATURE_RETHUNK )) -__text_gen_insn (ip,JMP32_INSN_OPCODE , ip, &__x86_return_thunk,JMP32_INSN_SIZE ); +__text_gen_insn (ip,JMP32_INSN_OPCODE , ip, x86_return_thunk,JMP32_INSN_SIZE );else memcpy (ip, retq,sizeof (retq));
arch/x86/kernel/static_call.c +1 -1
@@ -52,7 +52,7 @@ static void __ref __static_call_transforcase RET :if (cpu_feature_enabled (X86_FEATURE_RETHUNK )) - code =text_gen_insn (JMP32_INSN_OPCODE , insn, &__x86_return_thunk); + code =text_gen_insn (JMP32_INSN_OPCODE , insn, x86_return_thunk);else code = &retinsn;break ;
arch/x86/net/bpf_jit_comp.c +1 -1
@@ -430,7 +430,7 @@ static void emit_return(u8 **pprog, u8 *u8 *prog = *pprog;if (cpu_feature_enabled (X86_FEATURE_RETHUNK )) { -emit_jump (&prog, &__x86_return_thunk, ip); +emit_jump (&prog, x86_return_thunk, ip); }else {EMIT1 (0xC3 );/* ret */ if (IS_ENABLED (CONFIG_SLS ))
PATCH 27/38 x86/asm: Provide ALTERNATIVE_3
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Fairly straight forward adaptation/extention of ALTERNATIVE_2.
Required for call depth tracking.
--- arch/x86/include/asm/alternative.h | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)
arch/x86/include/asm/alternative.h +30 -3
@@ -367,6 +367,7 @@ static inline int alternatives_text_rese#define old_len 141b-140b#define new_len1 144f-143f#define new_len2 145f-144f +#define new_len3 146f-145f/* * gas compatible max based on the idea from:
@@ -374,7 +375,8 @@ static inline int alternatives_text_rese* * The additional "-" is needed because gas uses a "true" value of -1. */ -#define alt_max_short (a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b))))) +#define alt_max_2 (a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b))))) +#define alt_max_3 (a, b, c) (alt_max_2(alt_max_2(a, b), c))/*
@@ -386,8 +388,8 @@ static inline int alternatives_text_rese 140: \oldinstr 141: - .skip -((alt_max_short (new_len1, new_len2) - (old_len)) >0 ) * \ - (alt_max_short (new_len1, new_len2) - (old_len)),0x90 + .skip -((alt_max_2 (new_len1, new_len2) - (old_len)) >0 ) * \ + (alt_max_2 (new_len1, new_len2) - (old_len)),0x90 142: .pushsection .altinstructions,"a"
@@ -404,6 +406,31 @@ static inline int alternatives_text_rese .popsection .endm +.macro ALTERNATIVE_3 oldinstr, newinstr1, feature1, newinstr2, feature2, newinstr3, feature3 +140: + \oldinstr +141: + .skip -((alt_max_3(new_len1, new_len2, new_len3) - (old_len)) > 0) * \ + (alt_max_3(new_len1, new_len2, new_len3) - (old_len)),0x90 +142: + + .pushsection .altinstructions,"a" + altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f + altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f + altinstruction_entry 140b,145f,\feature3,142b-140b,146f-145f + .popsection + + .pushsection .altinstr_replacement,"ax" +143: + \newinstr1 +144: + \newinstr2 +145: + \newinstr3 +146: + .popsection +.endm + /* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY(oldinstr, feature, newinstr_yes, newinstr_no) \ ALTERNATIVE_2 oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \
PATCH 28/38 x86/retbleed: Add SKL return thunk
In reply to Thomas GleixnerTo address the Intel SKL RSB underflow issue in software it's required to do call depth tracking.
Provide a return thunk for call depth tracking on Intel SKL CPUs.
The tracking does not use a counter. It uses uses arithmetic shift right on call entry and logical shift left on return.
The depth tracking variable is initialized to 0x8000.... when the call depth is zero. The arithmetic shift right sign extends the MSB and saturates after the 12th call. The shift count is 5 so the tracking covers 12 nested calls. On return the variable is shifted left logically so it becomes zero again.
CALL RET
0: 0x8000000000000000 0x0000000000000000
1: 0xfc00000000000000 0xf000000000000000
...
11: 0xfffffffffffffff8 0xfffffffffffffc00
12: 0xffffffffffffffff 0xffffffffffffffe0
After a return buffer fill the depth is credited 12 calls before the next stuffing has to take place.
There is a inaccuracy for situations like this:
10 calls
5 returns
3 calls
4 returns
3 calls
....The shift count might cause this to be off by one in either direction, but there is still a cushion vs. the RSB depth. The algorithm does not claim to be perfect, but it should obfuscate the problem enough to make exploitation extremly difficult.
The theory behind this is:
RSB is a stack with depth 16 which is filled on every call. On the return path speculation "pops" entries to speculate down the call chain. Once the speculative RSB is empty it switches to other predictors, e.g. the Branch History Buffer, which can be mistrained by user space and misguide the speculation path to a gadget.
Call depth tracking is designed to break this speculation path by stuffing speculation trap calls into the RSB which are never getting a corresponding return executed. This stalls the prediction path until it gets resteered,
The assumption is that stuffing at the 12th return is sufficient to break the speculation before it hits the underflow and the fallback to the other predictors. Testing confirms that it works. Johannes, one of the retbleed researchers. tried to attack this approach but failed.
There is obviously no scientific proof that this will withstand future research progress, but all we can do right now is to speculate about it.
The SAR/SHL usage was suggested by Andi Kleen.
--- arch/x86/entry/entry_64.S | 10 +-- arch/x86/include/asm/nospec-branch.h | 114 +++++++++++++++++++++++++++++++++-- arch/x86/kernel/cpu/common.c | 5 + arch/x86/lib/retpoline.S | 30 +++++++++ 4 files changed, 149 insertions(+), 10 deletions(-)
arch/x86/entry/entry_64.S +6 -4
@@ -287,6 +287,7 @@ SYM_FUNC_END(__switch_to_asm) SYM_CODE_START(ret_from_fork) UNWIND_HINT_EMPTY ANNOTATE_NOENDBR // copy_thread + CALL_DEPTH_ACCOUNT movq %rax, %rdi call schedule_tail /* rdi: 'prev' task parameter */
@@ -331,7 +332,7 @@ SYM_CODE_START(xen_error_entry) UNWIND_HINT_FUNC PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8 - UNTRAIN_RET + UNTRAIN_RET_FROM_CALL RET SYM_CODE_END(xen_error_entry)
@@ -975,7 +976,7 @@ SYM_CODE_START(paranoid_entry) * CR3 above, keep the old value in a callee saved register. */ IBRS_ENTER save_reg=%r15 - UNTRAIN_RET + UNTRAIN_RET_FROM_CALL RET SYM_CODE_END(paranoid_entry)
@@ -1060,7 +1061,7 @@ SYM_CODE_START(error_entry) /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rax IBRS_ENTER - UNTRAIN_RET + UNTRAIN_RET_FROM_CALL leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ /* Put us onto the real thread stack. */
@@ -1095,6 +1096,7 @@ SYM_CODE_START(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY + CALL_DEPTH_ACCOUNT leaq 8(%rsp), %rax /* return pt_regs pointer */ ANNOTATE_UNRET_END RET
@@ -1113,7 +1115,7 @@ SYM_CODE_START(error_entry) FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=%rax IBRS_ENTER - UNTRAIN_RET + UNTRAIN_RET_FROM_CALL /* * Pretend that the exception came from user mode: set up pt_regs
arch/x86/include/asm/nospec-branch.h +108 -6
147 lines not on this page · raw message
arch/x86/kernel/cpu/common.c +5 -0
@@ -2002,6 +2002,11 @@ EXPORT_PER_CPU_SYMBOL(__preempt_count); DEFINE_PER_CPU(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK; +#ifdef CONFIG_CALL_DEPTH_TRACKING +DEFINE_PER_CPU (u64, __x86_call_depth); +EXPORT_PER_CPU_SYMBOL_GPL (__x86_call_depth); +#endif +static void wrmsrl_cstar (unsigned long val) {/*
arch/x86/lib/retpoline.S +30 -0
@@ -8,6 +8,7 @@ #include <asm/export.h> #include <asm/nospec-branch.h> #include <asm/unwind_hints.h> +#include <asm/percpu.h> #include <asm/frame.h> .section .text.__x86.indirect_thunk
@@ -140,3 +141,32 @@ SYM_FUNC_END(zen_untrain_ret) EXPORT_SYMBOL(__x86_return_thunk) #endif /* CONFIG_RETHUNK */ + +#ifdef CONFIG_CALL_DEPTH_TRACKING + + .align 64 +SYM_FUNC_START(__x86_return_skl) + ANNOTATE_NOENDBR + /* Keep the hotpath in a 16byte I-fetch */ + shlq $5, PER_CPU_VAR(__x86_call_depth) + jz 1f + ANNOTATE_UNRET_SAFE + ret + int3 +1: + .rept 16 + ANNOTATE_INTRA_FUNCTION_CALL + call 2f + int3 +2: + .endr + add $(8*16), %rsp + + CREDIT_CALL_DEPTH + + ANNOTATE_UNRET_SAFE + ret + int3 +SYM_FUNC_END(__x86_return_skl) + +#endif /* CONFIG_CALL_DEPTH_TRACKING */
PATCH 29/38 x86/retpoline: Add SKL retthunk retpolines
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Ensure that retpolines do the proper call accounting so that the return accounting works correctly.
Specifically; retpolines are used to replace both 'jmp *%reg' and 'call *%reg', however these two cases do not have the same accounting requirements. Therefore split things up and provide two different retpoline arrays for SKL.
The 'jmp *%reg' case needs no accounting, the __x86_indirect_jump_thunk_array[] covers this. The retpoline is changed to not use the return thunk; it's a simple call;ret construct.
[ strictly speaking it should do:
andq $(~0x1f), PER_CPU_VAR(__x86_call_depth)
but we can argue this can be covered by the fuzz we already have in the accounting depth (12) vs the RSB depth (16) ]
The 'call *%reg' case does need accounting, the __x86_indirect_call_thunk_array[] covers this. Again, this retpoline avoids the use of the return-thunk, in this case to avoid double accounting.
--- arch/x86/include/asm/nospec-branch.h | 12 +++++ arch/x86/kernel/alternative.c | 43 +++++++++++++++++++-- arch/x86/lib/retpoline.S | 71 +++++++++++++++++++++++++++++++---- arch/x86/net/bpf_jit_comp.c | 5 +- 4 files changed, 119 insertions(+), 12 deletions(-)
arch/x86/include/asm/nospec-branch.h +12 -0
@@ -252,6 +252,8 @@typedef u8 retpoline_thunk_t [RETPOLINE_THUNK_SIZE ];extern retpoline_thunk_t __x86_indirect_thunk_array[]; +extern retpoline_thunk_t __x86_indirect_call_thunk_array[]; +extern retpoline_thunk_t __x86_indirect_jump_thunk_array[];extern void __x86_return_thunk (void );extern void zen_untrain_ret (void );
@@ -283,6 +285,16 @@ static inline void x86_set_skl_return_th#include <asm/GEN-for-each-reg.h> #undef GEN +#define GEN (reg) \ + extern retpoline_thunk_t __x86_indirect_call_thunk_ ## reg; +#include <asm/GEN-for-each-reg.h> +#undef GEN + +#define GEN (reg) \ + extern retpoline_thunk_t __x86_indirect_jump_thunk_ ## reg; +#include <asm/GEN-for-each-reg.h> +#undef GEN +#ifdef CONFIG_X86_64 /*
arch/x86/kernel/alternative.c +40 -3
57 lines not on this page · raw message
arch/x86/lib/retpoline.S +63 -8
97 lines not on this page · raw message
arch/x86/net/bpf_jit_comp.c +4 -1
@@ -417,7 +417,10 @@ static void emit_indirect_jump(u8 **pproEMIT2 (0xFF ,0xE0 + reg); }else if (cpu_feature_enabled (X86_FEATURE_RETPOLINE )) {OPTIMIZER_HIDE_VAR (reg); -emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); +if (cpu_feature_enabled (X86_FEATURE_CALL_DEPTH )) +emit_jump (&prog, &__x86_indirect_jump_thunk_array[reg], ip); +else +emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else {EMIT2 (0xFF ,0xE0 + reg); }
PATCH 30/38 x86/retbleed: Add SKL call thunk
In reply to Thomas GleixnerAdd the actual SKL call thunk for call depth accounting.
--- arch/x86/kernel/callthunks.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/x86/kernel/callthunks.c +24 -0
@@ -52,6 +52,24 @@ struct thunk_desc {static struct thunk_desc callthunk_desc__ro_after_init ; +asm ( +".pushsection .rodata \n" +".global skl_call_thunk_template \n" +"skl_call_thunk_template: \n" + __stringify(INCREMENT_CALL_DEPTH)" \n" +".global skl_call_thunk_tail \n" +"skl_call_thunk_tail: \n" +".popsection \n" +); + +extern u8 skl_call_thunk_template[]; +extern u8 skl_call_thunk_tail[]; + +#define SKL_TMPL_SIZE \ + ((unsigned int)(skl_call_thunk_tail - skl_call_thunk_template)) +#define SKL_CALLTHUNK_CODE_SIZE (SKL_TMPL_SIZE + JMP32_INSN_SIZE + INT3_INSN_SIZE) +#define SKL_CALLTHUNK_SIZE roundup_pow_of_two(SKL_CALLTHUNK_CODE_SIZE) +struct thunk_mem {void *base;unsigned int size;
@@ -447,6 +465,12 @@ static __init noinline void callthunks_i {int ret; +if (cpu_feature_enabled (X86_FEATURE_CALL_DEPTH )) { + callthunk_desc.template = skl_call_thunk_template; + callthunk_desc.template_size =SKL_TMPL_SIZE ; + callthunk_desc.thunk_size =SKL_CALLTHUNK_SIZE ; + } +if (!callthunk_desc.template)return ;
PATCH 31/38 x86/calldepth: Add ret/call counting for debug
In reply to Thomas GleixnerAdd a debuigfs mechanism to validate the accounting, e.g. vs. call/ret balance and to gather statistics about the stuffing to call ratio.
--- arch/x86/include/asm/nospec-branch.h | 32 +++++++++++++++++++-- arch/x86/kernel/callthunks.c | 51 +++++++++++++++++++++++++++++++++++ arch/x86/lib/retpoline.S | 7 ++++ 3 files changed, 86 insertions(+), 4 deletions(-)
arch/x86/include/asm/nospec-branch.h +29 -3
@@ -77,6 +77,23 @@*/ #ifdef CONFIG_CALL_DEPTH_TRACKING + +#ifdef CONFIG_CALL_THUNKS_DEBUG +# define CALL_THUNKS_DEBUG_INC_CALLS \ + incq %gs:__x86_call_count; +# define CALL_THUNKS_DEBUG_INC_RETS \ + incq %gs:__x86_ret_count; +# define CALL_THUNKS_DEBUG_INC_STUFFS \ + incq %gs:__x86_stuffs_count; +# define CALL_THUNKS_DEBUG_INC_CTXSW \ + incq %gs:__x86_ctxsw_count; +#else +# define CALL_THUNKS_DEBUG_INC_CALLS +# define CALL_THUNKS_DEBUG_INC_RETS +# define CALL_THUNKS_DEBUG_INC_STUFFS +# define CALL_THUNKS_DEBUG_INC_CTXSW +#endif +#define CREDIT_CALL_DEPTH \ movq $-1, PER_CPU_VAR(__x86_call_depth);
@@ -88,10 +105,12 @@#define RESET_CALL_DEPTH_FROM_CALL \ mov $0xfc, %rax; \ shl $56, %rax; \ - movq %rax, PER_CPU_VAR(__x86_call_depth); + movq %rax, PER_CPU_VAR(__x86_call_depth); \ + CALL_THUNKS_DEBUG_INC_CALLS#define INCREMENT_CALL_DEPTH \ - sarq $5, %gs:__x86_call_depth + sarq $5, %gs:__x86_call_depth; \ + CALL_THUNKS_DEBUG_INC_CALLS#else #define CREDIT_CALL_DEPTH #define RESET_CALL_DEPTH
@@ -127,7 +146,8 @@dec reg; \ jnz 771b; \ \ -CREDIT_CALL_DEPTH +CREDIT_CALL_DEPTH \ + CALL_THUNKS_DEBUG_INC_CTXSW#ifdef __ASSEMBLY__
@@ -274,6 +294,12 @@ static inline void x86_set_skl_return_th }DECLARE_PER_CPU (u64, __x86_call_depth); +#ifdef CONFIG_CALL_THUNKS_DEBUG +DECLARE_PER_CPU (u64, __x86_call_count); +DECLARE_PER_CPU (u64, __x86_ret_count); +DECLARE_PER_CPU (u64, __x86_stuffs_count); +DECLARE_PER_CPU (u64, __x86_ctxsw_count); +#endif #else static inline void x86_set_skl_return_thunk (void ) {}#endif
arch/x86/kernel/callthunks.c +51 -0
66 lines not on this page · raw message
arch/x86/lib/retpoline.S +6 -1
@@ -202,13 +202,18 @@ EXPORT_SYMBOL(__x86_return_thunk) .align 64 SYM_FUNC_START(__x86_return_skl) ANNOTATE_NOENDBR - /* Keep the hotpath in a 16byte I-fetch */ + /* + * Keep the hotpath in a 16byte I-fetch for the non-debug + * case. + */ + CALL_THUNKS_DEBUG_INC_RETS shlq $5, PER_CPU_VAR(__x86_call_depth) jz 1f ANNOTATE_UNRET_SAFE ret int3 1: + CALL_THUNKS_DEBUG_INC_STUFFS .rept 16 ANNOTATE_INTRA_FUNCTION_CALL call 2f
PATCH 32/38 static_call: Add call depth tracking support
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
When indirect calls are switched to direct calls then it has to be ensured that the call target is not the function, but the call thunk when call depth tracking is enabled. But static calls are available before call thunks have been set up.
Ensure a second run through the static call patching code after call thunks have been created. When call thunks are not enabled this has no side effects.
--- arch/x86/include/asm/alternative.h | 5 +++++ arch/x86/kernel/callthunks.c | 37 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/static_call.c | 1 + include/linux/static_call.h | 2 ++ kernel/static_call_inline.c | 23 ++++++++++++++++++----- 5 files changed, 63 insertions(+), 5 deletions(-)
arch/x86/include/asm/alternative.h +5 -0
@@ -93,12 +93,17 @@ extern void callthunks_patch_builtin_calextern void callthunks_patch_module_calls (struct callthunk_sites *sites,struct module *mod);extern void callthunks_module_free (struct module *mod); +extern void *callthunks_translate_call_dest (void *dest);#else static __always_inline void callthunks_patch_builtin_calls(void) {} static __always_inline voidcallthunks_patch_module_calls (struct callthunk_sites *sites,struct module *mod) {} static __always_inline void callthunks_module_free(struct module *mod) { } +static __always_inline void *callthunks_translate_call_dest(void *dest) +{ +return dest; +}#endif #ifdef CONFIG_SMP
arch/x86/kernel/callthunks.c +37 -0
@@ -7,6 +7,7 @@#include <linux/memory.h> #include <linux/moduleloader.h> #include <linux/set_memory.h> +#include <linux/static_call.h> #include <linux/vmalloc.h> #include <asm/alternative.h>
@@ -492,6 +493,7 @@ static __init noinline void callthunks_iif (WARN_ON_ONCE (ret))return ; +static_call_force_reinit (); thunks_initialized = true; }
@@ -511,6 +513,41 @@ void __init callthunks_patch_builtin_calmutex_unlock (&text_mutex); } +static bool is_module_init_dest (void *dest) +{ +bool ret = false; + +#ifdef CONFIG_MODULES +struct module *mod; + +preempt_disable (); + mod =__module_address ((unsigned long )dest); +if (mod &&within_module_init ((unsigned long )dest, mod)) + ret = true; +preempt_enable (); +#endif +return ret; +} + +void *callthunks_translate_call_dest (void *dest) +{ +void *thunk; + +lockdep_assert_held (&text_mutex); + +if (!thunks_initialized ||skip_addr (dest)) +return dest; + + thunk =btree_lookup64 (&call_thunks, (unsigned long )dest); + +if (thunk) +return thunk; + +WARN_ON_ONCE (!is_kernel_inittext ((unsigned long )dest) && + !is_module_init_dest (dest)); +return dest; +} +#ifdef CONFIG_MODULES void noinline callthunks_patch_module_calls(struct callthunk_sites *cs,struct module *mod)
arch/x86/kernel/static_call.c +1 -0
@@ -34,6 +34,7 @@ static void __ref __static_call_transforswitch (type) {case CALL : + func =callthunks_translate_call_dest (func); code =text_gen_insn (CALL_INSN_OPCODE , insn, func);if (func == &__static_call_return0) { emulate = code;
include/linux/static_call.h +2 -0
@@ -162,6 +162,8 @@ extern void arch_static_call_transform(vextern int __init static_call_init (void ); +extern void static_call_force_reinit (void ); +struct static_call_mod {struct static_call_mod *next;struct module *mod;/* for vmlinux, mod == NULL */
kernel/static_call_inline.c +18 -5
@@ -15,7 +15,18 @@ extern struct static_call_site __start_sextern struct static_call_tramp_key __start_static_call_tramp_key[], __stop_static_call_tramp_key[]; -static bool static_call_initialized; +static int static_call_initialized; + +/* +* Must be called before early_initcall() to be effective. +*/ +void static_call_force_reinit (void ) +{ +if (WARN_ON_ONCE (!static_call_initialized)) +return ; + + static_call_initialized++; +}/* mutex to protect key modules/sites */ static DEFINE_MUTEX (static_call_mutex);
@@ -475,7 +486,8 @@ int __init static_call_init(void) {int ret; -if (static_call_initialized) +/* See static_call_force_reinit(). */ +if (static_call_initialized ==1 )return 0 ;cpus_read_lock ();
@@ -490,11 +502,12 @@ int __init static_call_init(void)BUG (); } - static_call_initialized = true; -#ifdef CONFIG_MODULES -register_module_notifier (&static_call_module_nb); +if (!static_call_initialized) +register_module_notifier (&static_call_module_nb);#endif + + static_call_initialized =1 ;return 0 ; }early_initcall (static_call_init);
PATCH 33/38 kallsyms: Take callthunks into account
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Similar to ftrace and bpf call thunks are creating symbols which are interesting for things like printing stack-traces, perf, live-patching and things like that.
Add the required functionality to the core and implement it in x86.
Callthunks will report the same function name as their target, but their module name will be "callthunk" or "callthunk:${modname}" for modules.
--- arch/x86/kernel/callthunks.c | 155 +++++++++++++++++++++++++++++++++++++++++++ include/linux/kallsyms.h | 24 ++++++ kernel/kallsyms.c | 23 ++++++ 3 files changed, 202 insertions(+)
arch/x86/kernel/callthunks.c +155 -0
@@ -4,6 +4,7 @@#include <linux/btree.h> #include <linux/debugfs.h> +#include <linux/kallsyms.h> #include <linux/memory.h> #include <linux/moduleloader.h> #include <linux/set_memory.h>
@@ -548,6 +549,160 @@ void *callthunks_translate_call_dest(voireturn dest; } +static bool is_module_callthunk (void *addr) +{ +bool ret = false; + +#ifdef CONFIG_MODULES +struct module *mod; + +preempt_disable (); + mod =__module_address ((unsigned long )addr); +if (mod &&within_module_thunk ((unsigned long )addr, mod)) + ret = true; +preempt_enable (); +#endif +return ret; +} + +static bool is_callthunk (void *addr) +{ +if (builtin_layout.base <= addr && + addr < builtin_layout.base + builtin_layout.size) +return true; +return is_module_callthunk (addr); +} + +static void *__callthunk_dest (void *addr) +{ +unsigned long mask = callthunk_desc.thunk_size -1 ; +void *thunk; + + thunk = (void *)((unsigned long )addr & ~mask); + thunk += callthunk_desc.template_size; +return jump_get_dest (thunk); +} + +static void *callthunk_dest (void *addr) +{ +if (!is_callthunk (addr)) +return NULL ; +return __callthunk_dest (addr); +} + +static void set_modname (char **modname,unsigned long addr) +{ +if (!modname || !IS_ENABLED (CONFIG_MODULES )) + *modname ="callthunk" ; + +#ifdef CONFIG_MODULES + else { +struct module * mod; + +preempt_disable (); + mod =__module_address (addr); + *modname = mod->callthunk_name; +preempt_enable (); + } +#endif +} + +const char * +callthunk_address_lookup (unsigned long addr,unsigned long *size, +unsigned long *off,char **modname,char *sym) +{ +unsigned long dest, mask = callthunk_desc.thunk_size -1 ; +const char *ret; + +if (!thunks_initialized) +return NULL ; + + dest = (unsigned long )callthunk_dest ((void *)addr); +if (!dest) +return NULL ; + + ret =kallsyms_lookup (dest, size, off, modname, sym); +if (!ret) +return NULL ; + + *off = addr & mask; + *size = callthunk_desc.thunk_size; + +set_modname (modname, addr); +return ret; +} + +static int get_module_thunk (char **modname,struct module_layout **layoutp, +unsigned int symthunk) +{ +#ifdef CONFIG_MODULES +extern struct list_head modules; +struct module *mod; +unsigned int size; + + symthunk -= (*layoutp)->text_size; +list_for_each_entry_rcu (mod, &modules, list) { +if (mod->state ==MODULE_STATE_UNFORMED ) +continue ; + + *layoutp = &mod->thunk_layout; + size = mod->thunk_layout.text_size; + +if (symthunk >= size) { + symthunk -= size; +continue ; + } + *modname = mod->callthunk_name; +return symthunk; + } +#endif +return -ERANGE ; +} + +int callthunk_get_kallsym (unsigned int symnum,unsigned long *value, +char *type,char *name,char *module_name, +int *exported) +{ +int symthunk = symnum * callthunk_desc.thunk_size; +struct module_layout *layout = &builtin_layout; +char *modname ="callthunk" ; +void *thunk, *dest; +int ret = -ERANGE ; + +if (!thunks_initialized) +return -ERANGE ; + +preempt_disable (); + +if (symthunk >= layout->text_size) { + symthunk =get_module_thunk (&modname, &layout, symthunk); +if (symthunk <0 ) + goto out; + } + + thunk = layout->base + symthunk; + dest =__callthunk_dest (thunk); + +if (!dest) { +strlcpy (name,"(unknown callthunk)" ,KSYM_NAME_LEN ); + ret =0 ; + goto out; + } + + ret =lookup_symbol_name ((unsigned long )dest, name); +if (ret) + goto out; + + *value = (unsigned long )thunk; + *exported =0 ; + *type ='t' ; +strlcpy (module_name, modname,MODULE_NAME_LEN ); + +out: +preempt_enable (); +return ret; +} +#ifdef CONFIG_MODULES void noinline callthunks_patch_module_calls(struct callthunk_sites *cs,struct module *mod)
include/linux/kallsyms.h +24 -0
@@ -65,6 +65,30 @@ static inline void *dereference_symbol_dreturn ptr; } +#ifdef CONFIG_CALL_THUNKS +extern const char * +callthunk_address_lookup (unsigned long addr,unsigned long *size, +unsigned long *off,char **modname,char *sym); +extern int callthunk_get_kallsym (unsigned int symnum,unsigned long *value, +char *type,char *name,char *module_name, +int *exported); +#else +static inline const char * +callthunk_address_lookup (unsigned long addr,unsigned long *size, +unsigned long *off,char **modname,char *sym) +{ +return NULL ; +} + +static inline +int callthunk_get_kallsym (unsigned int symnum,unsigned long *value, +char *type,char *name,char *module_name, +int *exported) +{ +return -1 ; +} +#endif +#ifdef CONFIG_KALLSYMS int kallsyms_on_each_symbol(int (*fn)(void *, const char *, struct module *, unsigned long),
kernel/kallsyms.c +23 -0
@@ -365,6 +365,10 @@ static const char *kallsyms_lookup_build ret =ftrace_mod_address_lookup (addr, symbolsize, offset, modname, namebuf); +if (!ret) + ret =callthunk_address_lookup (addr, symbolsize, + offset, modname, namebuf); + found:cleanup_symbol_name (namebuf);return ret;
@@ -578,6 +582,7 @@ struct kallsym_iter {loff_t pos_mod_end;loff_t pos_ftrace_mod_end;loff_t pos_bpf_end; +loff_t pos_callthunk_end;unsigned long value;unsigned int nameoff;/* If iterating in core kernel symbols. */ char type;
@@ -657,6 +662,20 @@ static int get_ksymbol_bpf(struct kallsyreturn 1 ; } +static int get_ksymbol_callthunk (struct kallsym_iter *iter) +{ +int ret =callthunk_get_kallsym (iter->pos - iter->pos_bpf_end, + &iter->value, &iter->type, + iter->name, iter->module_name, + &iter->exported); +if (ret <0 ) { + iter->pos_callthunk_end = iter->pos; +return 0 ; + } + +return 1 ; +} +/* * This uses "__builtin__kprobes" as a module name for symbols for pages * allocated for kprobes' purposes, even though "__builtin__kprobes" is not a
@@ -724,6 +743,10 @@ static int update_iter_mod(struct kallsy get_ksymbol_bpf(iter))return 1 ; +if ((!iter->pos_callthunk_end || iter->pos_callthunk_end > pos) && +get_ksymbol_callthunk (iter)) +return 1 ; +return get_ksymbol_kprobe (iter); }
PATCH 34/38 x86/orc: Make it callthunk aware
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Callthunks addresses on the stack would confuse the ORC unwinder. Handle them correctly and tell ORC to proceed further down the stack.
--- arch/x86/include/asm/alternative.h | 5 +++++ arch/x86/kernel/callthunks.c | 2 +- arch/x86/kernel/unwind_orc.c | 21 ++++++++++++++++++++- 3 files changed, 26 insertions(+), 2 deletions(-)
arch/x86/include/asm/alternative.h +5 -0
@@ -94,6 +94,7 @@ extern void callthunks_patch_module_call struct module *mod);extern void callthunks_module_free (struct module *mod);extern void *callthunks_translate_call_dest (void *dest); +extern bool is_callthunk (void *addr);#else static __always_inline void callthunks_patch_builtin_calls(void) {}static __always_inline void
@@ -104,6 +105,10 @@ static __always_inline void *callthunks_ {return dest; } +static __always_inline bool is_callthunk(void *addr) +{ +return false; +}#endif #ifdef CONFIG_SMP
arch/x86/kernel/callthunks.c +1 -1
@@ -565,7 +565,7 @@ static bool is_module_callthunk(void *adreturn ret; } -static bool is_callthunk (void *addr) +bool is_callthunk (void *addr) {if (builtin_layout.base <= addr && addr < builtin_layout.base + builtin_layout.size)
arch/x86/kernel/unwind_orc.c +20 -1
@@ -131,6 +131,21 @@ static struct orc_entry null_orc_entry = .type = UNWIND_HINT_TYPE_CALL }; +#ifdef CONFIG_CALL_THUNKS +static struct orc_entry *orc_callthunk_find (unsigned long ip) +{ +if (!is_callthunk ((void *)ip)) +return NULL ; + +return &null_orc_entry; +} +#else +static struct orc_entry *orc_callthunk_find (unsigned long ip) +{ +return NULL ; +} +#endif +/* Fake frame pointer entry -- used as a fallback for generated code */ static struct orc_entry orc_fp_entry = { .type =UNWIND_HINT_TYPE_CALL ,
@@ -184,7 +199,11 @@ static struct orc_entry *orc_find(unsignif (orc)return orc; -return orc_ftrace_find (ip); + orc =orc_ftrace_find (ip); +if (orc) +return orc; + +return orc_callthunk_find (ip); }#ifdef CONFIG_MODULES
PATCH 35/38 kprobes: Add callthunk blacklisting
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Callthunks are not safe for probing. Add them to the kprobes black listed areas.
--- arch/x86/kernel/callthunks.c | 5 ++++ kernel/kprobes.c | 52 +++++++++++++++++++++++++++---------------- 2 files changed, 38 insertions(+), 19 deletions(-)
arch/x86/kernel/callthunks.c +5 -0
@@ -6,6 +6,7 @@#include <linux/debugfs.h> #include <linux/kallsyms.h> #include <linux/memory.h> +#include <linux/kprobes.h> #include <linux/moduleloader.h> #include <linux/set_memory.h> #include <linux/static_call.h>
@@ -476,6 +477,7 @@ static __init_or_module int callthunks_s static __init noinline void callthunks_init(struct callthunk_sites *cs) { +unsigned long base, size;int ret;if (cpu_feature_enabled (X86_FEATURE_CALL_DEPTH )) {
@@ -494,6 +496,9 @@ static __init noinline void callthunks_iif (WARN_ON_ONCE (ret))return ; + base = (unsigned long )builtin_layout.base; + size = builtin_layout.size; +kprobe_add_area_blacklist (base, base + size);static_call_force_reinit (); thunks_initialized = true; }
kernel/kprobes.c +33 -19
85 lines not on this page · raw message
PATCH 36/38 x86/ftrace: Make it call depth tracking aware
In reply to Thomas GleixnerFrom: Peter Zijlstra <[email protected]>
Since ftrace has trampolines, don't use thunks for the __fentry__ site but instead require that every function called from there includes accounting. This very much includes all the direct-call functions.
Additionally, ftrace uses ROP tricks in two places:
- return_to_handler(), and
- ftrace_regs_caller() when pt_regs->orig_ax is set by a direct-call.
return_to_handler() already uses a retpoline to replace an indirect-jump to defeat IBT, since this is a jump-type retpoline, make sure there is no accounting done and ALTERNATIVE the RET into a ret.
ftrace_regs_caller() does much the same but currently causes an RSB imbalance by effectively doing a PUSH+RET combo, rebalance.
--- arch/x86/include/asm/nospec-branch.h | 8 +++++++ arch/x86/kernel/ftrace.c | 16 ++++++++++---- arch/x86/kernel/ftrace_64.S | 31 ++++++++++++++++++++++++++-- arch/x86/net/bpf_jit_comp.c | 6 +++++ kernel/trace/trace_selftest.c | 5 +++- samples/ftrace/ftrace-direct-modify.c | 2 + samples/ftrace/ftrace-direct-multi-modify.c | 2 + samples/ftrace/ftrace-direct-multi.c | 1 samples/ftrace/ftrace-direct-too.c | 1 samples/ftrace/ftrace-direct.c | 1 10 files changed, 66 insertions(+), 7 deletions(-)
arch/x86/include/asm/nospec-branch.h +8 -0
@@ -293,6 +293,11 @@ static inline void x86_set_skl_return_th x86_return_thunk = &__x86_return_skl; } +#define CALL_DEPTH_ACCOUNT \ + ALTERNATIVE("", \ + __stringify(INCREMENT_CALL_DEPTH), \ + X86_FEATURE_CALL_DEPTH) +DECLARE_PER_CPU (u64, __x86_call_depth);#ifdef CONFIG_CALL_THUNKS_DEBUG DECLARE_PER_CPU (u64, __x86_call_count);
@@ -302,6 +307,9 @@ DECLARE_PER_CPU(u64, __x86_ctxsw_count);#endif #else static inline void x86_set_skl_return_thunk (void ) {} + +#define CALL_DEPTH_ACCOUNT +#endif #ifdef CONFIG_RETPOLINE
arch/x86/kernel/ftrace.c +12 -4
@@ -69,6 +69,10 @@ static const char *ftrace_nop_replace(vostatic const char *ftrace_call_replace (unsigned long ip,unsigned long addr) { +/* +* No need to translate into a callthunk. The trampoline does +* the depth accounting itself. +*/ return text_gen_insn (CALL_INSN_OPCODE , (void *)ip, (void *)addr); }
@@ -316,7 +320,7 @@ create_trampoline(struct ftrace_ops *opsunsigned long size;unsigned long *ptr;void *trampoline; -void *ip; +void *ip, *dest;/* 48 8b 15 <offset> is movq <offset>(%rip), %rdx */ unsigned const char op_ref[] = {0x48 ,0x8b ,0x15 };unsigned const char retq[] = {RET_INSN_OPCODE ,INT3_INSN_OPCODE };
@@ -403,10 +407,14 @@ create_trampoline(struct ftrace_ops *ops/* put in the call to the function */ mutex_lock (&text_mutex); call_offset -= start_offset; +/* +* No need to translate into a callthunk. The trampoline does +* the depth accounting before the call already. +*/ + dest =ftrace_ops_get_func (ops);memcpy (trampoline + call_offset, -text_gen_insn (CALL_INSN_OPCODE , - trampoline + call_offset, -ftrace_ops_get_func (ops)),CALL_INSN_SIZE ); +text_gen_insn (CALL_INSN_OPCODE , trampoline + call_offset, dest), +CALL_INSN_SIZE );mutex_unlock (&text_mutex);/* ALLOC_TRAMP flags lets us know we created it */
arch/x86/kernel/ftrace_64.S +29 -2
@@ -132,6 +132,7 @@ #ifdef CONFIG_DYNAMIC_FTRACE SYM_FUNC_START(__fentry__) + CALL_DEPTH_ACCOUNT RET SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__)
@@ -140,6 +141,8 @@ SYM_FUNC_START(ftrace_caller) /* save_mcount_regs fills in first two parameters */ save_mcount_regs + CALL_DEPTH_ACCOUNT + /* Stack - skipping return address of ftrace_caller */ leaq MCOUNT_REG_SIZE+8(%rsp), %rcx movq %rcx, RSP(%rsp)
@@ -155,6 +158,9 @@ SYM_INNER_LABEL(ftrace_caller_op_ptr, SY /* Only ops with REGS flag set should have CS register set */ movq $0, CS(%rsp) + /* Account for the function call below */ + CALL_DEPTH_ACCOUNT + SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub
@@ -195,6 +201,8 @@ SYM_FUNC_START(ftrace_regs_caller) save_mcount_regs 8 /* save_mcount_regs fills in first two parameters */ + CALL_DEPTH_ACCOUNT + SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL) ANNOTATE_NOENDBR /* Load the ftrace_ops into the 3rd parameter */
@@ -225,6 +233,9 @@ SYM_INNER_LABEL(ftrace_regs_caller_op_pt /* regs go into 4th parameter */ leaq (%rsp), %rcx + /* Account for the function call below */ + CALL_DEPTH_ACCOUNT + SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub
@@ -280,7 +291,19 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, /* Restore flags */ popfq UNWIND_HINT_FUNC - jmp ftrace_epilogue + + /* + * Since we're effectively emulating a tail-call with PUSH;RET + * make sure we don't unbalance the RSB and mess up accounting. + */ + ANNOTATE_INTRA_FUNCTION_CALL + call 2f + int3 +2: + add $8, %rsp + ALTERNATIVE __stringify(RET), \ + __stringify(ANNOTATE_UNRET_SAFE; ret; int3), \ + X86_FEATURE_CALL_DEPTH SYM_FUNC_END(ftrace_regs_caller) STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller)
@@ -289,6 +312,8 @@ STACK_FRAME_NON_STANDARD_FP(ftrace_regs_ #else /* ! CONFIG_DYNAMIC_FTRACE */ SYM_FUNC_START(__fentry__) + CALL_DEPTH_ACCOUNT + cmpq $ftrace_stub, ftrace_trace_function jnz trace
@@ -345,6 +370,8 @@ SYM_CODE_START(return_to_handler) int3 .Ldo_rop: mov %rdi, (%rsp) - RET + ALTERNATIVE __stringify(RET), \ + __stringify(ANNOTATE_UNRET_SAFE; ret; int3), \ + X86_FEATURE_CALL_DEPTH SYM_CODE_END(return_to_handler) #endif
arch/x86/net/bpf_jit_comp.c +6 -0
@@ -12,6 +12,7 @@#include <linux/memory.h> #include <linux/sort.h> #include <asm/extable.h> +#include <asm/ftrace.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> #include <asm/text-patching.h>
@@ -2090,6 +2091,11 @@ int arch_prepare_bpf_trampoline(struct b prog = image;EMIT_ENDBR (); +/* +* This is the direct-call trampoline, as such it needs accounting +* for the __fentry__ call. +*/ +x86_call_depth_emit_accounting (&prog, __fentry__);EMIT1 (0x55 );/* push rbp */ EMIT3 (0x48 ,0x89 ,0xE5 );/* mov rbp, rsp */ EMIT4 (0x48 ,0x83 ,0xEC , stack_size);/* sub rsp, stack_size */
kernel/trace/trace_selftest.c +4 -1
@@ -785,7 +785,10 @@ static struct fgraph_ops fgraph_ops __in };#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS -noinline __noclone static void trace_direct_tramp(void) { } +noinline __noclone static void trace_direct_tramp(void) +{ + asm(CALL_DEPTH_ACCOUNT); +}#endif /*
samples/ftrace/ftrace-direct-modify.c +2 -0
@@ -34,6 +34,7 @@ asm (ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" +CALL_DEPTH_ACCOUNT " call my_direct_func1\n" " leave\n" " .size my_tramp1, .-my_tramp1\n"
@@ -45,6 +46,7 @@ asm (ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" +CALL_DEPTH_ACCOUNT " call my_direct_func2\n" " leave\n" ASM_RET
samples/ftrace/ftrace-direct-multi-modify.c +2 -0
@@ -32,6 +32,7 @@ asm (ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" +CALL_DEPTH_ACCOUNT " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func1\n"
@@ -46,6 +47,7 @@ asm (ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" +CALL_DEPTH_ACCOUNT " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func2\n"
samples/ftrace/ftrace-direct-multi.c +1 -0
@@ -27,6 +27,7 @@ asm (ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" +CALL_DEPTH_ACCOUNT " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func\n"
samples/ftrace/ftrace-direct-too.c +1 -0
@@ -29,6 +29,7 @@ asm (ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" +CALL_DEPTH_ACCOUNT " pushq %rdi\n" " pushq %rsi\n" " pushq %rdx\n"
samples/ftrace/ftrace-direct.c +1 -0
@@ -26,6 +26,7 @@ asm (ASM_ENDBR " pushq %rbp\n" " movq %rsp, %rbp\n" +CALL_DEPTH_ACCOUNT " pushq %rdi\n" " call my_direct_func\n" " popq %rdi\n"
Steven Rostedt
maintainerRe: [patch 36/38] x86/ftrace: Make it call depth tracking aware
Quoted 10 lines · On Sun, 17 Jul 2022 01:18:08 +0200 (CEST) Thomas Gleixner <[email protected]>…
(quoted diff) +5
-1@@ -280,7 +291,19 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, /* Restore flags */ popfq UNWIND_HINT_FUNC - jmp ftrace_epilogue + + /* + * Since we're effectively emulating a tail-call with PUSH;RET + * make sure we don't unbalance the RSB and mess up accounting. + */
Review SYM_INNER_LABEL(ftrace_regs_caller_end, · line 308
Quoted 10 lines
(quoted diff) +8
-0+ ANNOTATE_INTRA_FUNCTION_CALL + call 2f + int3 +2: + add $8, %rsp + ALTERNATIVE __stringify(RET), \ + __stringify(ANNOTATE_UNRET_SAFE; ret; int3), \ + X86_FEATURE_CALL_DEPTH SYM_FUNC_END(ftrace_regs_caller)
Would this code be simpler if we nuked the ftrace_epilogue altogether?
After commit 0c0593b45c9b ("x86/ftrace: Make function graph use ftrace directly"), the ftrace_epilogue is no longer needed. That was there to make sure all the trampolines would call the function graph tracer. But now that function graph tracing is just another ftrace caller, it's not needed anymore.
Something like the below. It booted and passed the ftrace kselftests.
Feel free to include this in your series.
-- Steve
From 533f10bd48ffbc4ee5d2a07f0a7fe99aeb1c823a Mon Sep 17 00:00:00 2001
From: "Steven Rostedt (Google)" <[email protected]>
Date: Mon, 18 Jul 2022 16:01:07 -0400
Subject: [PATCH] ftrace/x86: Remove jumps to ftrace_epilogue
The jumps to ftrace_epilogue were done as a way to make sure all the function tracing trampolines ended at the function graph trampoline, as the ftrace_epilogue was the location that it would handle that.
With the advent of function graph tracer now being just one of the callbacks of the function tracer there is no more requirement that all trampolines go to a single location.
Remove the jumps to the ftrace_epilogue and replace them with return statements.
Note, the ftrace_epilogue can probably be renamed to ftrace_stub and the weak logic for that could probably be removed. But lets leave that as a separate change.
--- arch/x86/kernel/ftrace_64.S | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)
arch/x86/kernel/ftrace_64.S +7 -9
@@ -173,7 +173,9 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR - jmp ftrace_epilogue + UNWIND_HINT_FUNC + ENDBR + RET SYM_FUNC_END(ftrace_caller); STACK_FRAME_NON_STANDARD_FP(ftrace_caller)
@@ -261,15 +263,9 @@ SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL) /* Restore flags */ popfq - /* - * As this jmp to ftrace_epilogue can be a short jump - * it must not be copied into the trampoline. - * The trampoline will add the code to jump - * to the return. - */ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR - jmp ftrace_epilogue + jmp 2f /* Swap the flags with orig_rax */ 1: movq MCOUNT_REG_SIZE(%rsp), %rdi
@@ -279,8 +275,10 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) restore_mcount_regs 8 /* Restore flags */ popfq +2: UNWIND_HINT_FUNC - jmp ftrace_epilogue + ENDBR + RET SYM_FUNC_END(ftrace_regs_caller) STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller)
Signature · 1 line
2.35.1
Peter Zijlstra
maintainerRe: [patch 36/38] x86/ftrace: Make it call depth tracking aware
Quoted 30 lines · Steven Rostedt
From
533f10bd48ffbc4ee5d2a07f0a7fe99aeb1c823aMon Sep 17 00:00:00 2001
From: "Steven Rostedt (Google)" <[email protected]>
Date: Mon, 18 Jul 2022 16:01:07 -0400
Subject: [PATCH] ftrace/x86: Remove jumps toftrace_epilogueThe jumps to
ftrace_epiloguewere done as a way to make sure all the function tracing trampolines ended at the function graph trampoline, as theftrace_epiloguewas the location that it would handle that.With the advent of function graph tracer now being just one of the callbacks of the function tracer there is no more requirement that all trampolines go to a single location.
Remove the jumps to the
ftrace_epilogueand replace them with return statements.Note, the
ftrace_epiloguecan probably be renamed toftrace_stuband the weak logic for that could probably be removed. But lets leave that as a separate change.Signed-off-by Steven Rostedt (Google) <[email protected]>--- arch/x86/kernel/ftrace_64.S | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)arch/x86/kernel/ftrace_64.S +0
-0
arch/x86/kernel/ftrace_64.S SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) · line 177
Quoted 6 lines
arch/x86/kernel/ftrace_64.S +2
-1@@ -173,7 +173,9 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR - jmp ftrace_epilogue + UNWIND_HINT_FUNC + ENDBR
Only the RET should do I think; you definitely don't need an ENDBR here nor do you need to override the unwind hint. Lemme try..
Yeah, the below is sufficient:
arch/x86/kernel/ftrace_64.S +3 -4
@@ -172,8 +172,7 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) */ SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR - - jmp ftrace_epilogue + RET SYM_FUNC_END(ftrace_caller); STACK_FRAME_NON_STANDARD_FP(ftrace_caller)
@@ -269,7 +268,7 @@ SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL) */ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR - jmp ftrace_epilogue + RET /* Swap the flags with orig_rax */ 1: movq MCOUNT_REG_SIZE(%rsp), %rdi
@@ -280,7 +279,7 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL) /* Restore flags */ popfq UNWIND_HINT_FUNC - jmp ftrace_epilogue + RET SYM_FUNC_END(ftrace_regs_caller) STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller)
Steven Rostedt
maintainerRe: [patch 36/38] x86/ftrace: Make it call depth tracking aware
Quoted 2 lines · On Tue, 19 Jul 2022 10:46:48 +0200 Peter Zijlstra <[email protected]> wrote:
Only the RET should do I think; you definitely don't need an ENDBR here nor do you need to override the unwind hint. Lemme try..
I'll replace with the RET and resend v2.
-- Steve
PATCH 37/38 x86/bpf: Emit call depth accounting if required
In reply to Thomas GleixnerEnsure that calls in BPF jitted programs are emitting call depth accounting when enabled to keep the call/return balanced. The return thunk jump is already injected due to the earlier retbleed mitigations.
--- arch/x86/include/asm/alternative.h | 6 +++++ arch/x86/kernel/callthunks.c | 19 ++++++++++++++++ arch/x86/net/bpf_jit_comp.c | 43 ++++++++++++++++++++++++------------- 3 files changed, 53 insertions(+), 15 deletions(-)
arch/x86/include/asm/alternative.h +6 -0
@@ -95,6 +95,7 @@ extern void callthunks_patch_module_callextern void callthunks_module_free (struct module *mod);extern void *callthunks_translate_call_dest (void *dest);extern bool is_callthunk (void *addr); +extern int x86_call_depth_emit_accounting (u8 **pprog,void *func);#else static __always_inline void callthunks_patch_builtin_calls(void) {}static __always_inline void
@@ -109,6 +110,11 @@ static __always_inline bool is_callthunk {return false; } +static __always_inline int x86_call_depth_emit_accounting(u8 **pprog, +void *func) +{ +return 0 ; +}#endif #ifdef CONFIG_SMP
arch/x86/kernel/callthunks.c +19 -0
@@ -706,6 +706,25 @@ int callthunk_get_kallsym(unsigned int sreturn ret; } +#ifdef CONFIG_BPF_JIT +int x86_call_depth_emit_accounting (u8 **pprog,void *func) +{ +unsigned int tmpl_size = callthunk_desc.template_size; +void *tmpl = callthunk_desc.template; + +if (!thunks_initialized) +return 0 ; + +/* Is function call target a thunk? */ +if (is_callthunk (func)) +return 0 ; + +memcpy (*pprog, tmpl, tmpl_size); + *pprog += tmpl_size; +return tmpl_size; +} +#endif +#ifdef CONFIG_MODULES void noinline callthunks_patch_module_calls(struct callthunk_sites *cs,struct module *mod)
arch/x86/net/bpf_jit_comp.c +28 -15
101 lines not on this page · raw message
Re: [patch 37/38] x86/bpf: Emit call depth accounting if required
Quoted 86 lines · Thomas Gleixner
Ensure that calls in BPF jitted programs are emitting call depth accounting when enabled to keep the call/return balanced. The return thunk jump is already injected due to the earlier retbleed mitigations.
Signed-off-by Thomas Gleixner <[email protected]>Cc Alexei Starovoitov <[email protected]>Cc Daniel Borkmann <[email protected]>--- arch/x86/include/asm/alternative.h | 6 +++++ arch/x86/kernel/callthunks.c | 19 ++++++++++++++++ arch/x86/net/bpf_jit_comp.c | 43 ++++++++++++++++++++++++------------- 3 files changed, 53 insertions(+), 15 deletions(-)arch/x86/include/asm/alternative.h +6
-0@@ -95,6 +95,7 @@ extern void callthunks_patch_module_callextern void callthunks_module_free (struct module *mod);extern void *callthunks_translate_call_dest (void *dest);extern bool is_callthunk (void *addr); +extern int x86_call_depth_emit_accounting (u8 **pprog,void *func);#else static __always_inline void callthunks_patch_builtin_calls(void) {}static __always_inline void@@ -109,6 +110,11 @@ static __always_inline bool is_callthunk {return false; } +static __always_inline int x86_call_depth_emit_accounting(u8 **pprog, +void *func) +{ +return 0 ; +}#endif #ifdef CONFIG_SMP arch/x86/kernel/callthunks.c +19
-0@@ -706,6 +706,25 @@ int callthunk_get_kallsym(unsigned int sreturn ret; } +#ifdef CONFIG_BPF_JIT +int x86_call_depth_emit_accounting (u8 **pprog,void *func) +{ +unsigned int tmpl_size = callthunk_desc.template_size; +void *tmpl = callthunk_desc.template; + +if (!thunks_initialized) +return 0 ; + +/* Is function call target a thunk? */ +if (is_callthunk (func)) +return 0 ; + +memcpy (*pprog, tmpl, tmpl_size); + *pprog += tmpl_size; +return tmpl_size; +} +#endif +#ifdef CONFIG_MODULES void noinline callthunks_patch_module_calls(struct callthunk_sites *cs,struct module *mod)arch/x86/net/bpf_jit_comp.c +8
-1@@ -340,6 +340,12 @@ static int emit_call(u8 **pprog, void *freturn emit_patch (pprog, func, ip,0xE8 ); } +static int emit_rsb_call (u8 **pprog,void *func,void *ip) +{ +x86_call_depth_emit_accounting (pprog, func); +return emit_patch (pprog, func, ip,0xE8 ); +} +static int emit_jump (u8 **pprog,void *func,void *ip) {return emit_patch (pprog, func, ip,0xE9 );@@ -1431,19 +1437,26 @@ st: if (is_imm8(insn->off))break ;/* call */ -case BPF_JMP |BPF_CALL : +case BPF_JMP |BPF_CALL : { +int offs;
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 1450
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +3
-1+ func = (u8 *) __bpf_call_base + imm32;if (tail_call_reachable) {/* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */ EMIT3_off32 (0x48 ,0x8B ,0x85 , -round_up (bpf_prog->aux->stack_depth,8 ) -8 ); -if (!imm32 ||emit_call (&prog, func, image + addrs[i -1 ] +7 )) +if (!imm32)return -EINVAL ; + offs =7 +x86_call_depth_emit_accounting (&prog, func);
It's a bit hard to read all the macro magic in patches 28-30, but I suspect the asm inside callthunk_desc.template
that will be emitted here before the call
will do
some math on %rax
movq %rax, PER_CPU_VAR(__x86_call_depth).
Only %rax register is scratched by the callthunk_desc, right? If so, it's ok for all cases except this one.
See the comment few lines above
after if (tail_call_reachable)
and commit ebf7d1f508a7 ("bpf, x64: rework pro/epilogue and tailcall handling in JIT")
We use %rax to keep the tail_call count.
The callthunk_desc would need to preserve %rax.
I guess extra push %rax/pop %rax would do it.
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +5
-1}else { -if (!imm32 ||emit_call (&prog, func, image + addrs[i -1 ])) +if (!imm32)return -EINVAL ; + offs =x86_call_depth_emit_accounting (&prog, func); } +if (emit_call (&prog, func, image + addrs[i -1 ] + offs)) +return -EINVAL ;break ; + }
Peter Zijlstra
maintainerRe: [patch 37/38] x86/bpf: Emit call depth accounting if required
Quoted 27 lines · Alexei Starovoitov
Quoted 7 lines · Thomas Gleixner
arch/x86/net/bpf_jit_comp.c +2
-1@@ -1431,19 +1437,26 @@ st: if (is_imm8(insn->off))break ;/* call */ -case BPF_JMP |BPF_CALL : +case BPF_JMP |BPF_CALL : { +int offs;arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 1450
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +3
-1+ func = (u8 *) __bpf_call_base + imm32;if (tail_call_reachable) {/* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */ EMIT3_off32 (0x48 ,0x8B ,0x85 , -round_up (bpf_prog->aux->stack_depth,8 ) -8 ); -if (!imm32 ||emit_call (&prog, func, image + addrs[i -1 ] +7 )) +if (!imm32)return -EINVAL ; + offs =7 +x86_call_depth_emit_accounting (&prog, func);It's a bit hard to read all the macro magic in patches 28-30, but I suspect the asm inside callthunk_desc.template
that will be emitted here before the call
will do
some math on %rax
movq %rax,PER_CPU_VAR(__x86_call_depth).
Quoted 9 lines
Only %rax register is scratched by the
callthunk_desc, right? If so, it's ok for all cases except this one.
See the comment few lines above
after if (tail_call_reachable)
and commitebf7d1f508a7("bpf, x64: rework pro/epilogue and tailcall handling in JIT")
We use %rax to keep thetail_callcount.
Thecallthunk_descwould need to preserve %rax.
I guess extra push %rax/pop %rax would do it.
The accounting template is basically:
sarq $5, PER_CPU_VAR(__x86_call_depth)
No registeres used (with debugging on it's a few more memops).
PATCH 38/38 x86/retbleed: Add call depth tracking mitigation
In reply to Thomas GleixnerThe fully secure mitigation for RSB underflow on Intel SKL CPUs is IBRS, which inflicts up to 30% penalty for pathological syscall heavy work loads.
Software based call depth tracking and RSB refill is not perfect, but reduces the attack surface massively. The penalty for the pathological case is about 8% which is still annoying but definitely more palatable than IBRS.
Add a retbleed=stuff command line option to enable the call depth tracking and software refill of the RSB.
This gives admins a choice. IBeeRS are safe and cause headaches, call depth tracking is considered to be s(t)ufficiently safe.
--- arch/x86/kernel/cpu/bugs.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/bugs.c +30 -2
@@ -784,6 +784,7 @@ enum retbleed_mitigation {RETBLEED_MITIGATION_IBPB ,RETBLEED_MITIGATION_IBRS ,RETBLEED_MITIGATION_EIBRS , + RETBLEED_MITIGATION_STUFF, }; enum retbleed_mitigation_cmd {
@@ -791,6 +792,7 @@ enum retbleed_mitigation_cmd {RETBLEED_CMD_AUTO ,RETBLEED_CMD_UNRET ,RETBLEED_CMD_IBPB , + RETBLEED_CMD_STUFF, }; const char * const retbleed_strings[] = {
@@ -799,6 +801,7 @@ const char * const retbleed_strings[] = [RETBLEED_MITIGATION_IBPB] = "Mitigation: IBPB", [RETBLEED_MITIGATION_IBRS] = "Mitigation: IBRS", [RETBLEED_MITIGATION_EIBRS] = "Mitigation: Enhanced IBRS", + [RETBLEED_MITIGATION_STUFF] = "Mitigation: Stuffing", }; static enum retbleed_mitigation retbleed_mitigation __ro_after_init =
@@ -828,6 +831,8 @@ static int __init retbleed_parse_cmdline retbleed_cmd =RETBLEED_CMD_UNRET ; }else if (!strcmp (str,"ibpb" )) { retbleed_cmd =RETBLEED_CMD_IBPB ; + }else if (!strcmp (str,"stuff" )) { + retbleed_cmd =RETBLEED_CMD_STUFF ; }else if (!strcmp (str,"nosmt" )) { retbleed_nosmt = true; }else {
@@ -876,6 +881,21 @@ static void __init retbleed_select_mitig }break ; +case RETBLEED_CMD_STUFF : +if (IS_ENABLED (CONFIG_CALL_DEPTH_TRACKING ) && + spectre_v2_enabled ==SPECTRE_V2_RETPOLINE ) { + retbleed_mitigation =RETBLEED_MITIGATION_STUFF ; + + }else { +if (IS_ENABLED (CONFIG_CALL_DEPTH_TRACKING )) +pr_err ("WARNING: retbleed=stuff depends on spectre_v2=retpoline\n" ); +else +pr_err ("WARNING: kernel not compiled with CALL_DEPTH_TRACKING.\n" ); + + goto do_cmd_auto; + } +break ; + do_cmd_auto:case RETBLEED_CMD_AUTO :default :
@@ -913,6 +933,12 @@ static void __init retbleed_select_mitig mitigate_smt = true;break ; +case RETBLEED_MITIGATION_STUFF : +setup_force_cpu_cap (X86_FEATURE_RETHUNK ); +setup_force_cpu_cap (X86_FEATURE_CALL_DEPTH ); +x86_set_skl_return_thunk (); +break ; +default :break ; }
@@ -923,7 +949,7 @@ static void __init retbleed_select_mitig/* * Let IBRS trump all on Intel without affecting the effects of the -* retbleed= cmdline option. +* retbleed= cmdline option except for call depth based stuffing */ if (boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) {switch (spectre_v2_enabled) {
@@ -936,7 +962,8 @@ static void __init retbleed_select_mitig retbleed_mitigation =RETBLEED_MITIGATION_EIBRS ;break ;default : -pr_err (RETBLEED_INTEL_MSG ); +if (retbleed_mitigation !=RETBLEED_MITIGATION_STUFF ) +pr_err (RETBLEED_INTEL_MSG ); } }
@@ -1361,6 +1388,7 @@ static void __init spectre_v2_select_mitif (IS_ENABLED (CONFIG_CPU_IBRS_ENTRY ) &&boot_cpu_has_bug (X86_BUG_RETBLEED ) && retbleed_cmd !=RETBLEED_CMD_OFF && + retbleed_cmd !=RETBLEED_CMD_STUFF &&boot_cpu_has (X86_FEATURE_IBRS ) && boot_cpu_data.x86_vendor ==X86_VENDOR_INTEL ) { mode =SPECTRE_V2_IBRS ;
From: Thomas Gleixner
Quoted 42 lines
Sent: 17 July 2022 00:17
Folks!Back in the good old spectre v2 days (2018) we decided to not use IBRS. In hindsight this might have been the wrong decision because it did not force people to come up with alternative approaches.
It was already discussed back then to try software based call depth accounting and RSB stuffing on underflow for Intel SKL[-X] systems to avoid the insane overhead of IBRS.
This has been tried in 2018 and was rejected due to the massive overhead and other shortcomings of the approach to put the accounting into each function prologue:
1) Text size increase which is inflicted on everyone. While CPUs are good in ignoring NOPs they still pollute the I-cache.
2) That results in tail call over-accounting which can be exploited.
Disabling tail calls is not an option either and adding a 10 byte padding in front of every direct call is even worse in terms of text size and I-cache impact. We also could patch calls past the accounting in the function prologue but that becomes a nightmare vs. ENDBR.
As IBRS is a performance horror show, Peter Zijstra and me revisited the call depth tracking approach and implemented it in a way which is hopefully more palatable and avoids the downsides of the original attempt.
We both unsurprisingly hate the result with a passion...
The way we approached this is:
1) objtool creates a list of function entry points and a list of direct call sites into new sections which can be discarded after init.
2) On affected machines, use the new sections, allocate module memory and create a call thunk per function (16 bytes without debug/statistics). Then patch all direct calls to invoke the thunk, which does the call accounting and then jumps to the original call site.
Quoted 3 lines
3) Utilize the retbleed return thunk mechanism by making the jump target run-time configurable. Add the accounting counterpart and stuff RSB on underflow in that alternate implementation.
What happens to indirect calls?
The above would imply that they miss the function entry thunk, but get the return one.
Won't this lead to mis-counting of the RSB?
I also thought that retpolines would trash the return stack? Using a single retpoline thunk would pretty much ensure that they are never correctly predicted from the BTB, but it only gives a single BTB entry that needs 'setting up' to get mis- prediction.
I'm also sure I managed to infer from a document of instruction timings and architectures that some x86 cpu actually used the BTB for normal conditional jumps?
Possibly to avoid passing the full %ip value all down the cpu pipeline.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 10 lines · David Laight
From: Thomas Gleixner
Quoted 4 lines
3) Utilize the retbleed return thunk mechanism by making the jump target run-time configurable. Add the accounting counterpart and stuff RSB on underflow in that alternate implementation.
What happens to indirect calls?
The above would imply that they miss the function entry thunk, but get the return one.
Won't this lead to mis-counting of the RSB?
That's accounted in the indirect call thunk. This mitigation requires retpolines enabled.
Quoted 1 line
I also thought that retpolines would trash the return stack?
No. They prevent that the CPU misspeculates an indirect call due to a mistrained BTB.
Quoted 4 lines
Using a single retpoline thunk would pretty much ensure that they are never correctly predicted from the BTB, but it only gives a single BTB entry that needs 'setting up' to get mis- prediction.
BTB != RSB
The intra function call in the retpoline is of course adding a RSB entry which points to the speculation trap, but that gets popped immediately after that by the return which goes to the called function.
But that does not prevent the RSB underflow problem. As I described the RSB is a stack with depth 16. Call pushs, ret pops. So if speculation is ahead and emptied the RSB while speculating down the rets then the next speculated RET will fall back to other prediction mechanism which is what the SKL specific retbleed variant exploits via BHB mistraining.
Quoted 3 lines
I'm also sure I managed to infer from a document of instruction timings and architectures that some x86 cpu actually used the BTB for normal conditional jumps?
That's relevant to the problem at hand in which way?
Thanks,
tglx
From: Thomas Gleixner
Quoted 9 lines
Sent: 17 July 2022 16:07
Quoted 6 lines · David Laight
From: Thomas Gleixner
Quoted 4 lines
3) Utilize the retbleed return thunk mechanism by making the jump target run-time configurable. Add the accounting counterpart and stuff RSB on underflow in that alternate implementation.
Quoted 7 lines
Quoted 4 lines
What happens to indirect calls?
The above would imply that they miss the function entry thunk, but get the return one.
Won't this lead to mis-counting of the RSB?That's accounted in the indirect call thunk. This mitigation requires retpolines enabled.
Thanks, that wasn't in the summary.
Quoted 11 lines
Quoted 1 line
I also thought that retpolines would trash the return stack?
No. They prevent that the CPU misspeculates an indirect call due to a mistrained BTB.
Quoted 4 lines
Using a single retpoline thunk would pretty much ensure that they are never correctly predicted from the BTB, but it only gives a single BTB entry that needs 'setting up' to get mis- prediction.
BTB != RSB
I was thinking about what happens after the RSB has underflowed. Which is when (I presume) the BTB based speculation happens.
Quoted 3 lines
The intra function call in the retpoline is of course adding a RSB entry which points to the speculation trap, but that gets popped immediately after that by the return which goes to the called function.
I'm remembering the 'active' instructions in a retpoline being 'push; ret'. Which is an RSB imbalance.
...
Quoted 5 lines
Quoted 3 lines
I'm also sure I managed to infer from a document of instruction timings and architectures that some x86 cpu actually used the BTB for normal conditional jumps?
That's relevant to the problem at hand in which way?
The next problem :-)
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 11 lines · David Laight
From: Thomas Gleixner
Quoted 1 line · David Laight
I was thinking about what happens after the RSB has underflowed. Which is when (I presume) the BTB based speculation happens.
Quoted 3 lines
The intra function call in the retpoline is of course adding a RSB entry which points to the speculation trap, but that gets popped immediately after that by the return which goes to the called function.
I'm remembering the 'active' instructions in a retpoline being 'push; ret'. Which is an RSB imbalance.
Looking at the code might help to remember correctly:
call 1f
speculation trap
1: mov %reg, %rsp
ret
Thanks,
tglx
Quoted 26 lines · Thomas Gleixner
The function alignment option does not work for that because it just guarantees that the next function entry is aligned, but the padding size depends on the position of the last instruction of the previous function which might be anything between 0 and padsize-1 obviously, which is not a good starting point to put 10 bytes of accounting code into it reliably.
I hacked up GCC to emit such padding and from first experimentation it brings quite some performance back.
IBRS stuff stuff(pad)
sockperf 14 bytes: -23.76% -19.26% -14.31%
sockperf 1472 bytes: -22.51% -18.40% -12.25%
microbench: +37.20% +18.46% +15.47%
hackbench: +21.24% +10.94% +10.12%For FIO I don't have numbers yet, but I expect FIO to get a significant gain too.
Quoted 1 line
From a quick survey it seems to have no impact for the case where the
thunks are not used. But that really needs some deep investigation and there is a potential conflict with the clang CFI efforts.
The kernel text size increases with a Debian config from 9.9M to 10.4M, so about 5%. If the thunk is not 16 byte aligned, the text size increase is about 3%, but it turned out that 16 byte aligned is slightly faster.
Quoted 7 lines
The 16 byte function alignment turned out to be beneficial in general even without the thunks. Not much of an improvement, but measurable. We should revisit this independent of these horrors.
The implementation falls back to the allocated thunks when padding is not available. I'll send out the GCC patch and the required kernel patch as a reply to this series after polishing it a bit.
Here it goes. GCC hackery first.
---
Subject: gcc: Add padding in front of function entry points
From: Thomas Gleixner <[email protected]>
Date: Fri, 15 Jul 2022 14:37:53 +0200
For testing purposes:
Add a 16 byte padding filled with int3 in front of each function entry so the kernel can put call depth accounting into it.
--- gcc/config/i386/i386.cc | 11 +++++++++++ gcc/config/i386/i386.h | 7 +++++++ gcc/config/i386/i386.opt | 4 ++++ gcc/doc/invoke.texi | 6 ++++++ 4 files changed, 28 insertions(+)
gcc/config/i386/i386.cc +11 -0
@@ -6182,6 +6182,17 @@ ix86_code_end (void) file_end_indicate_split_stack (); } +void +x86_asm_output_function_prefix (FILE *asm_out_file, + const char *fnname ATTRIBUTE_UNUSED) +{ + if (flag_force_function_padding) + { + fprintf (asm_out_file, "\t.align 16\n"); + fprintf (asm_out_file, "\t.skip 16,0xcc\n"); + } +} + /* Emit code for the SET_GOT patterns. */ const char *
gcc/config/i386/i386.h +7 -0
@@ -2860,6 +2860,13 @@ extern enum attr_cpu ix86_schedule;#define LIBGCC2_UNWIND_ATTRIBUTE __attribute__((target ("no-mmx,no-sse")))#endif +#include <stdio.h> +extern void +x86_asm_output_function_prefix (FILE *asm_out_file, + const char *fnname ATTRIBUTE_UNUSED); +#undef ASM_OUTPUT_FUNCTION_PREFIX +#define ASM_OUTPUT_FUNCTION_PREFIX x86_asm_output_function_prefix +/* Local variables: version-control: t
gcc/config/i386/i386.opt +4 -0
@@ -1064,6 +1064,10 @@ mindirect-branch= Target RejectNegative Joined Enum(indirect_branch) Var(ix86_indirect_branch) Init(indirect_branch_keep) Convert indirect call and jump to call and return thunks. +mforce-function-padding +Target Var(flag_force_function_padding) Init(0) +Put a 16 byte padding area before each function + mfunction-return= Target RejectNegative Joined Enum(indirect_branch) Var(ix86_function_return) Init(indirect_branch_keep) Convert function return to call and return thunk.
gcc/doc/invoke.texi +6 -0
@@ -1451,6 +1451,7 @@ See RS/6000 and PowerPC Options. -mindirect-branch=@var{choice} -mfunction-return=@var{choice} @gol -mindirect-branch-register -mharden-sls=@var{choice} @gol -mindirect-branch-cs-prefix -mneeded -mno-direct-extern-access} +-mforce-function-padding @gol @emph{x86 Windows Options} @gccoptlist{-mconsole -mcygwin -mno-cygwin -mdll @gol
@@ -32849,6 +32850,11 @@ Force all calls to functions to be indir when using Intel Processor Trace where it generates more precise timing information for function calls. +@item -mforce-function-padding +@opindex -mforce-function-padding +Force a 16 byte padding are before each function which allows run-time +code patching to put a special prologue before the function entry. + @item -mmanual-endbr @opindex mmanual-endbr Insert ENDBR instruction at function entry only via the @code{cf_check}
Quoted 5 lines · Thomas Gleixner
Quoted 3 lines
The implementation falls back to the allocated thunks when padding is not available. I'll send out the GCC patch and the required kernel patch as a reply to this series after polishing it a bit.
Here it goes. GCC hackery first.
And the kernel counterpart.
---
Subject: x06/callthunks: Put thunks into compiler provided padding area
From: Thomas Gleixner <[email protected]>
Date: Fri, 15 Jul 2022 16:12:47 +0200
- NOT FOR INCLUSION -
Let the compiler add a 16 byte padding in front of each function entry point and put the call depth accounting there. That avoids calling out into the module area and reduces ITLB pressure.
--- arch/x86/Kconfig | 14 ++++++ arch/x86/Makefile | 4 + arch/x86/kernel/callthunks.c | 99 ++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 115 insertions(+), 2 deletions(-)
arch/x86/Kconfig +14 -0
@@ -2440,6 +2440,9 @@ config CC_HAS_SLS config CC_HAS_RETURN_THUNK def_bool $(cc-option,-mfunction-return=thunk-extern) +config CC_HAS_PADDING + def_bool $(cc-option,-mforce-function-padding) + config HAVE_CALL_THUNKS def_bool y depends on RETHUNK && OBJTOOL
@@ -2512,6 +2515,17 @@ config CALL_DEPTH_TRACKING of this option is marginal as the call depth tracking is using run-time generated call thunks and call patching. +config CALL_THUNKS_IN_PADDING + bool "Put call depth into padding area before function" + depends on CALL_DEPTH_TRACKING && CC_HAS_PADDING + default n + help + Put the call depth accounting into a padding area before the + function entry. This padding area is generated by the + compiler. This increases text size by ~5%. For non affected + systems this space is unused. On affected SKL systems this + results in a significant performance gain. + config CALL_THUNKS_DEBUG bool "Enable call thunks and call depth tracking debugging" depends on CALL_DEPTH_TRACKING
arch/x86/Makefile +4 -0
@@ -197,6 +197,10 @@ ifdef CONFIG_SLS KBUILD_CFLAGS += -mharden-sls=all endif +ifdef CONFIG_CALL_THUNKS_IN_PADDING + KBUILD_CFLAGS += -mforce-function-padding +endif + KBUILD_LDFLAGS += -m elf_$(UTS_MACHINE) ifdef CONFIG_LTO_CLANG
arch/x86/kernel/callthunks.c +97 -2
153 lines not on this page · raw message
Quoted 4 lines · Thomas Gleixner
Let the compiler add a 16 byte padding in front of each function entry point and put the call depth accounting there. That avoids calling out into the module area and reduces ITLB pressure.
Ooh.
I actually like this a lot better.
Could we just say "use this instead if you have SKL and care about the issue?"
I don't hate your module thunk trick, but this does seem *so* much simpler, and if it performs better anyway, it really does seem like the better approach.
And people and distros who care would have an easy time adding that simple compiler patch instead.
I do think that for generality, the "-mforce-function-padding" option should perhaps take as an argument how much padding (and how much alignment) to force:
-mforce-function-padding=5:16
would force 5 bytes of minimum padding, and align functions to 16 bytes. It should be easy to generate (no more complexity than your current one) by just making the output do
.skip 5,0xcc
.palign 4,0xccand now you can specify that you only need X bytes of padding, for example.
Linus
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Thomas Gleixner
Let the compiler add a 16 byte padding in front of each function entry point and put the call depth accounting there. That avoids calling out into the module area and reduces ITLB pressure.
Ooh.
Quoted 7 lines
I actually like this a lot better.
Could we just say "use this instead if you have SKL and care about the issue?"
I don't hate your module thunk trick, but this does seem *so* much simpler, and if it performs better anyway, it really does seem like the better approach.
Yes, Peter and I came from avoiding a new compiler and the overhead for everyone when putting the padding into the code. We realized only when staring at the perf data that this padding in front of the function might be an acceptable solution. I did some more tests today on different machines with mitigations=off with kernels compiled with and without that padding. I couldn't find a single test case where the result was outside of the usual noise. But then my tests are definitely incomplete.
Quoted 9 lines
And people and distros who care would have an easy time adding that simple compiler patch instead.
I do think that for generality, the "-mforce-function-padding" option should perhaps take as an argument how much padding (and how much alignment) to force:
-mforce-function-padding=5:16
Quoted 8 lines
would force 5 bytes of minimum padding, and align functions to 16 bytes. It should be easy to generate (no more complexity than your current one) by just making the output do
.skip 5,0xcc .palign 4,0xccand now you can specify that you only need X bytes of padding, for example.
Yes, I know. But it was horrible enough to find the right spot in that gcc maze. Then I was happy that I figured how to add the boolean option. I let real compiler people take care of the rest. HJL???
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Thanks,
tglx
Quoted 8 lines · Thomas Gleixner
Yes, Peter and I came from avoiding a new compiler and the overhead for everyone when putting the padding into the code. We realized only when staring at the perf data that this padding in front of the function might be an acceptable solution. I did some more tests today on different machines with mitigations=off with kernels compiled with and without that padding. I couldn't find a single test case where the result was outside of the usual noise. But then my tests are definitely incomplete.
Well, it sounds like it most definitely isn't a huge and obvious problem.
Quoted 6 lines
Yes, I know. But it was horrible enough to find the right spot in that gcc maze. Then I was happy that I figured how to add the boolean option. I let real compiler people take care of the rest. HJL???
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Yeah, looking at the gcc sources (I have them locally because it helps with the gcc bug reports I've done over the years), that ASM_OUTPUT_FUNCTION_PREFIX is very convenient, but it's too late to do any inter-function alignment for, because it's already after the usual function-alignment output.
So I guess the padding thing is largely tied together with alignment of the function start, so that idea of having different padding and alignment bytes doesn't workl that well.
At least not in that ASM_OUTPUT_FUNCTION_PREFIX model, which is how the gcc patch ends up being so small.
Linus
Peter Zijlstra
maintainerQuoted 25 lines · Linus Torvalds
Quoted 8 lines · Thomas Gleixner
Yes, Peter and I came from avoiding a new compiler and the overhead for everyone when putting the padding into the code. We realized only when staring at the perf data that this padding in front of the function might be an acceptable solution. I did some more tests today on different machines with mitigations=off with kernels compiled with and without that padding. I couldn't find a single test case where the result was outside of the usual noise. But then my tests are definitely incomplete.
Well, it sounds like it most definitely isn't a huge and obvious problem.
Quoted 6 lines
Yes, I know. But it was horrible enough to find the right spot in that gcc maze. Then I was happy that I figured how to add the boolean option. I let real compiler people take care of the rest. HJL???
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Yeah, looking at the gcc sources (I have them locally because it helps with the gcc bug reports I've done over the years), that
ASM_OUTPUT_FUNCTION_PREFIXis very convenient, but it's too late to do any inter-function alignment for, because it's already after the usual function-alignment output.
Quoted 6 lines
So I guess the padding thing is largely tied together with alignment of the function start, so that idea of having different padding and alignment bytes doesn't workl that well.
At least not in that
ASM_OUTPUT_FUNCTION_PREFIXmodel, which is how the gcc patch ends up being so small.
FWIW, when I was poking at this last week, I found that -falign-function only seems to apply to the normal .text section and not to random other sections with text we create.
Or rather, I was seeind a lot of unaligned functions that all had custom sections despite explicitly using the (what I thought was a global) function alignment toggle.
Quoted 8 lines · Peter Zijlstra
FWIW, when I was poking at this last week, I found that -falign-function only seems to apply to the normal .text section and not to random other sections with text we create.
Or rather, I was seeind a lot of unaligned functions that all had custom sections despite explicitly using the (what I thought was a global) function alignment toggle.
Hmm. This triggers a memory..
I think we may have two different issues at play.
One is that I think our linker script only aligns code sections to 8 bytes by default. Grep for ALIGN_FUNCTION.
And I think that any .align directive (or .p2align) only aligns relative to that section, so if the section itself wasn't aligned, it doesn't help to have some alignment within the section.
I may be wrong.
But I can definitely see gcc not aligning functions too, and doing a
nm vmlinux | grep ' t ' | grep -v '0 t ' | grep -v '\.cold$' | sort
shows a _lot_ of them for me.
I think the main cause is that the ACPI code builds with
ccflags-y := -Os -D_LINUX -DBUILDING_ACPICA
and that '-Os' will disable all function alignment. I think there's a few other places that do that too.
I don't see the same effect in my clang build, so I think that -Os behavior is likely gcc-specific.
In my clang build, I do see a few unaligned function symbols, but they seem to be all our own assembler ones (eg "nested_nmi") and they seem to be intentional (ie that "nested_nmi" thing is in the middle of the "asm_exc_nmi" function, which is the real thing and which _is_ aligned).
Linus
Peter Zijlstra
maintainerQuoted 6 lines · Linus Torvalds
I think the main cause is that the ACPI code builds with
ccflags-y := -Os -D_LINUX -DBUILDING_ACPICA
and that '-Os' will disable all function alignment. I think there's a few other places that do that too.
Urgh, that's -Ostupid, right?
Peter Zijlstra
maintainerQuoted 13 lines · Thomas Gleixner
Quoted 13 lines
I do think that for generality, the "-mforce-function-padding" option should perhaps take as an argument how much padding (and how much alignment) to force:
-mforce-function-padding=5:16would force 5 bytes of minimum padding, and align functions to 16 bytes. It should be easy to generate (no more complexity than your current one) by just making the output do
.skip 5,0xcc .palign 4,0xcc
Quoted 8 lines
Quoted 1 line
and now you can specify that you only need X bytes of padding, for example.
Yes, I know. But it was horrible enough to find the right spot in that gcc maze. Then I was happy that I figured how to add the boolean option. I let real compiler people take care of the rest. HJL???
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Right, I need to go look at the latest kCFI patches, that sorta got side-tracked for working on all the retbleed muck :/
Basically kCFI wants to preface every (indirect callable) function with:
__cfi_\func:
int3
movl $0x12345678, %rax
int3
int3\func:
endbr
\func_direct:
Ofc, we can still put the whole:
sarq $5, PER_CPU_VAR(__x86_call_depth); jmp \func_direct
thing in front of that. But it does somewhat destroy the version I had that only needs the 10 bytes padding for the sarq.
Quoted 18 lines · Peter Zijlstra
Quoted 2 lines · Thomas Gleixner
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Right, I need to go look at the latest kCFI patches, that sorta got side-tracked for working on all the retbleed muck :/
Basically kCFI wants to preface every (indirect callable) function with:
__cfi_\func:
int3 movl $0x12345678, %rax int3 int3\func:
endbr
\func_direct:
Quoted 7 lines
Ofc, we can still put the whole:
sarq $5, PER_CPU_VAR(__x86_call_depth); jmp \func_directthing in front of that. But it does somewhat destroy the version I had that only needs the 10 bytes padding for the sarq.
Right, because it needs the jump. I was just chatting with Jaoa about that over IRC.
The jump slow things down. Jaoa has ideas and will reply soonish.
Thanks,
tglx
Quoted 24 lines · Thomas Gleixner
Quoted 23 lines · Peter Zijlstra
Quoted 2 lines · Thomas Gleixner
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Right, I need to go look at the latest kCFI patches, that sorta got side-tracked for working on all the retbleed muck :/
Basically kCFI wants to preface every (indirect callable) function with:
__cfi_\func:
int3 movl $0x12345678, %rax int3 int3\func:
endbr
\func_direct:Ofc, we can still put the whole:
sarq $5, PER_CPU_VAR(__x86_call_depth); jmp \func_direct
Quoted 7 lines
Quoted 2 lines
thing in front of that. But it does somewhat destroy the version I had that only needs the 10 bytes padding for the sarq.
Right, because it needs the jump. I was just chatting with Jaoa about that over IRC.
The jump slow things down. Jaoa has ideas and will reply soonish.
So, IIRC, kCFI will do something like this to validate call targets based on the hash as described on Peter's e-mail:
func_whatever:
... cmpl $0x\hash, -6(%rax) je 1f ud2 1: call *%rax ...
Thus the hash will be 6 bytes before the function entry point. Then we can get the compiler to emit a padding area before the __cfi_\func snippet and, during boot, if the CPU needs the call depth tracking mitigation, we:
- move the __cfi_func into the padding area
- patch the call depth tracking snippet ahead of it (overwriting the old __cfi_\func:)
- fix the cmpl offset in the caller
func_whatever:
... cmpl $0x\hash, -FIXED_OFFSET(%rax) je 1f ud2 1: call *%rax ...
This approach is very similar to what we discussed in the past for replacing kCFI with FineIBT if CET is available. Also, it would prevent the need for any jump and would keep the additional padding area in 10 bytes.
Tks,
Joao
Sami Tolvanen
maintainerQuoted 9 lines · Joao Moreira
Thus the hash will be 6 bytes before the function entry point. Then we can get the compiler to emit a padding area before the __cfi_\func snippet and, during boot, if the CPU needs the call depth tracking mitigation, we:
- move the__cfi_funcinto the padding area
- patch the call depth tracking snippet ahead of it (overwriting the old __cfi_\func:)
- fix the cmpl offset in the caller
Quoted 8 lines
func_whatever:... cmpl $0x\hash, -FIXED_OFFSET(%rax) je 1f ud2 1: call *%rax ...
The problem with this is that the cmpl instruction contains the full type hash, which means that any instruction that's FIXED_OFFSET from the cmpl is a valid indirect call target as far as KCFI is concerned. -6 was chosen specifically to make the ud2 the only possible target.
Sami
Quoted 4 lines
The problem with this is that the cmpl instruction contains the full type hash, which means that any instruction that's
FIXED_OFFSETfrom the cmpl is a valid indirect call target as far as KCFI is concerned. -6 was chosen specifically to make the ud2 the only possible target.
Ugh. The bitter truth. I'll think a bit further.
Quoted 19 lines · Sami Tolvanen
Quoted 9 lines · Joao Moreira
Thus the hash will be 6 bytes before the function entry point. Then we can get the compiler to emit a padding area before the __cfi_\func snippet and, during boot, if the CPU needs the call depth tracking mitigation, we:
- move the__cfi_funcinto the padding area
- patch the call depth tracking snippet ahead of it (overwriting the old __cfi_\func:)
- fix the cmpl offset in the callerQuoted 8 lines
func_whatever:... cmpl $0x\hash, -FIXED_OFFSET(%rax) je 1f ud2 1: call *%rax ...
Quoted 4 lines
The problem with this is that the cmpl instruction contains the full type hash, which means that any instruction that's
FIXED_OFFSETfrom the cmpl is a valid indirect call target as far as KCFI is concerned. -6 was chosen specifically to make the ud2 the only possible target.
But that's an implementation detail, right? Whatever we put in between will still be a fixed offset, no? It's a different offset, but that's what patching can deal with.
Thanks,
tglx
Quoted 4 lines · Thomas Gleixner
But that's an implementation detail, right? Whatever we put in between will still be a fixed offset, no? It's a different offset, but that's what patching can deal with.
No, what Sami is sayin that because the "cmpl" *inside* the function that checks the hash value will have that same (valid) hash value encoded as part of it, then you actually have *two* valid markers with that hash value.
You have the "real" marker before the function.
But you also have the "false" marker that is part of the hash check that is *inside* the function.
The "real marker + 6" points to the function head itself, and so is ok as a target (normal operation).
The "false marker + 6" points to the "UD2", and so is *also* ok as a target (bad guy trying to mis-use the false marker gets trapped by UD2).
Linus
Quoted 8 lines · Linus Torvalds
You have the "real" marker before the function.
But you also have the "false" marker that is part of the hash check that is *inside* the function.
The "real marker + 6" points to the function head itself, and so is ok as a target (normal operation).
Of course, one fix for that is to make the hash be only 24 bits, and make the int3 byte part of the value you check, and not have the same pattern in the checking code at all.
Honestly, I think that would be a better model - yes, you lose 8 bits of hash, but considering that apparently the current KCFI code *guarantees* that the hash pattern will exist even outside the actual target pattern, I think it's still a better model.
I also happen to believe that the kCFI code should have entirely different targets for direct jumps and for indirect jumps, but that's a separate issue. Maybe it already does that?
Linus
Peter Zijlstra
maintainerQuoted 3 lines · Linus Torvalds
I also happen to believe that the kCFI code should have entirely different targets for direct jumps and for indirect jumps, but that's a separate issue. Maybe it already does that?
kCFI is purely about indirect calls.
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Linus Torvalds
I also happen to believe that the kCFI code should have entirely different targets for direct jumps and for indirect jumps, but that's a separate issue. Maybe it already does that?
kCFI is purely about indirect calls.
So it already only adds the pattern to things that have their address taken, not all functions?
If so, that's simple enough to sort out: don't do any RSB stack adjustment for those thunks AT ALL.
Because they should just then end up with a jump to the "real" target, and that real target will do the RSB stack thing.
Linus
Quoted 9 lines · Linus Torvalds
So it already only adds the pattern to things that have their address taken, not all functions?
If so, that's simple enough to sort out: don't do any RSB stack adjustment for those thunks AT ALL.
Because they should just then end up with a jump to the "real" target, and that real target will do the RSB stack thing.
Put another way, let's say that you have a function that looks like this:
int silly(void)
{
return 0;
}and now you have two cases:
- the "direct callable version" of that function looks exactly the
way it always has looked, and gets the 16 bytes of padding for it, and the RSB counting can happen in that padding
- the "somebody took the address of this function" creates code that
has the hash marker before it, and has the hash check, and then does a "jmp silly" to actually jump to the real code.
So what the RSB counting does is just ignore that second case entirely as far as the RSB code generation goes. No need to have any padding for it at all, it has that (completely different) kCFI padding instead.
Instead, only the "real" silly function gets that RSB code, and the "jmp silly" from the kCFI thunk needs to be updated to point to the RSB thunk in front of it.
Yes, yes, it makes indirect calls slightly more expensive than direct calls (because that kCFI thing can't just fall through to the real thing), but considering all the *other* costs of indirect calls, the cost of having that one "jmp" instruction doesn't really seem to matter, does it?
Linus
Peter Zijlstra
maintainerQuoted 38 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds
So it already only adds the pattern to things that have their address taken, not all functions?
If so, that's simple enough to sort out: don't do any RSB stack adjustment for those thunks AT ALL.
Because they should just then end up with a jump to the "real" target, and that real target will do the RSB stack thing.
Put another way, let's say that you have a function that looks like this:
int silly(void) { return 0; }and now you have two cases:
- the "direct callable version" of that function looks exactly the
way it always has looked, and gets the 16 bytes of padding for it, and the RSB counting can happen in that padding- the "somebody took the address of this function" creates code that
has the hash marker before it, and has the hash check, and then does a "jmp silly" to actually jump to the real code.So what the RSB counting does is just ignore that second case entirely as far as the RSB code generation goes. No need to have any padding for it at all, it has that (completely different) kCFI padding instead.
Instead, only the "real" silly function gets that RSB code, and the "jmp silly" from the kCFI thunk needs to be updated to point to the RSB thunk in front of it.
Quoted 5 lines
Yes, yes, it makes indirect calls slightly more expensive than direct calls (because that kCFI thing can't just fall through to the real thing), but considering all the *other* costs of indirect calls, the cost of having that one "jmp" instruction doesn't really seem to matter, does it?
So it's like 2:15 am here, so I might not be following things right, but doesn't the above mean you have to play funny games with what a function pointer is?
That is, the content of a function pointer (address taken) no longer match the actual function? That gives grief with things like static_call(), ftrace and other things that write call instructions instead of doing indirect calls.
Quoted 4 lines · Peter Zijlstra
So it's like 2:15 am here, so I might not be following things right, but doesn't the above mean you have to play funny games with what a function pointer is?
Yes, but probably no more than compilers already do.
On x86, function pointers are simple, and just pointers to the first instruction of the function.
But that's actually not true in general, and several other architectures have *much* more complicated function pointers, where they are pointers to special "function descriptor blocks" etc.
So I bet gcc has all that infrastructure in place anyway.
And the whole "use a different address for a direct call than for an indirect call" is still much simpler than having an actual separate function descriptor thing.
At worst, you'd actually always generate the thunk for the indirect call case, and let the linker kill unused cases. The compiler wouldn't even have to know about the two cases, except to use a different names for the direct call case.
Do I claim it would be *pretty*? No. But I bet the existing CFI patches already do things like this anyway.
(I have llvm sources on my machine too, because I used to build my own clang from source back when I was testing the asm goto stuff. But unlike gcc, I've never really *looked* at llvm, so I'm not familiar with it at all, and I'm not going to try to figure out what the CFI code actually does, and instead just handwave widely while saying "I bet it already does this".)
Linus
Sami Tolvanen
maintainerQuoted 5 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds
So it already only adds the pattern to things that have their address taken, not all functions?
The preamble is added to address-taken static functions and all global functions, because those might be indirectly called from other translation units. With LTO, we could prune unnecessary preambles from non-address-taken globals too.
Quoted 15 lines
Quoted 5 lines
If so, that's simple enough to sort out: don't do any RSB stack adjustment for those thunks AT ALL.
Because they should just then end up with a jump to the "real" target, and that real target will do the RSB stack thing.
Put another way, let's say that you have a function that looks like this:
int silly(void) { return 0; }and now you have two cases:
Quoted 7 lines
- the "direct callable version" of that function looks exactly the
way it always has looked, and gets the 16 bytes of padding for it, and the RSB counting can happen in that padding- the "somebody took the address of this function" creates code that
has the hash marker before it, and has the hash check, and then does a "jmp silly" to actually jump to the real code.
Clang's current CFI implementation is somewhat similar to this. It creates separate thunks for address-taken functions and changes function addresses in C code to point to the thunks instead.
While this works, it creates painful situations when interacting with assembly (e.g. a function address taken in assembly cannot be used for indirect calls in C as it doesn't point to the thunk) and needs unpleasant hacks when we want take the actual function address in C (i.e. scattering the code with function_nocfi() calls).
I have to agree with Peter on this, I would rather avoid messing with function pointers in KCFI to avoid these issues.
Sami
Peter Zijlstra
maintainerQuoted 4 lines · Sami Tolvanen
Clang's current CFI implementation is somewhat similar to this. It creates separate thunks for address-taken functions and changes function addresses in C code to point to the thunks instead.
Quoted 8 lines
While this works, it creates painful situations when interacting with assembly (e.g. a function address taken in assembly cannot be used for indirect calls in C as it doesn't point to the thunk) and needs unpleasant hacks when we want take the actual function address in C (i.e. scattering the code with
function_nocfi()calls).I have to agree with Peter on this, I would rather avoid messing with function pointers in KCFI to avoid these issues.
It is either this; and I think I can avoid the worst of it (see below); or grow the indirect_callsites to obscure the immediate (as Linus suggested), there's around ~16k indirect callsites in a defconfig-ish kernel, so growing it isn't too horrible, but it isn't nice either.
The prettiest option to obscure the immediate at the callsite I could conjure up is something like:
kcfi_caller_linus:
movl $0x12345600, %r10d movb $0x78, %r10b cmpl %r10d, -OFFSET(%r11) je 1f ud2
1: call __x86_thunk_indirect_r11
Which comes to around 22 bytes (+5 over the original).
Joao suggested putting part of that in the retpoline thunk like:
kcfi_caller_joao:
movl $0x12345600, %r10d movb $0x78, %r10b call __x86_thunk_indirect_cfi
__x86_thunk_indirect_cfi:
cmpl %r10d, -OFFSET(%r11) je 1f ud2
1:
call 1f
int3
1:
mov %r11, (%rsp) ret int3
The only down-side there is that eIBRS hardware doesn't need retpolines (given we currently default to ignoring Spectre-BHB) and as such this doesn't really work nicely (we don't want to re-introduce funneling).
The other option I came up with, alluded to above, is below, and having written it out, I'm pretty sure I faviour just growing the indirect callsite as per Linus' option above.
Suppose:
indirect_callsite:
cmpl $0x12345678, -6(%r11) # 8 je 1f # 2 ud2 # 2 call __x86_indirect_thunk_r11 # 5 (-> .retpoline_sites)
__cfi_\func:
movl $0x12345678, %eax # 5 int3 # 1 int3 # 1 \func: # aligned 16 endbr # 4 nop12 # 12 call __fentry__ # 5 ...
And for functions that do not get their address taken:
\func: # aligned 16
nop16 # 16 call __fentry__ # 5 ...
Instead, extend the objtool .call_sites to also include tail-calls and for:
- regular (!SKL, !IBT) systems;
* patch all direct calls/jmps to +16 (.call_sites)
* static_call/ftrace/etc.. can triviall add the +16
* retpolines can do +16 for the indirect calls
* retutn thunks are patched to ret;int3 (.return_sites)
(indirect calls for eIBRS which don't use retpoline
simply eat the nops)
- SKL systems;
* patch the first 16 bytes into:
nop6 sarq $5, PER_CPU_VAR(__x86_call_depth)
* patch all direct calls to +6 (.call_sites)
* patch all direct jumps to +16 (.call_sites)
* static_call/ftrace adjust to +6/+16 depending on instruction type
* retpolines are split between call/jmp and do +6/+16 resp.
* return thunks are patches to x86_return_skl (.return_sites)
- IBT systes;
* patch the first 16 bytes to:
endbr # 4 xorl $0x12345678, %r10d # 7 je 1f # 2 ud2 # 2 nop # 1
1:
* patch the callsites to: (.retpoline_sites)
movl $0x12345678, %r10d # 7 call *$r11 # 3 nop7 # 7
* patch all the direct calls/jmps to +16 (.call_sites)
* static_call/ftrace/etc.. add +16
* retutn thunks are patched to ret;int3 (.return_sites)
Yes, frobbing the address for static_call/ftrace/etc.. is a bit horrible, but at least &sym remains exactly that address and not something magical.
Note: It is possible to shift the __fentry__ call, but that would mean that we loose alignment or get to carry .call_sites at runtime (and it is *huge*)
From: Peter Zijlstra
Quoted 1 line
Sent: 20 July 2022 22:13
...
Quoted 4 lines
The prettiest option to obscure the immediate at the callsite I could conjure up is something like:
kcfi_caller_linus:
Quoted 8 lines
movl $0x12345600, %r10d movb $0x78, %r10b cmpl %r10d, -OFFSET(%r11) je 1f ud21: call
__x86_thunk_indirect_r11Which comes to around 22 bytes (+5 over the original).
You'd be better doing:
movl $0x12345678-0xaa, %r10d addl $0xaa, %r10d
so that the immediate is obscured even if the low bits are zero.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
From: David Laight
Quoted 16 lines
Sent: 21 July 2022 09:22
From: Peter Zijlstra
Quoted 1 line
Sent: 20 July 2022 22:13
...
Quoted 11 lines
The prettiest option to obscure the immediate at the callsite I could conjure up is something like:
kcfi_caller_linus:movl $0x12345600, %r10d movb $0x78, %r10b cmpl %r10d, -OFFSET(%r11) je 1f ud21: call
__x86_thunk_indirect_r11
Quoted 6 lines
Quoted 1 line
Which comes to around 22 bytes (+5 over the original).
You'd be better doing:
movl $0x12345678-0xaa, %r10d addl $0xaa, %r10dso that the immediate is obscured even if the low bits are zero.
Actually, can't you use %eax instead of %r10d?
IIRC it is only used for the number of FP registers in a varargs call - and that isn't used in the kernel.
That removes the 3 'REG' prefixes and lets you use the 2-byte 04-xx instruction to add to %al.
Although I'm sure I remember something about a penalty for accessing %al just after the full register.
So the 3-byte sign extending add may be better.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Peter Zijlstra
maintainerQuoted 24 lines · Peter Zijlstra
Quoted 4 lines · Sami Tolvanen
Clang's current CFI implementation is somewhat similar to this. It creates separate thunks for address-taken functions and changes function addresses in C code to point to the thunks instead.
Quoted 8 lines
While this works, it creates painful situations when interacting with assembly (e.g. a function address taken in assembly cannot be used for indirect calls in C as it doesn't point to the thunk) and needs unpleasant hacks when we want take the actual function address in C (i.e. scattering the code with
function_nocfi()calls).I have to agree with Peter on this, I would rather avoid messing with function pointers in KCFI to avoid these issues.
It is either this; and I think I can avoid the worst of it (see below); or grow the
indirect_callsitesto obscure the immediate (as Linus suggested), there's around ~16k indirect callsites in a defconfig-ish kernel, so growing it isn't too horrible, but it isn't nice either.The prettiest option to obscure the immediate at the callsite I could conjure up is something like:
kcfi_caller_linus:
Quoted 8 lines
movl $0x12345600, %r10d movb $0x78, %r10b cmpl %r10d, -OFFSET(%r11) je 1f ud21: call
__x86_thunk_indirect_r11Which comes to around 22 bytes (+5 over the original).
My very firstest LLVM patch; except it explodes at runtime and I'm not sure where to start looking...
On top of sami-llvm/kcfi
If I comment out the orl and cmpl it compiles stuff, put either one back and it explodes in some very unhelpful message.
The idea is the above callthunk that makes any offset work by not having the full hash as an immediate and allow kCFI along with -fpatchable-function-entry=N,M and include M in the offset.
Specifically, I meant to use -fpatchable-function-entry=16,16, but alas, I never got that far.
Help ?
---
llvm/lib/Target/X86/X86AsmPrinter.cpp +0 -11
@@ -124,23 +124,12 @@ void X86AsmPrinter::emitKCFITypeId(const MachineFunction &MF, OutStreamer->emitSymbolAttribute(FnSym, MCSA_ELF_TypeFunction); OutStreamer->emitLabel(FnSym); - // Emit int3 padding to allow runtime patching of the preamble. - EmitAndCountInstruction(MCInstBuilder(X86::INT3)); - EmitAndCountInstruction(MCInstBuilder(X86::INT3)); - // Embed the type hash in the X86::MOV32ri instruction to avoid special // casing object file parsers. EmitAndCountInstruction(MCInstBuilder(X86::MOV32ri) .addReg(X86::EAX) .addImm(Type->getZExtValue())); - // The type hash is encoded in the last four bytes of the X86::MOV32ri - // instruction. Emit additional X86::INT3 padding to ensure the hash is - // at offset -6 from the function start to avoid potential call target - // gadgets in checks emitted by X86AsmPrinter::LowerKCFI_CHECK. - EmitAndCountInstruction(MCInstBuilder(X86::INT3)); - EmitAndCountInstruction(MCInstBuilder(X86::INT3)); - if (MAI->hasDotTypeDotSizeDirective()) { MCSymbol *EndSym = OutContext.createTempSymbol("cfi_func_end"); OutStreamer->emitLabel(EndSym);
llvm/lib/Target/X86/X86MCInstLower.cpp +25 -13
@@ -1340,22 +1340,34 @@ void X86AsmPrinter::LowerKCFI_CHECK(const MachineInstr &MI) { assert(std::next(MI.getIterator())->isCall() && "KCFI_CHECK not followed by a call instruction"); - // The type hash is encoded in the last four bytes of the X86::CMP32mi - // instruction. If we decided to place the hash immediately before - // indirect call targets (offset -4), the X86::JCC_1 instruction we'll - // emit next would be a potential indirect call target as it's preceded - // by a valid type hash. - // - // To avoid generating useful gadgets, X86AsmPrinter::emitKCFITypeId - // emits the type hash prefix at offset -6, which makes X86::TRAP the - // only possible target in this instruction sequence. - EmitAndCountInstruction(MCInstBuilder(X86::CMP32mi) + const Function &F = MF->getFunction(); + unsigned Imm = MI.getOperand(1).getImm(); + unsigned Num; + + if (F.hasFnAttribute("patchable-function-prefix")) { + if (F.getFnAttribute("patchable-function-prefix") + .getValueAsString() + .getAsInteger(10, Num)) + Num = 0; + } + + // movl $0x12345600, %r10d + EmitAndCountInstruction(MCInstBuilder(X86::MOV32ri) + .addReg(X86::R10) + .addImm(Imm & ~0xff)); + + // orl $0x78, %r10d + EmitAndCountInstruction(MCInstBuilder(X86::OR32ri8) + .addReg(X86::R10) + .addImm(Imm & 0xff)); + + // cmpl %r10, -off(%r11) + EmitAndCountInstruction(MCInstBuilder(X86::CMP32rm) + .addReg(X86::R10) .addReg(MI.getOperand(0).getReg()) .addImm(1) .addReg(X86::NoRegister) - .addImm(-6) - .addReg(X86::NoRegister) - .addImm(MI.getOperand(1).getImm())); + .addImm(-(Num + 4))); MCSymbol *Pass = OutContext.createTempSymbol(); EmitAndCountInstruction(
Peter Zijlstra
maintainerQuoted 4 lines · Peter Zijlstra
My very firstest LLVM patch; except it explodes at runtime and I'm not sure where to start looking...
On top of sami-llvm/kcfi
Thanks Sami!
this seems to work, let me go hack the kernel..
--- clang/lib/Driver/SanitizerArgs.cpp | 12 --------- llvm/lib/Target/X86/X86AsmPrinter.cpp | 11 -------- llvm/lib/Target/X86/X86MCInstLower.cpp | 47 ++++++++++++++++++++++------------ 3 files changed, 31 insertions(+), 39 deletions(-)
clang/lib/Driver/SanitizerArgs.cpp +0 -12
@@ -719,18 +719,6 @@ SanitizerArgs::SanitizerArgs(const ToolChain &TC, D.Diag(diag::err_drv_argument_not_allowed_with) << "-fsanitize=kcfi" << lastArgumentForMask(D, Args, SanitizerKind::CFI); - - if (Arg *A = Args.getLastArg(options::OPT_fpatchable_function_entry_EQ)) { - StringRef S = A->getValue(); - unsigned N, M; - // With -fpatchable-function-entry=N,M, where M > 0, - // llvm::AsmPrinter::emitFunctionHeader injects nops before the - // KCFI type identifier, which is currently unsupported. - if (!S.consumeInteger(10, N) && S.consume_front(",") && - !S.consumeInteger(10, M) && M > 0) - D.Diag(diag::err_drv_argument_not_allowed_with) - << "-fsanitize=kcfi" << A->getAsString(Args); - } } Stats = Args.hasFlag(options::OPT_fsanitize_stats,
llvm/lib/Target/X86/X86AsmPrinter.cpp +0 -11
@@ -124,23 +124,12 @@ void X86AsmPrinter::emitKCFITypeId(const MachineFunction &MF, OutStreamer->emitSymbolAttribute(FnSym, MCSA_ELF_TypeFunction); OutStreamer->emitLabel(FnSym); - // Emit int3 padding to allow runtime patching of the preamble. - EmitAndCountInstruction(MCInstBuilder(X86::INT3)); - EmitAndCountInstruction(MCInstBuilder(X86::INT3)); - // Embed the type hash in the X86::MOV32ri instruction to avoid special // casing object file parsers. EmitAndCountInstruction(MCInstBuilder(X86::MOV32ri) .addReg(X86::EAX) .addImm(Type->getZExtValue())); - // The type hash is encoded in the last four bytes of the X86::MOV32ri - // instruction. Emit additional X86::INT3 padding to ensure the hash is - // at offset -6 from the function start to avoid potential call target - // gadgets in checks emitted by X86AsmPrinter::LowerKCFI_CHECK. - EmitAndCountInstruction(MCInstBuilder(X86::INT3)); - EmitAndCountInstruction(MCInstBuilder(X86::INT3)); - if (MAI->hasDotTypeDotSizeDirective()) { MCSymbol *EndSym = OutContext.createTempSymbol("cfi_func_end"); OutStreamer->emitLabel(EndSym);
llvm/lib/Target/X86/X86MCInstLower.cpp +31 -16
53 lines not on this page · raw message
Quoted 2 lines · Peter Zijlstra
this seems to work, let me go hack the kernel..
Am I missing something?
Isn't this generating
movl $~IMM,%r10d
negl %r10d
cmpl %r10d,-4(%calldest)for the sequence?
That seems bogus for two reasons:
(a) 'neg' is not the opposite of '~'. Did you mean 'notl' or did you mean '-'?
Or am I missing something entirely?
(b) since you have that r10 use anyway, why can't you just generate the simpler
movl $-IMM,%r10d
addl -4(%calldest),%r10d
instead? You only need ZF anyway.
Maybe you need to add some "r10 is clobbered" thing, I don't know.
But again: I don't know llvm, so the above is basically me just doing the "pattern matching monkey" thing.
Linus
Peter Zijlstra
maintainerQuoted 12 lines · Linus Torvalds
Quoted 2 lines · Peter Zijlstra
this seems to work, let me go hack the kernel..
Am I missing something?
Isn't this generating
movl $~IMM,%r10d negl %r10d cmpl %r10d,-4(%calldest)
Quoted 7 lines
for the sequence?
That seems bogus for two reasons:
(a) 'neg' is not the opposite of '~'. Did you mean 'notl' or did you mean '-'?
Or am I missing something entirely?
No, you're right, I'm being daft again.
Quoted 6 lines
(b) since you have that r10 use anyway, why can't you just generate the simpler
movl $-IMM,%r10d addl -4(%calldest),%r10d instead? You only need ZF anyway.
Right, lemme see if I can wrangle llvm to generate that.
Quoted 1 line
Maybe you need to add some "r10 is clobbered" thing, I don't know.
R11,R11 are caller-saved, and since this is the actual call site, the caller must already have saved them or marked them clobbered.
Quoted 3 lines · Peter Zijlstra
R11,R11 are caller-saved, and since this is the actual call site, the caller must already have saved them or marked them clobbered.
Ok. I don't know the context, but I was thinking along the lines of the same hash value perhaps being used multiple times because it has the same function type. Then using the "addl" trick means that the hash value in %r10 will be changing and cannot be re-used.
But I guess this is *much* too late for those kinds of optimizations, as it literally just outputs the raw instruction sequence, and so the (negated) hash value will always be re-generated anyway, no re-use possible.
Linus
Quoted 4 lines
Ok. I don't know the context, but I was thinking along the lines of the same hash value perhaps being used multiple times because it has the same function type. Then using the "addl" trick means that the hash value in %r10 will be changing and cannot be re-used.
Fwiiw, even if %r10 value was not being destroyed by the "addl", the call right after the check implies that you cannot trust the contents of %r10 anymore (it may have been messed up within the called function).
Sami Tolvanen
maintainerQuoted 23 lines · Peter Zijlstra
Quoted 12 lines · Linus Torvalds
Quoted 2 lines · Peter Zijlstra
this seems to work, let me go hack the kernel..
Am I missing something?
Isn't this generating
movl $~IMM,%r10d negl %r10d cmpl %r10d,-4(%calldest)Quoted 7 lines
for the sequence?
That seems bogus for two reasons:
(a) 'neg' is not the opposite of '~'. Did you mean 'notl' or did you mean '-'?
Or am I missing something entirely?
No, you're right, I'm being daft again.
Quoted 8 lines
Quoted 6 lines
(b) since you have that r10 use anyway, why can't you just generate the simpler
movl $-IMM,%r10d addl -4(%calldest),%r10d instead? You only need ZF anyway.Right, lemme see if I can wrangle llvm to generate that.
That looks good to me. I updated my LLVM tree to generate this code for the checks:
https://github.com/samitolvanen/llvm-project/commits/kcfi
Sami
Peter Zijlstra
maintainerQuoted 4 lines · Sami Tolvanen
That looks good to me. I updated my LLVM tree to generate this code for the checks:
Thanks!
The alignment thing you added:
// Emit int3 padding before the type information to maintain alignment. // The X86::MOV32ri instruction we emit is 5 bytes long. uint64_t Padding = offsetToAlignment(5, MF.getAlignment()); while (Padding--)
EmitAndCountInstruction(MCInstBuilder(X86::INT3));
Doesn't seem to quite do what we want though.
When I use -fpatchable-function-entry=16,16 we effectively get a 32 byte prefix on every function:
0000000000000000 <__cfi___traceiter_sched_kthread_stop>:
0: cc int3
1: cc int3
2: cc int3
3: cc int3
4: cc int3
5: cc int3
6: cc int3
7: cc int3
8: cc int3
9: cc int3
a: cc int3
b: b8 26 b1 df 98 mov $0x98dfb126,%eax
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 90 nop
16: 90 nop
17: 90 nop
18: 90 nop
19: 90 nop
1a: 90 nop
1b: 90 nop
1c: 90 nop
1d: 90 nop
1e: 90 nop
1f: 90 nopAnd given the parameters, that's indeed the only option. However, given I can scribble the type thing just fine when moving to FineIBT and the whole Skylake depth tracking only needs 10 bytes, I figured I'd try:
-fpatchable-function-entry=11,11 instead. But that resulted in unalignment:
0000000000000000 <__cfi___traceiter_sched_kthread_stop>:
0: cc int3
1: cc int3
2: cc int3
3: cc int3
4: cc int3
5: cc int3
6: cc int3
7: cc int3
8: cc int3
9: cc int3
a: cc int3
b: b8 26 b1 df 98 mov $0x98dfb126,%eax
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 90 nop
16: 90 nop
17: 90 nop
18: 90 nop
19: 90 nop
1a: 90 nop000000000000001b <__traceiter_sched_kthread_stop>:
However, if I change clang like so:
llvm/lib/Target/X86/X86AsmPrinter.cpp | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
llvm/lib/Target/X86/X86AsmPrinter.cpp +7 -1
@@ -124,9 +124,15 @@ void X86AsmPrinter::emitKCFITypeId(const MachineFunction &MF, OutStreamer->emitSymbolAttribute(FnSym, MCSA_ELF_TypeFunction); OutStreamer->emitLabel(FnSym); + int64_t PrefixNops = 0; + (void)MF.getFunction() + .getFnAttribute("patchable-function-prefix") + .getValueAsString() + .getAsInteger(10, PrefixNops); + // Emit int3 padding before the type information to maintain alignment. // The X86::MOV32ri instruction we emit is 5 bytes long. - uint64_t Padding = offsetToAlignment(5, MF.getAlignment()); + uint64_t Padding = offsetToAlignment(5+PrefixNops, MF.getAlignment()); while (Padding--) EmitAndCountInstruction(MCInstBuilder(X86::INT3));
Then it becomes:
0000000000000000 <__cfi___traceiter_sched_kthread_stop>:
0: b8 26 b1 df 98 mov $0x98dfb126,%eax
5: 90 nop
6: 90 nop
7: 90 nop
8: 90 nop
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop0000000000000010 <__traceiter_sched_kthread_stop>:
and things are 'good' again, except for functions that don't get a kcfi preamble, those are unaligned... I couldn't find where the patchable-function-prefix nops are generated to fix this up :/
Also; could you perhaps add a switch to supress ENDBR for functions with a kCFI preamble ?
Sami Tolvanen
maintainerQuoted 2 lines · Peter Zijlstra
and things are 'good' again, except for functions that don't get a kcfi preamble, those are unaligned...
One way to fix this would be to just emit an empty KCFI preamble for non-address-taken functions when patchable-function-prefix > 0, so all the functions end up with the same alignment.
Note that Clang doesn't keep the function entry aligned with -fpatchable-function-entry=N,M, where M>0. It generates .p2align 4, 0x90 before the nops, but if you want to maintain alignment for the entry, you just have to tell it to generate the correct number of prefix nops.
Quoted 2 lines
I couldn't find where the patchable-function-prefix nops are generated to fix this up :/
It's all in AsmPrinter::emitFunctionHeader, look for emitNops.
Quoted 2 lines
Also; could you perhaps add a switch to supress ENDBR for functions with a kCFI preamble ?
I'm planning to do that in a follow-up patch. I would rather not add features that are not critical to the initial patch to avoid further delays in getting the compiler changes accepted.
Sami
From: Linus Torvalds
Quoted 1 line
Sent: 21 July 2022 19:07
...
Quoted 5 lines
(b) since you have that r10 use anyway, why can't you just generate the simpler
movl $-IMM,%r10d
addl -4(%calldest),%r10d
Quoted 8 lines
instead? You only need ZF anyway.
Maybe you need to add some "r10 is clobbered" thing, I don't know.
But again: I don't know llvm, so the above is basically me just doing the "pattern matching monkey" thing.
Linus
Since: "If the callee is a variadic function, then the number of floating point arguments passed to the function in vector registers must be provided by the caller in the AL register."
And that that never happens in the kernel you can use %eax instead of %r10d.
Even in userspace %al can be set non-zero after the signature check.
If you are willing to cut the signature down to 26 bits and then ensure that one of the bytes of -IMM (or ~IMM if you use xor) is 0xcc and jump back to that on error the check becomes:
movl $-IMM,%eax 1: addl -4(%calldest),%eax jnz 1b-1 // or -2, -3, -4 add $num_fp_args,%eax // If needed non-zero call %calldest
I think that adds 10 bytes to the call site. Although with retpoline thunks (and no fp varargs calls) all but the initial movl can go into the thunk.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Peter Zijlstra
maintainerQuoted 6 lines · David Laight
Since: "If the callee is a variadic function, then the number of floating point arguments passed to the function in vector registers must be provided by the caller in the AL register."
And that that never happens in the kernel you can use %eax instead of %r10d.
Except there's the AMD BTC thing and we should (compiler patch seems MIA) have an unconditional: 'xor %eax,%eax' in front of every function call.
(The official mitigation strategy was CALL; LFENCE IIRC, but that's so horrible nobody is actually considering that)
Yes, the suggested sequence ends with rax being zero, but since we start the speculation before that result is computed that's not good enough I suspect.
From: Peter Zijlstra
Quoted 8 lines
Sent: 22 July 2022 12:03
Quoted 4 lines · David Laight
Since: "If the callee is a variadic function, then the number of floating point arguments passed to the function in vector registers must be provided by the caller in the AL register."
Quoted 6 lines
Quoted 2 lines
And that that never happens in the kernel you can use %eax instead of %r10d.
Except there's the AMD BTC thing and we should (compiler patch seems MIA) have an unconditional: 'xor %eax,%eax' in front of every function call.
I've just read https://www.amd.com/system/files/documents/technical-guidance-for-mitigating-branch-type-confusion_v7_20220712.pdf
It doesn't seem to suggest clearing registers except as a vague 'might help' before a function return (to limit what the speculated code can do.
The only advantage I can think of for 'xor ax,ax' is that it is done as a register rename - and isn't dependant on older instructions. So it might reduce some pipeline stalls.
I'm guessing that someone might find a 'gadget' that depends on %eax and it may be possible to find somewhere that leaves an arbitrary value in it.
It is also about the only register that isn't live!
Quoted 6 lines
(The official mitigation strategy was CALL; LFENCE IIRC, but that's so horrible nobody is actually considering that)
Yes, the suggested sequence ends with rax being zero, but since we start the speculation before that result is computed that's not good enough I suspect.
The speculated code can't use the 'wrong' %eax value. The only problem is that reading from -4(%r11) is likely to be a D$ miss giving plenty of time for the cpu to execute 'crap'. But I'm not sure a later 'xor ax,ax' helps.
(OTOH this is all horrid and makes my brian hurt.)
AFAICT with BTC you 'just lose'.
I thought it was bad enough that some cpu used the BTB for predicted conditional jumps - but using it to decide 'this must be a branch instruction' seems especially broken.
Seems the best thing to do with those cpu is to run an embedded system with a busybox+buildroot userspace where almost everything runs as root :-)
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 7 lines · Peter Zijlstra
On Wed, Jul 20, 2022 at 11:13:16PM +0200, Peter Zijlstra wrote:
The idea is the above callthunk that makes any offset work by not having the full hash as an immediate and allow kCFI along with -fpatchable-function-entry=N,M and include M in the offset.Specifically, I meant to use -fpatchable-function-entry=16,16, but alas, I never got that far.
That's embarrasing. I missed that option in the maze of gcc options. That would have spared me to stare at gcc code :)
Quoted 5 lines · Linus Torvalds
Honestly, I think that would be a better model - yes, you lose 8 bits of hash, but considering that apparently the current KCFI code *guarantees* that the hash pattern will exist even outside the actual target pattern,
Gaah, I'm being stupid,. You still get the value collision, since the int3 byte pattern would just be part of the compare pattern.
You'd have to use some multi-instruction compare to avoid having the pattern in the instruction stream. Probably with another register. Like
movl -FIXED_OFFSET(%eax),%rdx
addl $ANTI_PATTERN,%rdx
je ok
so that the "compare" wouldn't use the same pattern value, but be an add with the negated pattern value instead.
The extra instruction is likely less of a problem than the extra register used.
Linus
Quoted 2 lines
The extra instruction is likely less of a problem than the extra register used.
FWIIW, per-ABI, R11 is a scratch-reg and should be usable without hard consequences in this scenario.
Joao
Sami Tolvanen
maintainerQuoted 5 lines · Joao Moreira
Quoted 3 lines
The extra instruction is likely less of a problem than the extra register used.
FWIIW, per-ABI, R11 is a scratch-reg and should be usable without hard consequences in this scenario.
Clang always uses r11 for the indirect call with retpolines, so we'd need to use another register. Nevertheless, splitting the constant into two instructions would solve the call target gadget issue.
Sami
Quoted 3 lines
Clang always uses r11 for the indirect call with retpolines, so we'd need to use another register. Nevertheless, splitting the constant into two instructions would solve the call target gadget issue.
Yeah, it clicked later yesterday. But, FWIIW, R10 is also considered a scratch register, although used for passing static chain pointers which I think is not a thing in kernel context. Last case scenario we can always do liveness analysis and I doubt we'll have a significant (if any) number of spills.
If we are comparing through registers, I would suggest using a sub instruction instead of a cmp, as this will destroy the contents of the register and prevent it from being re-used on further unprotected indirect branches, if any exists.
From: Linus Torvalds
Quoted 17 lines
Sent: 19 July 2022 01:02
Quoted 5 lines · Linus Torvalds
Honestly, I think that would be a better model - yes, you lose 8 bits of hash, but considering that apparently the current KCFI code *guarantees* that the hash pattern will exist even outside the actual target pattern,
Gaah, I'm being stupid,. You still get the value collision, since the int3 byte pattern would just be part of the compare pattern.
You'd have to use some multi-instruction compare to avoid having the pattern in the instruction stream. Probably with another register. Like
Quoted 8 lines
movl -FIXED_OFFSET(%eax),%rdx
addl $ANTI_PATTERN,%rdx
je okso that the "compare" wouldn't use the same pattern value, but be an add with the negated pattern value instead.
The extra instruction is likely less of a problem than the extra register used.
Shouldn't it be testing the value the caller supplied?
The extra instruction is likely to be one clock - I doubt it will sensibly run in parallel with code later in the function.
The larger costs are (probably) polluting the D$ with I addresses (already done by the caller) and the likely mispredicted 'je ok'. Unless the function has been recently called the 'je ok' gets static prediction.
While traditionally that would predict a forwards branch 'not taken' ISTR more recent Intel cpu just use the predictor output - ie random. Not at all sure about AMD cpu though.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 2 lines · David Laight
Shouldn't it be testing the value the caller supplied?
Actually, I'm just all confused.
All that verification code is *in* the caller, before the call - to verify that the target looks fine.
I think I was confused by the hash thunk above the function also being generated with a "cmpl $hash". And I don't even know why that is, and why it wasn't just the bare constant.
Linus
Sami Tolvanen
maintainerQuoted 4 lines · Linus Torvalds
Quoted 2 lines · David Laight
Shouldn't it be testing the value the caller supplied?
Quoted 8 lines
Actually, I'm just all confused.
All that verification code is *in* the caller, before the call - to verify that the target looks fine.
I think I was confused by the hash thunk above the function also being generated with a "cmpl $hash". And I don't even know why that is, and why it wasn't just the bare constant.
The preamble hash is encoded into an instruction just to avoid special casing objtool, which would otherwise get confused about the random bytes. On arm64, we just emit a bare constant before the function.
Sami
Quoted 4 lines · Sami Tolvanen
The preamble hash is encoded into an instruction just to avoid special casing objtool, which would otherwise get confused about the random bytes. On arm64, we just emit a bare constant before the function.
Ahh.
I think objtool would want to understand about kCFI anyway, so I think in the long run that hack isn't a goog idea.
But I get why you'd do it as a "do this as just a compiler thing and hide it from objtool" as a development strategy.
Linus
Sami Tolvanen
maintainerQuoted 6 lines · Linus Torvalds
Quoted 4 lines · Sami Tolvanen
The preamble hash is encoded into an instruction just to avoid special casing objtool, which would otherwise get confused about the random bytes. On arm64, we just emit a bare constant before the function.
Quoted 7 lines
Ahh.
I think objtool would want to understand about kCFI anyway, so I think in the long run that hack isn't a goog idea.
But I get why you'd do it as a "do this as just a compiler thing and hide it from objtool" as a development strategy.
I believe it was actually Peter's idea to use an instruction. :) In earlier revisions of KCFI, I did teach objtool about the preambles, but that was just so it can ignore them.
Sami
Peter Zijlstra
maintainerQuoted 12 lines · Sami Tolvanen
Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Sami Tolvanen
The preamble hash is encoded into an instruction just to avoid special casing objtool, which would otherwise get confused about the random bytes. On arm64, we just emit a bare constant before the function.
Ahh.
I think objtool would want to understand about kCFI anyway, so I think in the long run that hack isn't a goog idea.
Quoted 6 lines
Quoted 2 lines
But I get why you'd do it as a "do this as just a compiler thing and hide it from objtool" as a development strategy.
I believe it was actually Peter's idea to use an instruction. :) In earlier revisions of KCFI, I did teach objtool about the preambles, but that was just so it can ignore them.
Right; even if we teach objtool about kCFI, having text be actual instructions makes things much nicer. Objdump and friends also shit their pants if you put random bytes in. It only costs a single byte to encode the immediate, so why not.
Specifically, the encoding used is:
movl $0x12345678, %eax
and that is 0xb8 followed by the constant, but there's plenty other single byte ops that could be used.
Sami Tolvanen
maintainerQuoted 8 lines · Peter Zijlstra
Quoted 2 lines · Thomas Gleixner
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Right, I need to go look at the latest kCFI patches, that sorta got side-tracked for working on all the retbleed muck :/
Quoted 8 lines
Basically kCFI wants to preface every (indirect callable) function with:
__cfi_\func:
int3 movl $0x12345678, %rax int3 int3\func:
Yes, and in order to avoid scattering the code with call target gadgets, the preamble should remain immediately before the function.
Quoted 6 lines
Ofc, we can still put the whole:
sarq $5, PER_CPU_VAR(__x86_call_depth); jmp \func_directthing in front of that.
Sure, that would work.
Quoted 2 lines
But it does somewhat destroy the version I had that only needs the 10 bytes padding for the sarq.
There's also the question of how function alignment should work in the KCFI case. Currently, the __cfi_ preamble is 16-byte aligned, which obviously means the function itself isn't.
Sami
Quoted 28 lines · Sami Tolvanen
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Thomas Gleixner
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Right, I need to go look at the latest kCFI patches, that sorta got side-tracked for working on all the retbleed muck :/
Quoted 8 lines
Basically kCFI wants to preface every (indirect callable) function with:
__cfi_\func:
int3 movl $0x12345678, %rax int3 int3\func:
Yes, and in order to avoid scattering the code with call target gadgets, the preamble should remain immediately before the function.
Quoted 6 lines
Ofc, we can still put the whole:
sarq $5, PER_CPU_VAR(__x86_call_depth); jmp \func_directthing in front of that.
Quoted 8 lines
Sure, that would work.
Quoted 2 lines
But it does somewhat destroy the version I had that only needs the 10 bytes padding for the sarq.
There's also the question of how function alignment should work in the KCFI case. Currently, the
__cfi_preamble is 16-byte aligned, which obviously means the function itself isn't.
That's bad. The function entry should be 16 byte aligned and as I just learned for AMD the ideal alignment would be possibly 32 byte as that's their I-fetch width. But my experiments with 16 bytes alignment independent of the padding muck is benefitial for both AMD and Intel over the 4 byte alignment we have right now.
This really needs a lot of thought and performance analysis before we commit to anything here. Peter's an my investigations have shown how sensitive this is.
We can't just add stuff without taking the whole picture into account (independent of the proposed padding muck).
Thanks,
tglx
Sami Tolvanen
maintainerQuoted 39 lines · Thomas Gleixner
Quoted 28 lines · Sami Tolvanen
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Thomas Gleixner
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Right, I need to go look at the latest kCFI patches, that sorta got side-tracked for working on all the retbleed muck :/
Quoted 8 lines
Basically kCFI wants to preface every (indirect callable) function with:
__cfi_\func:
int3 movl $0x12345678, %rax int3 int3\func:
Yes, and in order to avoid scattering the code with call target gadgets, the preamble should remain immediately before the function.
Quoted 6 lines
Ofc, we can still put the whole:
sarq $5, PER_CPU_VAR(__x86_call_depth); jmp \func_directthing in front of that.
Quoted 8 lines
Sure, that would work.
Quoted 2 lines
But it does somewhat destroy the version I had that only needs the 10 bytes padding for the sarq.
There's also the question of how function alignment should work in the KCFI case. Currently, the
__cfi_preamble is 16-byte aligned, which obviously means the function itself isn't.
Quoted 5 lines
That's bad. The function entry should be 16 byte aligned and as I just learned for AMD the ideal alignment would be possibly 32 byte as that's their I-fetch width. But my experiments with 16 bytes alignment independent of the padding muck is benefitial for both AMD and Intel over the 4 byte alignment we have right now.
OK, that's what I thought. KCFI hasn't landed in Clang yet, so it shouldn't be a problem to fix this.
Sami
Quoted 4 lines · Thomas Gleixner
That's bad. The function entry should be 16 byte aligned and as I just learned for AMD the ideal alignment would be possibly 32 byte as that's their I-fetch width.
In general, the L1 cache line size is likely the only "ideal" alignment.
Even if (I think) intel fetches just 16 bytes per cycle from the L1 I$ when decoding, being cacheline aligned still means that the L2->L1 transfer for the beginning of the function starts out better.
But Intel's current optimization many actually end sup having special rules:
When executing code from the Decoded Icache, direct branches that
are mostly taken should have all their instruction bytes in a 64B cache line and nearer the end of that cache line. Their targets should be at or near the beginning of a 64B cache line.
When executing code from the legacy decode pipeline, direct
branches that are mostly taken should have all their instruction bytes in a 16B aligned chunk of memory and nearer the end of that 16B aligned chunk. Their targets should be at or near the beginning of a 16B aligned chunk of memory.
So note how the branch itself should be at the end of the chunk, but the branch _target_ should be at the beginning of the chunk. And the chunk size is 16 bytes for decoding new instructions, and 64 bytes for predecoded.
I suspect that for the kernel, and for the beginning of the function (ie the target case), the 16-byte thing is still the main thing. Because the L0 I$ ("uop cache", "Decoded Icache", whatever you want to call it) is probably too small for a lot of kernel loads where user space has flushed things in between system calls or faults.
Older versions of the intel code just said "All branch targets should be 16-byte aligned".
So I think 16 bytes for function alignment ends up being what we generally want, but yes, it's slowly changing. AMD fetches 32-byte chunks, and Intel has that 64-bit predecode thing.
Linus
Peter Zijlstra
maintainerQuoted 18 lines · Sami Tolvanen
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Thomas Gleixner
And we need input from the Clang folks because their CFI work also puts stuff in front of the function entry, which nicely collides.
Right, I need to go look at the latest kCFI patches, that sorta got side-tracked for working on all the retbleed muck :/
Quoted 8 lines
Basically kCFI wants to preface every (indirect callable) function with:
__cfi_\func:
int3 movl $0x12345678, %rax int3 int3\func:
Quoted 2 lines
Yes, and in order to avoid scattering the code with call target gadgets, the preamble should remain immediately before the function.
I think we have a little room, but yes, -6 is just right to hit the UD2.
Quoted 8 lines
Quoted 6 lines
Ofc, we can still put the whole:
sarq $5, PER_CPU_VAR(__x86_call_depth); jmp \func_directthing in front of that.
Sure, that would work.
So if we assume \func starts with ENDBR, and further assume we've fixed up every direct jmp/call to land at +4, we can overwrite the ENDBR with part of the SARQ, that leaves us 6 more byte, placing the immediate at -10 if I'm not mis-counting.
Now, the call sites are:
41 81 7b fa 78 56 34 12 cmpl $0x12345678, -6(%r11)
74 02 je 1f
0f 0b ud2
e8 00 00 00 00 1: call __x86_indirect_thunk_r11
That means the offset of +10 lands in the middle of the CALL instruction, and since we only have 16 thunks there is a limited number of byte patterns available there.
This really isn't as nice as the -6 but might just work well enough, hmm?
Quoted 6 lines
Quoted 2 lines
But it does somewhat destroy the version I had that only needs the 10 bytes padding for the sarq.
There's also the question of how function alignment should work in the KCFI case. Currently, the
__cfi_preamble is 16-byte aligned, which obviously means the function itself isn't.
That seems unfortunate, at least the intel parts have a 16 byte i-fetch window (IIRC) so aligning the actual instructions at 16 bytes gets you the best bang for the buck wrt ifetch (functions are random access and not sequential).
Also, since we're talking at least 4 bytes more padding over the 7 that are required by the kCFI scheme, the FineIBT alternative gets a little more room to breathe. I'm thinking we can have the FineIBT landing site at -16.
__fineibt_\func:
endbr64 # 4 xorl $0x12345678, %r10d # 7 je \func+4 # 2 ud2 # 2
\func:
nop4 ...
With the callsite looking like:
nop7 movl $0x12345678, %r10d # 7 call *%r11 # 3
or something like that (everything having IBT has eIBRS at the very least).
Quoted 23 lines · Peter Zijlstra
Quoted 9 lines · Sami Tolvanen
Quoted 6 lines · Peter Zijlstra
Ofc, we can still put the whole:
sarq $5, PER_CPU_VAR(__x86_call_depth); jmp \func_directthing in front of that.
Sure, that would work.
So if we assume \func starts with ENDBR, and further assume we've fixed up every direct jmp/call to land at +4, we can overwrite the ENDBR with part of the SARQ, that leaves us 6 more byte, placing the immediate at -10 if I'm not mis-counting.
Now, the call sites are:
41 81 7b fa 78 56 34 12 cmpl $0x12345678, -6(%r11)
74 02 je 1f
0f 0b ud2
e8 00 00 00 00 1: call__x86_indirect_thunk_r11
Quoted 6 lines
That means the offset of +10 lands in the middle of the CALL instruction, and since we only have 16 thunks there is a limited number of byte patterns available there.
This really isn't as nice as the -6 but might just work well enough, hmm?
So I added a 32byte padding and put the thunk at the start:
sarq $5, PER_CPU_VAR(__x86_call_depth);
jmp \func_directFor sockperf that costs about 1% performance vs. the 16 byte variant. For mitigations=off it's a ~0.5% drop.
That's on a SKL. Did not check on other systems yet.
Thanks,
tglx
Sami Tolvanen
maintainerQuoted 12 lines · Peter Zijlstra
So if we assume \func starts with ENDBR, and further assume we've fixed up every direct jmp/call to land at +4, we can overwrite the ENDBR with part of the SARQ, that leaves us 6 more byte, placing the immediate at -10 if I'm not mis-counting.
Now, the call sites are:
41 81 7b fa 78 56 34 12 cmpl $0x12345678, -6(%r11)
74 02 je 1f
0f 0b ud2
e8 00 00 00 00 1: call__x86_indirect_thunk_r11
Quoted 6 lines
That means the offset of +10 lands in the middle of the CALL instruction, and since we only have 16 thunks there is a limited number of byte patterns available there.
This really isn't as nice as the -6 but might just work well enough, hmm?
I agree, this is probably fine, or at least low enough risk.
Did you have thoughts about changing the check instruction sequence to split the hash into multiple instructions and thus avoiding this issue altogether?
Sami
Sami Tolvanen
maintainerQuoted 6 lines · Peter Zijlstra
That means the offset of +10 lands in the middle of the CALL instruction, and since we only have 16 thunks there is a limited number of byte patterns available there.
This really isn't as nice as the -6 but might just work well enough, hmm?
pcc pointed out that we can also just add two more ud2 instructions to the check sequence if we want to be safe, with the cost of extra four bytes per callsite.
Quoted 6 lines
Also, since we're talking at least 4 bytes more padding over the 7 that are required by the kCFI scheme, the FineIBT alternative gets a little more room to breathe. I'm thinking we can have the FineIBT landing site at -16.
__fineibt_\func:
Quoted 8 lines
endbr64 # 4 xorl $0x12345678, %r10d # 7 je \func+4 # 2 ud2 # 2\func:
nop4 ...
I assume this means the preamble must also be 16-byte aligned to avoid performance issues with the FineIBT alternative? Which means we'll have a 16-byte preamble preceded by the usual nop padding.
Sami
Quoted 16 lines · Thomas Gleixner
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Thomas Gleixner
Let the compiler add a 16 byte padding in front of each function entry point and put the call depth accounting there. That avoids calling out into the module area and reduces ITLB pressure.
Ooh.
Quoted 7 lines
I actually like this a lot better.
Could we just say "use this instead if you have SKL and care about the issue?"
I don't hate your module thunk trick, but this does seem *so* much simpler, and if it performs better anyway, it really does seem like the better approach.
Quoted 8 lines
Yes, Peter and I came from avoiding a new compiler and the overhead for everyone when putting the padding into the code. We realized only when staring at the perf data that this padding in front of the function might be an acceptable solution. I did some more tests today on different machines with mitigations=off with kernels compiled with and without that padding. I couldn't find a single test case where the result was outside of the usual noise. But then my tests are definitely incomplete.
Here are some performance numbers for FIO running on a SKX server with Intel Cold Stream SSD. Padding improves performance significantly.
Tested latest depth tracking code from Thomas:
https://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git/log/?h=depthtracking
(SHA1 714d29e3e7e3faac27142424ae2533163ddd3a46) latest gcc patch from Thomas is included at the end.
Baseline Baseline
read (kIOPs) Mean stdev mitigations=off retbleed=off CPU util
================================================================================
mitigations=off 356.33 6.35 0.00% 7.11% 98.93%
retbleed=off 332.67 5.51 -6.64% 0.00% 99.16%
retbleed=ibrs 242.00 5.57 -32.09% -27.25% 99.41%
retbleed=stuff (nopad) 281.67 4.62 -20.95% -15.33% 99.35%
retbleed=stuff (pad) 310.67 0.58 -12.82% -6.61% 99.29%
read/write Baseline Baseline
70/30 (kIOPs) Mean stdev mitigations=off retbleed=off CPU util
================================================================================
mitigations=off 340.60 8.12 0.00% 4.01% 96.80%
retbleed=off 327.47 8.03 -3.86% 0.00% 97.06%
retbleed=ibrs 239.47 0.75 -29.69% -26.87% 98.23%
retbleed=stuff (nopad) 275.20 0.69 -19.20% -15.96% 97.86%
retbleed=stuff (pad) 296.60 2.03 -12.92% -9.43% 97.14%
Baseline Baseline
write (kIOPs) Mean stdev mitigations=off retbleed=off CPU util
================================================================================
mitigations=off 299.33 4.04 0.00% 7.16% 93.51%
retbleed=off 279.33 7.51 -6.68% 0.00% 94.30%
retbleed=ibrs 231.33 0.58 -22.72% -17.18% 95.84%
retbleed=stuff (nopad) 257.67 0.58 -13.92% -7.76% 94.96%
retbleed=stuff (pad) 274.67 1.53 -8.24% -1.67% 94.31%
Tim
gcc patch from Thomas:
--- gcc/config/i386/i386.cc | 13 +++++++++++++ gcc/config/i386/i386.h | 7 +++++++ gcc/config/i386/i386.opt | 4 ++++ gcc/doc/invoke.texi | 6 ++++++ 4 files changed, 30 insertions(+)
gcc/config/i386/i386.cc +13 -0
@@ -6182,6 +6182,19 @@ ix86_code_end (void) file_end_indicate_split_stack (); } +void +x86_asm_output_function_prefix (FILE *asm_out_file, + const char *fnname ATTRIBUTE_UNUSED) +{ + if (force_function_padding) + { + fprintf (asm_out_file, "\t.align %d\n", + 1 << force_function_padding); + fprintf (asm_out_file, "\t.skip %d,0xcc\n", + 1 << force_function_padding); + } +} + /* Emit code for the SET_GOT patterns. */ const char *
gcc/config/i386/i386.h +7 -0
@@ -2860,6 +2860,13 @@ extern enum attr_cpu ix86_schedule;#define LIBGCC2_UNWIND_ATTRIBUTE __attribute__((target ("no-mmx,no-sse")))#endif +#include <stdio.h> +extern void +x86_asm_output_function_prefix (FILE *asm_out_file, + const char *fnname ATTRIBUTE_UNUSED); +#undef ASM_OUTPUT_FUNCTION_PREFIX +#define ASM_OUTPUT_FUNCTION_PREFIX x86_asm_output_function_prefix +/* Local variables: version-control: t
gcc/config/i386/i386.opt +4 -0
@@ -1064,6 +1064,10 @@ mindirect-branch= Target RejectNegative Joined Enum(indirect_branch) Var(ix86_indirect_branch) Init(indirect_branch_keep) Convert indirect call and jump to call and return thunks. +mforce-function-padding= +Target Joined UInteger Var(force_function_padding) Init(0) IntegerRange(0, 6) +Put a 2^$N byte padding area before each function + mfunction-return= Target RejectNegative Joined Enum(indirect_branch) Var(ix86_function_return) Init(indirect_branch_keep) Convert function return to call and return thunk.
gcc/doc/invoke.texi +6 -0
@@ -1451,6 +1451,7 @@ See RS/6000 and PowerPC Options. -mindirect-branch=@var{choice} -mfunction-return=@var{choice} @gol -mindirect-branch-register -mharden-sls=@var{choice} @gol -mindirect-branch-cs-prefix -mneeded -mno-direct-extern-access} +-mforce-function-padding @gol @emph{x86 Windows Options} @gccoptlist{-mconsole -mcygwin -mno-cygwin -mdll @gol
@@ -32849,6 +32850,11 @@ Force all calls to functions to be indir when using Intel Processor Trace where it generates more precise timing information for function calls. +@item -mforce-function-padding +@opindex -mforce-function-padding +Force a 16 byte padding are before each function which allows run-time +code patching to put a special prologue before the function entry. + @item -mmanual-endbr @opindex mmanual-endbr Insert ENDBR instruction at function entry only via the @code{cf_check}
Quoted 3 lines · Tim Chen
Here are some performance numbers for FIO running on a SKX server with Intel Cold Stream SSD. Padding improves performance significantly.
That certainly looks oh-so-much better than those disgusting ibrs numbers.
One thing that I wonder about - gcc already knows about leaf functions for other reasons (stack setup is often different anyway), and I wonder it it might be worth looking at making leaf functions avoid the whole depth counting, and just rely on a regular call/ret.
The whole call chain thing is already not entirely exact and is counting to a smaller value than the real RSB size.
And leaf functions are generally the smallest and most often called, so it might be noticeable on profiles and performance numbers to just say "we already know this is a leaf, there's no reason to increment the depth for this only to decrement it when it returns".
The main issue is obviously that the return instruction has to be a non-decrementing one too for the leaf function case, so it's not just "don't do the counting on entry", it's also a "don't do the usual rethunk on exit".
So I just wanted to raise this, not because it's hugely important, but just to make people think about it - I have these very clear memories of the whole "don't make leaf functions create a stack frame" having been a surprisingly big deal for some loads.
Of course, sometimes when I have clear memories, they turn out to be just some drug-induced confusion in the end. But I know people experimented with "-fno-omit-frame-pointer -momit-leaf-frame-pointer" and that it made a big difference (but caused some issue with pvops hidden in asm so that gcc incorrectly thought they were leaf functions when they weren't).
Linus
Quoted 7 lines · Thomas Gleixner
For 4 RET pathes randomized with randomize_kstack_offset=y and RSP bit 3, 6, 5:
IBRS stuff stuff(pad) confuse
microbench: +37.20% +18.46% +15.47% +7.46%sockperf 14 bytes: -23.76% -19.26% -14.31% -16.80% sockperf 1472 bytes: -22.51% -18.40% -12.25% -15.95%
Quoted 7 lines
So for the more randomized variant sockperf tanks and is already slower than stuffing with thunks in the compiler provided padding space.
I send out a patch in reply to this series which implements that variant, but there needs to be input from the security researchers how protective this is. If we could get away with 2 RET pathes (perhaps multiple instances with different bits), that would be amazing.
Here is goes.
---
Subject: x86/retbleed: Add confusion mitigation
From: Thomas Gleixner <[email protected]>
Date: Fri, 15 Jul 2022 11:41:05 +0200
- NOT FOR INCLUSION -
Experimental option to confuse the return path by randomization.
The following command line options enable this:
retbleed=confuse 4 return pathes
retbleed=confuse,4 4 return pathes
retbleed=confuse,3 3 return pathes
retbleed=confuse,2 2 return pathes
This need scrunity by security researchers.
--- arch/x86/Kconfig | 12 ++++++ arch/x86/include/asm/nospec-branch.h | 23 +++++++++++ arch/x86/kernel/cpu/bugs.c | 41 +++++++++++++++++++++ arch/x86/lib/retpoline.S | 68 +++++++++++++++++++++++++++++++++++ include/linux/randomize_kstack.h | 6 +++ kernel/entry/common.c | 3 + 6 files changed, 153 insertions(+)
arch/x86/Kconfig +12 -0
@@ -2538,6 +2538,18 @@ config CALL_THUNKS_DEBUG Only enable this, when you are debugging call thunks as this creates a noticable runtime overhead. If unsure say N. +config RETURN_CONFUSION + bool "Mitigate RSB underflow with return confusion" + depends on CPU_SUP_INTEL && RETHUNK && RANDOMIZE_KSTACK_OFFSET + default y + help + Compile the kernel with return path confusion to mitigate the + Intel SKL Return-Speculation-Buffer (RSB) underflow issue. The + mitigation is off by default and needs to be enabled on the + kernel command line via the retbleed=confuse option. For + non-affected systems the overhead of this option is marginal as + the return thunk jumps are patched to direct ret instructions. + config CPU_IBPB_ENTRY bool "Enable IBPB on kernel entry" depends on CPU_SUP_AMD
arch/x86/include/asm/nospec-branch.h +23 -0
@@ -312,6 +312,29 @@ static inline void x86_set_skl_return_th#endif +#ifdef CONFIG_RETURN_CONFUSION +extern void __x86_return_confused_skl2 (void ); +extern void __x86_return_confused_skl3 (void ); +extern void __x86_return_confused_skl4 (void ); + +static inline void x86_set_skl_confused_return_thunk (int which) +{ +switch (which) { +case 2 : + x86_return_thunk = &__x86_return_confused_skl2; +break ; +case 3 : + x86_return_thunk = &__x86_return_confused_skl3; +break ; +case 4 : + x86_return_thunk = &__x86_return_confused_skl4; +break ; + } +} +#else +static inline void x86_set_skl_confused_return_thunk (void ) { } +#endif +#ifdef CONFIG_RETPOLINE #define GEN (reg) \
arch/x86/kernel/cpu/bugs.c +41 -0
97 lines not on this page · raw message
arch/x86/lib/retpoline.S +68 -0
71 lines not on this page · raw message
include/linux/randomize_kstack.h +6 -0
@@ -84,9 +84,15 @@ DECLARE_PER_CPU(u32, kstack_offset);raw_cpu_write (kstack_offset, offset); \ } \ } while (0) + +#define random_kstack_offset_enabled () \ + static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset) +#else /* CONFIG_RANDOMIZE_KSTACK_OFFSET */ #define add_random_kstack_offset () do { } while (0)#define choose_random_kstack_offset (rand) do { } while (0) +#define random_kstack_offset_enabled () false#endif /* CONFIG_RANDOMIZE_KSTACK_OFFSET */ #endif
kernel/entry/common.c +3 -0
@@ -298,6 +298,7 @@ void syscall_exit_to_user_mode_work(stru noinstr void irqentry_enter_from_user_mode(struct pt_regs *regs) { +add_random_kstack_offset ();__enter_from_user_mode (regs); }
@@ -444,6 +445,8 @@ irqentry_state_t noinstr irqentry_nmi_en {irqentry_state_t irq_state; +if (user_mode (regs)) +add_random_kstack_offset (); irq_state.lockdep =lockdep_hardirqs_enabled ();__nmi_enter ();
Virt Call depth tracking mitigation
In reply to Thomas GleixnerQuoted 5 lines · Thomas Gleixner
As IBRS is a performance horror show, Peter Zijstra and me revisited the call depth tracking approach and implemented it in a way which is hopefully more palatable and avoids the downsides of the original attempt.
We both unsurprisingly hate the result with a passion...
And I hate to add more problems, but here we go.
Under virt, it's not just SMI's which might run behind your back. Regular interrupts/etc can probably be hand-waved away in the same way that SMIs are.
Hypercalls however are a different matter.
Xen and HyperV both have hypercall pages, where the hypervisor provides some executable code for the guest kernel to use.
Under the current scheme, the calls into the hypercall pages get accounted, as objtool can see them, but the ret's don't. This imbalance is exasperated because some hypercalls are called in loops.
Worse however, it opens a hole where branch history is calculable and the ret can reliably underflow. This occurs when there's a minimal call depth in Linux to get to the hypercall, and then a call depth of >16 in the hypervisor.
The only variable in these cases is how much user control there is of the registers, and I for one am not feeling lucky in face of the current research.
The only solution I see here is for Linux to ret-thunk the hypercall page too. Under Xen, the hypercall page is mutable by the guest and there is room to turn every ret into a jmp, but obviously none of this is covered by any formal ABI, and this probably needs more careful consideration than the short time I've put towards it.
That said, after a return from the hypervisor, Linux has no idea what state the RSB is in, so the only safe course of action is to re-stuff.
CC'ing the HyperV folk for input on their side.
~Andrew
Re: Virt Call depth tracking mitigation
Quoted 5 lines · Andrew Cooper
Quoted 4 lines · Thomas Gleixner
As IBRS is a performance horror show, Peter Zijstra and me revisited the call depth tracking approach and implemented it in a way which is hopefully more palatable and avoids the downsides of the original attempt.
Quoted 7 lines
Quoted 1 line
We both unsurprisingly hate the result with a passion...
And I hate to add more problems, but here we go.
Under virt, it's not just SMI's which might run behind your back. Regular interrupts/etc can probably be hand-waved away in the same way that SMIs are.
You mean host side interrupts, right?
Quoted 8 lines
Hypercalls however are a different matter.
Xen and HyperV both have hypercall pages, where the hypervisor provides some executable code for the guest kernel to use.
Under the current scheme, the calls into the hypercall pages get accounted, as objtool can see them, but the ret's don't. This imbalance is exasperated because some hypercalls are called in loops.
Bah.
Quoted 9 lines
Worse however, it opens a hole where branch history is calculable and the ret can reliably underflow. This occurs when there's a minimal call depth in Linux to get to the hypercall, and then a call depth of >16 in the hypervisor.
The only variable in these cases is how much user control there is of the registers, and I for one am not feeling lucky in face of the current research.
Quoted 5 lines
The only solution I see here is for Linux to ret-thunk the hypercall page too. Under Xen, the hypercall page is mutable by the guest and there is room to turn every ret into a jmp, but obviously none of this is covered by any formal ABI, and this probably needs more careful consideration than the short time I've put towards it.
Well, that makes the guest side "safe", but isn't a deep hypercall > 16 already underflowing in the hypervisor code before it returns to the guest?
Quoted 2 lines
That said, after a return from the hypervisor, Linux has no idea what state the RSB is in, so the only safe course of action is to re-stuff.
Indeed.
Another proof for my claim that virt creates more problems than it solves.
Thanks,
tglx
Re: Virt Call depth tracking mitigation
Quoted 6 lines · Thomas Gleixner
Quoted 5 lines · Andrew Cooper
Quoted 4 lines · Thomas Gleixner
As IBRS is a performance horror show, Peter Zijstra and me revisited the call depth tracking approach and implemented it in a way which is hopefully more palatable and avoids the downsides of the original attempt.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
We both unsurprisingly hate the result with a passion...
And I hate to add more problems, but here we go.
Under virt, it's not just SMI's which might run behind your back. Regular interrupts/etc can probably be hand-waved away in the same way that SMIs are.
You mean host side interrupts, right?
Yes.
Quoted 20 lines
Quoted 9 lines
Hypercalls however are a different matter.
Xen and HyperV both have hypercall pages, where the hypervisor provides some executable code for the guest kernel to use.
Under the current scheme, the calls into the hypercall pages get accounted, as objtool can see them, but the ret's don't. This imbalance is exasperated because some hypercalls are called in loops.
Bah.
Quoted 9 lines
Worse however, it opens a hole where branch history is calculable and the ret can reliably underflow. This occurs when there's a minimal call depth in Linux to get to the hypercall, and then a call depth of >16 in the hypervisor.
The only variable in these cases is how much user control there is of the registers, and I for one am not feeling lucky in face of the current research.
Quoted 8 lines
Quoted 5 lines
The only solution I see here is for Linux to ret-thunk the hypercall page too. Under Xen, the hypercall page is mutable by the guest and there is room to turn every ret into a jmp, but obviously none of this is covered by any formal ABI, and this probably needs more careful consideration than the short time I've put towards it.
Well, that makes the guest side "safe", but isn't a deep hypercall > 16 already underflowing in the hypervisor code before it returns to the guest?
Yeah, but that's the hypervisor's problem to deal with, in whatever manner it sees fit.
And if the hypervisor is using IBeeRS then the first ret in guest context will underflow.
Quoted 6 lines
Quoted 2 lines
That said, after a return from the hypervisor, Linux has no idea what state the RSB is in, so the only safe course of action is to re-stuff.
Indeed.
Another proof for my claim that virt creates more problems than it solves.
So how did you like debugging the gsbase crash on native hardware. :)
~Andrew
Re: Virt Call depth tracking mitigation
Quoted 10 lines · Andrew Cooper
Quoted 3 lines · Thomas Gleixner
Well, that makes the guest side "safe", but isn't a deep hypercall > 16 already underflowing in the hypervisor code before it returns to the guest?
Yeah, but that's the hypervisor's problem to deal with, in whatever manner it sees fit.
And if the hypervisor is using IBeeRS then the first ret in guest context will underflow.
I have a look tomorrow.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
That said, after a return from the hypervisor, Linux has no idea what state the RSB is in, so the only safe course of action is to re-stuff.
Indeed.
Another proof for my claim that virt creates more problems than it solves.
So how did you like debugging the gsbase crash on native hardware. :)
First of all I said it's creating more problems than it solves, which means it solves some problems.
But more important, I'm not a wimp.
Thanks,
tglx
RE: Virt Call depth tracking mitigation
In reply to Andrew CooperFrom: Andrew Cooper <[email protected]> Sent: Tuesday, July 19, 2022 3:25 AM
Quoted 38 lines · Thomas Gleixner
Quoted 5 lines
As IBRS is a performance horror show, Peter Zijstra and me revisited the call depth tracking approach and implemented it in a way which is hopefully more palatable and avoids the downsides of the original attempt.
We both unsurprisingly hate the result with a passion...
And I hate to add more problems, but here we go.
Under virt, it's not just SMI's which might run behind your back. Regular interrupts/etc can probably be hand-waved away in the same way that SMIs are.
Hypercalls however are a different matter.
Xen and HyperV both have hypercall pages, where the hypervisor provides some executable code for the guest kernel to use.
Under the current scheme, the calls into the hypercall pages get accounted, as objtool can see them, but the ret's don't. This imbalance is exasperated because some hypercalls are called in loops.
Worse however, it opens a hole where branch history is calculable and the ret can reliably underflow. This occurs when there's a minimal call depth in Linux to get to the hypercall, and then a call depth of >16 in the hypervisor.
The only variable in these cases is how much user control there is of the registers, and I for one am not feeling lucky in face of the current research.
The only solution I see here is for Linux to ret-thunk the hypercall page too. Under Xen, the hypercall page is mutable by the guest and there is room to turn every ret into a jmp, but obviously none of this is covered by any formal ABI, and this probably needs more careful consideration than the short time I've put towards it.
Quoted 4 lines
That said, after a return from the hypervisor, Linux has no idea what state the RSB is in, so the only safe course of action is to re-stuff.
CC'ing the HyperV folk for input on their side.
In Hyper-V, the hypercall page is *not* writable by the guest. Quoting from Section 3.13 in the Hyper-V TLFS:
The hypercall page appears as an "overlay" to the GPA space; that is, it covers whatever else is mapped to the GPA range. Its contents are readable and executable by the guest. Attempts to write to the hypercall page will result in a protection (#GP) exception.
And:
After the interface has been established, the guest can initiate a hypercall. To do so, it populates the registers per the hypercall protocol and issues a CALL to the beginning of the hypercall page. The guest should assume the hypercall page performs the equivalent of a near return (0xC3) to return to the caller. As such, the hypercall must be invoked with a valid stack.
Michael
Peter Zijlstra
maintainerRe: Virt Call depth tracking mitigation
Quoted 8 lines · Michael Kelley (LINUX)
In Hyper-V, the hypercall page is *not* writable by the guest. Quoting from Section 3.13 in the Hyper-V TLFS:
The hypercall page appears as an "overlay" to the GPA space; that is, it covers whatever else is mapped to the GPA range. Its contents are readable and executable by the guest. Attempts to write to the hypercall page will result in a protection (#GP) exception.
Quoted 8 lines
And:
After the interface has been established, the guest can initiate a hypercall. To do so, it populates the registers per the hypercall protocol and issues a CALL to the beginning of the hypercall page. The guest should assume the hypercall page performs the equivalent of a near return (
0xC3) to return to the caller. As such, the hypercall must be invoked with a valid stack.
I'm hoping that these days you're following that 0xc3 with a 0xcc at the very least ?
IIRC the whole hyper-v thing is negotiated using (virtual) MSRs, would it be possible to write the address of a return thunk into an MSR and have the hypervisor rewrite the hypercall page accordingly?
This is needed for the AMD jmp2ret thing anyway. Or you get to eat an IBPB before every hypercall, which I'm guessing your performance people aren't keen on.
Steven Rostedt
maintainerI just ran my ftrace test suite against v5.19-rc7 to see if anything pops up. And one of my boot tests failed with:
[ 2.459713] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 1024 [ 2.460712] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 1024, 1GB 4 [ 2.461712] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 2.462713] Spectre V2 : Mitigation: Retpolines [ 2.464712] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 2.465712] Speculative Store Bypass: Vulnerable [ 2.466713] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 2.467712] SRBDS: Vulnerable: No microcode [ 2.488002] ------------[ cut here ]------------ [ 2.488712] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/alternative.c:558 apply_returns+0xa3/0x1ec [ 2.489712] Modules linked in: [ 2.490712] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.19.0-rc7-test #65 [ 2.491712] Hardware name: MSI MS-7823/CSM-H87M-G43 (MS-7823), BIOS V1.6 02/22/2014 [ 2.492712] RIP: 0010:apply_returns+0xa3/0x1ec [ 2.493712] Code: 0f b6 5d a2 48 63 45 88 48 01 c3 4c 01 e3 48 89 da 4c 89 e7 e8 c2 25 00 00 84 c0 0f 85 1f 01 00 00 48 81 fb c0 49 40 a1 74 07 <0f> 0b e9 0f 01 00 00 83 3d a1 83 4f 02 00 74 24 0f b6 55 a2 48 63 [ 2.494712] RSP: 0000:ffffffffa1e03df0 EFLAGS: 00010206 [ 2.495711] RAX: 0000000000000000 RBX: ffffffffa173e8ad RCX: 0000000000000001 [ 2.496711] RDX: 0000000000000003 RSI: ffffffffa161b20c RDI: ffffffffa173e8ad [ 2.497711] RBP: ffffffffa1e03ea8 R08: 00000000fffffff1 R09: 000000000000000f [ 2.498711] R10: ffffffffa1e03db8 R11: 0000000000000b03 R12: ffffffffa173e8a8 [ 2.499711] R13: ffffffffa2550d30 R14: 0000000000000000 R15: ffffffffa1e32138 [ 2.500712] FS: 0000000000000000(0000) GS:ffff8e8796400000(0000) knlGS:0000000000000000 [ 2.501711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2.502711] CR2: ffff8e879edff000 CR3: 0000000014e2a001 CR4: 00000000001706f0 [ 2.503712] Call Trace: [ 2.504712] <TASK> [ 2.505721] alternative_instructions+0x39/0xe9 [ 2.506712] check_bugs+0x310/0x330 [ 2.507712] start_kernel+0x605/0x63e [ 2.508715] x86_64_start_reservations+0x24/0x2a [ 2.509712] x86_64_start_kernel+0x8d/0x97 [ 2.510713] secondary_startup_64_no_verify+0xe0/0xeb [ 2.511719] </TASK> [ 2.512712] irq event stamp: 142170 [ 2.513711] hardirqs last enabled at (142180): [<ffffffffa014427c>] __up_console_sem+0x4b/0x53 [ 2.514711] hardirqs last disabled at (142189): [<ffffffffa014425c>] __up_console_sem+0x2b/0x53 [ 2.515712] softirqs last enabled at (8013): [<ffffffffa1400389>] __do_softirq+0x389/0x3c8 [ 2.516711] softirqs last disabled at (8006): [<ffffffffa00e2daa>] __irq_exit_rcu+0x72/0xcb [ 2.517711] ---[ end trace 0000000000000000 ]--- [ 2.529074] Freeing SMP alternatives memory: 44K [ 2.633924] smpboot: CPU0: Intel(R) Core(TM) i3-4130 CPU @ 3.40GHz (family: 0x6, model: 0x3c, stepping: 0x3) [ 2.636420] cblist_init_generic: Setting adjustable number of callback queues. [ 2.636712] cblist_init_generic: Setting shift to 2 and lim to 1. [ 2.637821] cblist_init_generic: Setting shift to 2 and lim to 1. [ 2.638822] Running RCU-tasks wait API self tests [ 2.742759] Performance Events: PEBS fmt2+, Haswell events, 16-deep LBR, full-width counters, Intel PMU driver. [ 2.743718] ... version: 3 [ 2.744712] ... bit width: 48 [ 2.745712] ... generic registers: 4 [ 2.746712] ... value mask: 0000ffffffffffff [ 2.747712] ... max period: 00007fffffffffff [ 2.748712] ... fixed-purpose events: 3 [ 2.749712] ... event mask: 000000070000000f [ 2.751038] Estimated ratio of average max frequency by base frequency (times 1024): 1024 [ 2.751848] rcu: Hierarchical SRCU implementation. [ 2.755405] smp: Bringing up secondary CPUs ... [ 2.756470] x86: Booting SMP configuration: [ 2.757713] .... node #0, CPUs: #1 [ 1.309155] numa_add_cpu cpu 1 node 0: mask now 0-1 [ 2.767842] #2 [ 1.309155] numa_add_cpu cpu 2 node 0: mask now 0-2 [ 2.774816] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details. [ 2.777343] #3 [ 1.309155] numa_add_cpu cpu 3 node 0: mask now 0-3 [ 2.784230] smp: Brought up 1 node, 4 CPUs [ 2.784713] smpboot: Max logical packages: 1
Attached is the config.
-- Steve
Quoted 2 lines · Steven Rostedt
[ 2.488712] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/alternative.c:558 apply_returns+0xa3/0x1ec
That warning is kind of annoying, in how it doesn't actually give any information about where the problem is.
I do note that we only fix up JMP32_INSN_OPCODE, and I wonder if we have a "jmp __x86_return_thunk" that is close enough to the return thunk that it actually uses a byte offset?
But that WARN_ON_ONCE() should probably be changed to actually give some information about where the problem is.
The silly thing is, there's even debug output in that function that you could enable, but it will enable output for the *normal* case, not for the WARN_ON_ONCE() case or the "we didn't do anything" case. That seems a bit backwards.
Linus
Peter Zijlstra
maintainerQuoted 7 lines · Linus Torvalds
Quoted 2 lines · Steven Rostedt
[ 2.488712] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/alternative.c:558 apply_returns+0xa3/0x1ecThat warning is kind of annoying, in how it doesn't actually give any information about where the problem is.
Quoted 6 lines
I do note that we only fix up
JMP32_INSN_OPCODE, and I wonder if we have a "jmp__x86_return_thunk" that is close enough to the return thunk that it actually uses a byte offset?But that
WARN_ON_ONCE()should probably be changed to actually give some information about where the problem is.
There's a patch for that:
https://lkml.kernel.org/r/[email protected]
it seems to have gotten lost, let me go queue that.
Steven Rostedt
maintainerQuoted 5 lines · On Wed, 20 Jul 2022 19:24:32 +0200 Peter Zijlstra <[email protected]> wrote:
There's a patch for that:
https://lkml.kernel.org/r/[email protected]
it seems to have gotten lost, let me go queue that.
With this applied, I now have:
[ 2.442118] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 2.443117] SRBDS: Vulnerable: No microcode [ 2.463339] ------------[ cut here ]------------ [ 2.464117] missing return thunk: lkdtm_rodata_do_nothing+0x0/0x8-lkdtm_rodata_do_nothing+0x5/0x8: e9 00 00 00 00 [ 2.464128] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/alternative.c:558 apply_returns+0xcb/0x219 [ 2.466117] Modules linked in: [ 2.467117] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.19.0-rc7-test+ #66 [ 2.468117] Hardware name: MSI MS-7823/CSM-H87M-G43 (MS-7823), BIOS V1.6 02/22/2014 [ 2.469117] RIP: 0010:apply_returns+0xcb/0x219 [ 2.470117] Code: 80 3d d1 32 06 02 00 75 59 49 89 d8 b9 05 00 00 00 4c 89 e2 48 89 de 48 c7 c7 63 98 b4 ae c6 05 b3 32 06 02 01 e8 f8 4f fb 00 <0f> 0b eb 34 44 0f b6 65 a2 31 c0 48 8d 55 c1 c6 45 c0 c3 48 89 d7 [ 2.471117] RSP: 0000:ffffffffaee03df0 EFLAGS: 00010286 [ 2.472117] RAX: 0000000000000000 RBX: ffffffffae73e8a8 RCX: ffffffffae056641 [ 2.473117] RDX: ffffffffaee03d78 RSI: 0000000000000001 RDI: 00000000ffffffff [ 2.474117] RBP: ffffffffaee03ea8 R08: 0000000000000000 R09: 00000000ffffffea [ 2.475117] R10: 000000000000001f R11: ffffffffaee03a4d R12: ffffffffae73e8ad [ 2.476117] R13: ffffffffaf550d30 R14: 0000000000000000 R15: ffffffffaee32138 [ 2.477117] FS: 0000000000000000(0000) GS:ffff9d71d6400000(0000) knlGS:0000000000000000 [ 2.478117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
-- Steve
Quoted 2 lines · Steven Rostedt
[ 2.464117] missing return thunk: lkdtm_rodata_do_nothing+0x0/0x8-lkdtm_rodata_do_nothing+0x5/0x8: e9 00 00 00 00
Well, that looks like a "jmp" instruction that has never been relocated.
The 'e9' is 'jmp', the four zeros after it are either "I'm jumping to the next instruction" or "I haven't been filled in".
I'm assuming it's the second case.
That lkdtm_rodata_do_nothing thing is odd, and does
OBJCOPYFLAGS_rodata_objcopy.o := \
--rename-section.noinstr.text=.rodata,alloc,readonly,load,contents
to put the code in an odd section. I'm assuming this hackery is related to it then not getting relocated.
Linus
Steven Rostedt
maintainerQuoted 11 lines · On Wed, 20 Jul 2022 11:07:26 -0700 Linus Torvalds…
Quoted 2 lines · Steven Rostedt
[ 2.464117] missing return thunk: lkdtm_rodata_do_nothing+0x0/0x8-lkdtm_rodata_do_nothing+0x5/0x8: e9 00 00 00 00Well, that looks like a "jmp" instruction that has never been relocated.
The 'e9' is 'jmp', the four zeros after it are either "I'm jumping to the next instruction" or "I haven't been filled in".
I'm assuming it's the second case.
Quoted 9 lines
That
lkdtm_rodata_do_nothingthing is odd, and doesOBJCOPYFLAGS_rodata_objcopy.o := \ --rename-section.noinstr.text=.rodata,alloc,readonly,load,contents
to put the code in an odd section. I'm assuming this hackery is related to it then not getting relocated.
Right, because this looks to be some magic being done for testing purposes:
static void lkdtm_EXEC_RODATA(void)
{
execute_location(dereference_function_descriptor(lkdtm_rodata_do_nothing),
CODE_AS_IS);
}static void *setup_function_descriptor(func_desc_t *fdesc, void *dst)
{
if (!have_function_descriptors())
return dst;
memcpy(fdesc, do_nothing, sizeof(*fdesc));
fdesc->addr = (unsigned long)dst;
barrier();
return fdesc;
}static noinline void execute_location(void *dst, bool write)
{
void (*func)(void);
func_desc_t fdesc;
void *do_nothing_text = dereference_function_descriptor(do_nothing);
pr_info("attempting ok execution at %px\n", do_nothing_text);
do_nothing();
if (write == CODE_WRITE) {
memcpy(dst, do_nothing_text, EXEC_SIZE);
flush_icache_range((unsigned long)dst,
(unsigned long)dst + EXEC_SIZE);
}
pr_info("attempting bad execution at %px\n", dst);
func = setup_function_descriptor(&fdesc, dst);
func();
pr_err("FAIL: func returned\n");
}And that appears that it wants to crash, as the code is located in readonly data.
OBJCOPYFLAGS_rodata_objcopy.o := \
--rename-section .noinstr.text=.rodata,alloc,readonly,load,contents
And because the alternatives fixup tries to write to it, and fails due to it being readonly, I'm guessing we get this warning.
Thus, is there a way to keep this file from being entered into the return_sites section?
-- Steve
Quoted 3 lines · Steven Rostedt
Thus, is there a way to keep this file from being entered into the
return_sitessection?
I think the whole concept is broken.
Testing known-broken code on the expectation that "this won't work anyway, so we can jump off to code that is broken" is not acceptable.
*If* the test were to fail, it would start executing random code that hasn't been relocated or fixed up properly.
So I think the whole concept is broken. It relies on the compiler generating code that can work in a read-only data section, and it's not clear that that is even physically possible (ie the data section might be far enough away from a code section that any relocation just fundamentally cannot happen).
I think it worked purely by mistake, because the code was simple enough that it didn't need any relocation at all before. But even without RETHUNK, that was never guaranteed, because any random tracing or debug code or whatever could have made even that empty function have code in it that just fundamentally wouldn't work in a non-text section.
So honestly, I think that test should be removed as a "we used this, it happened to work almost by mistake, but it doesn't work any more and it is unfixably broken".
Maybe somebody can come up with an entirely different way to do that test that isn't so broken, but if so, I think it's going to be using some other machinery (eg bpf and explicitly marking it read-only and non-executable), and removing this broken model is the right thing regardless.
So unless somebody has some one-liner workaround, I really suspect the fix is to remove all this. The amount of hackery to make it work in the first place is kind of disgusting anyway.
Since this was a WARN_ONCE(), can you make sure that with this case removed, nothing else triggers?
Linus
drivers/misc/lkdtm/Makefile | 11 ----------- drivers/misc/lkdtm/lkdtm.h | 3 --- drivers/misc/lkdtm/perms.c | 7 ------- drivers/misc/lkdtm/rodata.c | 11 ----------- 4 files changed, 32 deletions(-)
drivers/misc/lkdtm/Makefile +0 -11
@@ -6,21 +6,10 @@ lkdtm-$(CONFIG_LKDTM) += bugs.o lkdtm-$(CONFIG_LKDTM) += heap.o lkdtm-$(CONFIG_LKDTM) += perms.o lkdtm-$(CONFIG_LKDTM) += refcount.o -lkdtm-$(CONFIG_LKDTM) += rodata_objcopy.o lkdtm-$(CONFIG_LKDTM) += usercopy.o lkdtm-$(CONFIG_LKDTM) += stackleak.o lkdtm-$(CONFIG_LKDTM) += cfi.o lkdtm-$(CONFIG_LKDTM) += fortify.o lkdtm-$(CONFIG_PPC_64S_HASH_MMU) += powerpc.o -KASAN_SANITIZE_rodata.o := n KASAN_SANITIZE_stackleak.o := n -KCOV_INSTRUMENT_rodata.o := n -CFLAGS_REMOVE_rodata.o += $(CC_FLAGS_LTO) - -OBJCOPYFLAGS := -OBJCOPYFLAGS_rodata_objcopy.o := \ - --rename-section .noinstr.text=.rodata,alloc,readonly,load,contents -targets += rodata.o rodata_objcopy.o -$(obj)/rodata_objcopy.o: $(obj)/rodata.o FORCE - $(call if_changed,objcopy)
drivers/misc/lkdtm/lkdtm.h +0 -3
@@ -94,7 +94,4 @@ void __init lkdtm_perms_init(void);void __init lkdtm_usercopy_init (void ); void __exit lkdtm_usercopy_exit(void); -/* Special declaration for function-in-rodata. */ -void lkdtm_rodata_do_nothing (void ); -#endif
drivers/misc/lkdtm/perms.c +0 -7
@@ -191,12 +191,6 @@ static void lkdtm_EXEC_VMALLOC(void)vfree (vmalloc_area); } -static void lkdtm_EXEC_RODATA (void ) -{ -execute_location (dereference_function_descriptor (lkdtm_rodata_do_nothing), -CODE_AS_IS ); -} -static void lkdtm_EXEC_USERSPACE (void ) {unsigned long user_addr;
@@ -280,7 +274,6 @@ static struct crashtype crashtypes[] = {CRASHTYPE (EXEC_STACK ),CRASHTYPE (EXEC_KMALLOC ),CRASHTYPE (EXEC_VMALLOC ), -CRASHTYPE (EXEC_RODATA ),CRASHTYPE (EXEC_USERSPACE ),CRASHTYPE (EXEC_NULL ), CRASHTYPE(ACCESS_USERSPACE),
drivers/misc/lkdtm/rodata.c deleted +0 -11
@@ -1,11 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* -* This includes functions that are meant to live entirely in .rodata -* (via objcopy tricks), to validate the non-executability of .rodata. -*/ -#include "lkdtm.h" - -void noinstr lkdtm_rodata_do_nothing(void) -{ -/* Does nothing. We just want an architecture agnostic "return". */ -}
Steven Rostedt
maintainerQuoted 6 lines · On Wed, 20 Jul 2022 11:43:37 -0700 Linus Torvalds…
So unless somebody has some one-liner workaround, I really suspect the fix is to remove all this. The amount of hackery to make it work in the first place is kind of disgusting anyway.
Since this was a
WARN_ONCE(), can you make sure that with this case removed, nothing else triggers?
Actually, this fixes it too:
(and this config boots to completion without warnings).
I'll add this to my full test suite and see if it finishes.
-- Steve
arch/x86/Makefile +1 -0
@@ -25,6 +25,7 @@ endif ifdef CONFIG_RETHUNK RETHUNK_CFLAGS := -mfunction-return=thunk-extern RETPOLINE_CFLAGS += $(RETHUNK_CFLAGS) +export RETHUNK_CFLAGS endif export RETPOLINE_CFLAGS
drivers/misc/lkdtm/Makefile +1 -1
@@ -16,7 +16,7 @@ lkdtm-$(CONFIG_PPC_64S_HASH_MMU) += powerpc.o KASAN_SANITIZE_rodata.o := n KASAN_SANITIZE_stackleak.o := n KCOV_INSTRUMENT_rodata.o := n -CFLAGS_REMOVE_rodata.o += $(CC_FLAGS_LTO) +CFLAGS_REMOVE_rodata.o += $(CC_FLAGS_LTO) $(RETHUNK_CFLAGS) OBJCOPYFLAGS := OBJCOPYFLAGS_rodata_objcopy.o := \
Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Steven Rostedt
[ 2.464117] missing return thunk: lkdtm_rodata_do_nothing+0x0/0x8-lkdtm_rodata_do_nothing+0x5/0x8: e9 00 00 00 00Well, that looks like a "jmp" instruction that has never been relocated.
Peter, Josh, and I drilled down into this recently[1] and discussed some solutions[2].
This test is doing what's expected: it needed an arch-agnostic way to do a "return", and when the way to do that changed, it also changed (which would normally be good, but in this case broke it). It's been happily being used as part of the per-section architectural behavior testing[3] of execution-vs-expected-memory-permissions for quite a long while now.
I'd rather not remove it (or do it dynamically) since the point is to test what has been generated by the toolchain/build process and stuffed into the .rodata section. i.e. making sure gadgets there can't be executed, that the boot-time section permission-setting works correctly, etc. Before the retbleed mitigation, this test worked for all architectures; I'd hate to regress it. :(
-Kees
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] e.g. https://linux.kernelci.org/test/plan/id/62d61ee8ef31e0f0faa39bff/
Signature · 1 line
Kees Cook
Steven Rostedt
maintainerQuoted 6 lines · On Wed, 20 Jul 2022 12:36:38 -0700 Kees Cook <[email protected]> wrote:
I'd rather not remove it (or do it dynamically) since the point is to test what has been generated by the toolchain/build process and stuffed into the .rodata section. i.e. making sure gadgets there can't be executed, that the boot-time section permission-setting works correctly, etc. Before the retbleed mitigation, this test worked for all architectures; I'd hate to regress it. :(
If you haven't noticed my reply, I wasn't able to come up with a one line workaround, but I was able to come up with a two line workaround. Hopefully that will be good enough to keep your little feature.
https://lore.kernel.org/all/[email protected]/
I'm currently running it under my entire ftrace test suite. If it passes, I'll submit a formal patch.
-- Steve
Peter Zijlstra
maintainerQuoted 23 lines · Kees Cook
Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Steven Rostedt
[ 2.464117] missing return thunk: lkdtm_rodata_do_nothing+0x0/0x8-lkdtm_rodata_do_nothing+0x5/0x8: e9 00 00 00 00Well, that looks like a "jmp" instruction that has never been relocated.
Peter, Josh, and I drilled down into this recently[1] and discussed some solutions[2].
This test is doing what's expected: it needed an arch-agnostic way to do a "return", and when the way to do that changed, it also changed (which would normally be good, but in this case broke it). It's been happily being used as part of the per-section architectural behavior testing[3] of execution-vs-expected-memory-permissions for quite a long while now.
I'd rather not remove it (or do it dynamically) since the point is to test what has been generated by the toolchain/build process and stuffed into the .rodata section. i.e. making sure gadgets there can't be executed, that the boot-time section permission-setting works correctly, etc. Before the retbleed mitigation, this test worked for all architectures; I'd hate to regress it. :(
Quoted 5 lines
-Kees
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] e.g. https://linux.kernelci.org/test/plan/id/62d61ee8ef31e0f0faa39bff/
Josh posted this:
which I picked up today; barring robot fail I'll push it to x86/urgent tomorrow.