x86: Pile o' FS/GS changes

From: Borislav Petkov <[email protected]>

Erratum 88 affects old AMD K8s, where a SWAPGS fails to cause an input dependency on GS. Therefore, we need to MFENCE before it.

But that MFENCE is expensive and unnecessary on the remaining x86 CPUs out there so patch it out on the CPUs which don't require it.

Signed-off-by Borislav Petkov <[email protected]>

Cc <[email protected]>

Signed-off-by Andy Lutomirski <[email protected]

---
 arch/x86/entry/entry_64.S          | 2 +-
 arch/x86/include/asm/cpufeatures.h | 2 ++
 arch/x86/kernel/cpu/amd.c          | 1 +
 3 files changed, 4 insertions(+), 1 deletion(-)

arch/x86/entry/entry_64.S +1 -1

index 858b555e274b..64d2033d1e49 100644
@@ -783,7 +783,7 @@ ENTRY(native_load_gs_index)
        SWAPGS
 gs_change:
        movl    %edi, %gs
-2:     mfence                                  /* workaround */
+2:     ALTERNATIVE "", "mfence", X86_BUG_SWAPGS_FENCE
        SWAPGS
        popfq
        ret

arch/x86/include/asm/cpufeatures.h +2 -0

index 2a052302bc43..7bfb6b70c745 100644
@@ -295,6 +295,8 @@
 #define X86_BUG_CLFLUSH_MONITOR        X86_BUG(7) /* AAI65, CLFLUSH required before MONITOR */
 #define X86_BUG_SYSRET_SS_ATTRS        X86_BUG(8) /* SYSRET doesn't fix up SS attrs */
 #define X86_BUG_NULL_SEG       X86_BUG(9) /* Nulling a selector preserves the base */
+#define X86_BUG_SWAPGS_FENCE   X86_BUG(10) /* SWAPGS without input dep on GS */
+
 
 #ifdef CONFIG_X86_32
 /*

arch/x86/kernel/cpu/amd.c +1 -1

index 6e47e3a916f1..b7cc9efe08b5 100644
@@ -632,6 +632,7 @@ static void init_amd_k8(struct cpuinfo_x86 *c)
         */
        msr_set_bit(MSR_K7_HWCR, 6);
 #endif
+       set_cpu_bug(c, X86_BUG_SWAPGS_FENCE);
 }
 
 static void init_amd_gh(struct cpuinfo_x86 *c)
--

2.5.5

Ingo, etc: we should probably apply one of those patches with a -stable tag (to mitigate the otherwise potentially unpleasant performance regression in here), but I don't really care which one. Andi's has a name for the bug that seems nicer by one character to me, but it would have to be (trivally) rebased.

ARCH_GET_FS and ARCH_GET_GS attempted to figure out the fsbase and gsbase respectively from saved thread state. This was wrong: fsbase and gsbase live in registers while a thread is running, not in memory.

Quoted 5 lines

ARCH_GET_FS and ARCH_GET_GS attempted to figure out the fsbase and gsbase respectively from saved thread state. This was wrong: fsbase and gsbase live in registers while a thread is running, not in memory.

So I'm wondering, the current code looks totally broken,what user-space code can possibly use this? I checked glibc and Wine, and neither of them does. Wine uses ARCH_SET_GS and glibc uses ARCH_SET_FS, but that's all - neither actually tries to use the ARCH_GET_* reading APIs.

So for backporting purposes I'd be much happier about simply returning -EINVAL or -ENOSYS, and we could re-introduce this code in v4.7.

Quoted 4 lines · Andy Lutomirski

ARCH_GET_FS and ARCH_GET_GS attempted to figure out the fsbase and gsbase respectively from saved thread state. This was wrong: fsbase and gsbase live in registers while a thread is running, not in memory.

So I'm wondering, the current code looks totally broken,what user-space code can possibly use this? I checked glibc and Wine, and neither of them does. Wine uses ARCH_SET_GS and glibc uses ARCH_SET_FS, but that's all - neither actually tries to use the ARCH_GET_* reading APIs.

So for backporting purposes I'd be much happier about simply returning -EINVAL or -ENOSYS, and we could re-introduce this code in v4.7.

Hi all-

This whole mess is intended for x86/urgent. It fixes several bugs.

It's probably a tiny performance regression on some workloads on Intel CPUs. It's probably varies between a less tiny regression and a small speedup on newer AMD CPUs. It's a bigger regression on AMD K8.

The AMD code could be further sped up by adding a new 'clear_gs' pvop with a better native implementation. I don't particularly want to deal with a backported new pvop, so I didn't do that. Maybe I'll do it later on for new kernels, though.

Hi all-

This whole mess is intended for x86/urgent. It fixes several bugs.

It's probably a tiny performance regression on some workloads on Intel CPUs. It's probably varies between a less tiny regression and a small speedup on newer AMD CPUs. It's a bigger regression on AMD K8.

The AMD code could be further sped up by adding a new 'clear_gs' pvop with a better native implementation. I don't particularly want to deal with a backported new pvop, so I didn't do that. Maybe I'll do it later on for new kernels, though.

(For people who care about such things, there are probably three bugs in here depending how you count. Somewhere between one and two of them were independently discovered by Rudolf Marek.)

Andy Lutomirski (5):
selftests/x86: Test the FSBASE/GSBASE API and context switching x86/arch_prctl: Fix ARCH_GET_FS and ARCH_GET_GS x86/cpu: Probe the behavior of nulling out a segment at boot time x86/switch_to: Rewrite the FS and GS context switch code x86/cpu: Move X86_BUG_ESPFIX initialization to generic_identify

Borislav Petkov (2):
x86/cpu: Add Erratum 88 detection on AMD
x86/entry: Make gs_change a local label

 arch/x86/entry/entry_64.S              |  12 +-
 arch/x86/include/asm/cpufeatures.h     |   3 +
 arch/x86/kernel/cpu/amd.c              |   1 +
 arch/x86/kernel/cpu/common.c           |  71 ++++--
 arch/x86/kernel/process_64.c           | 167 ++++++++------
 tools/testing/selftests/x86/Makefile   |   1 +
 tools/testing/selftests/x86/fsgsbase.c | 398 +++++++++++++++++++++++++++++++++
 7 files changed, 560 insertions(+), 93 deletions(-)
 create mode 100644 tools/testing/selftests/x86/fsgsbase.c