x86/mm: INVPCID support

Commit-ID:  060a402a1ddb551455ee410de2eadd3349f2801b
Gitweb:     http://git.kernel.org/tip/060a402a1ddb551455ee410de2eadd3349f2801b
Author:     Andy Lutomirski <[email protected]>
AuthorDate: Fri, 29 Jan 2016 11:42:57 -0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 9 Feb 2016 13:36:10 +0100

x86/mm: Add INVPCID helpers

This adds helpers for each of the four currently-specified INVPCID modes.

(quoted diff) +6 -0

+static inline void __invpcid(unsigned long pcid, unsigned long addr,
+			     unsigned long type)
+{
+	u64 desc[2] = { pcid, addr };
+
+	/*

Hi,

On Wed, 10 Feb 2016, Borislav Petkov wrote:

arch/x86/include/asm/tlbflush.h static inline void __invpcid(unsigned long pcid, unsigned long addr, · line 26

Quoted 8 lines

arch/x86/include/asm/tlbflush.h +1 -1

@@ -23,7 +23,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr,
 	 * invpcid (%rcx), %rax in long mode.
 	 */
 	asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01"
-		      : : "m" (desc), "a" (type), "c" (desc) : "memory");
+		      : : "m" (*desc), "a" (type), "c" (desc) : "memory");

That still doesn't do what you want. Arrays in C are funny. *desc is exactly equivalent to desc[0], _not_ to the whole array,

indeed there's no C syntax to name an lvalue of array type in normal expressions. You need to jump through hoops for this:

"m" (*(struct {unsigned long x[2];} *)desc)

It'd probably be easier to simply declare the descriptor as a struct, rather than an array, then the original syntax would have been mostly correct:

  struct {u64 d[2];} desc = { pcid, addr };
  asm ... "m" (desc), "c" (&desc)

Quoted 4 lines · Michael Matz

Hi,

On Wed, 10 Feb 2016, Borislav Petkov wrote:

Quoted 11 lines

arch/x86/include/asm/tlbflush.h static inline void __invpcid(unsigned long pcid, unsigned long addr, · line 26

Quoted 8 lines

arch/x86/include/asm/tlbflush.h +1 -1

@@ -23,7 +23,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr,
     * invpcid (%rcx), %rax in long mode.
     */
    asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01"
-                 : : "m" (desc), "a" (type), "c" (desc) : "memory");
+                 : : "m" (*desc), "a" (type), "c" (desc) : "memory");

That still doesn't do what you want. Arrays in C are funny. *desc is exactly equivalent to desc[0], _not_ to the whole array,

Doh!

Quoted 4 lines

indeed there's no C syntax to name an lvalue of array type in normal expressions. You need to jump through hoops for this:

"m" (*(struct {unsigned long x[2];} *)desc)

Aha! That's why we wrapped the array in clwb() in a struct too, btw:

static inline void clwb(volatile void *__p)

{
        volatile struct { char x[64]; } *p = __p;
        ...

Quoted 6 lines

It'd probably be easier to simply declare the descriptor as a struct, rather than an array, then the original syntax would have been mostly correct:

  struct {u64 d[2];} desc = { pcid, addr };
  asm ... "m" (desc), "c" (&desc)

Sounds better. Done. How does that below look like?

Thanks Micha!

---
From: Borislav Petkov <[email protected]>
Date: Wed, 10 Feb 2016 12:53:48 +0100
Subject: [PATCH -v1.1] x86/mm: Fix INVPCID asm constraint
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

So we want to specify the dependency on both @pcid and @addr so that the compiler doesn't reorder accesses to them *before* the TLB flush. But for that to work, we need to express this properly in the inline asm and deref the whole desc array, not the pointer to it. See clwb() for an example.

This fixes the build error on 32-bit:

arch/x86/include/asm/tlbflush.h: In function ‘__invpcid’:
arch/x86/include/asm/tlbflush.h:26:18: error: memory input 0 is not directly addressable

Boris, I think you already have these prerequisites queued up:

http://lkml.kernel.org/g/[email protected]
http://lkml.kernel.org/g/[email protected]

This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.)

I think we should consider these patches separately from the rest of the PCID stuff -- they barely interact, and this part is much simpler and is useful on its own.

Changes from v2:
- Add macros for the INVPCID mode numbers.
- Add a changelog message for the chicken bit.

v1 was exactly identical to patches 2-4 of the PCID RFC series. Andy Lutomirski (3):
x86/mm: Add INVPCID helpers
x86/mm: Add a noinvpcid option to turn off INVPCID x86/mm: If INVPCID is available, use it to flush global mappings

 Documentation/kernel-parameters.txt |  2 ++
 arch/x86/include/asm/tlbflush.h     | 57 +++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/common.c        | 16 +++++++++++
 3 files changed, 75 insertions(+)