x86/mm: INVPCID support
Subsystems (3)
v2 cover x86/mm: INVPCID support
Ingo, before applying this, please apply these two KASAN fixes:
http://lkml.kernel.org/g/[email protected]
http://lkml.kernel.org/g/[email protected]
Without those fixes, this series will trigger a KASAN bug.
This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.)
I think we should consider these patches separately from the rest of the PCID stuff -- they barely interact, and this part is much simpler and is useful on its own.
This is exactly identical to patches 2-4 of the PCID RFC series.
Andy Lutomirski (3):
x86/mm: Add INVPCID helpers
x86/mm: Add a noinvpcid option to turn off INVPCID x86/mm: If INVPCID is available, use it to flush global mappings
Documentation/kernel-parameters.txt | 2 ++ arch/x86/include/asm/tlbflush.h | 50 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++ 3 files changed, 68 insertions(+)
Signature · 7 lines
2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 1/3 x86/mm: Add INVPCID helpers
This adds helpers for each of the four currently-specified INVPCID modes.
--- arch/x86/include/asm/tlbflush.h | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
arch/x86/include/asm/tlbflush.h +41 -0
@@ -7,6 +7,47 @@#include <asm/processor.h> #include <asm/special_insns.h> +static inline void __invpcid (unsigned long pcid,unsigned long addr, +unsigned long type) +{ +u64 desc[2 ] = { pcid, addr }; + +/* +* The memory clobber is because the whole point is to invalidate +* stale TLB entries and, especially if we're flushing global +* mappings, we don't want the compiler to reorder any subsequent +* memory accesses before the TLB flush. +*/ + asmvolatile ( +".byte 0x66, 0x0f, 0x38, 0x82, 0x01" /* invpcid (%cx), %ax */ + : :"m" (desc),"a" (type),"c" (desc) :"memory" ); +} + +/* Flush all mappings for a given pcid and addr, not including globals. */ +static inline void invpcid_flush_one (unsigned long pcid, +unsigned long addr) +{ +__invpcid (pcid, addr,0 ); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invpcid_flush_single_context (unsigned long pcid) +{ +__invpcid (pcid,0 ,1 ); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invpcid_flush_everything (void ) +{ +__invpcid (0 ,0 ,2 ); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invpcid_flush_all_nonglobals (void ) +{ +__invpcid (0 ,0 ,3 ); +} +#ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else
Signature · 7 lines
2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Borislav Petkov
maintainerRe: [PATCH v2 1/3] x86/mm: Add INVPCID helpers
Quoted 18 lines · Andy Lutomirski
This adds helpers for each of the four currently-specified INVPCID modes.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/include/asm/tlbflush.h | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)arch/x86/include/asm/tlbflush.h +2
-0@@ -7,6 +7,47 @@#include <asm/processor.h> #include <asm/special_insns.h> +static inline void __invpcid (unsigned long pcid,unsigned long addr, +unsigned long type)
arch/x86/include/asm/tlbflush.h line 21
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+{ +u64 desc[2 ] = { pcid, addr }; + +/* +* The memory clobber is because the whole point is to invalidate +* stale TLB entries and, especially if we're flushing global +* mappings, we don't want the compiler to reorder any subsequent +* memory accesses before the TLB flush. +*/ + asm volatile (
Yeah, no need for that linebreak here:
asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01"reads fine too.
arch/x86/include/asm/tlbflush.h line 25
Quoted 4 lines
arch/x86/include/asm/tlbflush.h +4
-0+ ".byte 0x66, 0x0f, 0x38, 0x82, 0x01" /* invpcid (%cx), %ax */ + : : "m" (desc), "a" (type), "c" (desc) : "memory"); +} +
Please add defines for the invalidation types:
#define INVPCID_TYPE_INDIVIDUAL 0 #define INVPCID_TYPE_SINGLE_CTXT 1 #define INVPCID_TYPE_ALL 2 #define INVPCID_TYPE_ALL_NON_GLOBAL 3
and add macros:
#define invpcid_flush_one(pcid, addr) __invpcid(pcid, addr, INVPCID_TYPE_INDIVIDUAL)
...
and so on.
Oh, and the "flush everything" macro I'd call invpcid_flush_all() like tlb_flush_all().
Thanks.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.PATCH v2 2/3 x86/mm: Add a noinvpcid option to turn off INVPCID
In reply to Andy Lutomirski--- Documentation/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+)
Documentation/kernel-parameters.txt +2 -0
@@ -2508,6 +2508,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nointroute [IA-64] + noinvpcid [X86] Disable the INVPCID cpu feature. + nojitter [IA-64] Disables jitter checking for ITC timers. no-kvmclock [X86,KVM] Disable paravirtualized KVM clock driver
arch/x86/kernel/cpu/common.c +16 -0
@@ -162,6 +162,22 @@ static int __init x86_mpx_setup(char *s) }__setup ("nompx" , x86_mpx_setup); +static int __init x86_noinvpcid_setup (char *s) +{ +/* noinvpcid doesn't accept parameters */ +if (s) +return -EINVAL ; + +/* do not emit a message if the feature is not present */ +if (!boot_cpu_has (X86_FEATURE_INVPCID )) +return 0 ; + +setup_clear_cpu_cap (X86_FEATURE_INVPCID ); +pr_info ("noinvpcid: INVPCID feature disabled\n" ); +return 0 ; +} +early_param ("noinvpcid" , x86_noinvpcid_setup); +#ifdef CONFIG_X86_32 static int cachesize_override =-1 ;static int disable_x86_serial_nr =1 ;
Signature · 7 lines
2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Borislav Petkov
maintainerRe: [PATCH v2 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID
On Mon, Jan 25, 2016 at 10:37:43AM -0800, Andy Lutomirski wrote:
<--- Commit message please, albeit a trivial one like "Add a chicken bit ..."
Quoted 5 lines
Signed-off-by Andy Lutomirski <[email protected]>--- Documentation/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++ 2 files changed, 18 insertions(+)
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.PATCH v2 3/3 x86/mm: If INVPCID is available, use it to flush global mappings
In reply to Andy LutomirskiOn my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns.
--- arch/x86/include/asm/tlbflush.h | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/include/asm/tlbflush.h +9 -0
@@ -145,6 +145,15 @@ static inline void __native_flush_tlb_global(void) {unsigned long flags; +if (static_cpu_has_safe (X86_FEATURE_INVPCID )) { +/* +* Using INVPCID is considerably faster than a pair of writes +* to CR4 sandwiched inside an IRQ flag save/restore. +*/ +invpcid_flush_everything (); +return ; + } +/* * Read-modify-write to CR4 - protect it from preemption and * from interrupts. (Use the raw variant because this code can
Signature · 7 lines
2.5.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Borislav Petkov
maintainerRe: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings
Quoted 4 lines · Andy Lutomirski
On my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns.
FWIW, I ran your microbenchmark on the IVB laptop I have here 3 times and some of the numbers from each run are pretty unstable. Not that it means a whole lot - the thing doesn't have INVPCID support.
I'm just questioning the microbenchmark and whether we should be rather doing those measurements with a real benchmark, whatever that means. My limited experience says that measuring TLB performance is hard.
./context_switch_latency 0 thread same
use_xstate = 0
Using threads
1: 100000 iters at 2676.2 ns/switch
2: 100000 iters at 2700.2 ns/switch
3: 100000 iters at 2656.1 ns/switch
./context_switch_latency 0 thread different
use_xstate = 0
Using threads
1: 100000 iters at 5174.8 ns/switch
2: 100000 iters at 5140.5 ns/switch
3: 100000 iters at 5292.9 ns/switch
./context_switch_latency 0 process same
use_xstate = 0
Using a subprocess
1: 100000 iters at 2361.2 ns/switch
2: 100000 iters at 2332.2 ns/switch
3: 100000 iters at 3436.9 ns/switch
./context_switch_latency 0 process different
use_xstate = 0
Using a subprocess
1: 100000 iters at 4713.6 ns/switch
2: 100000 iters at 4957.5 ns/switch
3: 100000 iters at 5012.2 ns/switch
./context_switch_latency 1 thread same
use_xstate = 1
Using threads
1: 100000 iters at 2505.6 ns/switch
2: 100000 iters at 2483.1 ns/switch
3: 100000 iters at 2479.7 ns/switch
./context_switch_latency 1 thread different
use_xstate = 1
Using threads
1: 100000 iters at 5245.9 ns/switch
2: 100000 iters at 5241.1 ns/switch
3: 100000 iters at 5220.3 ns/switch
./context_switch_latency 1 process same
use_xstate = 1
Using a subprocess
1: 100000 iters at 2329.8 ns/switch
2: 100000 iters at 2350.2 ns/switch
3: 100000 iters at 2500.9 ns/switch
./context_switch_latency 1 process different
use_xstate = 1
Using a subprocess
1: 100000 iters at 4970.7 ns/switch
2: 100000 iters at 5034.0 ns/switch
3: 100000 iters at 4991.6 ns/switch
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Re: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings
Quoted 63 lines · Borislav Petkov
Quoted 4 lines · Andy Lutomirski
On my Skylake laptop, INVPCID function 2 (flush absolutely everything) takes about 376ns, whereas saving flags, twiddling CR4.PGE to flush global mappings, and restoring flags takes about 539ns.
FWIW, I ran your microbenchmark on the IVB laptop I have here 3 times and some of the numbers from each run are pretty unstable. Not that it means a whole lot - the thing doesn't have INVPCID support.
I'm just questioning the microbenchmark and whether we should be rather doing those measurements with a real benchmark, whatever that means. My limited experience says that measuring TLB performance is hard.
./context_switch_latency 0 thread same
use_xstate= 0
Using threads
1: 100000 iters at 2676.2 ns/switch
2: 100000 iters at 2700.2 ns/switch
3: 100000 iters at 2656.1 ns/switch./context_switch_latency 0 thread different
use_xstate= 0
Using threads
1: 100000 iters at 5174.8 ns/switch
2: 100000 iters at 5140.5 ns/switch
3: 100000 iters at 5292.9 ns/switch./context_switch_latency 0 process same
use_xstate= 0
Using a subprocess
1: 100000 iters at 2361.2 ns/switch
2: 100000 iters at 2332.2 ns/switch
3: 100000 iters at 3436.9 ns/switch./context_switch_latency 0 process different
use_xstate= 0
Using a subprocess
1: 100000 iters at 4713.6 ns/switch
2: 100000 iters at 4957.5 ns/switch
3: 100000 iters at 5012.2 ns/switch./context_switch_latency 1 thread same
use_xstate= 1
Using threads
1: 100000 iters at 2505.6 ns/switch
2: 100000 iters at 2483.1 ns/switch
3: 100000 iters at 2479.7 ns/switch./context_switch_latency 1 thread different
use_xstate= 1
Using threads
1: 100000 iters at 5245.9 ns/switch
2: 100000 iters at 5241.1 ns/switch
3: 100000 iters at 5220.3 ns/switch./context_switch_latency 1 process same
use_xstate= 1
Using a subprocess
1: 100000 iters at 2329.8 ns/switch
2: 100000 iters at 2350.2 ns/switch
3: 100000 iters at 2500.9 ns/switch
Quoted 7 lines
./context_switch_latency 1 process different
use_xstate= 1
Using a subprocess
1: 100000 iters at 4970.7 ns/switch
2: 100000 iters at 5034.0 ns/switch
3: 100000 iters at 4991.6 ns/switch
I'll fiddle with that benchmark a little bit. Maybe I can make it suck less. If anyone knows a good non-micro benchmark for this, let me know. I refuse to use dbus as my benchmark :)
FWIW, I benchmarked cr4 vs invpcid by adding a prctl and calling it in a loop. If Ingo's fpu benchmark thing ever lands, I'll gladly send a patch to add TLB flushes to it.
--Andy
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Borislav Petkov
maintainerRe: [PATCH v2 3/3] x86/mm: If INVPCID is available, use it to flush global mappings
Quoted 3 lines · Andy Lutomirski
I'll fiddle with that benchmark a little bit. Maybe I can make it suck less. If anyone knows a good non-micro benchmark for this, let me know.
Yeah, I don't know of a good one. The TLB and all those intermediary walker caches modern x86 CPUs have are really good. So it is hard to measure any improvements there. I guess in this particular case, if one can't measure slowdowns and the code is simple enough, then we're good enough. In theory, we are carefully killing less TLB entries and the related cached page walker data so that should be a good thing...
Quoted 1 line
I refuse to use dbus as my benchmark :)
Ha!
Quoted 2 lines
FWIW, I benchmarked cr4 vs invpcid by adding a prctl and calling it in a loop.
Apparently INVPCID is faster than the two CR4 writes.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 23 lines · Andy Lutomirski
Ingo, before applying this, please apply these two KASAN fixes:
http://lkml.kernel.org/g/[email protected]
http://lkml.kernel.org/g/[email protected]Without those fixes, this series will trigger a KASAN bug.
This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.)
I think we should consider these patches separately from the rest of the PCID stuff -- they barely interact, and this part is much simpler and is useful on its own.
This is exactly identical to patches 2-4 of the PCID RFC series.
Andy Lutomirski (3):
x86/mm: Add INVPCID helpers
x86/mm: Add a noinvpcid option to turn off INVPCID x86/mm: If INVPCID is available, use it to flush global mappings
Quoted 4 lines
Documentation/kernel-parameters.txt | 2 ++ arch/x86/include/asm/tlbflush.h | 50 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 16 ++++++++++++ 3 files changed, 68 insertions(+)
Ok, I'll pick these up tomorrow unless there are objections.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: several messages
Quoted 4 lines · Andy Lutomirski
This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.)
Haswell and Broadwell have it. No idea about ivy bridge.
Thanks,
tglx
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Borislav Petkov
maintainerRe: several messages
Quoted 7 lines · Thomas Gleixner
Quoted 4 lines · Andy Lutomirski
This is a straightforward speedup on Ivy Bridge and newer, IIRC. (I tested on Skylake. INVPCID is not available on Sandy Bridge. I don't have Ivy Bridge, Haswell or Broadwell to test on, so I could be wrong as to when the feature was introduced.)
Haswell and Broadwell have it. No idea about ivy bridge.
I have an IVB model 58. It doesn't have it:
CPUID_0x00000007: EAX=0x00000000, EBX=0x00000281, ECX=0x00000000, EDX=0x00000000
INVPCID should be EBX[10].
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.