PCID and improved laziness
Subsystems (8)
Andy Lutomirski
maintainerv4 cover PCID and improved laziness
*** Ingo, even if this misses 4.13, please apply the first patch before *** the merge window.
There are three performance benefits here:
1. TLB flushing is slow. (I.e. the flush itself takes a while.) This avoids many of them when switching tasks by using PCID. In a stupid little benchmark I did, it saves about 100ns on my laptop per context switch. I'll try to improve that benchmark.
2. Mms that have been used recently on a given CPU might get to keep their TLB entries alive across process switches with this patch set. TLB fills are pretty fast on modern CPUs, but they're even faster when they don't happen.
3. Lazy TLB is way better. We used to do two stupid things when we ran kernel threads: we'd send IPIs to flush user contexts on their CPUs and then we'd write to CR3 for no particular reason as an excuse to stop further IPIs. With this patch, we do neither.
This will, in general, perform suboptimally if paravirt TLB flushing is in use (currently just Xen, I think, but Hyper-V is in the works). The code is structured so we could fix it in one of two ways: we could take a spinlock when touching the percpu state so we can update it remotely after a paravirt flush, or we could be more careful about our exactly how we access the state and use cmpxchg16b to do atomic remote updates. (On SMP systems without cmpxchg16b, we'd just skip the optimization entirely.)
This is still missing a final comment-only patch to add overall documentation for the whole thing, but I didn't want to block sending the maybe-hopefully-final code on that.
This is based on tip:x86/mm. The branch is here if you want to play:
https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/log/?h=x86/pcid
In general, performance seems to exceed my expectations. Here are some performance numbers copy-and-pasted from the changelogs for "Rework lazy TLB mode and TLB freshness" and "Try to preserve old TLB entries using PCID":
MADV_DONTNEED; touch the page; switch CPUs using sched_setaffinity. In an unpatched kernel, MADV_DONTNEED will send an IPI to the previous CPU. This is intended to be a nearly worst-case test.
patched: 13.4Aus unpatched: 21.6Aus
Vitaly's pthread_mmap microbenchmark with 8 threads (on four cores), nrounds = 100, 256M data
patched: 1.1 seconds or so unpatched: 1.9 seconds or so
ping-pong between two mms on the same CPU using eventfd:
patched: 1.22Aus
patched, nopcid: 1.33Aus
unpatched: 1.34Aus
Same ping-pong, but now touch 512 pages (all zero-page to minimize cache misses) each iteration. dTLB misses are measured by dtlb_load_misses.miss_causes_a_walk:
patched: 1.8Aus 11M dTLB misses
patched, nopcid: 6.2Aus, 207M dTLB misses
unpatched: 6.1Aus, 190M dTLB misses
MADV_DONTNEED; touch the page; switch CPUs using sched_setaffinity. In an unpatched kernel, MADV_DONTNEED will send an IPI to the previous CPU. This is intended to be a nearly worst-case test.
patched: 13.4Aus
unpatched: 21.6Aus
Changes from v3:
- Lots more acks.
- Move comment deletion to the beginning.
- Misc cleanups from lots of reviewers.
Changes from v2:
- Add some Acks
- Move the reentrancy issue to the beginning.
(I also sent the same patch as a standalone fix -- it's just in here
so that this series applies to x86/mm.)
- Fix some comments.
Changes from RFC:
- flush_tlb_func_common() no longer gets reentered (Nadav)
- Fix ASID corruption on unlazying (kbuild bot)
- Move Xen init to the right place
- Misc cleanups
Andy Lutomirski (10):
x86/mm: Don't reenter flush_tlb_func_common()
x86/mm: Delete a big outdated comment about TLB flushing x86/mm: Give each mm TLB flush generation a unique ID x86/mm: Track the TLB's tlb_gen and update the flushing algorithm x86/mm: Rework lazy TLB mode and TLB freshness tracking x86/mm: Stop calling leave_mm() in idle code
x86/mm: Disable PCID on 32-bit kernels
x86/mm: Add nopcid to turn off PCID
x86/mm: Enable CR4.PCIDE on supported systems
x86/mm: Try to preserve old TLB entries using PCID
Documentation/admin-guide/kernel-parameters.txt | 2 + arch/ia64/include/asm/acpi.h | 2 - arch/x86/include/asm/acpi.h | 2 - arch/x86/include/asm/disabled-features.h | 4 +- arch/x86/include/asm/mmu.h | 25 +- arch/x86/include/asm/mmu_context.h | 15 +- arch/x86/include/asm/processor-flags.h | 2 + arch/x86/include/asm/tlbflush.h | 87 +++++- arch/x86/kernel/cpu/bugs.c | 8 + arch/x86/kernel/cpu/common.c | 40 +++ arch/x86/mm/init.c | 2 +- arch/x86/mm/tlb.c | 382 ++++++++++++++++-------- arch/x86/xen/enlighten_pv.c | 6 + arch/x86/xen/mmu_pv.c | 5 +- drivers/acpi/processor_idle.c | 2 - drivers/idle/intel_idle.c | 9 +- 16 files changed, 446 insertions(+), 147 deletions(-)
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andy Lutomirski
maintainerPATCH v4 1/10 x86/mm: Don't reenter flush_tlb_func_common()
It was historically possible to have two concurrent TLB flushes targetting the same CPU: one initiated locally and one initiated remotely. This can now cause an OOPS in leave_mm() at arch/x86/mm/tlb.c:47:
if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
BUG();with this call trace:
flush_tlb_func_local arch/x86/mm/tlb.c:239 [inline] flush_tlb_mm_range+0x26d/0x370 arch/x86/mm/tlb.c:317
Without reentrancy, this OOPS is impossible: leave_mm() is only called if we're not in TLBSTATE_OK, but then we're unexpectedly in TLBSTATE_OK in leave_mm().
This can be caused by flush_tlb_func_remote() happening between the two checks and calling leave_mm(), resulting in two consecutive leave_mm() calls on the same CPU with no intervening switch_mm() calls.
We never saw this OOPS before because the old leave_mm() implementation didn't put us back in TLBSTATE_OK, so the assertion didn't fire.
Nadav noticed the reentrancy issue in a different context, but neither of us realized that it caused a problem yet.
--- arch/x86/mm/tlb.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
arch/x86/mm/tlb.c +15 -2
@@ -192,6 +192,9 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,static void flush_tlb_func_common (const struct flush_tlb_info *f,bool local,enum tlb_flush_reason reason) { +/* This code cannot presently handle being reentered. */ +VM_WARN_ON (!irqs_disabled ()); +if (this_cpu_read (cpu_tlbstate.state) !=TLBSTATE_OK ) {leave_mm (smp_processor_id ());return ;
@@ -297,8 +300,13 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, info.end =TLB_FLUSH_ALL ; } -if (mm ==this_cpu_read (cpu_tlbstate.loaded_mm)) +if (mm ==this_cpu_read (cpu_tlbstate.loaded_mm)) { +VM_WARN_ON (irqs_disabled ()); +local_irq_disable ();flush_tlb_func_local (&info,TLB_LOCAL_MM_SHOOTDOWN ); +local_irq_enable (); + } +if (cpumask_any_but (mm_cpumask (mm), cpu) < nr_cpu_ids)flush_tlb_others (mm_cpumask (mm), &info);put_cpu ();
@@ -354,8 +362,13 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)int cpu =get_cpu (); -if (cpumask_test_cpu (cpu, &batch->cpumask)) +if (cpumask_test_cpu (cpu, &batch->cpumask)) { +VM_WARN_ON (irqs_disabled ()); +local_irq_disable ();flush_tlb_func_local (&info,TLB_LOCAL_SHOOTDOWN ); +local_irq_enable (); + } +if (cpumask_any_but (&batch->cpumask, cpu) < nr_cpu_ids)flush_tlb_others (&batch->cpumask, &info);cpumask_clear (&batch->cpumask);
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Don't reenter flush_tlb_func_common()
x86/mm: Don't reenter flush_tlb_func_common()
Commit-ID: bc0d5a89fbe3c83ac45438d7ba88309f4713615d Gitweb: http://git.kernel.org/tip/bc0d5a89fbe3c83ac45438d7ba88309f4713615d Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:13 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 30 Jun 2017 10:12:35 +0200
x86/mm: Don't reenter flush_tlb_func_common()
It was historically possible to have two concurrent TLB flushes targetting the same CPU: one initiated locally and one initiated remotely. This can now cause an OOPS in leave_mm() at arch/x86/mm/tlb.c:47:
if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
BUG();with this call trace:
flush_tlb_func_local arch/x86/mm/tlb.c:239 [inline] flush_tlb_mm_range+0x26d/0x370 arch/x86/mm/tlb.c:317
Without reentrancy, this OOPS is impossible: leave_mm() is only called if we're not in TLBSTATE_OK, but then we're unexpectedly in TLBSTATE_OK in leave_mm().
This can be caused by flush_tlb_func_remote() happening between the two checks and calling leave_mm(), resulting in two consecutive leave_mm() calls on the same CPU with no intervening switch_mm() calls.
We never saw this OOPS before because the old leave_mm() implementation didn't put us back in TLBSTATE_OK, so the assertion didn't fire.
Nadav noticed the reentrancy issue in a different context, but neither of us realized that it caused a problem yet.
--- arch/x86/mm/tlb.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
arch/x86/mm/tlb.c +15 -2
@@ -192,6 +192,9 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,static void flush_tlb_func_common (const struct flush_tlb_info *f,bool local,enum tlb_flush_reason reason) { +/* This code cannot presently handle being reentered. */ +VM_WARN_ON (!irqs_disabled ()); +if (this_cpu_read (cpu_tlbstate.state) !=TLBSTATE_OK ) {leave_mm (smp_processor_id ());return ;
@@ -297,8 +300,13 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, info.end =TLB_FLUSH_ALL ; } -if (mm ==this_cpu_read (cpu_tlbstate.loaded_mm)) +if (mm ==this_cpu_read (cpu_tlbstate.loaded_mm)) { +VM_WARN_ON (irqs_disabled ()); +local_irq_disable ();flush_tlb_func_local (&info,TLB_LOCAL_MM_SHOOTDOWN ); +local_irq_enable (); + } +if (cpumask_any_but (mm_cpumask (mm), cpu) < nr_cpu_ids)flush_tlb_others (mm_cpumask (mm), &info);put_cpu ();
@@ -354,8 +362,13 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)int cpu =get_cpu (); -if (cpumask_test_cpu (cpu, &batch->cpumask)) +if (cpumask_test_cpu (cpu, &batch->cpumask)) { +VM_WARN_ON (irqs_disabled ()); +local_irq_disable ();flush_tlb_func_local (&info,TLB_LOCAL_SHOOTDOWN ); +local_irq_enable (); + } +if (cpumask_any_but (&batch->cpumask, cpu) < nr_cpu_ids)flush_tlb_others (&batch->cpumask, &info);cpumask_clear (&batch->cpumask);
Andy Lutomirski
maintainerPATCH v4 2/10 x86/mm: Delete a big outdated comment about TLB flushing
In reply to Andy LutomirskiThe comment describes the old explicit IPI-based flush logic, which is long gone.
--- arch/x86/mm/tlb.c | 36 ------------------------------------ 1 file changed, 36 deletions(-)
arch/x86/mm/tlb.c +0 -36
@@ -153,42 +153,6 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,switch_ldt (real_prev, next); } -/* -* The flush IPI assumes that a thread switch happens in this order: -* [cpu0: the cpu that switches] -* 1) switch_mm() either 1a) or 1b) -* 1a) thread switch to a different mm -* 1a1) set cpu_tlbstate to TLBSTATE_OK -* Now the tlb flush NMI handler flush_tlb_func won't call leave_mm -* if cpu0 was in lazy tlb mode. -* 1a2) update cpu active_mm -* Now cpu0 accepts tlb flushes for the new mm. -* 1a3) cpu_set(cpu, new_mm->cpu_vm_mask); -* Now the other cpus will send tlb flush ipis. -* 1a4) change cr3. -* 1a5) cpu_clear(cpu, old_mm->cpu_vm_mask); -* Stop ipi delivery for the old mm. This is not synchronized with -* the other cpus, but flush_tlb_func ignore flush ipis for the wrong -* mm, and in the worst case we perform a superfluous tlb flush. -* 1b) thread switch without mm change -* cpu active_mm is correct, cpu0 already handles flush ipis. -* 1b1) set cpu_tlbstate to TLBSTATE_OK -* 1b2) test_and_set the cpu bit in cpu_vm_mask. -* Atomically set the bit [other cpus will start sending flush ipis], -* and test the bit. -* 1b3) if the bit was 0: leave_mm was called, flush the tlb. -* 2) switch %%esp, ie current -* -* The interrupt must handle 2 special cases: -* - cr3 is changed before %%esp, ie. it cannot use current->{active_,}mm. -* - the cpu performs speculative tlb reads, i.e. even if the cpu only -* runs in kernel space, the cpu could load tlb entries for user space -* pages. -* -* The good news is that cpu_tlbstate is local to each cpu, no -* write/read ordering problems. -*/ -static void flush_tlb_func_common (const struct flush_tlb_info *f,bool local,enum tlb_flush_reason reason) {
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Delete a big outdated comment about TLB flushing
x86/mm: Delete a big outdated comment about TLB flushing
Commit-ID: 8781fb7e9749da424e01daacd14834b674658c63 Gitweb: http://git.kernel.org/tip/8781fb7e9749da424e01daacd14834b674658c63 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:14 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 30 Jun 2017 10:12:35 +0200
x86/mm: Delete a big outdated comment about TLB flushing
The comment describes the old explicit IPI-based flush logic, which is long gone.
--- arch/x86/mm/tlb.c | 36 ------------------------------------ 1 file changed, 36 deletions(-)
arch/x86/mm/tlb.c +0 -36
@@ -153,42 +153,6 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,switch_ldt (real_prev, next); } -/* -* The flush IPI assumes that a thread switch happens in this order: -* [cpu0: the cpu that switches] -* 1) switch_mm() either 1a) or 1b) -* 1a) thread switch to a different mm -* 1a1) set cpu_tlbstate to TLBSTATE_OK -* Now the tlb flush NMI handler flush_tlb_func won't call leave_mm -* if cpu0 was in lazy tlb mode. -* 1a2) update cpu active_mm -* Now cpu0 accepts tlb flushes for the new mm. -* 1a3) cpu_set(cpu, new_mm->cpu_vm_mask); -* Now the other cpus will send tlb flush ipis. -* 1a4) change cr3. -* 1a5) cpu_clear(cpu, old_mm->cpu_vm_mask); -* Stop ipi delivery for the old mm. This is not synchronized with -* the other cpus, but flush_tlb_func ignore flush ipis for the wrong -* mm, and in the worst case we perform a superfluous tlb flush. -* 1b) thread switch without mm change -* cpu active_mm is correct, cpu0 already handles flush ipis. -* 1b1) set cpu_tlbstate to TLBSTATE_OK -* 1b2) test_and_set the cpu bit in cpu_vm_mask. -* Atomically set the bit [other cpus will start sending flush ipis], -* and test the bit. -* 1b3) if the bit was 0: leave_mm was called, flush the tlb. -* 2) switch %%esp, ie current -* -* The interrupt must handle 2 special cases: -* - cr3 is changed before %%esp, ie. it cannot use current->{active_,}mm. -* - the cpu performs speculative tlb reads, i.e. even if the cpu only -* runs in kernel space, the cpu could load tlb entries for user space -* pages. -* -* The good news is that cpu_tlbstate is local to each cpu, no -* write/read ordering problems. -*/ -static void flush_tlb_func_common (const struct flush_tlb_info *f,bool local,enum tlb_flush_reason reason) {
Andy Lutomirski
maintainerPATCH v4 3/10 x86/mm: Give each mm TLB flush generation a unique ID
In reply to Andy LutomirskiThis adds two new variables to mmu_context_t: ctx_id and tlb_gen. ctx_id uniquely identifies the mm_struct and will never be reused. For a given mm_struct (and hence ctx_id), tlb_gen is a monotonic count of the number of times that a TLB flush has been requested. The pair (ctx_id, tlb_gen) can be used as an identifier for TLB flush actions and will be used in subsequent patches to reliably determine whether all needed TLB flushes have occurred on a given CPU.
This patch is split out for ease of review. By itself, it has no real effect other than creating and updating the new variables.
--- arch/x86/include/asm/mmu.h | 25 +++++++++++++++++++++++-- arch/x86/include/asm/mmu_context.h | 6 ++++++ arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ arch/x86/mm/tlb.c | 6 ++++-- 4 files changed, 51 insertions(+), 4 deletions(-)
arch/x86/include/asm/mmu.h +23 -2
@@ -3,12 +3,28 @@#include <linux/spinlock.h> #include <linux/mutex.h> +#include <linux/atomic.h> /* -* The x86 doesn't have a mmu context, but -* we put the segment information here. +* x86 has arch-specific MMU state beyond what lives in mm_struct. */ typedef struct { + /* + * ctx_id uniquely identifies this mm_struct. A ctx_id will never + * be reused, and zero is not a valid ctx_id. + */ + u64 ctx_id; + + /* + * Any code that needs to do any sort of TLB flushing for this + * mm will first make its changes to the page tables, then + * increment tlb_gen, then flush. This lets the low-level + * flushing code keep track of what needs flushing. + * + * This is not used on Xen PV. + */ + atomic64_t tlb_gen; + #ifdef CONFIG_MODIFY_LDT_SYSCALL struct ldt_struct *ldt; #endif
@@ -37,6 +53,11 @@ typedef struct { #endif } mm_context_t; +#define INIT_MM_CONTEXT (mm) \ + .context = { \ + .ctx_id = 1, \ + } +void leave_mm (int cpu);#endif /* _ASM_X86_MMU_H */
arch/x86/include/asm/mmu_context.h +6 -0
@@ -12,6 +12,9 @@#include <asm/tlbflush.h> #include <asm/paravirt.h> #include <asm/mpx.h> + +extern atomic64_t last_mm_ctx_id; +#ifndef CONFIG_PARAVIRT static inline void paravirt_activate_mm (struct mm_struct *prev,struct mm_struct *next)
@@ -132,6 +135,9 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)static inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) { + mm->context.ctx_id =atomic64_inc_return (&last_mm_ctx_id); +atomic64_set (&mm->context.tlb_gen,0 ); +#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS if (cpu_feature_enabled (X86_FEATURE_OSPKE )) {/* pkey 0 is the default and always allocated */
arch/x86/include/asm/tlbflush.h +18 -0
@@ -57,6 +57,23 @@ static inline void invpcid_flush_all_nonglobals(void)__invpcid (0 ,0 ,INVPCID_TYPE_ALL_NON_GLOBAL ); } +static inline u64 inc_mm_tlb_gen (struct mm_struct *mm) +{ +u64 new_tlb_gen; + +/* +* Bump the generation count. This also serves as a full barrier +* that synchronizes with switch_mm(): callers are required to order +* their read of mm_cpumask after their writes to the paging +* structures. +*/ +smp_mb__before_atomic (); + new_tlb_gen =atomic64_inc_return (&mm->context.tlb_gen); +smp_mb__after_atomic (); + +return new_tlb_gen; +} +#ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else
@@ -262,6 +279,7 @@ void native_flush_tlb_others(const struct cpumask *cpumask,static inline void arch_tlbbatch_add_mm (struct arch_tlbflush_unmap_batch *batch,struct mm_struct *mm) { +inc_mm_tlb_gen (mm);cpumask_or (&batch->cpumask, &batch->cpumask,mm_cpumask (mm)); }
arch/x86/mm/tlb.c +4 -2
@@ -28,6 +28,8 @@* Implement flush IPI by CALL_FUNCTION_VECTOR, Alex Shi */ +atomic64_t last_mm_ctx_id =ATOMIC64_INIT (1 ); +void leave_mm (int cpu) {struct mm_struct *loaded_mm =this_cpu_read (cpu_tlbstate.loaded_mm);
@@ -250,8 +252,8 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, cpu =get_cpu (); -/* Synchronize with switch_mm. */ -smp_mb (); +/* This is also a barrier that synchronizes with switch_mm(). */ +inc_mm_tlb_gen (mm);/* Should we flush just the requested range? */ if ((end != TLB_FLUSH_ALL) &&
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Give each mm TLB flush generation a unique ID
x86/mm: Give each mm TLB flush generation a unique ID
Commit-ID: f39681ed0f48498b80455095376f11535feea332 Gitweb: http://git.kernel.org/tip/f39681ed0f48498b80455095376f11535feea332 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:15 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 5 Jul 2017 10:52:56 +0200
x86/mm: Give each mm TLB flush generation a unique ID
This adds two new variables to mmu_context_t: ctx_id and tlb_gen. ctx_id uniquely identifies the mm_struct and will never be reused. For a given mm_struct (and hence ctx_id), tlb_gen is a monotonic count of the number of times that a TLB flush has been requested. The pair (ctx_id, tlb_gen) can be used as an identifier for TLB flush actions and will be used in subsequent patches to reliably determine whether all needed TLB flushes have occurred on a given CPU.
This patch is split out for ease of review. By itself, it has no real effect other than creating and updating the new variables.
--- arch/x86/include/asm/mmu.h | 25 +++++++++++++++++++++++-- arch/x86/include/asm/mmu_context.h | 6 ++++++ arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ arch/x86/mm/tlb.c | 6 ++++-- 4 files changed, 51 insertions(+), 4 deletions(-)
arch/x86/include/asm/mmu.h +23 -2
@@ -3,12 +3,28 @@#include <linux/spinlock.h> #include <linux/mutex.h> +#include <linux/atomic.h> /* -* The x86 doesn't have a mmu context, but -* we put the segment information here. +* x86 has arch-specific MMU state beyond what lives in mm_struct. */ typedef struct { + /* + * ctx_id uniquely identifies this mm_struct. A ctx_id will never + * be reused, and zero is not a valid ctx_id. + */ + u64 ctx_id; + + /* + * Any code that needs to do any sort of TLB flushing for this + * mm will first make its changes to the page tables, then + * increment tlb_gen, then flush. This lets the low-level + * flushing code keep track of what needs flushing. + * + * This is not used on Xen PV. + */ + atomic64_t tlb_gen; + #ifdef CONFIG_MODIFY_LDT_SYSCALL struct ldt_struct *ldt; #endif
@@ -37,6 +53,11 @@ typedef struct { #endif } mm_context_t; +#define INIT_MM_CONTEXT (mm) \ + .context = { \ + .ctx_id = 1, \ + } +void leave_mm (int cpu);#endif /* _ASM_X86_MMU_H */
arch/x86/include/asm/mmu_context.h +6 -0
@@ -12,6 +12,9 @@#include <asm/tlbflush.h> #include <asm/paravirt.h> #include <asm/mpx.h> + +extern atomic64_t last_mm_ctx_id; +#ifndef CONFIG_PARAVIRT static inline void paravirt_activate_mm (struct mm_struct *prev,struct mm_struct *next)
@@ -132,6 +135,9 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)static inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) { + mm->context.ctx_id =atomic64_inc_return (&last_mm_ctx_id); +atomic64_set (&mm->context.tlb_gen,0 ); +#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS if (cpu_feature_enabled (X86_FEATURE_OSPKE )) {/* pkey 0 is the default and always allocated */
arch/x86/include/asm/tlbflush.h +18 -0
@@ -57,6 +57,23 @@ static inline void invpcid_flush_all_nonglobals(void)__invpcid (0 ,0 ,INVPCID_TYPE_ALL_NON_GLOBAL ); } +static inline u64 inc_mm_tlb_gen (struct mm_struct *mm) +{ +u64 new_tlb_gen; + +/* +* Bump the generation count. This also serves as a full barrier +* that synchronizes with switch_mm(): callers are required to order +* their read of mm_cpumask after their writes to the paging +* structures. +*/ +smp_mb__before_atomic (); + new_tlb_gen =atomic64_inc_return (&mm->context.tlb_gen); +smp_mb__after_atomic (); + +return new_tlb_gen; +} +#ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else
@@ -262,6 +279,7 @@ void native_flush_tlb_others(const struct cpumask *cpumask,static inline void arch_tlbbatch_add_mm (struct arch_tlbflush_unmap_batch *batch,struct mm_struct *mm) { +inc_mm_tlb_gen (mm);cpumask_or (&batch->cpumask, &batch->cpumask,mm_cpumask (mm)); }
arch/x86/mm/tlb.c +4 -2
@@ -28,6 +28,8 @@* Implement flush IPI by CALL_FUNCTION_VECTOR, Alex Shi */ +atomic64_t last_mm_ctx_id =ATOMIC64_INIT (1 ); +void leave_mm (int cpu) {struct mm_struct *loaded_mm =this_cpu_read (cpu_tlbstate.loaded_mm);
@@ -250,8 +252,8 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, cpu =get_cpu (); -/* Synchronize with switch_mm. */ -smp_mb (); +/* This is also a barrier that synchronizes with switch_mm(). */ +inc_mm_tlb_gen (mm);/* Should we flush just the requested range? */ if ((end != TLB_FLUSH_ALL) &&
Peter Zijlstra
maintainerRe: [tip:x86/mm] x86/mm: Give each mm TLB flush generation a unique ID
arch/x86/include/asm/mmu_context.h static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) · line 138
Quoted 5 lines · tip-bot for Andy Lutomirski
arch/x86/include/asm/mmu_context.h +1
-0@@ -132,6 +135,9 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)static inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) { + mm->context.ctx_id =atomic64_inc_return (&last_mm_ctx_id);
You could use atomic64_inc_return_relaxed() here; but since its x86 specific there is no difference.
Quoted 17 lines
arch/x86/include/asm/mmu_context.h +2
-0+atomic64_set (&mm->context.tlb_gen,0 ); +#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS if (cpu_feature_enabled (X86_FEATURE_OSPKE )) {/* pkey 0 is the default and always allocated */ arch/x86/include/asm/tlbflush.h +3
-0@@ -57,6 +57,23 @@ static inline void invpcid_flush_all_nonglobals(void)__invpcid (0 ,0 ,INVPCID_TYPE_ALL_NON_GLOBAL ); } +static inline u64 inc_mm_tlb_gen (struct mm_struct *mm) +{ +u64 new_tlb_gen;
arch/x86/include/asm/tlbflush.h static inline void invpcid_flush_all_nonglobals(void) · line 72
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+ +/* +* Bump the generation count. This also serves as a full barrier +* that synchronizes with switch_mm(): callers are required to order +* their read of mm_cpumask after their writes to the paging +* structures. +*/ +smp_mb__before_atomic (); + new_tlb_gen =atomic64_inc_return (&mm->context.tlb_gen); +smp_mb__after_atomic ();
that's wrong... smp_mb__{before,after}_atomic() are entirely superfluous here:
- they're no-ops on x86
- atomic_*_return() is already fully serializing
Andy Lutomirski
maintainerPATCH v4 4/10 x86/mm: Track the TLB's tlb_gen and update the flushing algorithm
In reply to Andy LutomirskiThere are two kernel features that would benefit from tracking how up-to-date each CPU's TLB is in the case where IPIs aren't keeping it up to date in real time:
- Lazy mm switching currently works by switching to init_mm when it would otherwise flush. This is wasteful: there isn't fundamentally any need to update CR3 at all when going lazy or when returning from lazy mode, nor is there any need to receive flush IPIs at all. Instead, we should just stop trying to keep the TLB coherent when we go lazy and, when unlazying, check whether we missed any flushes.
- PCID will let us keep recent user contexts alive in the TLB. If we start doing this, we need a way to decide whether those contexts are up to date.
On some paravirt systems, remote TLBs can be flushed without IPIs. This won't update the target CPUs' tlb_gens, which may cause unnecessary local flushes later on. We can address this if it becomes a problem by carefully updating the target CPU's tlb_gen directly.
By itself, this patch is a very minor optimization that avoids unnecessary flushes when multiple TLB flushes targetting the same CPU race. The complexity in this patch would not be worth it on its own, but it will enable improved lazy TLB tracking and PCID.
--- arch/x86/include/asm/tlbflush.h | 43 +++++++++++++++-- arch/x86/mm/tlb.c | 102 +++++++++++++++++++++++++++++++++++++--- 2 files changed, 135 insertions(+), 10 deletions(-)
arch/x86/include/asm/tlbflush.h +40 -3
@@ -82,6 +82,11 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm)#define __flush_tlb_single (addr) __native_flush_tlb_single(addr)#endif +struct tlb_context { +u64 ctx_id; +u64 tlb_gen; +}; +struct tlb_state {/* * cpu_tlbstate.loaded_mm should match CR3 whenever interrupts
@@ -97,6 +102,21 @@ struct tlb_state {* disabling interrupts when modifying either one. */ unsigned long cr4; + +/* +* This is a list of all contexts that might exist in the TLB. +* Since we don't yet use PCID, there is only one context. +* +* For each context, ctx_id indicates which mm the TLB's user +* entries came from. As an invariant, the TLB will never +* contain entries that are out-of-date as when that mm reached +* the tlb_gen in the list. +* +* To be clear, this means that it's legal for the TLB code to +* flush the TLB without updating tlb_gen. This can happen +* (for now, at least) due to paravirt remote flushes. +*/ +struct tlb_context ctxs[1 ]; }; DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate);
@@ -248,9 +268,26 @@ static inline void __flush_tlb_one(unsigned long addr)* and page-granular flushes are available only on i486 and up. */ struct flush_tlb_info { -struct mm_struct *mm; -unsigned long start; -unsigned long end; +/* +* We support several kinds of flushes. +* +* - Fully flush a single mm. .mm will be set, .end will be +* TLB_FLUSH_ALL, and .new_tlb_gen will be the tlb_gen to +* which the IPI sender is trying to catch us up. +* +* - Partially flush a single mm. .mm will be set, .start and +* .end will indicate the range, and .new_tlb_gen will be set +* such that the changes between generation .new_tlb_gen-1 and +* .new_tlb_gen are entirely contained in the indicated range. +* +* - Fully flush all mms whose tlb_gens have been updated. .mm +* will be NULL, .end will be TLB_FLUSH_ALL, and .new_tlb_gen +* will be zero. +*/ +struct mm_struct *mm; +unsigned long start; +unsigned long end; +u64 new_tlb_gen; };#define local_flush_tlb () __flush_tlb()
arch/x86/mm/tlb.c +95 -7
140 lines not on this page · raw message
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Track the TLB's tlb_gen and update the flushing algorithm
x86/mm: Track the TLB's tlb_gen and update the flushing algorithm
Commit-ID: b0579ade7cd82391360e959cc844e50a160e8a96 Gitweb: http://git.kernel.org/tip/b0579ade7cd82391360e959cc844e50a160e8a96 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:16 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 5 Jul 2017 10:52:56 +0200
x86/mm: Track the TLB's tlb_gen and update the flushing algorithm
There are two kernel features that would benefit from tracking how up-to-date each CPU's TLB is in the case where IPIs aren't keeping it up to date in real time:
- Lazy mm switching currently works by switching to init_mm when it would otherwise flush. This is wasteful: there isn't fundamentally any need to update CR3 at all when going lazy or when returning from lazy mode, nor is there any need to receive flush IPIs at all. Instead, we should just stop trying to keep the TLB coherent when we go lazy and, when unlazying, check whether we missed any flushes.
- PCID will let us keep recent user contexts alive in the TLB. If we start doing this, we need a way to decide whether those contexts are up to date.
On some paravirt systems, remote TLBs can be flushed without IPIs. This won't update the target CPUs' tlb_gens, which may cause unnecessary local flushes later on. We can address this if it becomes a problem by carefully updating the target CPU's tlb_gen directly.
By itself, this patch is a very minor optimization that avoids unnecessary flushes when multiple TLB flushes targetting the same CPU race. The complexity in this patch would not be worth it on its own, but it will enable improved lazy TLB tracking and PCID.
--- arch/x86/include/asm/tlbflush.h | 43 +++++++++++++++-- arch/x86/mm/tlb.c | 102 +++++++++++++++++++++++++++++++++++++--- 2 files changed, 135 insertions(+), 10 deletions(-)
arch/x86/include/asm/tlbflush.h +40 -3
@@ -82,6 +82,11 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm)#define __flush_tlb_single (addr) __native_flush_tlb_single(addr)#endif +struct tlb_context { +u64 ctx_id; +u64 tlb_gen; +}; +struct tlb_state {/* * cpu_tlbstate.loaded_mm should match CR3 whenever interrupts
@@ -97,6 +102,21 @@ struct tlb_state {* disabling interrupts when modifying either one. */ unsigned long cr4; + +/* +* This is a list of all contexts that might exist in the TLB. +* Since we don't yet use PCID, there is only one context. +* +* For each context, ctx_id indicates which mm the TLB's user +* entries came from. As an invariant, the TLB will never +* contain entries that are out-of-date as when that mm reached +* the tlb_gen in the list. +* +* To be clear, this means that it's legal for the TLB code to +* flush the TLB without updating tlb_gen. This can happen +* (for now, at least) due to paravirt remote flushes. +*/ +struct tlb_context ctxs[1 ]; }; DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate);
@@ -248,9 +268,26 @@ static inline void __flush_tlb_one(unsigned long addr)* and page-granular flushes are available only on i486 and up. */ struct flush_tlb_info { -struct mm_struct *mm; -unsigned long start; -unsigned long end; +/* +* We support several kinds of flushes. +* +* - Fully flush a single mm. .mm will be set, .end will be +* TLB_FLUSH_ALL, and .new_tlb_gen will be the tlb_gen to +* which the IPI sender is trying to catch us up. +* +* - Partially flush a single mm. .mm will be set, .start and +* .end will indicate the range, and .new_tlb_gen will be set +* such that the changes between generation .new_tlb_gen-1 and +* .new_tlb_gen are entirely contained in the indicated range. +* +* - Fully flush all mms whose tlb_gens have been updated. .mm +* will be NULL, .end will be TLB_FLUSH_ALL, and .new_tlb_gen +* will be zero. +*/ +struct mm_struct *mm; +unsigned long start; +unsigned long end; +u64 new_tlb_gen; };#define local_flush_tlb () __flush_tlb()
arch/x86/mm/tlb.c +95 -7
140 lines not on this page · raw message
Andy Lutomirski
maintainerPATCH v4 5/10 x86/mm: Rework lazy TLB mode and TLB freshness tracking
In reply to Andy Lutomirskix86's lazy TLB mode used to be fairly weak -- it would switch to init_mm the first time it tried to flush a lazy TLB. This meant an unnecessary CR3 write and, if the flush was remote, an unnecessary IPI.
Rewrite it entirely. When we enter lazy mode, we simply remove the CPU from mm_cpumask. This means that we need a way to figure out whether we've missed a flush when we switch back out of lazy mode. I use the tlb_gen machinery to track whether a context is up to date.
Note to reviewers: this patch, my itself, looks a bit odd. I'm using an array of length 1 containing (ctx_id, tlb_gen) rather than just storing tlb_gen, and making it at array isn't necessary yet. I'm doing this because the next few patches add PCID support, and, with PCID, we need ctx_id, and the array will end up with a length greater than 1. Making it an array now means that there will be less churn and therefore less stress on your eyeballs.
NB: This is dubious but, AFAICT, still correct on Xen and UV. xen_exit_mmap() uses mm_cpumask() for nefarious purposes and this patch changes the way that mm_cpumask() works. This should be okay, since Xen *also* iterates all online CPUs to find all the CPUs it needs to twiddle.
The UV tlbflush code is rather dated and should be changed.
Here are some benchmark results, done on a Skylake laptop at 2.3 GHz (turbo off, intel_pstate requesting max performance) under KVM with the guest using idle=poll (to avoid artifacts when bouncing between CPUs). I haven't done any real statistics here -- I just ran them in a loop and picked the fastest results that didn't look like outliers. Unpatched means commit a4eb8b993554, so all the bookkeeping overhead is gone.
MADV_DONTNEED; touch the page; switch CPUs using sched_setaffinity. In an unpatched kernel, MADV_DONTNEED will send an IPI to the previous CPU. This is intended to be a nearly worst-case test.
patched: 13.4Aus unpatched: 21.6Aus
Vitaly's pthread_mmap microbenchmark with 8 threads (on four cores), nrounds = 100, 256M data
patched: 1.1 seconds or so unpatched: 1.9 seconds or so
The sleepup on Vitaly's test appearss to be because it spends a lot of time blocked on mmap_sem, and this patch avoids sending IPIs to blocked CPUs.
--- arch/x86/include/asm/mmu_context.h | 6 +- arch/x86/include/asm/tlbflush.h | 4 - arch/x86/mm/init.c | 1 - arch/x86/mm/tlb.c | 197 ++++++++++++++++++++++--------------- arch/x86/xen/mmu_pv.c | 5 +- 5 files changed, 124 insertions(+), 89 deletions(-)
arch/x86/include/asm/mmu_context.h +4 -2
@@ -128,8 +128,10 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next)static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) { -if (this_cpu_read (cpu_tlbstate.state) ==TLBSTATE_OK ) -this_cpu_write (cpu_tlbstate.state,TLBSTATE_LAZY ); +int cpu =smp_processor_id (); + +if (cpumask_test_cpu (cpu,mm_cpumask (mm))) +cpumask_clear_cpu (cpu,mm_cpumask (mm)); } static inline int init_new_context(struct task_struct *tsk,
arch/x86/include/asm/tlbflush.h +0 -4
@@ -95,7 +95,6 @@ struct tlb_state {* mode even if we've already switched back to swapper_pg_dir. */ struct mm_struct *loaded_mm; -int state;/* * Access to this CR4 shadow and to H/W CR4 is protected by
@@ -310,9 +309,6 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a)void native_flush_tlb_others (const struct cpumask *cpumask,const struct flush_tlb_info *info); -#define TLBSTATE_OK 1 -#define TLBSTATE_LAZY 2 -static inline void arch_tlbbatch_add_mm (struct arch_tlbflush_unmap_batch *batch,struct mm_struct *mm) {
arch/x86/mm/init.c +0 -1
@@ -812,7 +812,6 @@ void __init zone_sizes_init(void) DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate) = { .loaded_mm = &init_mm, - .state =0 , .cr4 = ~0UL ,/* fail hard if we screw up cr4 shadow initialization */ };EXPORT_SYMBOL_GPL (cpu_tlbstate);
arch/x86/mm/tlb.c +118 -79
265 lines not on this page · raw message
arch/x86/xen/mmu_pv.c +2 -3
@@ -1005,14 +1005,12 @@ static void xen_drop_mm_ref(struct mm_struct *mm)/* Get the "official" set of cpus referring to our pagetable. */ if (!alloc_cpumask_var (&mask,GFP_ATOMIC )) {for_each_online_cpu (cpu) { -if (!cpumask_test_cpu (cpu,mm_cpumask (mm)) - &&per_cpu (xen_current_cr3, cpu) !=__pa (mm->pgd)) +if (per_cpu (xen_current_cr3, cpu) !=__pa (mm->pgd))continue ;smp_call_function_single (cpu, drop_mm_ref_this_cpu, mm,1 ); }return ; } -cpumask_copy (mask,mm_cpumask (mm));/* * It's possible that a vcpu may have a stale reference to our
@@ -1021,6 +1019,7 @@ static void xen_drop_mm_ref(struct mm_struct *mm)* look at its actual current cr3 value, and force it to flush * if needed. */ +cpumask_clear (mask);for_each_online_cpu (cpu) {if (per_cpu (xen_current_cr3, cpu) ==__pa (mm->pgd))cpumask_set_cpu (cpu, mask);
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Rework lazy TLB mode and TLB freshness tracking
x86/mm: Rework lazy TLB mode and TLB freshness tracking
Commit-ID: 94b1b03b519b81c494900cb112aa00ed205cc2d9 Gitweb: http://git.kernel.org/tip/94b1b03b519b81c494900cb112aa00ed205cc2d9 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:17 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 5 Jul 2017 10:52:57 +0200
x86/mm: Rework lazy TLB mode and TLB freshness tracking
x86's lazy TLB mode used to be fairly weak -- it would switch to init_mm the first time it tried to flush a lazy TLB. This meant an unnecessary CR3 write and, if the flush was remote, an unnecessary IPI.
Rewrite it entirely. When we enter lazy mode, we simply remove the CPU from mm_cpumask. This means that we need a way to figure out whether we've missed a flush when we switch back out of lazy mode. I use the tlb_gen machinery to track whether a context is up to date.
Note to reviewers: this patch, my itself, looks a bit odd. I'm using an array of length 1 containing (ctx_id, tlb_gen) rather than just storing tlb_gen, and making it at array isn't necessary yet. I'm doing this because the next few patches add PCID support, and, with PCID, we need ctx_id, and the array will end up with a length greater than 1. Making it an array now means that there will be less churn and therefore less stress on your eyeballs.
NB: This is dubious but, AFAICT, still correct on Xen and UV. xen_exit_mmap() uses mm_cpumask() for nefarious purposes and this patch changes the way that mm_cpumask() works. This should be okay, since Xen *also* iterates all online CPUs to find all the CPUs it needs to twiddle.
The UV tlbflush code is rather dated and should be changed.
Here are some benchmark results, done on a Skylake laptop at 2.3 GHz (turbo off, intel_pstate requesting max performance) under KVM with the guest using idle=poll (to avoid artifacts when bouncing between CPUs). I haven't done any real statistics here -- I just ran them in a loop and picked the fastest results that didn't look like outliers. Unpatched means commit a4eb8b993554, so all the bookkeeping overhead is gone.
MADV_DONTNEED; touch the page; switch CPUs using sched_setaffinity. In an unpatched kernel, MADV_DONTNEED will send an IPI to the previous CPU. This is intended to be a nearly worst-case test.
patched: 13.4µs
unpatched: 21.6µs
Vitaly's pthread_mmap microbenchmark with 8 threads (on four cores), nrounds = 100, 256M data
patched: 1.1 seconds or so
unpatched: 1.9 seconds or so
The sleepup on Vitaly's test appearss to be because it spends a lot of time blocked on mmap_sem, and this patch avoids sending IPIs to blocked CPUs.
--- arch/x86/include/asm/mmu_context.h | 6 +- arch/x86/include/asm/tlbflush.h | 4 - arch/x86/mm/init.c | 1 - arch/x86/mm/tlb.c | 197 ++++++++++++++++++++++--------------- arch/x86/xen/mmu_pv.c | 5 +- 5 files changed, 124 insertions(+), 89 deletions(-)
arch/x86/include/asm/mmu_context.h +4 -2
@@ -128,8 +128,10 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next)static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) { -if (this_cpu_read (cpu_tlbstate.state) ==TLBSTATE_OK ) -this_cpu_write (cpu_tlbstate.state,TLBSTATE_LAZY ); +int cpu =smp_processor_id (); + +if (cpumask_test_cpu (cpu,mm_cpumask (mm))) +cpumask_clear_cpu (cpu,mm_cpumask (mm)); } static inline int init_new_context(struct task_struct *tsk,
arch/x86/include/asm/tlbflush.h +0 -4
@@ -95,7 +95,6 @@ struct tlb_state {* mode even if we've already switched back to swapper_pg_dir. */ struct mm_struct *loaded_mm; -int state;/* * Access to this CR4 shadow and to H/W CR4 is protected by
@@ -310,9 +309,6 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a)void native_flush_tlb_others (const struct cpumask *cpumask,const struct flush_tlb_info *info); -#define TLBSTATE_OK 1 -#define TLBSTATE_LAZY 2 -static inline void arch_tlbbatch_add_mm (struct arch_tlbflush_unmap_batch *batch,struct mm_struct *mm) {
arch/x86/mm/init.c +0 -1
@@ -812,7 +812,6 @@ void __init zone_sizes_init(void) DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate) = { .loaded_mm = &init_mm, - .state =0 , .cr4 = ~0UL ,/* fail hard if we screw up cr4 shadow initialization */ };EXPORT_SYMBOL_GPL (cpu_tlbstate);
arch/x86/mm/tlb.c +118 -79
265 lines not on this page · raw message
arch/x86/xen/mmu_pv.c +2 -3
@@ -1005,14 +1005,12 @@ static void xen_drop_mm_ref(struct mm_struct *mm)/* Get the "official" set of cpus referring to our pagetable. */ if (!alloc_cpumask_var (&mask,GFP_ATOMIC )) {for_each_online_cpu (cpu) { -if (!cpumask_test_cpu (cpu,mm_cpumask (mm)) - &&per_cpu (xen_current_cr3, cpu) !=__pa (mm->pgd)) +if (per_cpu (xen_current_cr3, cpu) !=__pa (mm->pgd))continue ;smp_call_function_single (cpu, drop_mm_ref_this_cpu, mm,1 ); }return ; } -cpumask_copy (mask,mm_cpumask (mm));/* * It's possible that a vcpu may have a stale reference to our
@@ -1021,6 +1019,7 @@ static void xen_drop_mm_ref(struct mm_struct *mm)* look at its actual current cr3 value, and force it to flush * if needed. */ +cpumask_clear (mask);for_each_online_cpu (cpu) {if (per_cpu (xen_current_cr3, cpu) ==__pa (mm->pgd))cpumask_set_cpu (cpu, mask);
Andy Lutomirski
maintainerPATCH v4 6/10 x86/mm: Stop calling leave_mm() in idle code
In reply to Andy LutomirskiNow that lazy TLB suppresses all flush IPIs (as opposed to all but the first), there's no need to leave_mm() when going idle.
This means we can get rid of the rcuidle hack in switch_mm_irqs_off() and we can unexport leave_mm().
This also removes acpi_unlazy_tlb() from the x86 and ia64 headers, since it has no callers any more.
--- arch/ia64/include/asm/acpi.h | 2 -- arch/x86/include/asm/acpi.h | 2 -- arch/x86/mm/tlb.c | 20 +++----------------- drivers/acpi/processor_idle.c | 2 -- drivers/idle/intel_idle.c | 9 ++++----- 5 files changed, 7 insertions(+), 28 deletions(-)
arch/ia64/include/asm/acpi.h +0 -2
@@ -112,8 +112,6 @@ static inline void arch_acpi_set_pdc_bits(u32 *buf) buf[2 ] |=ACPI_PDC_EST_CAPABILITY_SMP ; } -#define acpi_unlazy_tlb (x) -#ifdef CONFIG_ACPI_NUMA extern cpumask_t early_cpu_possible_map;#define for_each_possible_early_cpu (cpu) \
arch/x86/include/asm/acpi.h +0 -2
@@ -150,8 +150,6 @@ static inline void disable_acpi(void) { }extern int x86_acpi_numa_init (void );#endif /* CONFIG_ACPI_NUMA */ -#define acpi_unlazy_tlb (x) leave_mm(x) -#ifdef CONFIG_ACPI_APEI static inline pgprot_t arch_apei_get_mem_attribute (phys_addr_t addr) {
arch/x86/mm/tlb.c +3 -17
@@ -50,7 +50,6 @@ void leave_mm(int cpu)switch_mm (NULL , &init_mm,NULL ); } -EXPORT_SYMBOL_GPL (leave_mm);void switch_mm (struct mm_struct *prev,struct mm_struct *next,struct task_struct *tsk)
@@ -117,15 +116,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,this_cpu_write (cpu_tlbstate.ctxs[0 ].tlb_gen, next_tlb_gen);write_cr3 (__pa (next->pgd)); - -/* -* This gets called via leave_mm() in the idle path -* where RCU functions differently. Tracing normally -* uses RCU, so we have to call the tracepoint -* specially here. -*/ -trace_tlb_flush_rcuidle (TLB_FLUSH_ON_TASK_SWITCH , -TLB_FLUSH_ALL ); +trace_tlb_flush (TLB_FLUSH_ON_TASK_SWITCH , +TLB_FLUSH_ALL ); }/*
@@ -167,13 +159,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,this_cpu_write (cpu_tlbstate.loaded_mm, next);write_cr3 (__pa (next->pgd)); -/* -* This gets called via leave_mm() in the idle path where RCU -* functions differently. Tracing normally uses RCU, so we -* have to call the tracepoint specially here. -*/ -trace_tlb_flush_rcuidle (TLB_FLUSH_ON_TASK_SWITCH , -TLB_FLUSH_ALL ); +trace_tlb_flush (TLB_FLUSH_ON_TASK_SWITCH ,TLB_FLUSH_ALL ); }load_mm_cr4 (next);
drivers/acpi/processor_idle.c +0 -2
@@ -708,8 +708,6 @@ static DEFINE_RAW_SPINLOCK(c3_lock);static void acpi_idle_enter_bm (struct acpi_processor *pr,struct acpi_processor_cx *cx,bool timer_bc) { -acpi_unlazy_tlb (smp_processor_id ()); -/* * Must be done before busmaster disable as we might need to * access HPET !
drivers/idle/intel_idle.c +4 -5
@@ -912,16 +912,15 @@ static __cpuidle int intel_idle(struct cpuidle_device *dev,struct cpuidle_state *state = &drv->states[index];unsigned long eax =flg2MWAIT (state->flags);unsigned int cstate; -int cpu =smp_processor_id (); cstate = (((eax) >>MWAIT_SUBSTATE_SIZE ) &MWAIT_CSTATE_MASK ) +1 ;/* -* leave_mm() to avoid costly and often unnecessary wakeups -* for flushing the user TLB's associated with the active mm. +* NB: if CPUIDLE_FLAG_TLB_FLUSHED is set, this idle transition +* will probably flush the TLB. It's not guaranteed to flush +* the TLB, though, so it's not clear that we can do anything +* useful with this knowledge. */ -if (state->flags &CPUIDLE_FLAG_TLB_FLUSHED ) -leave_mm (cpu);if (!(lapic_timer_reliable_states & (1 << (cstate))))tick_broadcast_enter ();
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Stop calling leave_mm() in idle code
x86/mm: Stop calling leave_mm() in idle code
Commit-ID: 43858b4f25cf0adc5c2ca9cf5ce5fdf2532941e5 Gitweb: http://git.kernel.org/tip/43858b4f25cf0adc5c2ca9cf5ce5fdf2532941e5 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:18 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 5 Jul 2017 10:52:57 +0200
x86/mm: Stop calling leave_mm() in idle code
Now that lazy TLB suppresses all flush IPIs (as opposed to all but the first), there's no need to leave_mm() when going idle.
This means we can get rid of the rcuidle hack in switch_mm_irqs_off() and we can unexport leave_mm().
This also removes acpi_unlazy_tlb() from the x86 and ia64 headers, since it has no callers any more.
--- arch/ia64/include/asm/acpi.h | 2 -- arch/x86/include/asm/acpi.h | 2 -- arch/x86/mm/tlb.c | 20 +++----------------- drivers/acpi/processor_idle.c | 2 -- drivers/idle/intel_idle.c | 9 ++++----- 5 files changed, 7 insertions(+), 28 deletions(-)
arch/ia64/include/asm/acpi.h +0 -2
@@ -112,8 +112,6 @@ static inline void arch_acpi_set_pdc_bits(u32 *buf) buf[2 ] |=ACPI_PDC_EST_CAPABILITY_SMP ; } -#define acpi_unlazy_tlb (x) -#ifdef CONFIG_ACPI_NUMA extern cpumask_t early_cpu_possible_map;#define for_each_possible_early_cpu (cpu) \
arch/x86/include/asm/acpi.h +0 -2
@@ -150,8 +150,6 @@ static inline void disable_acpi(void) { }extern int x86_acpi_numa_init (void );#endif /* CONFIG_ACPI_NUMA */ -#define acpi_unlazy_tlb (x) leave_mm(x) -#ifdef CONFIG_ACPI_APEI static inline pgprot_t arch_apei_get_mem_attribute (phys_addr_t addr) {
arch/x86/mm/tlb.c +3 -17
@@ -50,7 +50,6 @@ void leave_mm(int cpu)switch_mm (NULL , &init_mm,NULL ); } -EXPORT_SYMBOL_GPL (leave_mm);void switch_mm (struct mm_struct *prev,struct mm_struct *next,struct task_struct *tsk)
@@ -117,15 +116,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,this_cpu_write (cpu_tlbstate.ctxs[0 ].tlb_gen, next_tlb_gen);write_cr3 (__pa (next->pgd)); - -/* -* This gets called via leave_mm() in the idle path -* where RCU functions differently. Tracing normally -* uses RCU, so we have to call the tracepoint -* specially here. -*/ -trace_tlb_flush_rcuidle (TLB_FLUSH_ON_TASK_SWITCH , -TLB_FLUSH_ALL ); +trace_tlb_flush (TLB_FLUSH_ON_TASK_SWITCH , +TLB_FLUSH_ALL ); }/*
@@ -167,13 +159,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,this_cpu_write (cpu_tlbstate.loaded_mm, next);write_cr3 (__pa (next->pgd)); -/* -* This gets called via leave_mm() in the idle path where RCU -* functions differently. Tracing normally uses RCU, so we -* have to call the tracepoint specially here. -*/ -trace_tlb_flush_rcuidle (TLB_FLUSH_ON_TASK_SWITCH , -TLB_FLUSH_ALL ); +trace_tlb_flush (TLB_FLUSH_ON_TASK_SWITCH ,TLB_FLUSH_ALL ); }load_mm_cr4 (next);
drivers/acpi/processor_idle.c +0 -2
@@ -708,8 +708,6 @@ static DEFINE_RAW_SPINLOCK(c3_lock);static void acpi_idle_enter_bm (struct acpi_processor *pr,struct acpi_processor_cx *cx,bool timer_bc) { -acpi_unlazy_tlb (smp_processor_id ()); -/* * Must be done before busmaster disable as we might need to * access HPET !
drivers/idle/intel_idle.c +4 -5
@@ -912,16 +912,15 @@ static __cpuidle int intel_idle(struct cpuidle_device *dev,struct cpuidle_state *state = &drv->states[index];unsigned long eax =flg2MWAIT (state->flags);unsigned int cstate; -int cpu =smp_processor_id (); cstate = (((eax) >>MWAIT_SUBSTATE_SIZE ) &MWAIT_CSTATE_MASK ) +1 ;/* -* leave_mm() to avoid costly and often unnecessary wakeups -* for flushing the user TLB's associated with the active mm. +* NB: if CPUIDLE_FLAG_TLB_FLUSHED is set, this idle transition +* will probably flush the TLB. It's not guaranteed to flush +* the TLB, though, so it's not clear that we can do anything +* useful with this knowledge. */ -if (state->flags &CPUIDLE_FLAG_TLB_FLUSHED ) -leave_mm (cpu);if (!(lapic_timer_reliable_states & (1 << (cstate))))tick_broadcast_enter ();
Andy Lutomirski
maintainerPATCH v4 7/10 x86/mm: Disable PCID on 32-bit kernels
In reply to Andy Lutomirski32-bit kernels on new hardware will see PCID in CPUID, but PCID can only be used in 64-bit mode. Rather than making all PCID code conditional, just disable the feature on 32-bit builds.
--- arch/x86/include/asm/disabled-features.h | 4 +++- arch/x86/kernel/cpu/bugs.c | 8 ++++++++ 2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/disabled-features.h +3 -1
@@ -21,11 +21,13 @@# define DISABLE_K6_MTRR (1<<(X86_FEATURE_K6_MTRR & 31))# define DISABLE_CYRIX_ARR (1<<(X86_FEATURE_CYRIX_ARR & 31))# define DISABLE_CENTAUR_MCR (1<<(X86_FEATURE_CENTAUR_MCR & 31)) +# define DISABLE_PCID 0#else # define DISABLE_VME 0# define DISABLE_K6_MTRR 0# define DISABLE_CYRIX_ARR 0# define DISABLE_CENTAUR_MCR 0 +# define DISABLE_PCID (1<<(X86_FEATURE_PCID & 31))#endif /* CONFIG_X86_64 */ #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
@@ -49,7 +51,7 @@#define DISABLED_MASK1 0#define DISABLED_MASK2 0#define DISABLED_MASK3 (DISABLE_CYRIX_ARR|DISABLE_CENTAUR_MCR|DISABLE_K6_MTRR) -#define DISABLED_MASK4 0 +#define DISABLED_MASK4 (DISABLE_PCID)#define DISABLED_MASK5 0#define DISABLED_MASK6 0#define DISABLED_MASK7 0
arch/x86/kernel/cpu/bugs.c +8 -0
@@ -21,6 +21,14 @@void __init check_bugs (void ) { +#ifdef CONFIG_X86_32 +/* +* Regardless of whether PCID is enumerated, the SDM says +* that it can't be enabled in 32-bit mode. +*/ +setup_clear_cpu_cap (X86_FEATURE_PCID ); +#endif +identify_boot_cpu ();if (!IS_ENABLED (CONFIG_SMP )) {
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Disable PCID on 32-bit kernels
x86/mm: Disable PCID on 32-bit kernels
Commit-ID: cba4671af7550e008f7a7835f06df0763825bf3e Gitweb: http://git.kernel.org/tip/cba4671af7550e008f7a7835f06df0763825bf3e Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:19 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 5 Jul 2017 10:52:57 +0200
x86/mm: Disable PCID on 32-bit kernels
32-bit kernels on new hardware will see PCID in CPUID, but PCID can only be used in 64-bit mode. Rather than making all PCID code conditional, just disable the feature on 32-bit builds.
--- arch/x86/include/asm/disabled-features.h | 4 +++- arch/x86/kernel/cpu/bugs.c | 8 ++++++++ 2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/disabled-features.h +3 -1
@@ -21,11 +21,13 @@# define DISABLE_K6_MTRR (1<<(X86_FEATURE_K6_MTRR & 31))# define DISABLE_CYRIX_ARR (1<<(X86_FEATURE_CYRIX_ARR & 31))# define DISABLE_CENTAUR_MCR (1<<(X86_FEATURE_CENTAUR_MCR & 31)) +# define DISABLE_PCID 0#else # define DISABLE_VME 0# define DISABLE_K6_MTRR 0# define DISABLE_CYRIX_ARR 0# define DISABLE_CENTAUR_MCR 0 +# define DISABLE_PCID (1<<(X86_FEATURE_PCID & 31))#endif /* CONFIG_X86_64 */ #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
@@ -49,7 +51,7 @@#define DISABLED_MASK1 0#define DISABLED_MASK2 0#define DISABLED_MASK3 (DISABLE_CYRIX_ARR|DISABLE_CENTAUR_MCR|DISABLE_K6_MTRR) -#define DISABLED_MASK4 0 +#define DISABLED_MASK4 (DISABLE_PCID)#define DISABLED_MASK5 0#define DISABLED_MASK6 0#define DISABLED_MASK7 0
arch/x86/kernel/cpu/bugs.c +8 -0
@@ -21,6 +21,14 @@void __init check_bugs (void ) { +#ifdef CONFIG_X86_32 +/* +* Regardless of whether PCID is enumerated, the SDM says +* that it can't be enabled in 32-bit mode. +*/ +setup_clear_cpu_cap (X86_FEATURE_PCID ); +#endif +identify_boot_cpu ();if (!IS_ENABLED (CONFIG_SMP )) {
Andy Lutomirski
maintainerPATCH v4 8/10 x86/mm: Add nopcid to turn off PCID
In reply to Andy LutomirskiThe parameter is only present on x86_64 systems to save a few bytes, as PCID is always disabled on x86_32.
--- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +2 -0
@@ -2648,6 +2648,8 @@ nopat [X86] Disable PAT (page attribute table extension of pagetables) support. + nopcid [X86-64] Disable the PCID cpu feature. + norandmaps Don't use address space randomization. Equivalent to echo 0 > /proc/sys/kernel/randomize_va_space
arch/x86/kernel/cpu/common.c +18 -0
@@ -168,6 +168,24 @@ static int __init x86_mpx_setup(char *s) }__setup ("nompx" , x86_mpx_setup); +#ifdef CONFIG_X86_64 +static int __init x86_pcid_setup (char *s) +{ +/* require an exact match without trailing characters */ +if (strlen (s)) +return 0 ; + +/* do not emit a message if the feature is not present */ +if (!boot_cpu_has (X86_FEATURE_PCID )) +return 1 ; + +setup_clear_cpu_cap (X86_FEATURE_PCID ); +pr_info ("nopcid: PCID feature disabled\n" ); +return 1 ; +} +__setup ("nopcid" , x86_pcid_setup); +#endif +static int __init x86_noinvpcid_setup (char *s) {/* noinvpcid doesn't accept parameters */
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Add the 'nopcid' boot option to turn off PCID
x86/mm: Add the 'nopcid' boot option to turn off PCID
Commit-ID: 0790c9aad84901ca1bdc14746175549c8b5da215 Gitweb: http://git.kernel.org/tip/0790c9aad84901ca1bdc14746175549c8b5da215 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:20 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 5 Jul 2017 10:52:57 +0200
x86/mm: Add the 'nopcid' boot option to turn off PCID
The parameter is only present on x86_64 systems to save a few bytes, as PCID is always disabled on x86_32.
--- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/common.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +2 -0
@@ -2657,6 +2657,8 @@ nopat [X86] Disable PAT (page attribute table extension of pagetables) support. + nopcid [X86-64] Disable the PCID cpu feature. + norandmaps Don't use address space randomization. Equivalent to echo 0 > /proc/sys/kernel/randomize_va_space
arch/x86/kernel/cpu/common.c +18 -0
@@ -168,6 +168,24 @@ static int __init x86_mpx_setup(char *s) }__setup ("nompx" , x86_mpx_setup); +#ifdef CONFIG_X86_64 +static int __init x86_pcid_setup (char *s) +{ +/* require an exact match without trailing characters */ +if (strlen (s)) +return 0 ; + +/* do not emit a message if the feature is not present */ +if (!boot_cpu_has (X86_FEATURE_PCID )) +return 1 ; + +setup_clear_cpu_cap (X86_FEATURE_PCID ); +pr_info ("nopcid: PCID feature disabled\n" ); +return 1 ; +} +__setup ("nopcid" , x86_pcid_setup); +#endif +static int __init x86_noinvpcid_setup (char *s) {/* noinvpcid doesn't accept parameters */
Andy Lutomirski
maintainerPATCH v4 9/10 x86/mm: Enable CR4.PCIDE on supported systems
In reply to Andy LutomirskiWe can use PCID if the CPU has PCID and PGE and we're not on Xen.
By itself, this has no effect. The next patch will start using PCID.
--- arch/x86/include/asm/tlbflush.h | 8 ++++++++ arch/x86/kernel/cpu/common.c | 22 ++++++++++++++++++++++ arch/x86/xen/enlighten_pv.c | 6 ++++++ 3 files changed, 36 insertions(+)
arch/x86/include/asm/tlbflush.h +8 -0
@@ -243,6 +243,14 @@ static inline void __flush_tlb_all(void)__flush_tlb_global (); else__flush_tlb (); + +/* +* Note: if we somehow had PCID but not PGE, then this wouldn't work -- +* we'd end up flushing kernel translations for the current ASID but +* we might fail to flush kernel translations for other cached ASIDs. +* +* To avoid this issue, we force PCID off if PGE is off. +*/ }static inline void __flush_tlb_one (unsigned long addr)
arch/x86/kernel/cpu/common.c +22 -0
@@ -329,6 +329,25 @@ static __always_inline void setup_smap(struct cpuinfo_x86 *c) } } +static void setup_pcid (struct cpuinfo_x86 *c) +{ +if (cpu_has (c,X86_FEATURE_PCID )) { +if (cpu_has (c,X86_FEATURE_PGE )) { +cr4_set_bits (X86_CR4_PCIDE ); + }else { +/* +* flush_tlb_all(), as currently implemented, won't +* work if PCID is on but PGE is not. Since that +* combination doesn't exist on real hardware, there's +* no reason to try to fully support it, but it's +* polite to avoid corrupting data if we're on +* an improperly configured VM. +*/ +clear_cpu_cap (c,X86_FEATURE_PCID ); + } + } +} +/* * Protection Keys are not available in 32-bit mode. */
@@ -1143,6 +1162,9 @@ static void identify_cpu(struct cpuinfo_x86 *c)setup_smep (c);setup_smap (c); +/* Set up PCID */ +setup_pcid (c); +/* * The vendor-specific functions might have changed features. * Now we do "generic changes."
arch/x86/xen/enlighten_pv.c +6 -0
@@ -295,6 +295,12 @@ static void __init xen_init_capabilities(void)setup_clear_cpu_cap (X86_FEATURE_ACC );setup_clear_cpu_cap (X86_FEATURE_X2APIC ); +/* +* Xen PV would need some work to support PCID: CR3 handling as well +* as xen_flush_tlb_others() would need updating. +*/ +setup_clear_cpu_cap (X86_FEATURE_PCID ); +if (!xen_initial_domain ())setup_clear_cpu_cap (X86_FEATURE_ACPI );
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Andy Lutomirski
merge bot[tip:x86/mm] x86/mm: Enable CR4.PCIDE on supported systems
x86/mm: Enable CR4.PCIDE on supported systems
Commit-ID: 660da7c9228f685b2ebe664f9fd69aaddcc420b5 Gitweb: http://git.kernel.org/tip/660da7c9228f685b2ebe664f9fd69aaddcc420b5 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 29 Jun 2017 08:53:21 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 5 Jul 2017 10:52:58 +0200
x86/mm: Enable CR4.PCIDE on supported systems
We can use PCID if the CPU has PCID and PGE and we're not on Xen.
By itself, this has no effect. A followup patch will start using PCID.
--- arch/x86/include/asm/tlbflush.h | 8 ++++++++ arch/x86/kernel/cpu/common.c | 22 ++++++++++++++++++++++ arch/x86/xen/enlighten_pv.c | 6 ++++++ 3 files changed, 36 insertions(+)
arch/x86/include/asm/tlbflush.h +8 -0
@@ -243,6 +243,14 @@ static inline void __flush_tlb_all(void)__flush_tlb_global (); else__flush_tlb (); + +/* +* Note: if we somehow had PCID but not PGE, then this wouldn't work -- +* we'd end up flushing kernel translations for the current ASID but +* we might fail to flush kernel translations for other cached ASIDs. +* +* To avoid this issue, we force PCID off if PGE is off. +*/ }static inline void __flush_tlb_one (unsigned long addr)
arch/x86/kernel/cpu/common.c +22 -0
@@ -329,6 +329,25 @@ static __always_inline void setup_smap(struct cpuinfo_x86 *c) } } +static void setup_pcid (struct cpuinfo_x86 *c) +{ +if (cpu_has (c,X86_FEATURE_PCID )) { +if (cpu_has (c,X86_FEATURE_PGE )) { +cr4_set_bits (X86_CR4_PCIDE ); + }else { +/* +* flush_tlb_all(), as currently implemented, won't +* work if PCID is on but PGE is not. Since that +* combination doesn't exist on real hardware, there's +* no reason to try to fully support it, but it's +* polite to avoid corrupting data if we're on +* an improperly configured VM. +*/ +clear_cpu_cap (c,X86_FEATURE_PCID ); + } + } +} +/* * Protection Keys are not available in 32-bit mode. */
@@ -1143,6 +1162,9 @@ static void identify_cpu(struct cpuinfo_x86 *c)setup_smep (c);setup_smap (c); +/* Set up PCID */ +setup_pcid (c); +/* * The vendor-specific functions might have changed features. * Now we do "generic changes."
arch/x86/xen/enlighten_pv.c +6 -0
@@ -295,6 +295,12 @@ static void __init xen_init_capabilities(void)setup_clear_cpu_cap (X86_FEATURE_ACC );setup_clear_cpu_cap (X86_FEATURE_X2APIC ); +/* +* Xen PV would need some work to support PCID: CR3 handling as well +* as xen_flush_tlb_others() would need updating. +*/ +setup_clear_cpu_cap (X86_FEATURE_PCID ); +if (!xen_initial_domain ())setup_clear_cpu_cap (X86_FEATURE_ACPI );
Andy Lutomirski
maintainerPATCH v4 10/10 x86/mm: Try to preserve old TLB entries using PCID
In reply to Andy LutomirskiPCID is a "process context ID" -- it's what other architectures call an address space ID. Every non-global TLB entry is tagged with a PCID, only TLB entries that match the currently selected PCID are used, and we can switch PGDs without flushing the TLB. x86's PCID is 12 bits.
This is an unorthodox approach to using PCID. x86's PCID is far too short to uniquely identify a process, and we can't even really uniquely identify a running process because there are monster systems with over 4096 CPUs. To make matters worse, past attempts to use all 12 PCID bits have resulted in slowdowns instead of speedups.
This patch uses PCID differently. We use a PCID to identify a recently-used mm on a per-cpu basis. An mm has no fixed PCID binding at all; instead, we give it a fresh PCID each time it's loaded except in cases where we want to preserve the TLB, in which case we reuse a recent value.
Here are some benchmark results, done on a Skylake laptop at 2.3 GHz (turbo off, intel_pstate requesting max performance) under KVM with the guest using idle=poll (to avoid artifacts when bouncing between CPUs). I haven't done any real statistics here -- I just ran them in a loop and picked the fastest results that didn't look like outliers. Unpatched means commit a4eb8b993554, so all the bookkeeping overhead is gone.
ping-pong between two mms on the same CPU using eventfd:
patched: 1.22Aus
patched, nopcid: 1.33Aus
unpatched: 1.34Aus
Same ping-pong, but now touch 512 pages (all zero-page to minimize cache misses) each iteration. dTLB misses are measured by dtlb_load_misses.miss_causes_a_walk:
patched: 1.8Aus 11M dTLB misses
patched, nopcid: 6.2Aus, 207M dTLB misses
unpatched: 6.1Aus, 190M dTLB misses
--- arch/x86/include/asm/mmu_context.h | 3 ++ arch/x86/include/asm/processor-flags.h | 2 + arch/x86/include/asm/tlbflush.h | 18 +++++++- arch/x86/mm/init.c | 1 + arch/x86/mm/tlb.c | 82 +++++++++++++++++++++++++++------- 5 files changed, 88 insertions(+), 18 deletions(-)
arch/x86/include/asm/mmu_context.h +3 -0
@@ -300,6 +300,9 @@ static inline unsigned long __get_current_cr3_fast(void) {unsigned long cr3 =__pa (this_cpu_read (cpu_tlbstate.loaded_mm)->pgd); +if (static_cpu_has (X86_FEATURE_PCID )) + cr3 |=this_cpu_read (cpu_tlbstate.loaded_mm_asid); +/* For now, be very restrictive about when this can be called. */ VM_WARN_ON (in_nmi () || !in_atomic ());
arch/x86/include/asm/processor-flags.h +2 -0
@@ -35,6 +35,7 @@/* Mask off the address space ID bits. */ #define CR3_ADDR_MASK 0x7FFFFFFFFFFFF000ull#define CR3_PCID_MASK 0xFFFull +#define CR3_NOFLUSH (1UL << 63)#else /* * CR3_ADDR_MASK needs at least bits 31:5 set on PAE systems, and we save
@@ -42,6 +43,7 @@*/ #define CR3_ADDR_MASK 0xFFFFFFFFull#define CR3_PCID_MASK 0ull +#define CR3_NOFLUSH 0#endif #endif /* _ASM_X86_PROCESSOR_FLAGS_H */
arch/x86/include/asm/tlbflush.h +16 -2
@@ -82,6 +82,12 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm)#define __flush_tlb_single (addr) __native_flush_tlb_single(addr)#endif +/* +* 6 because 6 should be plenty and struct tlb_state will fit in +* two cache lines. +*/ +#define TLB_NR_DYN_ASIDS 6 +struct tlb_context {u64 ctx_id;u64 tlb_gen;
@@ -95,6 +101,8 @@ struct tlb_state {* mode even if we've already switched back to swapper_pg_dir. */ struct mm_struct *loaded_mm; +u16 loaded_mm_asid; +u16 next_asid;/* * Access to this CR4 shadow and to H/W CR4 is protected by
@@ -104,7 +112,8 @@ struct tlb_state {/* * This is a list of all contexts that might exist in the TLB. -* Since we don't yet use PCID, there is only one context. +* There is one per ASID that we use, and the ASID (what the +* CPU calls PCID) is the index into ctxts. * * For each context, ctx_id indicates which mm the TLB's user * entries came from. As an invariant, the TLB will never
@@ -114,8 +123,13 @@ struct tlb_state {* To be clear, this means that it's legal for the TLB code to * flush the TLB without updating tlb_gen. This can happen * (for now, at least) due to paravirt remote flushes. +* +* NB: context 0 is a bit special, since it's also used by +* various bits of init code. This is fine -- code that +* isn't aware of PCID will end up harmlessly flushing +* context 0. */ -struct tlb_context ctxs[1 ]; +struct tlb_context ctxs[TLB_NR_DYN_ASIDS ]; }; DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate);
arch/x86/mm/init.c +1 -0
@@ -812,6 +812,7 @@ void __init zone_sizes_init(void) DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate) = { .loaded_mm = &init_mm, + .next_asid =1 , .cr4 = ~0UL ,/* fail hard if we screw up cr4 shadow initialization */ };EXPORT_SYMBOL_GPL (cpu_tlbstate);
arch/x86/mm/tlb.c +66 -16
150 lines not on this page · raw message
Signature · 7 lines
2.9.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
Quoted 12 lines · Andy Lutomirski
ping-pong between two mms on the same CPU using eventfd:
patched: 1.22µs
patched, nopcid: 1.33µs
unpatched: 1.34µsSame ping-pong, but now touch 512 pages (all zero-page to minimize cache misses) each iteration. dTLB misses are measured by dtlb_load_misses.miss_causes_a_walk:
patched: 1.8µs 11M dTLB misses
patched, nopcid: 6.2µs, 207M dTLB misses
unpatched: 6.1µs, 190M dTLB misses
Quoted 2 lines
Reviewed-by Nadav Amit <[email protected]>Signed-off-by Andy Lutomirski <[email protected]>
Peter Zijlstra
maintainerRe: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
In reply to Andy Lutomirskiarch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, · line 142
Quoted 4 lines · Andy Lutomirski
arch/x86/mm/tlb.c +0
-0@@ -104,18 +140,20 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* Resume remote flushes and then read tlb_gen. */ cpumask_set_cpu (cpu,mm_cpumask (next));
Barriers should have a comment... what is being ordered here against what?
Quoted 19 lines
arch/x86/mm/tlb.c +5
-0+smp_mb__after_atomic (); next_tlb_gen =atomic64_read (&next->context.tlb_gen); +if (this_cpu_read (cpu_tlbstate.ctxs[prev_asid].tlb_gen) < + next_tlb_gen) {/* * Ideally, we'd have a flush_tlb() variant that * takes the known CR3 value as input. This would * be faster on Xen PV and on hypothetical CPUs * on which INVPCID is fast. */ +this_cpu_write (cpu_tlbstate.ctxs[prev_asid].tlb_gen, next_tlb_gen); +write_cr3 (__pa (next->pgd) | prev_asid);trace_tlb_flush (TLB_FLUSH_ON_TASK_SWITCH ,TLB_FLUSH_ALL ); }
arch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, · line 193
Quoted 4 lines
arch/x86/mm/tlb.c +1
-0@@ -152,14 +190,25 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,* Start remote flushes and then read tlb_gen. */ cpumask_set_cpu (cpu,mm_cpumask (next)); +smp_mb__after_atomic ();
idem
Quoted 11 lines
arch/x86/mm/tlb.c +8
-0next_tlb_gen =atomic64_read (&next->context.tlb_gen); +choose_new_asid (next, next_tlb_gen, &new_asid, &need_flush); +if (need_flush) { +this_cpu_write (cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); +this_cpu_write (cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); +write_cr3 (__pa (next->pgd) | new_asid); +trace_tlb_flush (TLB_FLUSH_ON_TASK_SWITCH , +TLB_FLUSH_ALL ); + }else {
arch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, · line 214
Quoted 10 lines
arch/x86/mm/tlb.c +7
-0+/* The new ASID is already up to date. */ +write_cr3 (__pa (next->pgd) | new_asid |CR3_NOFLUSH ); +trace_tlb_flush (TLB_FLUSH_ON_TASK_SWITCH ,0 ); + } + +this_cpu_write (cpu_tlbstate.loaded_mm, next); +this_cpu_write (cpu_tlbstate.loaded_mm_asid, new_asid); }load_mm_cr4 (next);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andy Lutomirski
maintainerRe: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
Quoted 8 lines · Peter Zijlstra
arch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, · line 142
Quoted 4 lines · Andy Lutomirski
arch/x86/mm/tlb.c +0
-0@@ -104,18 +140,20 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* Resume remote flushes and then read tlb_gen. */ cpumask_set_cpu (cpu,mm_cpumask (next));Barriers should have a comment... what is being ordered here against what?
How's this comment?
/*
* Resume remote flushes and then read tlb_gen. We need to do
* it in this order: any inc_mm_tlb_gen() caller that writes a
* larger tlb_gen than we read here must see our cpu set in
* mm_cpumask() so that it will know to flush us. The barrier
* here synchronizes with inc_mm_tlb_gen().
*/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
Quoted 12 lines · Andy Lutomirski
Quoted 8 lines · Peter Zijlstra
arch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, · line 142
Quoted 4 lines · Andy Lutomirski
arch/x86/mm/tlb.c +0
-0@@ -104,18 +140,20 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* Resume remote flushes and then read tlb_gen. */ cpumask_set_cpu (cpu,mm_cpumask (next));Barriers should have a comment... what is being ordered here against what?
How's this comment?
Quoted 7 lines
/*
* Resume remote flushes and then readtlb_gen. We need to do
* it in this order: anyinc_mm_tlb_gen()caller that writes a
* largertlb_genthan we read here must see our cpu set in
*mm_cpumask()so that it will know to flush us. The barrier
* here synchronizes withinc_mm_tlb_gen().
*/
Slightly confusing, you mean this, right?
cpumask_set_cpu(cpu, mm_cpumask()); inc_mm_tlb_gen(); MB MB next_tlb_gen = atomic64_read(&next->context.tlb_gen); flush_tlb_others(mm_cpumask());
which seems to make sense.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
Quoted 24 lines · Peter Zijlstra
Quoted 12 lines · Andy Lutomirski
Quoted 8 lines · Peter Zijlstra
arch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, · line 142
Quoted 4 lines · Andy Lutomirski
arch/x86/mm/tlb.c +0
-0@@ -104,18 +140,20 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* Resume remote flushes and then read tlb_gen. */ cpumask_set_cpu (cpu,mm_cpumask (next));Barriers should have a comment... what is being ordered here against what?
How's this comment?
Quoted 7 lines
/*
* Resume remote flushes and then readtlb_gen. We need to do
* it in this order: anyinc_mm_tlb_gen()caller that writes a
* largertlb_genthan we read here must see our cpu set in
*mm_cpumask()so that it will know to flush us. The barrier
* here synchronizes withinc_mm_tlb_gen().
*/Slightly confusing, you mean this, right?
Quoted 8 lines
cpumask_set_cpu(cpu, mm_cpumask()); inc_mm_tlb_gen(); MB MB next_tlb_gen = atomic64_read(&next->context.tlb_gen); flush_tlb_others(mm_cpumask());which seems to make sense.
Btw., I'll wait for a v5 iteration before applying this last patch to tip:x86/mm.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andy Lutomirski
maintainerRe: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
Quoted 29 lines · Ingo Molnar
Quoted 26 lines · Peter Zijlstra
Quoted 12 lines · Andy Lutomirski
Quoted 8 lines · Peter Zijlstra
arch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, · line 142
Quoted 4 lines · Andy Lutomirski
arch/x86/mm/tlb.c +0
-0@@ -104,18 +140,20 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* Resume remote flushes and then read tlb_gen. */ cpumask_set_cpu (cpu,mm_cpumask (next));Barriers should have a comment... what is being ordered here against what?
How's this comment?
Quoted 7 lines
/*
* Resume remote flushes and then readtlb_gen. We need to do
* it in this order: anyinc_mm_tlb_gen()caller that writes a
* largertlb_genthan we read here must see our cpu set in
*mm_cpumask()so that it will know to flush us. The barrier
* here synchronizes withinc_mm_tlb_gen().
*/Slightly confusing, you mean this, right?
cpumask_set_cpu(cpu, mm_cpumask()); inc_mm_tlb_gen();
Quoted 8 lines
Quoted 6 lines
MB MB next_tlb_gen = atomic64_read(&next->context.tlb_gen); flush_tlb_others(mm_cpumask());which seems to make sense.
Btw., I'll wait for a v5 iteration before applying this last patch to tip:x86/mm.
I'll send it shortly. I think I'll also add a patch to factor out the flush calls a bit more to prepare for Mel's upcoming fix.
Quoted 4 lines
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
In reply to Andy LutomirskiQuoted 17 lines · Andy Lutomirski
arch/x86/mm/tlb.c +17
-0+static void choose_new_asid (struct mm_struct *next,u64 next_tlb_gen, +u16 *new_asid,bool *need_flush) +{ +u16 asid; + +if (!static_cpu_has (X86_FEATURE_PCID )) { + *new_asid =0 ; + *need_flush = true; +return ; + } + +for (asid =0 ; asid <TLB_NR_DYN_ASIDS ; asid++) { +if (this_cpu_read (cpu_tlbstate.ctxs[asid].ctx_id) != + next->context.ctx_id) +continue ; + + *new_asid = asid;
arch/x86/mm/tlb.c line 59
Quoted 10 lines
arch/x86/mm/tlb.c +10
-0+ *need_flush = (this_cpu_read (cpu_tlbstate.ctxs[asid].tlb_gen) < + next_tlb_gen); +return ; + } + +/* +* We don't currently own an ASID slot on this CPU. +* Allocate a slot. +*/ + *new_asid =this_cpu_add_return (cpu_tlbstate.next_asid,1 ) -1 ;
So this basically RR the ASID slots. Have you tried slightly more complex replacement policies like CLOCK ?
arch/x86/mm/tlb.c line 65
Quoted 6 lines
arch/x86/mm/tlb.c +6
-0+if (*new_asid >=TLB_NR_DYN_ASIDS ) { + *new_asid =0 ; +this_cpu_write (cpu_tlbstate.next_asid,1 ); + } + *need_flush = true; +}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andy Lutomirski
maintainerRe: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · Andy Lutomirski
arch/x86/mm/tlb.c +22
-0+static void choose_new_asid (struct mm_struct *next,u64 next_tlb_gen, +u16 *new_asid,bool *need_flush) +{ +u16 asid; + +if (!static_cpu_has (X86_FEATURE_PCID )) { + *new_asid =0 ; + *need_flush = true; +return ; + } + +for (asid =0 ; asid <TLB_NR_DYN_ASIDS ; asid++) { +if (this_cpu_read (cpu_tlbstate.ctxs[asid].ctx_id) != + next->context.ctx_id) +continue ; + + *new_asid = asid; + *need_flush = (this_cpu_read (cpu_tlbstate.ctxs[asid].tlb_gen) < + next_tlb_gen); +return ; + } +
Quoted 8 lines
arch/x86/mm/tlb.c line 59
Quoted 5 lines
arch/x86/mm/tlb.c +5
-0+/* +* We don't currently own an ASID slot on this CPU. +* Allocate a slot. +*/ + *new_asid =this_cpu_add_return (cpu_tlbstate.next_asid,1 ) -1 ;So this basically RR the ASID slots. Have you tried slightly more complex replacement policies like CLOCK ?
No, mainly because I'm lazy and because CLOCK requires scavenging a bit. (Which we can certainly do, but it will further complicate the code.) It could be worth playing with better replacement algorithms as a followup, though.
I've also considered a slight elaboration of RR in which we make sure not to reuse the most recent ASID slot, which would guarantee that, if we switch from task A to B and back to A, we don't flush on the way back to A. (Currently, if B is not in the cache, there's a 1/6 chance we'll flush on the way back.)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerRe: [PATCH v4 10/10] x86/mm: Try to preserve old TLB entries using PCID
Quoted 38 lines · Andy Lutomirski
Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · Andy Lutomirski
(quoted diff) +22
-0+static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, + u16 *new_asid, bool *need_flush) +{ + u16 asid; + + if (!static_cpu_has(X86_FEATURE_PCID)) { + *new_asid = 0; + *need_flush = true; + return; + } + + for (asid = 0; asid < TLB_NR_DYN_ASIDS; asid++) { + if (this_cpu_read(cpu_tlbstate.ctxs[asid].ctx_id) != + next->context.ctx_id) + continue; + + *new_asid = asid; + *need_flush = (this_cpu_read(cpu_tlbstate.ctxs[asid].tlb_gen) < + next_tlb_gen); + return; + } +Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ /* + * We don't currently own an ASID slot on this CPU. + * Allocate a slot. + */ + *new_asid = this_cpu_add_return(cpu_tlbstate.next_asid, 1) - 1;So this basically RR the ASID slots. Have you tried slightly more complex replacement policies like CLOCK ?
No, mainly because I'm lazy and because CLOCK requires scavenging a bit. (Which we can certainly do, but it will further complicate the code.) It could be worth playing with better replacement algorithms as a followup, though.
Quoted 5 lines
I've also considered a slight elaboration of RR in which we make sure not to reuse the most recent ASID slot, which would guarantee that, if we switch from task A to B and back to A, we don't flush on the way back to A. (Currently, if B is not in the cache, there's a 1/6 chance we'll flush on the way back.)
How's this?
--- arch/x86/include/asm/mmu.h | 2 +- arch/x86/include/asm/mmu_context.h | 2 +- arch/x86/include/asm/tlbflush.h | 2 +- arch/x86/mm/init.c | 2 +- arch/x86/mm/tlb.c | 47 ++++++++++++++++++++++++++------------ 5 files changed, 36 insertions(+), 19 deletions(-)
arch/x86/include/asm/mmu.h +1 -1
@@ -55,7 +55,7 @@ typedef struct {#define INIT_MM_CONTEXT (mm) \ .context = { \ - .ctx_id = 1, \ + .ctx_id = 2, \ }void leave_mm (int cpu);
arch/x86/include/asm/mmu_context.h +1 -1
@@ -137,7 +137,7 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)static inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) { - mm->context.ctx_id =atomic64_inc_return (&last_mm_ctx_id); + mm->context.ctx_id =atomic64_add_return (2 , &last_mm_ctx_id);atomic64_set (&mm->context.tlb_gen,0 );#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
arch/x86/include/asm/tlbflush.h +1 -1
@@ -102,7 +102,7 @@ struct tlb_state {*/ struct mm_struct *loaded_mm;u16 loaded_mm_asid; -u16 next_asid; +u16 last_asid;/* * Access to this CR4 shadow and to H/W CR4 is protected by
arch/x86/mm/init.c +1 -1
@@ -812,7 +812,7 @@ void __init zone_sizes_init(void) DEFINE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate) = { .loaded_mm = &init_mm, - .next_asid =1 , + .last_asid =0 , .cr4 = ~0UL ,/* fail hard if we screw up cr4 shadow initialization */ };EXPORT_SYMBOL_GPL (cpu_tlbstate);
arch/x86/mm/tlb.c +32 -15
85 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 15 lines · Andy Lutomirski
*** Ingo, even if this misses 4.13, please apply the first patch before *** the merge window.
There are three performance benefits here:
1. TLB flushing is slow. (I.e. the flush itself takes a while.) This avoids many of them when switching tasks by using PCID. In a stupid little benchmark I did, it saves about 100ns on my laptop per context switch. I'll try to improve that benchmark.
2. Mms that have been used recently on a given CPU might get to keep their TLB entries alive across process switches with this patch set. TLB fills are pretty fast on modern CPUs, but they're even faster when they don't happen.
Quoted 4 lines
3. Lazy TLB is way better. We used to do two stupid things when we ran kernel threads: we'd send IPIs to flush user contexts on their CPUs and then we'd write to CR3 for no particular reason as an excuse to stop further IPIs. With this patch, we do neither.
Heads up, I'm gonna queue this for a run on SUSE's performance test grid.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 16 lines · Matt Fleming
Quoted 15 lines · Andy Lutomirski
*** Ingo, even if this misses 4.13, please apply the first patch before *** the merge window.
There are three performance benefits here:
1. TLB flushing is slow. (I.e. the flush itself takes a while.) This avoids many of them when switching tasks by using PCID. In a stupid little benchmark I did, it saves about 100ns on my laptop per context switch. I'll try to improve that benchmark.
2. Mms that have been used recently on a given CPU might get to keep their TLB entries alive across process switches with this patch set. TLB fills are pretty fast on modern CPUs, but they're even faster when they don't happen.
Quoted 7 lines
Quoted 4 lines
3. Lazy TLB is way better. We used to do two stupid things when we ran kernel threads: we'd send IPIs to flush user contexts on their CPUs and then we'd write to CR3 for no particular reason as an excuse to stop further IPIs. With this patch, we do neither.
Heads up, I'm gonna queue this for a run on SUSE's performance test grid.
FWIW, I didn't see any change in performance with this series on a PCID-capable machine. On the plus side, I didn't see any weird-looking bugs either.
Are your benchmarks available anywhere?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andy Lutomirski
maintainerQuoted 22 lines · Matt Fleming
Quoted 21 lines · Matt Fleming
Quoted 19 lines · Andy Lutomirski
*** Ingo, even if this misses 4.13, please apply the first patch before *** the merge window.
There are three performance benefits here:
1. TLB flushing is slow. (I.e. the flush itself takes a while.) This avoids many of them when switching tasks by using PCID. In a stupid little benchmark I did, it saves about 100ns on my laptop per context switch. I'll try to improve that benchmark.
2. Mms that have been used recently on a given CPU might get to keep their TLB entries alive across process switches with this patch set. TLB fills are pretty fast on modern CPUs, but they're even faster when they don't happen.
3. Lazy TLB is way better. We used to do two stupid things when we ran kernel threads: we'd send IPIs to flush user contexts on their CPUs and then we'd write to CR3 for no particular reason as an excuse to stop further IPIs. With this patch, we do neither.
Quoted 8 lines
Quoted 2 lines
Heads up, I'm gonna queue this for a run on SUSE's performance test grid.
FWIW, I didn't see any change in performance with this series on a PCID-capable machine. On the plus side, I didn't see any weird-looking bugs either.
Are your benchmarks available anywhere?
https://git.kernel.org/pub/scm/linux/kernel/git/luto/misc-tests.git/
I did:
$ ./context_switch_latency_64 0 process same
and
$ ./madvise_bounce_64 10k [IIRC -- it might have been a different loop count]
--Andy
Quoted 6 lines · Andy Lutomirski
https://git.kernel.org/pub/scm/linux/kernel/git/luto/misc-tests.git/
I did:
$ ./context_switch_latency_64 0 process same
Ah, that's better. I see about a 3.3% speedup with your patches when running the context-switch benchmark.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · Andy Lutomirski
*** Ingo, even if this misses 4.13, please apply the first patch before *** the merge window.
Andy Lutomirski (10):
x86/mm: Don't reenterflush_tlb_func_common()
x86/mm: Delete a big outdated comment about TLB flushing x86/mm: Give each mm TLB flush generation a unique ID x86/mm: Track the TLB'stlb_genand update the flushing algorithm x86/mm: Rework lazy TLB mode and TLB freshness tracking x86/mm: Stop callingleave_mm()in idle code
x86/mm: Disable PCID on 32-bit kernels
x86/mm: Add nopcid to turn off PCID
x86/mm: Enable CR4.PCIDE on supported systems
x86/mm: Try to preserve old TLB entries using PCID
So this series is really nice, and the first two patches are already upstream, and I've just applied all but the final patch to tip:x86/mm (out of caution - I'm a wimp).
That should already offer some improvements and enables the CR4 bit - but doesn't actually use the PCID hardware yet.
I'll push it all out when it passes testing.
If it's all super stable I plan to tempt Linus with a late merge window pull request for all these preparatory patches. (Unless he objects that is. Hint, hint.)
Any objections?
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Ingo Molnar
If it's all super stable I plan to tempt Linus with a late merge window pull request for all these preparatory patches. (Unless he objects that is. Hint, hint.)
I don't think I'll object. At some point the best testing is "lots of users".
TLB issues are a bitch to debug, but at the same time this is clearly a "..but at some point we need to bite the bullet" case. I doubt the series is going to get a lot better.
But yes, please do give it as much testing as humanly possible even without the wider coverage by random people.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 24 lines · Ingo Molnar
Quoted 14 lines · Andy Lutomirski
*** Ingo, even if this misses 4.13, please apply the first patch before *** the merge window.
Andy Lutomirski (10):
x86/mm: Don't reenterflush_tlb_func_common()
x86/mm: Delete a big outdated comment about TLB flushing x86/mm: Give each mm TLB flush generation a unique ID x86/mm: Track the TLB'stlb_genand update the flushing algorithm x86/mm: Rework lazy TLB mode and TLB freshness tracking x86/mm: Stop callingleave_mm()in idle code
x86/mm: Disable PCID on 32-bit kernels
x86/mm: Add nopcid to turn off PCID
x86/mm: Enable CR4.PCIDE on supported systems
x86/mm: Try to preserve old TLB entries using PCIDSo this series is really nice, and the first two patches are already upstream, and I've just applied all but the final patch to tip:x86/mm (out of caution - I'm a wimp).
That should already offer some improvements and enables the CR4 bit - but doesn't actually use the PCID hardware yet.
Quoted 7 lines
I'll push it all out when it passes testing.
If it's all super stable I plan to tempt Linus with a late merge window pull request for all these preparatory patches. (Unless he objects that is. Hint, hint.)
Any objections?
What was the final verdict here? I have a patch ready that should be layered on top which will need a backport. PCID support does not appear to have made it in this merge window so I'm wondering if I should send the patch as-is for placement on top of Andy's work or go with the backport and apply a follow-on patch after Andy's work gets merged.
Thanks.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 30 lines · Mel Gorman
Quoted 29 lines · Ingo Molnar
Quoted 14 lines · Andy Lutomirski
*** Ingo, even if this misses 4.13, please apply the first patch before *** the merge window.
Andy Lutomirski (10):
x86/mm: Don't reenterflush_tlb_func_common()
x86/mm: Delete a big outdated comment about TLB flushing x86/mm: Give each mm TLB flush generation a unique ID x86/mm: Track the TLB'stlb_genand update the flushing algorithm x86/mm: Rework lazy TLB mode and TLB freshness tracking x86/mm: Stop callingleave_mm()in idle code
x86/mm: Disable PCID on 32-bit kernels
x86/mm: Add nopcid to turn off PCID
x86/mm: Enable CR4.PCIDE on supported systems
x86/mm: Try to preserve old TLB entries using PCIDSo this series is really nice, and the first two patches are already upstream, and I've just applied all but the final patch to tip:x86/mm (out of caution - I'm a wimp).
That should already offer some improvements and enables the CR4 bit - but doesn't actually use the PCID hardware yet.
I'll push it all out when it passes testing.
If it's all super stable I plan to tempt Linus with a late merge window pull request for all these preparatory patches. (Unless he objects that is. Hint, hint.)
Quoted 8 lines
Quoted 2 lines
Any objections?
What was the final verdict here? I have a patch ready that should be layered on top which will need a backport. PCID support does not appear to have made it in this merge window so I'm wondering if I should send the patch as-is for placement on top of Andy's work or go with the backport and apply a follow-on patch after Andy's work gets merged.
It's en route for v4.14 - it narrowly missed v4.13.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Ingo Molnar
Quoted 8 lines
Quoted 7 lines
I'll push it all out when it passes testing.
If it's all super stable I plan to tempt Linus with a late merge window pull request for all these preparatory patches. (Unless he objects that is. Hint, hint.)
Any objections?
Quoted 8 lines
Quoted 5 lines
What was the final verdict here? I have a patch ready that should be layered on top which will need a backport. PCID support does not appear to have made it in this merge window so I'm wondering if I should send the patch as-is for placement on top of Andy's work or go with the backport and apply a follow-on patch after Andy's work gets merged.
It's en route for v4.14 - it narrowly missed v4.13.
Grand. I sent out a version that doesn't depend on Andy's work to Andrew as it's purely a mm patch. If that passes inspection then I'll send a follow-on patch to apply on top of the PCID work.
Thanks Ingo.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
A message that is not in the archive (sent off-list or not yet received): CALBSrqDW6pGjHxOmzfnkY_KoNeH6F=pTb8-tJ8r-zbu4prw9HQ@mail.gmail.com
Quoted 49 lines
From: Andy Lutomirski <[email protected]>
Date: Thu, Jun 29, 2017 at 8:53 AM
Subject: [PATCH v4 00/10] PCID and improved laziness
To: [email protected]
Cc: [email protected], Borislav Petkov <[email protected]>, Linus Torvalds <[email protected]>, Andrew Morton <[email protected]>, Mel Gorman <[email protected]>, "[email protected]" <[email protected]>, Nadav Amit <[email protected]>, Rik van Riel <[email protected]>, Dave Hansen <[email protected]>, Arjan van de Ven <[email protected]>, Peter Zijlstra <[email protected]>, Andy Lutomirski <[email protected]>*** Ingo, even if this misses 4.13, please apply the first patch before
*** the merge window.There are three performance benefits here:
1. TLB flushing is slow. (I.e. the flush itself takes a while.) This avoids many of them when switching tasks by using PCID. In a stupid little benchmark I did, it saves about 100ns on my laptop per context switch. I'll try to improve that benchmark.
2. Mms that have been used recently on a given CPU might get to keep their TLB entries alive across process switches with this patch set. TLB fills are pretty fast on modern CPUs, but they're even faster when they don't happen.
3. Lazy TLB is way better. We used to do two stupid things when we ran kernel threads: we'd send IPIs to flush user contexts on their CPUs and then we'd write to CR3 for no particular reason as an
excuse
to stop further IPIs. With this patch, we do neither.This will, in general, perform suboptimally if paravirt TLB flushing is in use (currently just Xen, I think, but Hyper-V is in the works). The code is structured so we could fix it in one of two ways: we could take a spinlock when touching the percpu state so we can update it remotely after a paravirt flush, or we could be more careful about our exactly how we access the state and use cmpxchg16b to do atomic remote updates. (On SMP systems without cmpxchg16b, we'd just skip the optimization entirely.)
This is still missing a final comment-only patch to add overall documentation for the whole thing, but I didn't want to block sending the maybe-hopefully-final code on that.
Quoted 9 lines
This is based on tip:x86/mm. The branch is here if you want to play:
https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/log/?h=x86/pcidIn general, performance seems to exceed my expectations. Here are some performance numbers copy-and-pasted from the changelogs for "Rework lazy TLB mode and TLB freshness" and "Try to preserve old TLB entries using PCID":
Hi Andy,
I have booted Linus's tree (8fac2f96ab86b0e14ec4e42851e21e9b518bdc55) on Skylake server and noticed that it reboots automatically.
When I booted the same kernel with command line arg "nopcid" it works fine. Please find below a snippet of dmesg. Please let me know if you need more info to debug.
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.13.0+ root=UUID=3b8e9636-6e23-4785-a4e2-5954bfe86fd9 ro console=tty0 console=ttyS0,115200n8 [ 0.000000] log_buf_len individual max cpu contribution: 4096 bytes [ 0.000000] log_buf_len total cpu_extra contributions: 258048 bytes [ 0.000000] log_buf_len min size: 262144 bytes [ 0.000000] log_buf_len: 524288 bytes [ 0.000000] early log buf free: 212560(81%) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] ------------[ cut here ]------------ [ 0.000000] WARNING: CPU: 0 PID: 0 at arch/x86/mm/tlb.c:245 initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.13.0+ #5 [ 0.000000] task: ffffffff8960f480 task.stack: ffffffff89600000 [ 0.000000] RIP: 0010:initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.000000] RSP: 0000:ffffffff89603e60 EFLAGS: 00010046 [ 0.000000] RAX: 00000000000406b0 RBX: ffff9f1700a17880 RCX: ffffffff8965de60 [ 0.000000] RDX: 0000008383a0a000 RSI: 000000000960a000 RDI: 0000008383a0a000 [ 0.000000] RBP: ffffffff89603e60 R08: 0000000000000000 R09: 0000ffffffffffff [ 0.000000] R10: ffffffff89603ee8 R11: ffffffff0000ffff R12: 0000000000000000 [ 0.000000] R13: ffff9f1700a0c3e0 R14: ffffffff8960f480 R15: 0000000000000000 [ 0.000000] FS: 0000000000000000(0000) GS:ffff9f1700a00000(0000) knlGS:0000000000000000 [ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.000000] CR2: ffff9fa7bffff000 CR3: 0000008383a0a000 CR4: 00000000000406b0 [ 0.000000] Call Trace: [ 0.000000] cpu_init+0x206/0x4f0 [ 0.000000] ? __set_pte_vaddr+0x1d/0x30 [ 0.000000] trap_init+0x3e/0x50 [ 0.000000] ? trap_init+0x3e/0x50 [ 0.000000] start_kernel+0x1e2/0x3f2 [ 0.000000] x86_64_start_reservations+0x24/0x26 [ 0.000000] x86_64_start_kernel+0x6f/0x72 [ 0.000000] secondary_startup_64+0xa5/0xa5 [ 0.000000] Code: de 00 48 01 f0 48 39 c7 0f 85 92 00 00 00 48 8b 05 ee e2 ee 00 a9 00 00 02 00 74 11 65 48 8b 05 8b 9d 7c 77 a9 00 00 02 00 75 02 <0f> ff 48 81 e2 00 f0 ff ff 0f 22 da 65 66 c7 05 66 9d 7c 77 00 [ 0.000000] ---[ end trace c258f2d278fe031f ]--- [ 0.000000] Memory: 791050356K/803934656K available (9585K kernel code, 1313K rwdata, 3000K rodata, 1176K init, 680K bss, 12884300K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=64, Nodes=4 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU event tracing is enabled. [ 0.000000] NR_IRQS: 4352, nr_irqs: 3928, preallocated irqs: 16 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns [ 0.001000] tsc: Detected 2000.000 MHz processor [ 0.002000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4000.00 BogoMIPS (lpj=2000000) [ 0.003003] pid_max: default: 65536 minimum: 512 [ 0.004030] ACPI: Core revision 20170728 [ 0.091853] ACPI: 6 ACPI AML tables successfully acquired and loaded [ 0.094143] Security Framework initialized [ 0.095004] SELinux: Initializing. [ 0.145612] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) [ 0.170544] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.172699] Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.174441] Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.176351] CPU: Physical Processor ID: 0 [ 0.177003] CPU: Processor Core ID: 0 [ 0.178007] ENERGY_PERF_BIAS: Set to 'normal', was 'performance' [ 0.179003] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) [ 0.180013] mce: CPU supports 20 MCE banks [ 0.181018] CPU0: Thermal monitoring enabled (TM1) [ 0.182057] process: using mwait in idle threads [ 0.183005] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8 [ 0.184003] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4 [ 0.185223] Freeing SMP alternatives memory: 36K [ 0.193912] smpboot: Max logical packages: 8 [ 0.194017] Switched APIC routing to physical flat. [ 0.196496] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.206252] smpboot: CPU0: Intel(R) Xeon(R) Platinum 8164 CPU @ 2.00GHz (family: 0x6, model: 0x55, stepping: 0x4) [ 0.207131] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver. [ 0.208003] ... version: 4 [ 0.209001] ... bit width: 48 [ 0.210001] ... generic registers: 4 [ 0.211001] ... value mask: 0000ffffffffffff [ 0.212001] ... max period: 00007fffffffffff [ 0.213001] ... fixed-purpose events: 3 [ 0.214001] ... event mask: 000000070000000f [ 0.215078] Hierarchical SRCU implementation. [ 0.216867] smp: Bringing up secondary CPUs ... [ 0.217085] x86: Booting SMP configuration: [ 0.218001] .... node #0, CPUs: #1 [ 0.001000] ------------[ cut here ]------------ [ 0.001000] WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:245 initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.001000] Modules linked in: [ 0.001000] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.13.0+ #5 [ 0.001000] task: ffff9f16fa393e40 task.stack: ffffaf0e98afc000 [ 0.001000] RIP: 0010:initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.001000] RSP: 0000:ffffaf0e98affeb0 EFLAGS: 00010046 [ 0.001000] RAX: 00000000000000a0 RBX: ffff9f1700a57880 RCX: ffffffff8965de60 [ 0.001000] RDX: 0000008383a0a000 RSI: 000000000960a000 RDI: 0000008383a0a000 [ 0.001000] RBP: ffffaf0e98affeb0 R08: 0000000000000000 R09: 0000000000000000 [ 0.001000] R10: ffffaf0e98affe78 R11: ffffaf0e98affdb6 R12: 0000000000000001 [ 0.001000] R13: ffff9f1700a4c3e0 R14: ffff9f16fa393e40 R15: 0000000000000001 [ 0.001000] FS: 0000000000000000(0000) GS:ffff9f1700a40000(0000) knlGS:0000000000000000 [ 0.001000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.001000] CR2: 0000000000000000 CR3: 0000008383a0a000 CR4: 00000000000000a0 [ 0.001000] invalid opcode: 0000 [#1] SMP [ 0.001000] Modules linked in: [ 0.001000] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.13.0+ #5 [ 0.001000] task: ffff9f16fa393e40 task.stack: ffffaf0e98afc000 [ 0.001000] RIP: 0010:__show_regs+0x255/0x290 [ 0.001000] RSP: 0000:ffffaf0e98affbc0 EFLAGS: 00010002 [ 0.001000] RAX: 0000000000000018 RBX: 0000000000000000 RCX: 0000000000000000 [ 0.001000] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff898a978c [ 0.001000] RBP: ffffaf0e98affc10 R08: 0000000000000001 R09: 0000000000000373 [ 0.001000] R10: ffffffff8884fb8c R11: ffffffff898ab7cd R12: 00000000ffff0ff0 [ 0.001000] R13: 0000000000000400 R14: ffff9f1700a40000 R15: 0000000000000000 [ 0.001000] FS: 0000000000000000(0000) GS:ffff9f1700a40000(0000) knlGS:0000000000000000 [ 0.001000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.001000] CR2: 0000000000000000 CR3: 0000008383a0a000 CR4: 00000000000000a0 --------------------<snip>-------------------------------------- [ 0.001000] invalid opcode: 0000 [#20] SMP [ 0.001000] Modules linked in: [ 0.001000] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.13.0+ #5 [ 0.001000] task: ffff9f16fa393e40 task.stack: ffffaf0e98afc000 [ 0.001000] RIP: 0010:__show_regs+0x255/0x290 [ 0.001000] RSP: 0000:ffffaf0e98afc788 EFLAGS: 00010002 [ 0.001000] RAX: 0000000000000018 RBX: 0000000000000000 RCX: 0000000000000000 [ 0.001000] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff898a978c [ 0.001000] RBP: ffffaf0e98afc7d8 R08: 0000000000000001 R09: 0000000000000490 [ 0.001000] R10: ffffffff88818785 R11: ffffffff898ab7cd R12: 00000000ffff0ff0 [ 0.001000] R13: 0000000000000400 R14: ffff9f1700a40000 R15: 0000000000000000 [ 0.001000] FS: 0000000000000000(0000) GS:ffff9f1700a40000(0000) knlGS:0000000000000000 [ 0.001000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.001000] CR2: 0000000000000000 CR3: 0000008383a0a000 CR4:
00000000000000a0
Force an S5 exit path.
Regards,
Sai
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 239 lines · Sai Praneeth Prakhya
Quoted 49 lines
From: Andy Lutomirski <[email protected]>
Date: Thu, Jun 29, 2017 at 8:53 AM
Subject: [PATCH v4 00/10] PCID and improved laziness
To: [email protected]
Cc: [email protected], Borislav Petkov <[email protected]>, Linus Torvalds <[email protected]>, Andrew Morton <[email protected]>, Mel Gorman <[email protected]>, "[email protected]" <[email protected]>, Nadav Amit <[email protected]>, Rik van Riel <[email protected]>, Dave Hansen <[email protected]>, Arjan van de Ven <[email protected]>, Peter Zijlstra <[email protected]>, Andy Lutomirski <[email protected]>*** Ingo, even if this misses 4.13, please apply the first patch before
*** the merge window.There are three performance benefits here:
1. TLB flushing is slow. (I.e. the flush itself takes a while.) This avoids many of them when switching tasks by using PCID. In a stupid little benchmark I did, it saves about 100ns on my laptop per context switch. I'll try to improve that benchmark.
2. Mms that have been used recently on a given CPU might get to keep their TLB entries alive across process switches with this patch set. TLB fills are pretty fast on modern CPUs, but they're even faster when they don't happen.
3. Lazy TLB is way better. We used to do two stupid things when we ran kernel threads: we'd send IPIs to flush user contexts on their CPUs and then we'd write to CR3 for no particular reason as an
excuse
to stop further IPIs. With this patch, we do neither.This will, in general, perform suboptimally if paravirt TLB flushing is in use (currently just Xen, I think, but Hyper-V is in the works). The code is structured so we could fix it in one of two ways: we could take a spinlock when touching the percpu state so we can update it remotely after a paravirt flush, or we could be more careful about our exactly how we access the state and use cmpxchg16b to do atomic remote updates. (On SMP systems without cmpxchg16b, we'd just skip the optimization entirely.)
This is still missing a final comment-only patch to add overall documentation for the whole thing, but I didn't want to block sending the maybe-hopefully-final code on that.
Quoted 9 lines
This is based on tip:x86/mm. The branch is here if you want to play:
https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/log/?h=x86/pcidIn general, performance seems to exceed my expectations. Here are some performance numbers copy-and-pasted from the changelogs for "Rework lazy TLB mode and TLB freshness" and "Try to preserve old TLB entries using PCID":
Hi Andy,
I have booted Linus's tree (
8fac2f96ab86b0e14ec4e42851e21e9b518bdc55) on Skylake server and noticed that it reboots automatically.When I booted the same kernel with command line arg "nopcid" it works fine. Please find below a snippet of dmesg. Please let me know if you need more info to debug.
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.13.0+ root=UUID=3b8e9636-6e23-4785-a4e2-5954bfe86fd9 ro console=tty0 console=ttyS0,115200n8 [ 0.000000] log_buf_len individual max cpu contribution: 4096 bytes [ 0.000000] log_buf_len total cpu_extra contributions: 258048 bytes [ 0.000000] log_buf_len min size: 262144 bytes [ 0.000000] log_buf_len: 524288 bytes [ 0.000000] early log buf free: 212560(81%) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] ------------[ cut here ]------------ [ 0.000000] WARNING: CPU: 0 PID: 0 at arch/x86/mm/tlb.c:245 initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.13.0+ #5 [ 0.000000] task: ffffffff8960f480 task.stack: ffffffff89600000 [ 0.000000] RIP: 0010:initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.000000] RSP: 0000:ffffffff89603e60 EFLAGS: 00010046 [ 0.000000] RAX: 00000000000406b0 RBX: ffff9f1700a17880 RCX: ffffffff8965de60 [ 0.000000] RDX: 0000008383a0a000 RSI: 000000000960a000 RDI: 0000008383a0a000 [ 0.000000] RBP: ffffffff89603e60 R08: 0000000000000000 R09: 0000ffffffffffff [ 0.000000] R10: ffffffff89603ee8 R11: ffffffff0000ffff R12: 0000000000000000 [ 0.000000] R13: ffff9f1700a0c3e0 R14: ffffffff8960f480 R15: 0000000000000000 [ 0.000000] FS: 0000000000000000(0000) GS:ffff9f1700a00000(0000) knlGS:0000000000000000 [ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.000000] CR2: ffff9fa7bffff000 CR3: 0000008383a0a000 CR4: 00000000000406b0 [ 0.000000] Call Trace: [ 0.000000] cpu_init+0x206/0x4f0 [ 0.000000] ? __set_pte_vaddr+0x1d/0x30 [ 0.000000] trap_init+0x3e/0x50 [ 0.000000] ? trap_init+0x3e/0x50 [ 0.000000] start_kernel+0x1e2/0x3f2 [ 0.000000] x86_64_start_reservations+0x24/0x26 [ 0.000000] x86_64_start_kernel+0x6f/0x72 [ 0.000000] secondary_startup_64+0xa5/0xa5 [ 0.000000] Code: de 00 48 01 f0 48 39 c7 0f 85 92 00 00 00 48 8b 05 ee e2 ee 00 a9 00 00 02 00 74 11 65 48 8b 05 8b 9d 7c 77 a9 00 00 02 00 75 02 <0f> ff 48 81 e2 00 f0 ff ff 0f 22 da 65 66 c7 05 66 9d 7c 77 00 [ 0.000000] ---[ end trace c258f2d278fe031f ]--- [ 0.000000] Memory: 791050356K/803934656K available (9585K kernel code, 1313K rwdata, 3000K rodata, 1176K init, 680K bss, 12884300K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=64, Nodes=4 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU event tracing is enabled. [ 0.000000] NR_IRQS: 4352, nr_irqs: 3928, preallocated irqs: 16 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns [ 0.001000] tsc: Detected 2000.000 MHz processor [ 0.002000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4000.00 BogoMIPS (lpj=2000000) [ 0.003003] pid_max: default: 65536 minimum: 512 [ 0.004030] ACPI: Core revision 20170728 [ 0.091853] ACPI: 6 ACPI AML tables successfully acquired and loaded [ 0.094143] Security Framework initialized [ 0.095004] SELinux: Initializing. [ 0.145612] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) [ 0.170544] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.172699] Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.174441] Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.176351] CPU: Physical Processor ID: 0 [ 0.177003] CPU: Processor Core ID: 0 [ 0.178007] ENERGY_PERF_BIAS: Set to 'normal', was 'performance' [ 0.179003] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) [ 0.180013] mce: CPU supports 20 MCE banks [ 0.181018] CPU0: Thermal monitoring enabled (TM1) [ 0.182057] process: using mwait in idle threads [ 0.183005] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8 [ 0.184003] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4 [ 0.185223] Freeing SMP alternatives memory: 36K [ 0.193912] smpboot: Max logical packages: 8 [ 0.194017] Switched APIC routing to physical flat. [ 0.196496] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.206252] smpboot: CPU0: Intel(R) Xeon(R) Platinum 8164 CPU @ 2.00GHz (family: 0x6, model: 0x55, stepping: 0x4) [ 0.207131] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver. [ 0.208003] ... version: 4 [ 0.209001] ... bit width: 48 [ 0.210001] ... generic registers: 4 [ 0.211001] ... value mask: 0000ffffffffffff [ 0.212001] ... max period: 00007fffffffffff [ 0.213001] ... fixed-purpose events: 3 [ 0.214001] ... event mask: 000000070000000f [ 0.215078] Hierarchical SRCU implementation. [ 0.216867] smp: Bringing up secondary CPUs ... [ 0.217085] x86: Booting SMP configuration: [ 0.218001] .... node #0, CPUs: #1 [ 0.001000] ------------[ cut here ]------------ [ 0.001000] WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:245 initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.001000] Modules linked in: [ 0.001000] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.13.0+ #5 [ 0.001000] task: ffff9f16fa393e40 task.stack: ffffaf0e98afc000 [ 0.001000] RIP: 0010:initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.001000] RSP: 0000:ffffaf0e98affeb0 EFLAGS: 00010046 [ 0.001000] RAX: 00000000000000a0 RBX: ffff9f1700a57880 RCX: ffffffff8965de60 [ 0.001000] RDX: 0000008383a0a000 RSI: 000000000960a000 RDI: 0000008383a0a000 [ 0.001000] RBP: ffffaf0e98affeb0 R08: 0000000000000000 R09: 0000000000000000 [ 0.001000] R10: ffffaf0e98affe78 R11: ffffaf0e98affdb6 R12: 0000000000000001 [ 0.001000] R13: ffff9f1700a4c3e0 R14: ffff9f16fa393e40 R15: 0000000000000001 [ 0.001000] FS: 0000000000000000(0000) GS:ffff9f1700a40000(0000) knlGS:0000000000000000 [ 0.001000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.001000] CR2: 0000000000000000 CR3: 0000008383a0a000 CR4: 00000000000000a0 [ 0.001000] invalid opcode: 0000 [#1] SMP [ 0.001000] Modules linked in: [ 0.001000] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.13.0+ #5 [ 0.001000] task: ffff9f16fa393e40 task.stack: ffffaf0e98afc000 [ 0.001000] RIP: 0010:__show_regs+0x255/0x290 [ 0.001000] RSP: 0000:ffffaf0e98affbc0 EFLAGS: 00010002 [ 0.001000] RAX: 0000000000000018 RBX: 0000000000000000 RCX: 0000000000000000 [ 0.001000] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff898a978c [ 0.001000] RBP: ffffaf0e98affc10 R08: 0000000000000001 R09: 0000000000000373 [ 0.001000] R10: ffffffff8884fb8c R11: ffffffff898ab7cd R12: 00000000ffff0ff0 [ 0.001000] R13: 0000000000000400 R14: ffff9f1700a40000 R15: 0000000000000000 [ 0.001000] FS: 0000000000000000(0000) GS:ffff9f1700a40000(0000) knlGS:0000000000000000 [ 0.001000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.001000] CR2: 0000000000000000 CR3: 0000008383a0a000 CR4: 00000000000000a0 --------------------<snip>-------------------------------------- [ 0.001000] invalid opcode: 0000 [#20] SMP [ 0.001000] Modules linked in: [ 0.001000] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.13.0+ #5 [ 0.001000] task: ffff9f16fa393e40 task.stack: ffffaf0e98afc000 [ 0.001000] RIP: 0010:__show_regs+0x255/0x290 [ 0.001000] RSP: 0000:ffffaf0e98afc788 EFLAGS: 00010002 [ 0.001000] RAX: 0000000000000018 RBX: 0000000000000000 RCX: 0000000000000000 [ 0.001000] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff898a978c [ 0.001000] RBP: ffffaf0e98afc7d8 R08: 0000000000000001 R09: 0000000000000490 [ 0.001000] R10: ffffffff88818785 R11: ffffffff898ab7cd R12: 00000000ffff0ff0 [ 0.001000] R13: 0000000000000400 R14: ffff9f1700a40000 R15: 0000000000000000 [ 0.001000] FS: 0000000000000000(0000) GS:ffff9f1700a40000(0000) knlGS:0000000000000000 [ 0.001000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.001000] CR2: 0000000000000000 CR3: 0000008383a0a000 CR4:
Quoted 2 lines
00000000000000a0
Force an S5 exit path.
I'm on my way to LPC, so I can't easily work on this right this instant.
Can you try this branch, though?
Quoted 5 lines
Regards,
Sai
I'm on my way to LPC, so I can't easily work on this right this instant.
Can you try this branch, though?
Hi Andy,
Good news! This branch works. I have also applied “x86/mm/64: Initialize CR4.PCIDE early” patch on top of Linus’s tree and it fixes the issue.
Regards,
Sai
Quoted 5 lines · Andy Lutomirski
I'm on my way to LPC, so I can't easily work on this right this instant.
Can you try this branch, though?
Any objections against me applying these fixes directly and getting them to Linus later today, to not widen the window of breakage any further?
I'll also apply:
x86/mm/64: Initialize CR4.PCIDE early
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Andy Lutomirski
I'm on my way to LPC, so I can't easily work on this right this instant.
Can you try this branch, though?
Quoted 8 lines
Any objections against me applying these fixes directly and getting them to Linus later today, to not widen the window of breakage any further?
I'll also apply:
x86/mm/64: Initialize CR4.PCIDE early
So now that I've looked at the branch, I think I'll apply these three:
cb88ae619b4c: x86/mm/64: Initialize CR4.PCIDE early 6ec68017784f: x86/hibernate/64: Mask off CR3's PCID bits in the saved CR3 5d1298206d7b: x86/mm: Get rid of VM_BUG_ON in switch_tlb_irqs_off()
The first one is that likely fixed the crash that Sai Praneeth Prakhya reported.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 159 lines
Hi Andy,
I have booted Linus's tree (
8fac2f96ab86b0e14ec4e42851e21e9b518bdc55) on Skylake server and noticed that it reboots automatically.When I booted the same kernel with command line arg "nopcid" it works fine. Please find below a snippet of dmesg. Please let me know if you need more info to debug.
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.13.0+ root=UUID=3b8e9636-6e23-4785-a4e2-5954bfe86fd9 ro console=tty0 console=ttyS0,115200n8 [ 0.000000] log_buf_len individual max cpu contribution: 4096 bytes [ 0.000000] log_buf_len total cpu_extra contributions: 258048 bytes [ 0.000000] log_buf_len min size: 262144 bytes [ 0.000000] log_buf_len: 524288 bytes [ 0.000000] early log buf free: 212560(81%) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] ------------[ cut here ]------------ [ 0.000000] WARNING: CPU: 0 PID: 0 at arch/x86/mm/tlb.c:245 initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.13.0+ #5 [ 0.000000] task: ffffffff8960f480 task.stack: ffffffff89600000 [ 0.000000] RIP: 0010:initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.000000] RSP: 0000:ffffffff89603e60 EFLAGS: 00010046 [ 0.000000] RAX: 00000000000406b0 RBX: ffff9f1700a17880 RCX: ffffffff8965de60 [ 0.000000] RDX: 0000008383a0a000 RSI: 000000000960a000 RDI: 0000008383a0a000 [ 0.000000] RBP: ffffffff89603e60 R08: 0000000000000000 R09: 0000ffffffffffff [ 0.000000] R10: ffffffff89603ee8 R11: ffffffff0000ffff R12: 0000000000000000 [ 0.000000] R13: ffff9f1700a0c3e0 R14: ffffffff8960f480 R15: 0000000000000000 [ 0.000000] FS: 0000000000000000(0000) GS:ffff9f1700a00000(0000) knlGS:0000000000000000 [ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.000000] CR2: ffff9fa7bffff000 CR3: 0000008383a0a000 CR4: 00000000000406b0 [ 0.000000] Call Trace: [ 0.000000] cpu_init+0x206/0x4f0 [ 0.000000] ? __set_pte_vaddr+0x1d/0x30 [ 0.000000] trap_init+0x3e/0x50 [ 0.000000] ? trap_init+0x3e/0x50 [ 0.000000] start_kernel+0x1e2/0x3f2 [ 0.000000] x86_64_start_reservations+0x24/0x26 [ 0.000000] x86_64_start_kernel+0x6f/0x72 [ 0.000000] secondary_startup_64+0xa5/0xa5 [ 0.000000] Code: de 00 48 01 f0 48 39 c7 0f 85 92 00 00 00 48 8b 05 ee e2 ee 00 a9 00 00 02 00 74 11 65 48 8b 05 8b 9d 7c 77 a9 00 00 02 00 75 02 <0f> ff 48 81 e2 00 f0 ff ff 0f 22 da 65 66 c7 05 66 9d 7c 77 00 [ 0.000000] ---[ end trace c258f2d278fe031f ]--- [ 0.000000] Memory: 791050356K/803934656K available (9585K kernel code, 1313K rwdata, 3000K rodata, 1176K init, 680K bss, 12884300K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=64, Nodes=4 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU event tracing is enabled. [ 0.000000] NR_IRQS: 4352, nr_irqs: 3928, preallocated irqs: 16 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns [ 0.001000] tsc: Detected 2000.000 MHz processor [ 0.002000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4000.00 BogoMIPS (lpj=2000000) [ 0.003003] pid_max: default: 65536 minimum: 512 [ 0.004030] ACPI: Core revision 20170728 [ 0.091853] ACPI: 6 ACPI AML tables successfully acquired and loaded [ 0.094143] Security Framework initialized [ 0.095004] SELinux: Initializing. [ 0.145612] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) [ 0.170544] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.172699] Mount-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.174441] Mountpoint-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.176351] CPU: Physical Processor ID: 0 [ 0.177003] CPU: Processor Core ID: 0 [ 0.178007] ENERGY_PERF_BIAS: Set to 'normal', was 'performance' [ 0.179003] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) [ 0.180013] mce: CPU supports 20 MCE banks [ 0.181018] CPU0: Thermal monitoring enabled (TM1) [ 0.182057] process: using mwait in idle threads [ 0.183005] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8 [ 0.184003] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4 [ 0.185223] Freeing SMP alternatives memory: 36K [ 0.193912] smpboot: Max logical packages: 8 [ 0.194017] Switched APIC routing to physical flat. [ 0.196496] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.206252] smpboot: CPU0: Intel(R) Xeon(R) Platinum 8164 CPU @ 2.00GHz (family: 0x6, model: 0x55, stepping: 0x4) [ 0.207131] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver. [ 0.208003] ... version: 4 [ 0.209001] ... bit width: 48 [ 0.210001] ... generic registers: 4 [ 0.211001] ... value mask: 0000ffffffffffff [ 0.212001] ... max period: 00007fffffffffff [ 0.213001] ... fixed-purpose events: 3 [ 0.214001] ... event mask: 000000070000000f [ 0.215078] Hierarchical SRCU implementation. [ 0.216867] smp: Bringing up secondary CPUs ... [ 0.217085] x86: Booting SMP configuration: [ 0.218001] .... node #0, CPUs: #1 [ 0.001000] ------------[ cut here ]------------ [ 0.001000] WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:245 initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.001000] Modules linked in: [ 0.001000] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.13.0+ #5 [ 0.001000] task: ffff9f16fa393e40 task.stack: ffffaf0e98afc000 [ 0.001000] RIP: 0010:initialize_tlbstate_and_flush+0x6c/0xf0 [ 0.001000] RSP: 0000:ffffaf0e98affeb0 EFLAGS: 00010046 [ 0.001000] RAX: 00000000000000a0 RBX: ffff9f1700a57880 RCX: ffffffff8965de60 [ 0.001000] RDX: 0000008383a0a000 RSI: 000000000960a000 RDI: 0000008383a0a000 [ 0.001000] RBP: ffffaf0e98affeb0 R08: 0000000000000000 R09: 0000000000000000 [ 0.001000] R10: ffffaf0e98affe78 R11: ffffaf0e98affdb6 R12: 0000000000000001 [ 0.001000] R13: ffff9f1700a4c3e0 R14: ffff9f16fa393e40 R15: 0000000000000001 [ 0.001000] FS: 0000000000000000(0000) GS:ffff9f1700a40000(0000) knlGS:0000000000000000 [ 0.001000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.001000] CR2: 0000000000000000 CR3: 0000008383a0a000 CR4: 00000000000000a0 [ 0.001000] invalid opcode: 0000 [#1] SMP [ 0.001000] Modules linked in: [ 0.001000] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.13.0+ #5 [ 0.001000] task: ffff9f16fa393e40 task.stack: ffffaf0e98afc000 [ 0.001000] RIP: 0010:__show_regs+0x255/0x290 [ 0.001000] RSP: 0000:ffffaf0e98affbc0 EFLAGS: 00010002 [ 0.001000] RAX: 0000000000000018 RBX: 0000000000000000 RCX: 0000000000000000 [ 0.001000] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff898a978c [ 0.001000] RBP: ffffaf0e98affc10 R08: 0000000000000001 R09: 0000000000000373 [ 0.001000] R10: ffffffff8884fb8c R11: ffffffff898ab7cd R12: 00000000ffff0ff0 [ 0.001000] R13: 0000000000000400 R14: ffff9f1700a40000 R15: 0000000000000000 [ 0.001000] FS: 0000000000000000(0000) GS:ffff9f1700a40000(0000) knlGS:0000000000000000 [ 0.001000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.001000] CR2: 0000000000000000 CR3: 0000008383a0a000 CR4:
Quoted 1 line
00000000000000a0
Hi All,
Since I didn't find conversation between Andy and myself on the mailing list, I wanted to keep this thread updated. So, sorry! for spamming your inbox if it's a duplicate.
Andy has asked me to try a patch that's already in his git repo and it fixes the issue.
Regards,
Sai
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>