powerpc/64s: fix for CPU hotplug vs mm_cpumask bug
Subsystems (3)
Nicholas Piggin
reviewercover powerpc/64s: fix for CPU hotplug vs mm_cpumask bug
This fixes a race in powerpc mm_cpumask code, I hope the core kernel patch looks okay and we could take it through the powerpc tree with an ack from someone (Peter or Thomas, perhaps?)
Thanks,
Nick
Nicholas Piggin (2):
kernel/cpu: add arch override for clear_tasks_mm_cpumask() mm handling powerpc/64s: Trim offlined CPUs from mm_cpumasks
arch/powerpc/include/asm/book3s/64/mmu.h | 12 ++++++++++++ arch/powerpc/mm/book3s64/mmu_context.c | 20 ++++++++++++++++++++ arch/powerpc/platforms/powermac/smp.c | 2 ++ arch/powerpc/platforms/powernv/smp.c | 3 +++ arch/powerpc/platforms/pseries/hotplug-cpu.c | 3 +++ kernel/cpu.c | 6 +++++- 6 files changed, 45 insertions(+), 1 deletion(-)
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH 1/2 kernel/cpu: add arch override for clear_tasks_mm_cpumask() mm handling
powerpc keeps a counter in the mm which counts bits set in mm_cpumask as well as other things. This means it can't use generic code to clear bits out of the mask and doesn't adjust the arch specific counter.
Add an arch override allowing powerpc to use clear_tasks_mm_cpumask().
--- kernel/cpu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/cpu.c +5 -1
@@ -815,6 +815,10 @@ void __init cpuhp_threads_init(void) }#ifdef CONFIG_HOTPLUG_CPU +#ifndef arch_clear_mm_cpumask_cpu +#define arch_clear_mm_cpumask_cpu (cpu, mm) cpumask_clear_cpu(cpu, mm_cpumask(mm)) +#endif +/** * clear_tasks_mm_cpumask - Safely clear tasks' mm_cpumask for a CPU * @cpu: a CPU id
@@ -850,7 +854,7 @@ void clear_tasks_mm_cpumask(int cpu) t =find_lock_task_mm (p);if (!t)continue ; -cpumask_clear_cpu (cpu,mm_cpumask (t->mm)); +arch_clear_mm_cpumask_cpu (cpu, t->mm);task_unlock (t); }rcu_read_unlock ();
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH 2/2 powerpc/64s: Trim offlined CPUs from mm_cpumasks
In reply to Nicholas PigginWhen offlining a CPU, powerpc/64s does not flush TLBs, rather it just leaves the CPU set in mm_cpumasks, so it continues to receive TLBIEs to manage its TLBs.
However the exit_flush_lazy_tlbs() function expects that after returning, all CPUs (except self) have flushed TLBs for that mm, in which case TLBIEL can be used for this flush. This breaks for offline CPUs because they don't get the IPI to flush their TLB. This can lead to stale translations.
Fix this by clearing the CPU from mm_cpumasks, then flushing all TLBs before going offline.
These offlined CPU bits stuck in the cpumask also prevents the cpumask from being trimmed back to local mode, which means continual broadcast IPIs or TLBIEs are needed for TLB flushing. This patch prevents that situation too.
--- arch/powerpc/include/asm/book3s/64/mmu.h | 12 ++++++++++++ arch/powerpc/mm/book3s64/mmu_context.c | 20 ++++++++++++++++++++ arch/powerpc/platforms/powermac/smp.c | 2 ++ arch/powerpc/platforms/powernv/smp.c | 3 +++ arch/powerpc/platforms/pseries/hotplug-cpu.c | 3 +++ 5 files changed, 40 insertions(+)
arch/powerpc/include/asm/book3s/64/mmu.h +12 -0
@@ -242,6 +242,18 @@ extern void radix_init_pseries(void);static inline void radix_init_pseries (void ) { };#endif +#ifdef CONFIG_HOTPLUG_CPU +#define arch_clear_mm_cpumask_cpu (cpu, mm) \ + do { \ + if (cpumask_test_cpu(cpu, mm_cpumask(mm))) { \ + atomic_dec(&(mm)->context.active_cpus); \ + cpumask_clear_cpu(cpu, mm_cpumask(mm)); \ + } \ + } while (0) + +void cleanup_cpu_mmu_context (void ); +#endif +static inline int get_user_context (mm_context_t *ctx,unsigned long ea) {int index = ea >>MAX_EA_BITS_PER_CONTEXT ;
arch/powerpc/mm/book3s64/mmu_context.c +20 -0
@@ -17,6 +17,7 @@#include <linux/export.h> #include <linux/gfp.h> #include <linux/slab.h> +#include <linux/cpu.h> #include <asm/mmu_context.h> #include <asm/pgalloc.h>
@@ -307,3 +308,22 @@ void radix__switch_mmu_context(struct mm_struct *prev, struct mm_struct *next) isync(); }#endif + +/** +* cleanup_cpu_mmu_context - Clean up MMU details for this CPU (newly offlined) +* +* This clears the CPU from mm_cpumask for all processes, and then flushes the +* local TLB to ensure TLB coherency in case the CPU is onlined again. +* +* KVM guest translations are not necessarily flushed here. If KVM started +* using mm_cpumask or the Linux APIs which do, this would have to be resolved. +*/ +#ifdef CONFIG_HOTPLUG_CPU +void cleanup_cpu_mmu_context (void ) +{ +int cpu =smp_processor_id (); + +clear_tasks_mm_cpumask (cpu); +tlbiel_all (); +} +#endif
arch/powerpc/platforms/powermac/smp.c +2 -0
@@ -911,6 +911,8 @@ static int smp_core99_cpu_disable(void)mpic_cpu_set_priority (0xf ); +cleanup_cpu_mmu_context (); +return 0 ; }
arch/powerpc/platforms/powernv/smp.c +3 -0
@@ -143,6 +143,9 @@ static int pnv_smp_cpu_disable(void)xive_smp_disable_cpu (); elsexics_migrate_irqs_away (); + +cleanup_cpu_mmu_context (); +return 0 ; }
arch/powerpc/platforms/pseries/hotplug-cpu.c +3 -0
@@ -90,6 +90,9 @@ static int pseries_cpu_disable(void)xive_smp_disable_cpu (); elsexics_migrate_irqs_away (); + +cleanup_cpu_mmu_context (); +return 0 ; }
Signature · 1 line
2.23.0
Re: [PATCH 1/2] kernel/cpu: add arch override for clear_tasks_mm_cpumask() mm handling
In reply to Nicholas PigginQuoted 7 lines · Nicholas Piggin
powerpc keeps a counter in the mm which counts bits set in
mm_cpumaskas well as other things. This means it can't use generic code to clear bits out of the mask and doesn't adjust the arch specific counter.Add an arch override allowing powerpc to use
clear_tasks_mm_cpumask().Signed-off-by Nicholas Piggin <[email protected]>
Seems reasonable enough..
Quoted 31 lines
--- kernel/cpu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)kernel/cpu.c +5
-1@@ -815,6 +815,10 @@ void __init cpuhp_threads_init(void) }#ifdef CONFIG_HOTPLUG_CPU +#ifndef arch_clear_mm_cpumask_cpu +#define arch_clear_mm_cpumask_cpu (cpu, mm) cpumask_clear_cpu(cpu, mm_cpumask(mm)) +#endif +/** * clear_tasks_mm_cpumask - Safely clear tasks' mm_cpumask for a CPU * @cpu: a CPU id @@ -850,7 +854,7 @@ void clear_tasks_mm_cpumask(int cpu) t =find_lock_task_mm (p);if (!t)continue ; -cpumask_clear_cpu (cpu,mm_cpumask (t->mm)); +arch_clear_mm_cpumask_cpu (cpu, t->mm);task_unlock (t); }rcu_read_unlock ();Signature · 1 line
2.23.0
Re: [PATCH 2/2] powerpc/64s: Trim offlined CPUs from mm_cpumasks
In reply to Nicholas PigginHi Nicholas,
Quoted 14 lines · Nicholas Piggin
When offlining a CPU, powerpc/64s does not flush TLBs, rather it just leaves the CPU set in
mm_cpumasks, so it continues to receive TLBIEs to manage its TLBs.However the
exit_flush_lazy_tlbs()function expects that after returning, all CPUs (except self) have flushed TLBs for that mm, in which case TLBIEL can be used for this flush. This breaks for offline CPUs because they don't get the IPI to flush their TLB. This can lead to stale translations.Fix this by clearing the CPU from
mm_cpumasks, then flushing all TLBs before going offline.
Quoted 6 lines
These offlined CPU bits stuck in the cpumask also prevents the cpumask from being trimmed back to local mode, which means continual broadcast IPIs or TLBIEs are needed for TLB flushing. This patch prevents that situation too.
Signed-off-by Nicholas Piggin <[email protected]>
Thanks for your patch!
arch/powerpc/platforms/powermac/smp.c static int smp_core99_cpu_disable(void) · line 915
Quoted 8 lines
arch/powerpc/platforms/powermac/smp.c +2
-0@@ -911,6 +911,8 @@ static int smp_core99_cpu_disable(void)mpic_cpu_set_priority (0xf ); +cleanup_cpu_mmu_context (); +
I guess this change broke pmac32_defconfig+SMP in v5.10-rc7?
arch/powerpc/platforms/powermac/smp.c: error: implicit declaration of function 'cleanup_cpu_mmu_context' [-Werror=implicit-function-declaration]: => 914:2
Quoted 32 lines
arch/powerpc/platforms/powermac/smp.c +0
-0return 0 ; }arch/powerpc/platforms/powernv/smp.c +3
-0@@ -143,6 +143,9 @@ static int pnv_smp_cpu_disable(void)xive_smp_disable_cpu (); elsexics_migrate_irqs_away (); + +cleanup_cpu_mmu_context (); +return 0 ; }arch/powerpc/platforms/pseries/hotplug-cpu.c +3
-1@@ -90,6 +90,9 @@ static int pseries_cpu_disable(void)xive_smp_disable_cpu (); elsexics_migrate_irqs_away (); + +cleanup_cpu_mmu_context (); +return 0 ; } --
Quoted 2 lines
2.23.0
--
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Nicholas Piggin
reviewerRe: [PATCH 2/2] powerpc/64s: Trim offlined CPUs from mm_cpumasks
Excerpts from Geert Uytterhoeven's message of December 10, 2020 7:06 pm:
Quoted 37 lines
Hi Nicholas,
Quoted 14 lines · Nicholas Piggin
When offlining a CPU, powerpc/64s does not flush TLBs, rather it just leaves the CPU set in
mm_cpumasks, so it continues to receive TLBIEs to manage its TLBs.However the
exit_flush_lazy_tlbs()function expects that after returning, all CPUs (except self) have flushed TLBs for that mm, in which case TLBIEL can be used for this flush. This breaks for offline CPUs because they don't get the IPI to flush their TLB. This can lead to stale translations.Fix this by clearing the CPU from
mm_cpumasks, then flushing all TLBs before going offline.Quoted 6 lines
These offlined CPU bits stuck in the cpumask also prevents the cpumask from being trimmed back to local mode, which means continual broadcast IPIs or TLBIEs are needed for TLB flushing. This patch prevents that situation too.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for your patch!
arch/powerpc/platforms/powermac/smp.c static int smp_core99_cpu_disable(void) · line 915
Quoted 8 lines
arch/powerpc/platforms/powermac/smp.c +2
-0@@ -911,6 +911,8 @@ static int smp_core99_cpu_disable(void)mpic_cpu_set_priority (0xf ); +cleanup_cpu_mmu_context (); +I guess this change broke pmac32_defconfig+SMP in v5.10-rc7?
Quoted 5 lines
arch/powerpc/platforms/powermac/smp.c: error: implicit declaration of function 'cleanup_cpu_mmu_context' [-Werror=implicit-function-declaration]: => 914:2
Hey, yeah it does thanks for catching it. This patch fixes it for me
---
From a9b5ec92ffac975e81c6d7db6ff2b1486b2723f7 Mon Sep 17 00:00:00 2001
From: Nicholas Piggin <[email protected]>
Date: Mon, 14 Dec 2020 13:52:39 +1000
Subject: [PATCH] powerpc/32s: Fix cleanup_cpu_mmu_context() compile bug
32s has no tlbiel_all() defined, so just disable the cleanup with a comment.
--- arch/powerpc/platforms/powermac/smp.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/powerpc/platforms/powermac/smp.c +9 -0
@@ -911,7 +911,16 @@ static int smp_core99_cpu_disable(void)mpic_cpu_set_priority (0xf ); +/* +* Would be nice for consistency if all platforms clear mm_cpumask and +* flush TLBs on unplug, but the TLB invalidation bug described in +* commit 01b0f0eae081 ("powerpc/64s: Trim offlined CPUs from +* mm_cpumasks") only applies to 64s and for now we only have the TLB +* flush code for that platform. +*/ +#ifdef CONFIG_PPC64 cleanup_cpu_mmu_context (); +#endif return 0 ; }
Signature · 1 line
2.23.0
Michael Ellerman
reviewerRe: [PATCH 2/2] powerpc/64s: Trim offlined CPUs from mm_cpumasks
Quoted 46 lines · Nicholas Piggin
Excerpts from Geert Uytterhoeven's message of December 10, 2020 7:06 pm:
Quoted 37 lines
Hi Nicholas,
Quoted 14 lines · Nicholas Piggin
When offlining a CPU, powerpc/64s does not flush TLBs, rather it just leaves the CPU set in
mm_cpumasks, so it continues to receive TLBIEs to manage its TLBs.However the
exit_flush_lazy_tlbs()function expects that after returning, all CPUs (except self) have flushed TLBs for that mm, in which case TLBIEL can be used for this flush. This breaks for offline CPUs because they don't get the IPI to flush their TLB. This can lead to stale translations.Fix this by clearing the CPU from
mm_cpumasks, then flushing all TLBs before going offline.Quoted 6 lines
These offlined CPU bits stuck in the cpumask also prevents the cpumask from being trimmed back to local mode, which means continual broadcast IPIs or TLBIEs are needed for TLB flushing. This patch prevents that situation too.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for your patch!
arch/powerpc/platforms/powermac/smp.c static int smp_core99_cpu_disable(void) · line 915
Quoted 8 lines
arch/powerpc/platforms/powermac/smp.c +2
-0@@ -911,6 +911,8 @@ static int smp_core99_cpu_disable(void)mpic_cpu_set_priority (0xf ); +cleanup_cpu_mmu_context (); +I guess this change broke pmac32_defconfig+SMP in v5.10-rc7?
Quoted 5 lines
arch/powerpc/platforms/powermac/smp.c: error: implicit declaration of function 'cleanup_cpu_mmu_context' [-Werror=implicit-function-declaration]: => 914:2Hey, yeah it does thanks for catching it. This patch fixes it for me
Quoted 8 lines
---
Froma9b5ec92ffac975e81c6d7db6ff2b1486b2723f7Mon Sep 17 00:00:00 2001
From: Nicholas Piggin <[email protected]>
Date: Mon, 14 Dec 2020 13:52:39 +1000
Subject: [PATCH] powerpc/32s: Fixcleanup_cpu_mmu_context()compile bug32s has no
tlbiel_all()defined, so just disable the cleanup with a comment.
Or what about just:
arch/powerpc/include/asm/book3s/32/mmu-hash.h +1 -0
@@ -94,6 +94,7 @@ typedef struct { } mm_context_t;void update_bats (void ); +static inline void cleanup_cpu_mmu_context (void ) { };/* patch sites */ extern s32 patch__hash_page_A0, patch__hash_page_A1, patch__hash_page_A2;
cheers
Nicholas Piggin
reviewerRe: [PATCH 2/2] powerpc/64s: Trim offlined CPUs from mm_cpumasks
Excerpts from Michael Ellerman's message of December 14, 2020 8:43 pm:
Quoted 53 lines
Quoted 52 lines · Nicholas Piggin
Excerpts from Geert Uytterhoeven's message of December 10, 2020 7:06 pm:
Quoted 37 lines
Hi Nicholas,
Quoted 14 lines · Nicholas Piggin
When offlining a CPU, powerpc/64s does not flush TLBs, rather it just leaves the CPU set in
mm_cpumasks, so it continues to receive TLBIEs to manage its TLBs.However the
exit_flush_lazy_tlbs()function expects that after returning, all CPUs (except self) have flushed TLBs for that mm, in which case TLBIEL can be used for this flush. This breaks for offline CPUs because they don't get the IPI to flush their TLB. This can lead to stale translations.Fix this by clearing the CPU from
mm_cpumasks, then flushing all TLBs before going offline.Quoted 6 lines
These offlined CPU bits stuck in the cpumask also prevents the cpumask from being trimmed back to local mode, which means continual broadcast IPIs or TLBIEs are needed for TLB flushing. This patch prevents that situation too.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for your patch!
arch/powerpc/platforms/powermac/smp.c static int smp_core99_cpu_disable(void) · line 915
Quoted 8 lines
arch/powerpc/platforms/powermac/smp.c +2
-0@@ -911,6 +911,8 @@ static int smp_core99_cpu_disable(void)mpic_cpu_set_priority (0xf ); +cleanup_cpu_mmu_context (); +I guess this change broke pmac32_defconfig+SMP in v5.10-rc7?
Quoted 5 lines
arch/powerpc/platforms/powermac/smp.c: error: implicit declaration of function 'cleanup_cpu_mmu_context' [-Werror=implicit-function-declaration]: => 914:2Hey, yeah it does thanks for catching it. This patch fixes it for me
---
Froma9b5ec92ffac975e81c6d7db6ff2b1486b2723f7Mon Sep 17 00:00:00 2001
From: Nicholas Piggin <[email protected]>
Date: Mon, 14 Dec 2020 13:52:39 +1000
Subject: [PATCH] powerpc/32s: Fixcleanup_cpu_mmu_context()compile bug
Quoted 4 lines
Quoted 2 lines
32s has no
tlbiel_all()defined, so just disable the cleanup with a comment.Or what about just:
That works, I kind of wanted it in there explicit that we don't clean up on 32s. I don't mind if you prefer this though.
Thanks,
Nick
Quoted 18 lines
arch/powerpc/include/asm/book3s/32/mmu-hash.h +1
-0@@ -94,6 +94,7 @@ typedef struct { } mm_context_t;void update_bats (void ); +static inline void cleanup_cpu_mmu_context (void ) { };/* patch sites */ extern s32 patch__hash_page_A0, patch__hash_page_A1, patch__hash_page_A2;cheers
Michael Ellerman
reviewerRe: [PATCH 2/2] powerpc/64s: Trim offlined CPUs from mm_cpumasks
Quoted 54 lines · Nicholas Piggin
Excerpts from Michael Ellerman's message of December 14, 2020 8:43 pm:
Quoted 53 lines
Quoted 52 lines · Nicholas Piggin
Excerpts from Geert Uytterhoeven's message of December 10, 2020 7:06 pm:
Quoted 37 lines
Hi Nicholas,
Quoted 14 lines · Nicholas Piggin
When offlining a CPU, powerpc/64s does not flush TLBs, rather it just leaves the CPU set in
mm_cpumasks, so it continues to receive TLBIEs to manage its TLBs.However the
exit_flush_lazy_tlbs()function expects that after returning, all CPUs (except self) have flushed TLBs for that mm, in which case TLBIEL can be used for this flush. This breaks for offline CPUs because they don't get the IPI to flush their TLB. This can lead to stale translations.Fix this by clearing the CPU from
mm_cpumasks, then flushing all TLBs before going offline.Quoted 6 lines
These offlined CPU bits stuck in the cpumask also prevents the cpumask from being trimmed back to local mode, which means continual broadcast IPIs or TLBIEs are needed for TLB flushing. This patch prevents that situation too.
Signed-off-by Nicholas Piggin <[email protected]>Thanks for your patch!
arch/powerpc/platforms/powermac/smp.c static int smp_core99_cpu_disable(void) · line 915
Quoted 8 lines
arch/powerpc/platforms/powermac/smp.c +2
-0@@ -911,6 +911,8 @@ static int smp_core99_cpu_disable(void)mpic_cpu_set_priority (0xf ); +cleanup_cpu_mmu_context (); +I guess this change broke pmac32_defconfig+SMP in v5.10-rc7?
Quoted 5 lines
arch/powerpc/platforms/powermac/smp.c: error: implicit declaration of function 'cleanup_cpu_mmu_context' [-Werror=implicit-function-declaration]: => 914:2Hey, yeah it does thanks for catching it. This patch fixes it for me
---
Froma9b5ec92ffac975e81c6d7db6ff2b1486b2723f7Mon Sep 17 00:00:00 2001
From: Nicholas Piggin <[email protected]>
Date: Mon, 14 Dec 2020 13:52:39 +1000
Subject: [PATCH] powerpc/32s: Fixcleanup_cpu_mmu_context()compile bug
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
32s has no
tlbiel_all()defined, so just disable the cleanup with a comment.Or what about just:
That works, I kind of wanted it in there explicit that we don't clean up on 32s. I don't mind if you prefer this though.
OK. I'll merge my version because I can do that without needing to merge master in order to get the broken commit into my tree.
cheers