three cpu hotplug fixes
cover three cpu hotplug fixes
Three fixes for some very rare to find cpu hotplug bugs.
PATCH 1/3 printk: fix wake_up_klogd() vs cpu hotplug
From: Heiko Carstens <[email protected]>
wake_up_klogd() may get called from preemtible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.
No idea why that behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.
Let's use get_cpu_var() instead which disables preemption and makes sure that the outlined scenario cannot happen.
--- kernel/printk.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/printk.c +4 -2
@@ -1087,8 +1087,10 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void ) { -if (waitqueue_active (&log_wait)) -__raw_get_cpu_var (printk_pending) =1 ; +if (waitqueue_active (&log_wait)) { +get_cpu_var (printk_pending) =1 ; +put_cpu_var (printk_pending); + } }/**
PATCH 2/3 nohz: fix printk_needs_cpu() return value on offline cpus
In reply to Heiko CarstensFrom: Heiko Carstens <[email protected]>
This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.
printk_needs_cpu() may return 1 if called on offline cpus. When a cpu gets offlined it schedules the idle process which, before killing its own cpu, will call tick_nohz_stop_sched_tick().
That function in turn will call printk_needs_cpu() in order to check if the local tick can be disabled. On offline cpus this function should naturally return 0 since regardless if the tick gets disabled or not the cpu will be dead short after. That is besides the fact that __cpu_disable() should already have made sure that no interrupts on the offlined cpu will be delivered anyway.
In this case it prevents tick_nohz_stop_sched_tick() to call select_nohz_load_balancer(). No idea if that really is a problem. However what made me debug this is that on 2.6.32 the function get_nohz_load_balancer() is used within __mod_timer() to select a cpu on which a timer gets enqueued. If printk_needs_cpu() returns 1 then the nohz_load_balancer cpu doesn't get updated when a cpu gets offlined. It may contain the cpu number of an offline cpu. In turn timers get enqueued on an offline cpu and not very surprisingly they never expire and cause system hangs.
This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.
Easiest way to fix this is just to test if the current cpu is offline and call printk_tick() directly which clears the condition.
Alternatively I tried a cpu hotplug notifier which would clear the condition, however between calling the notifier function and printk_needs_cpu() something could have called printk() again and the problem is back again. This seems to be the safest fix.
--- kernel/printk.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/printk.c +2 -0
@@ -1082,6 +1082,8 @@ void printk_tick(void)int printk_needs_cpu (int cpu) { +if (unlikely (cpu_is_offline (cpu))) +printk_tick ();return per_cpu (printk_pending, cpu); }
PATCH 3/3 nohz/s390: fix arch_needs_cpu() return value on offline cpus
In reply to Heiko CarstensFrom: Heiko Carstens <[email protected]>
This fixes the same problem as described in the patch "nohz: fix printk_needs_cpu() return value on offline cpus" for the arch_needs_cpu() primitive.
This specific bug was indrocuded with 3c5d92a0 "nohz: Introduce arch_needs_cpu".
In this case a cpu hotplug notifier is used to fix the issue in order to keep the normal/fast path small. All we need to do is to clear the condition that makes arch_needs_cpu() return 1 since it is just a performance improvement which is supposed to keep the local tick running for a short period if a cpu goes idle. Nothing special needs to be done except for clearing the condition.
--- arch/s390/kernel/vtime.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
arch/s390/kernel/vtime.c +19 -0
@@ -19,6 +19,7 @@#include <linux/kernel_stat.h> #include <linux/rcupdate.h> #include <linux/posix-timers.h> +#include <linux/cpu.h> #include <asm/s390_ext.h> #include <asm/timer.h>
@@ -566,6 +567,23 @@ void init_cpu_vtimer(void)__ctl_set_bit (0 ,10 ); } +static int __cpuinit s390_nohz_notify (struct notifier_block *self, +unsigned long action,void *hcpu) +{ +struct s390_idle_data *idle; +long cpu = (long ) hcpu; + + idle = &per_cpu (s390_idle, cpu); +switch (action) { +case CPU_DYING : +case CPU_DYING_FROZEN : + idle->nohz_delay =0 ; +default : +break ; + } +return NOTIFY_OK ; +} +void __init vtime_init (void ) {/* request the cpu timer external interrupt */
@@ -574,5 +592,6 @@ void __init vtime_init(void)/* Enable cpu timer interrupts on the boot cpu. */ init_cpu_vtimer (); +cpu_notifier (s390_nohz_notify,0 ); }
Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug
In reply to Heiko CarstensQuoted 27 lines · Heiko Carstens
plain text document attachment (001_printk_preempt.diff)
From: Heiko Carstens <[email protected]>
wake_up_klogd()may get called from preemtible context but uses__raw_get_cpu_var()to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.No idea why that behaviour was introduced with
fa33507a"printk: robustify printk, fix #2" which was supposed to fix a "usingsmp_processor_id()in preemptible" warning.Let's use
get_cpu_var()instead which disables preemption and makes sure that the outlined scenario cannot happen.Signed-off-by Heiko Carstens <[email protected]>--- kernel/printk.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/printk.c +0
-0@@ -1087,8 +1087,10 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void )
kernel/printk.c int printk_needs_cpu(int cpu) · line 1096
Quoted 10 lines
kernel/printk.c +4
-2{ -if (waitqueue_active (&log_wait)) -__raw_get_cpu_var (printk_pending) =1 ; +if (waitqueue_active (&log_wait)) { +get_cpu_var (printk_pending) =1 ; +put_cpu_var (printk_pending); + } }/**
But but but, the cpu can still be offlined between writing this state and the next tick happening, right?
Re: [patch 2/3] nohz: fix printk_needs_cpu() return value on offline cpus
In reply to Heiko CarstensQuoted 46 lines · Heiko Carstens
plain text document attachment (002_printk_needs_cpu.diff)
From: Heiko Carstens <[email protected]>This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.
printk_needs_cpu()may return 1 if called on offline cpus. When a cpu gets offlined it schedules the idle process which, before killing its own cpu, will calltick_nohz_stop_sched_tick().
That function in turn will callprintk_needs_cpu()in order to check if the local tick can be disabled. On offline cpus this function should naturally return 0 since regardless if the tick gets disabled or not the cpu will be dead short after. That is besides the fact that__cpu_disable()should already have made sure that no interrupts on the offlined cpu will be delivered anyway.In this case it prevents
tick_nohz_stop_sched_tick()to callselect_nohz_load_balancer(). No idea if that really is a problem. However what made me debug this is that on 2.6.32 the functionget_nohz_load_balancer()is used within__mod_timer()to select a cpu on which a timer gets enqueued. Ifprintk_needs_cpu()returns 1 then thenohz_load_balancercpu doesn't get updated when a cpu gets offlined. It may contain the cpu number of an offline cpu. In turn timers get enqueued on an offline cpu and not very surprisingly they never expire and cause system hangs.This has been observed 2.6.32 kernels. On current kernels
__mod_timer()usesget_nohz_timer_target()which doesn't have that problem. However there might be other problems because of the too early exittick_nohz_stop_sched_tick()in case a cpu goes offline.Easiest way to fix this is just to test if the current cpu is offline and call
printk_tick()directly which clears the condition.Alternatively I tried a cpu hotplug notifier which would clear the condition, however between calling the notifier function and
printk_needs_cpu()something could have calledprintk()again and the problem is back again. This seems to be the safest fix.Signed-off-by Heiko Carstens <[email protected]>--- kernel/printk.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/printk.c void printk_tick(void) · line 1089
Quoted 8 lines
kernel/printk.c +2
-0@@ -1082,6 +1082,8 @@ void printk_tick(void)int printk_needs_cpu (int cpu) { +if (unlikely (cpu_is_offline (cpu))) +printk_tick ();return per_cpu (printk_pending, cpu); }
Nice,.. applied.
Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug
In reply to Peter ZijlstraQuoted 36 lines · Peter Zijlstra
Quoted 35 lines · Heiko Carstens
plain text document attachment (001_printk_preempt.diff)
From: Heiko Carstens <[email protected]>
wake_up_klogd()may get called from preemtible context but uses__raw_get_cpu_var()to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.No idea why that behaviour was introduced with
fa33507a"printk: robustify printk, fix #2" which was supposed to fix a "usingsmp_processor_id()in preemptible" warning.Let's use
get_cpu_var()instead which disables preemption and makes sure that the outlined scenario cannot happen.Signed-off-by Heiko Carstens <[email protected]>--- kernel/printk.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/printk.c +4
-2@@ -1087,8 +1087,10 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void ) { -if (waitqueue_active (&log_wait)) -__raw_get_cpu_var (printk_pending) =1 ; +if (waitqueue_active (&log_wait)) { +get_cpu_var (printk_pending) =1 ; +put_cpu_var (printk_pending); + } }
Quoted 5 lines
kernel/printk.c int printk_needs_cpu(int cpu) · line 1096
Quoted 2 lines
kernel/printk.c +0
-0/** But but but, the cpu can still be offlined between writing this state and the next tick happening, right?
Yes, that's what the second patch would fix as a side effect.
Re: [patch 3/3] nohz/s390: fix arch_needs_cpu() return value on offline cpus
In reply to Heiko CarstensQuoted 8 lines · Heiko Carstens
plain text document attachment (003_arch_needs_cpu.diff)
From: Heiko Carstens <[email protected]>This fixes the same problem as described in the patch "nohz: fix
printk_needs_cpu()return value on offline cpus" for thearch_needs_cpu()primitive.
This specific bug was indrocuded with3c5d92a0"nohz: Introducearch_needs_cpu".
Quoted 8 lines
In this case a cpu hotplug notifier is used to fix the issue in order to keep the normal/fast path small. All we need to do is to clear the condition that makes
arch_needs_cpu()return 1 since it is just a performance improvement which is supposed to keep the local tick running for a short period if a cpu goes idle. Nothing special needs to be done except for clearing the condition.Signed-off-by Heiko Carstens <[email protected]>
OK, and s390 seems to be the only architecture making use of that interface.
Did you audit all the other *_needs_cpu() interfaces for this same problem?
Anyway,
Will you take this through the s390?
Quoted 46 lines
--- arch/s390/kernel/vtime.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)arch/s390/kernel/vtime.c +19
-0@@ -19,6 +19,7 @@#include <linux/kernel_stat.h> #include <linux/rcupdate.h> #include <linux/posix-timers.h> +#include <linux/cpu.h> #include <asm/s390_ext.h> #include <asm/timer.h> @@ -566,6 +567,23 @@ void init_cpu_vtimer(void)__ctl_set_bit (0 ,10 ); } +static int __cpuinit s390_nohz_notify (struct notifier_block *self, +unsigned long action,void *hcpu) +{ +struct s390_idle_data *idle; +long cpu = (long ) hcpu; + + idle = &per_cpu (s390_idle, cpu); +switch (action) { +case CPU_DYING : +case CPU_DYING_FROZEN : + idle->nohz_delay =0 ; +default : +break ; + } +return NOTIFY_OK ; +} +void __init vtime_init (void ) {/* request the cpu timer external interrupt */ @@ -574,5 +592,6 @@ void __init vtime_init(void)/* Enable cpu timer interrupts on the boot cpu. */ init_cpu_vtimer (); +cpu_notifier (s390_nohz_notify,0 ); }
Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug
In reply to Heiko CarstensQuoted 4 lines · Heiko Carstens
Quoted 2 lines
But but but, the cpu can still be offlined between writing this state and the next tick happening, right?
Yes, that's what the second patch would fix as a side effect.
Ah, right, ok took them both.
Re: [patch 3/3] nohz/s390: fix arch_needs_cpu() return value on offline cpus
In reply to Peter ZijlstraQuoted 15 lines · Peter Zijlstra
Quoted 14 lines · Heiko Carstens
plain text document attachment (003_arch_needs_cpu.diff)
From: Heiko Carstens <[email protected]>This fixes the same problem as described in the patch "nohz: fix
printk_needs_cpu()return value on offline cpus" for thearch_needs_cpu()primitive.
This specific bug was indrocuded with3c5d92a0"nohz: Introducearch_needs_cpu".In this case a cpu hotplug notifier is used to fix the issue in order to keep the normal/fast path small. All we need to do is to clear the condition that makes
arch_needs_cpu()return 1 since it is just a performance improvement which is supposed to keep the local tick running for a short period if a cpu goes idle. Nothing special needs to be done except for clearing the condition.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Heiko Carstens <[email protected]>OK, and s390 seems to be the only architecture making use of that interface.
Did you audit all the other
*_needs_cpu()interfaces for this same problem?
There's only rcu_needs_cpu() left and rcu indeed has a cpu hotplug notifier. So there _shouldn't_ be any problems with that.
Quoted 3 lines
Acked-by Peter Zijlstra <[email protected]>Will you take this through the s390?
Yes. Thanks!
Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug
In reply to Heiko CarstensQuoted 26 lines · Heiko Carstens
pièce jointe document texte brut (001_printk_preempt.diff)
From: Heiko Carstens <[email protected]>
wake_up_klogd()may get called from preemtible context but uses__raw_get_cpu_var()to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.No idea why that behaviour was introduced with
fa33507a"printk: robustify printk, fix #2" which was supposed to fix a "usingsmp_processor_id()in preemptible" warning.Let's use
get_cpu_var()instead which disables preemption and makes sure that the outlined scenario cannot happen.Signed-off-by Heiko Carstens <[email protected]>--- kernel/printk.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/printk.c +0
-0@@ -1087,8 +1087,10 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void )
kernel/printk.c int printk_needs_cpu(int cpu) · line 1096
Quoted 10 lines
kernel/printk.c +4
-2{ -if (waitqueue_active (&log_wait)) -__raw_get_cpu_var (printk_pending) =1 ; +if (waitqueue_active (&log_wait)) { +get_cpu_var (printk_pending) =1 ; +put_cpu_var (printk_pending); + } }
Please use :
this_cpu_write(printk_pending, 1);
It is faster on x86, and does the right thing too.
Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug
Quoted 13 lines · Eric Dumazet
Review
Quoted 9 lines · Heiko Carstens
void
wake_up_klogd(void)
{(quoted diff) +4
-2- if (waitqueue_active(&log_wait)) - __raw_get_cpu_var(printk_pending) = 1; + if (waitqueue_active(&log_wait)) { + get_cpu_var(printk_pending) = 1; + put_cpu_var(printk_pending); + } }Please use :
Quoted 3 lines
this_cpu_write(printk_pending, 1);It is faster on x86, and does the right thing too.
Ah, right. I wasn't aware that such a thing even exists. Updated patch below:
Subject: [PATCH] printk: fix wake_up_klogd() vs cpu hotplug
From: Heiko Carstens <[email protected]>
wake_up_klogd() may get called from preemtible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.
No idea why that behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.
Let's use this_cpu_write() instead which disables preemption and makes sure that the outlined scenario cannot happen.
--- kernel/printk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/printk.c +1 -1
@@ -1088,7 +1088,7 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void ) {if (waitqueue_active (&log_wait)) -__raw_get_cpu_var (printk_pending) =1 ; +this_cpu_write (printk_pending,1 ); }/**
Eric Dumazet
Acked-byRe: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug
Quoted 48 lines · Heiko Carstens
Quoted 13 lines · Eric Dumazet
Review
Quoted 9 lines · Heiko Carstens
void
wake_up_klogd(void)
{(quoted diff) +4
-2- if (waitqueue_active(&log_wait)) - __raw_get_cpu_var(printk_pending) = 1; + if (waitqueue_active(&log_wait)) { + get_cpu_var(printk_pending) = 1; + put_cpu_var(printk_pending); + } }Please use :
Quoted 3 lines
this_cpu_write(printk_pending, 1);It is faster on x86, and does the right thing too.
Ah, right. I wasn't aware that such a thing even exists. Updated patch below:
Subject: [PATCH] printk: fix
wake_up_klogd()vs cpu hotplugFrom: Heiko Carstens <[email protected]>
wake_up_klogd()may get called from preemtible context but uses__raw_get_cpu_var()to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.No idea why that behaviour was introduced with
fa33507a"printk: robustify printk, fix #2" which was supposed to fix a "usingsmp_processor_id()in preemptible" warning.Let's use
this_cpu_write()instead which disables preemption and makes sure that the outlined scenario cannot happen.Signed-off-by Heiko Carstens <[email protected]>--- kernel/printk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/printk.c int printk_needs_cpu(int cpu) · line 1094
Quoted 8 lines
kernel/printk.c +1
-1@@ -1088,7 +1088,7 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void ) {if (waitqueue_active (&log_wait)) -__raw_get_cpu_var (printk_pending) =1 ; +this_cpu_write (printk_pending,1 ); }/**
[tip:sched/urgent] printk: Fix wake_up_klogd() vs cpu hotplug
In reply to Heiko CarstensCommit-ID: 49f4138346b3cec2706adff02658fe27ceb1e46f Gitweb: http://git.kernel.org/tip/49f4138346b3cec2706adff02658fe27ceb1e46f Author: Heiko Carstens <[email protected]> AuthorDate: Fri, 26 Nov 2010 13:42:47 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 26 Nov 2010 15:03:11 +0100
printk: Fix wake_up_klogd() vs cpu hotplug
wake_up_klogd() may get called from preemptible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.
This buggy behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.
Let's use this_cpu_write() instead which disables preemption and makes sure that the outlined scenario cannot happen.
LKML-Reference: <[email protected]>
--- kernel/printk.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/printk.c +1 -1
@@ -1088,7 +1088,7 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void ) {if (waitqueue_active (&log_wait)) -__raw_get_cpu_var (printk_pending) =1 ; +this_cpu_write (printk_pending,1 ); }/**
[tip:sched/urgent] nohz: Fix printk_needs_cpu() return value on offline cpus
In reply to Heiko CarstensCommit-ID: 61ab25447ad6334a74e32f60efb135a3467223f8 Gitweb: http://git.kernel.org/tip/61ab25447ad6334a74e32f60efb135a3467223f8 Author: Heiko Carstens <[email protected]> AuthorDate: Fri, 26 Nov 2010 13:00:59 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 26 Nov 2010 15:03:12 +0100
nohz: Fix printk_needs_cpu() return value on offline cpus
This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.
printk_needs_cpu() may return 1 if called on offline cpus. When a cpu gets offlined it schedules the idle process which, before killing its own cpu, will call tick_nohz_stop_sched_tick(). That function in turn will call printk_needs_cpu() in order to check if the local tick can be disabled. On offline cpus this function should naturally return 0 since regardless if the tick gets disabled or not the cpu will be dead short after. That is besides the fact that __cpu_disable() should already have made sure that no interrupts on the offlined cpu will be delivered anyway.
In this case it prevents tick_nohz_stop_sched_tick() to call select_nohz_load_balancer(). No idea if that really is a problem. However what made me debug this is that on 2.6.32 the function get_nohz_load_balancer() is used within __mod_timer() to select a cpu on which a timer gets enqueued. If printk_needs_cpu() returns 1 then the nohz_load_balancer cpu doesn't get updated when a cpu gets offlined. It may contain the cpu number of an offline cpu. In turn timers get enqueued on an offline cpu and not very surprisingly they never expire and cause system hangs.
This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.
Easiest way to fix this is just to test if the current cpu is offline and call printk_tick() directly which clears the condition.
Alternatively I tried a cpu hotplug notifier which would clear the condition, however between calling the notifier function and printk_needs_cpu() something could have called printk() again and the problem is back again. This seems to be the safest fix.
LKML-Reference: <[email protected]>
--- kernel/printk.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/printk.c +2 -0
@@ -1082,6 +1082,8 @@ void printk_tick(void)int printk_needs_cpu (int cpu) { +if (unlikely (cpu_is_offline (cpu))) +printk_tick ();return per_cpu (printk_pending, cpu); }
PATCH 1/1 printk: use this_cpu_{read|write} api on printk_pending
Le vendredi 26 novembre 2010 à 15:02 +0000, tip-bot for Heiko Carstens a écrit :
Quoted 7 lines
Commit-ID: 61ab25447ad6334a74e32f60efb135a3467223f8 Gitweb: http://git.kernel.org/tip/61ab25447ad6334a74e32f60efb135a3467223f8 Author: Heiko Carstens <[email protected]> AuthorDate: Fri, 26 Nov 2010 13:00:59 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 26 Nov 2010 15:03:12 +0100
...
kernel/printk.c void printk_tick(void) · line 1089
Quoted 13 lines
kernel/printk.c +2
-0@@ -1082,6 +1082,8 @@ void printk_tick(void)int printk_needs_cpu (int cpu) { +if (unlikely (cpu_is_offline (cpu))) +printk_tick ();return per_cpu (printk_pending, cpu); }
The unlikely() was not needed, since we already have :
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))
Anyway, this patch implies 'cpu' is always the current cpu.
If so, we can have a followup patch to use modern this_cpu api.
Thanks
[PATCH] printk: use this_cpu_{read|write} api on printk_pending
__get_cpu_var() is a bit inefficient, lets use __this_cpu_read() and __this_cpu_write() to manipulate printk_pending.
printk_needs_cpu(cpu) is called only for the current cpu :
Use faster __this_cpu_read().
remove the redondant unlikely on (cpu_is_offline(cpu)) test
# size kernel/printk.o* text data bss dec hex filename 9942 756 263488 274186 42f0a kernel/printk.o.new 9990 756 263488 274234 42f3a kernel/printk.o.old
--- kernel/printk.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/printk.c +4 -4
@@ -1074,17 +1074,17 @@ static DEFINE_PER_CPU(int, printk_pending);void printk_tick (void ) { -if (__get_cpu_var (printk_pending)) { -__get_cpu_var (printk_pending) =0 ; +if (__this_cpu_read (printk_pending)) { +__this_cpu_write (printk_pending,0 );wake_up_interruptible (&log_wait); } }int printk_needs_cpu (int cpu) { -if (unlikely (cpu_is_offline (cpu))) +if (cpu_is_offline (cpu))printk_tick (); -return per_cpu (printk_pending, cpu); +return __this_cpu_read (printk_pending); }void wake_up_klogd (void )
Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending
Quoted 8 lines · Eric Dumazet
[PATCH] printk: use this_cpu_{read|write} api on
printk_pending
__get_cpu_var()is a bit inefficient, lets use__this_cpu_read()and__this_cpu_write()to manipulateprintk_pending.
printk_needs_cpu(cpu)is called only for the current cpu :
Use faster__this_cpu_read().
Quoted 6 lines
remove the redondant unlikely on (
cpu_is_offline(cpu)) test# size kernel/printk.o* text data bss dec hex filename 9942 756 263488 274186 42f0a kernel/printk.o.new 9990 756 263488 274234 42f3a kernel/printk.o.old
Thanks, queued!
Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending
In reply to Eric Dumazetkernel/printk.c static DEFINE_PER_CPU(int, printk_pending); · line 1081
Quoted 13 lines · Eric Dumazet
kernel/printk.c +2
-2@@ -1074,17 +1074,17 @@ static DEFINE_PER_CPU(int, printk_pending);void printk_tick (void ) { -if (__get_cpu_var (printk_pending)) { -__get_cpu_var (printk_pending) =0 ; +if (__this_cpu_read (printk_pending)) { +__this_cpu_write (printk_pending,0 );wake_up_interruptible (&log_wait); } }
The above looks ok.
kernel/printk.c static DEFINE_PER_CPU(int, printk_pending); · line 1088
Quoted 8 lines
kernel/printk.c +2
-2int printk_needs_cpu (int cpu) { -if (unlikely (cpu_is_offline (cpu))) +if (cpu_is_offline (cpu))printk_tick (); -return per_cpu (printk_pending, cpu); +return __this_cpu_read (printk_pending); }
This only works if printk_needs_cpu is always passed the current cpu. The check for cpu_is_offline suggests that the function is also used for other cpus. The __this_cpu_read() will then get the printk_pending stat for the current cpu and not for the specified cpu.
Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending
Quoted 6 lines · Christoph Lameter
This only works if
printk_needs_cpuis always passed the current cpu. The check forcpu_is_offlinesuggests that the function is also used for other cpus. The__this_cpu_read()will then get theprintk_pendingstat for the current cpu and not for the specified cpu.
I guess you read the changelog too fast ;)
I posted this patch after noticing printk_needs_cpu() was always used for the current cpu.
We have other functions around, that always work for the current cpu.
We pass cpu as a parameter, mostly because smp_processor_id() was a bit expensive in old kernels, and is still expensive because of sanity tests.
Now we have this_cpu_ api, we probably can do a cleanup to avoid now useless parameter passing.
If we had this_cpu_is_online() function, we could avoid (int cpu) param in printk_needs_cpu()
Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending
Quoted 1 line · Eric Dumazet
I guess you read the changelog too fast ;)
No I only read the code.... ;-)
Quoted 2 lines
I posted this patch after noticing
printk_needs_cpu()was always used for the current cpu.
Ok.
Quoted 1 line
We have other functions around, that always work for the current cpu.
Ok.
Quoted 3 lines
We pass cpu as a parameter, mostly because
smp_processor_id()was a bit expensive in old kernels, and is still expensive because of sanity tests.
I thought sanity tests do not apply for performance testing scenarios?
smp_processor_id() is usually a simple per cpu data access and very fast.
Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending
Quoted 2 lines · Christoph Lameter
I thought sanity tests do not apply for performance testing scenarios?
CONFIG_DEBUG_PREEMPT
Quoted 2 lines
smp_processor_id()is usually a simple per cpu data access and very fast.
Sure, but when its already in a register, its better to use the register ;)
Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Christoph Lameter
I thought sanity tests do not apply for performance testing scenarios?
CONFIG_DEBUG_PREEMPT
You really enable that option for performance testing?
Quoted 5 lines
Quoted 2 lines
smp_processor_id()is usually a simple per cpu data access and very fast.Sure, but when its already in a register, its better to use the register ;)
Well that increases register pressure in the caller. There may be no need to handle the current processor number in the caller.
Re: [patch 3/3] nohz/s390: fix arch_needs_cpu() return value on offline cpus
In reply to Peter ZijlstraQuoted 15 lines · Peter Zijlstra
Quoted 14 lines · Heiko Carstens
plain text document attachment (003_arch_needs_cpu.diff)
From: Heiko Carstens <[email protected]>This fixes the same problem as described in the patch "nohz: fix
printk_needs_cpu()return value on offline cpus" for thearch_needs_cpu()primitive.
This specific bug was indrocuded with3c5d92a0"nohz: Introducearch_needs_cpu".In this case a cpu hotplug notifier is used to fix the issue in order to keep the normal/fast path small. All we need to do is to clear the condition that makes
arch_needs_cpu()return 1 since it is just a performance improvement which is supposed to keep the local tick running for a short period if a cpu goes idle. Nothing special needs to be done except for clearing the condition.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Heiko Carstens <[email protected]>OK, and s390 seems to be the only architecture making use of that interface.
Did you audit all the other
*_needs_cpu()interfaces for this same problem?
Yes I did, but guess what.. get_next_timer_interrupt() bites us as well *sigh*.
Here is yet another nohz vs cpu hotplug patch which should also go into stable.
Subject: [PATCH] nohz: fix get_next_timer_interrupt() vs cpu hotplug
From: Heiko Carstens <[email protected]>
This fixes a bug as seen on 2.6.32 based kernels where timers got enqueued on offline cpus.
If a cpu goes offline it might still have pending timers. These will be migrated during CPU_DEAD handling after the cpu is offline. However while the cpu is going offline it will schedule the idle task which will then call tick_nohz_stop_sched_tick().
That function in turn will call get_next_timer_intterupt() to figure out if the tick of the cpu can be stopped or not. If it turns out that the next tick is just one jiffy off (delta_jiffies == 1) tick_nohz_stop_sched_tick() incorrectly assumes that the tick should not stop and takes an early exit and thus it won't update the load balancer cpu.
Just afterwards the cpu will be killed and the load balancer cpu could be the offline cpu.
On 2.6.32 based kernel get_nohz_load_balancer() gets called to decide on which cpu a timer should be enqueued (see __mod_timer()). Which leads to the possibility that timers get enqueued on an offline cpu. These will never expire and can cause a system hang.
This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.
The easiest and probably safest fix seems to be to let get_next_timer_interrupt() just lie and let it say there isn't any pending timer if the current cpu is offline.
I also thought of moving migrate_[hr]timers() from CPU_DEAD to CPU_DYING, but seeing that there already have been fixes at least in the hrtimer code in this area I'm afraid that this could add new subtle bugs.
---
kernel/timer.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/timer.c +6 -0
@@ -1252,6 +1252,12 @@ unsigned long get_next_timer_interrupt(unsigned long now)struct tvec_base *base =__get_cpu_var (tvec_bases);unsigned long expires; +/* +* Pretend that there is no timer pending if the cpu is offline. +* Possible pending timers will be migrated later to an active cpu. +*/ +if (cpu_is_offline (smp_processor_id ())) +return now +NEXT_TIMER_MAX_DELTA ;spin_lock (&base->lock);if (time_before_eq (base->next_timer, base->timer_jiffies)) base->next_timer =__next_timer_interrupt (base);
Re: [patch 3/3] nohz/s390: fix arch_needs_cpu() return value on offline cpus
Quoted 37 lines · Heiko Carstens
Subject: [PATCH] nohz: fix
get_next_timer_interrupt()vs cpu hotplugFrom: Heiko Carstens <[email protected]>
This fixes a bug as seen on 2.6.32 based kernels where timers got enqueued on offline cpus.
If a cpu goes offline it might still have pending timers. These will be migrated during
CPU_DEADhandling after the cpu is offline. However while the cpu is going offline it will schedule the idle task which will then calltick_nohz_stop_sched_tick().
That function in turn will callget_next_timer_intterupt()to figure out if the tick of the cpu can be stopped or not. If it turns out that the next tick is just one jiffy off (delta_jiffies== 1)tick_nohz_stop_sched_tick()incorrectly assumes that the tick should not stop and takes an early exit and thus it won't update the load balancer cpu.
Just afterwards the cpu will be killed and the load balancer cpu could be the offline cpu.
On 2.6.32 based kernelget_nohz_load_balancer()gets called to decide on which cpu a timer should be enqueued (see__mod_timer()). Which leads to the possibility that timers get enqueued on an offline cpu. These will never expire and can cause a system hang.This has been observed 2.6.32 kernels. On current kernels
__mod_timer()usesget_nohz_timer_target()which doesn't have that problem. However there might be other problems because of the too early exittick_nohz_stop_sched_tick()in case a cpu goes offline.The easiest and probably safest fix seems to be to let
get_next_timer_interrupt()just lie and let it say there isn't any pending timer if the current cpu is offline.
I also thought of moving migrate_[hr]timers() fromCPU_DEADtoCPU_DYING, but seeing that there already have been fixes at least in the hrtimer code in this area I'm afraid that this could add new subtle bugs.
Quoted 3 lines
Signed-off-by Heiko Carstens <[email protected]>---
Thanks Heiko, I queued this one as well.
Re: [stable] [patch 2/3] nohz: fix printk_needs_cpu() return value on offline cpus
In reply to Peter ZijlstraQuoted 48 lines · Peter Zijlstra
Quoted 47 lines · Heiko Carstens
plain text document attachment (002_printk_needs_cpu.diff)
From: Heiko Carstens <[email protected]>This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.
printk_needs_cpu()may return 1 if called on offline cpus. When a cpu gets offlined it schedules the idle process which, before killing its own cpu, will calltick_nohz_stop_sched_tick().
That function in turn will callprintk_needs_cpu()in order to check if the local tick can be disabled. On offline cpus this function should naturally return 0 since regardless if the tick gets disabled or not the cpu will be dead short after. That is besides the fact that__cpu_disable()should already have made sure that no interrupts on the offlined cpu will be delivered anyway.In this case it prevents
tick_nohz_stop_sched_tick()to callselect_nohz_load_balancer(). No idea if that really is a problem. However what made me debug this is that on 2.6.32 the functionget_nohz_load_balancer()is used within__mod_timer()to select a cpu on which a timer gets enqueued. Ifprintk_needs_cpu()returns 1 then thenohz_load_balancercpu doesn't get updated when a cpu gets offlined. It may contain the cpu number of an offline cpu. In turn timers get enqueued on an offline cpu and not very surprisingly they never expire and cause system hangs.This has been observed 2.6.32 kernels. On current kernels
__mod_timer()usesget_nohz_timer_target()which doesn't have that problem. However there might be other problems because of the too early exittick_nohz_stop_sched_tick()in case a cpu goes offline.Easiest way to fix this is just to test if the current cpu is offline and call
printk_tick()directly which clears the condition.Alternatively I tried a cpu hotplug notifier which would clear the condition, however between calling the notifier function and
printk_needs_cpu()something could have calledprintk()again and the problem is back again. This seems to be the safest fix.Signed-off-by Heiko Carstens <[email protected]>--- kernel/printk.c | 2 ++ 1 file changed, 2 insertions(+)kernel/printk.c +0
-0@@ -1082,6 +1082,8 @@ void printk_tick(void)
Quoted 9 lines
kernel/printk.c void printk_tick(void) · line 1089
Quoted 7 lines
kernel/printk.c +2
-0int printk_needs_cpu (int cpu) { +if (unlikely (cpu_is_offline (cpu))) +printk_tick ();return per_cpu (printk_pending, cpu); }Nice,.. applied.
Is this going to make it into .37, or is it going to wait until .38?
thanks,
greg k-h
Re: [stable] [patch 2/3] nohz: fix printk_needs_cpu() return value on offline cpus
Quoted 7 lines · Greg KH
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Heiko Carstens
plain text document attachment (002_printk_needs_cpu.diff)
From: Heiko Carstens <[email protected]>This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.
[...]
Quoted 2 lines
Is this going to make it into .37, or is it going to wait until .38?
I hope it will go into .37. The same should be true for the get_next_timer_interrupt() which addresses the same problem. Peter, Ingo?
Re: [stable] [patch 2/3] nohz: fix printk_needs_cpu() return value on offline cpus
Quoted 8 lines · Heiko Carstens
Quoted 7 lines · Greg KH
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Heiko Carstens
plain text document attachment (002_printk_needs_cpu.diff)
From: Heiko Carstens <[email protected]>This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.
Quoted 7 lines
[...]
Quoted 2 lines
Is this going to make it into .37, or is it going to wait until .38?
I hope it will go into .37. The same should be true for the
get_next_timer_interrupt()which addresses the same problem. Peter, Ingo?
Ingo just send the pull request to Linus for two of the three patches, I'll try and feed him the last one today.
[tip:sched/urgent] nohz: Fix get_next_timer_interrupt() vs cpu hotplug
In reply to Heiko CarstensCommit-ID: dbd87b5af055a0cc9bba17795c9a2b0d17795389 Gitweb: http://git.kernel.org/tip/dbd87b5af055a0cc9bba17795c9a2b0d17795389 Author: Heiko Carstens <[email protected]> AuthorDate: Wed, 1 Dec 2010 10:11:09 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 8 Dec 2010 20:15:07 +0100
nohz: Fix get_next_timer_interrupt() vs cpu hotplug
This fixes a bug as seen on 2.6.32 based kernels where timers got enqueued on offline cpus.
If a cpu goes offline it might still have pending timers. These will be migrated during CPU_DEAD handling after the cpu is offline. However while the cpu is going offline it will schedule the idle task which will then call tick_nohz_stop_sched_tick().
That function in turn will call get_next_timer_intterupt() to figure out if the tick of the cpu can be stopped or not. If it turns out that the next tick is just one jiffy off (delta_jiffies == 1) tick_nohz_stop_sched_tick() incorrectly assumes that the tick should not stop and takes an early exit and thus it won't update the load balancer cpu.
Just afterwards the cpu will be killed and the load balancer cpu could be the offline cpu.
On 2.6.32 based kernel get_nohz_load_balancer() gets called to decide on which cpu a timer should be enqueued (see __mod_timer()). Which leads to the possibility that timers get enqueued on an offline cpu. These will never expire and can cause a system hang.
This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.
The easiest and probably safest fix seems to be to let get_next_timer_interrupt() just lie and let it say there isn't any pending timer if the current cpu is offline.
I also thought of moving migrate_[hr]timers() from CPU_DEAD to CPU_DYING, but seeing that there already have been fixes at least in the hrtimer code in this area I'm afraid that this could add new subtle bugs.
LKML-Reference: <[email protected]>
--- kernel/timer.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
kernel/timer.c +6 -0
@@ -1252,6 +1252,12 @@ unsigned long get_next_timer_interrupt(unsigned long now)struct tvec_base *base =__get_cpu_var (tvec_bases);unsigned long expires; +/* +* Pretend that there is no timer pending if the cpu is offline. +* Possible pending timers will be migrated later to an active cpu. +*/ +if (cpu_is_offline (smp_processor_id ())) +return now +NEXT_TIMER_MAX_DELTA ;spin_lock (&base->lock);if (time_before_eq (base->next_timer, base->timer_jiffies)) base->next_timer =__next_timer_interrupt (base);
[tip:sched/core] printk: Use this_cpu_{read|write} api on printk_pending
In reply to Eric DumazetCommit-ID: 40dc11ffb35e8c4e8fa71092048e0f8de9db758c Gitweb: http://git.kernel.org/tip/40dc11ffb35e8c4e8fa71092048e0f8de9db758c Author: Eric Dumazet <[email protected]> AuthorDate: Fri, 26 Nov 2010 17:22:16 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 8 Dec 2010 20:16:01 +0100
printk: Use this_cpu_{read|write} api on printk_pending
__get_cpu_var() is a bit inefficient, lets use __this_cpu_read() and __this_cpu_write() to manipulate printk_pending.
printk_needs_cpu(cpu) is called only for the current cpu :
Use faster __this_cpu_read().
Remove the redundant unlikely on (cpu_is_offline(cpu)) test:
# size kernel/printk.o* text data bss dec hex filename 9942 756 263488 274186 42f0a kernel/printk.o.new 9990 756 263488 274234 42f3a kernel/printk.o.old
LKML-Reference: <1290788536.2855.237.camel@edumazet-laptop>
--- kernel/printk.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/printk.c +4 -4
@@ -1074,17 +1074,17 @@ static DEFINE_PER_CPU(int, printk_pending);void printk_tick (void ) { -if (__get_cpu_var (printk_pending)) { -__get_cpu_var (printk_pending) =0 ; +if (__this_cpu_read (printk_pending)) { +__this_cpu_write (printk_pending,0 );wake_up_interruptible (&log_wait); } }int printk_needs_cpu (int cpu) { -if (unlikely (cpu_is_offline (cpu))) +if (cpu_is_offline (cpu))printk_tick (); -return per_cpu (printk_pending, cpu); +return __this_cpu_read (printk_pending); }void wake_up_klogd (void )
Re: [tip:sched/core] printk: Use this_cpu_{read|write} api on printk_pending
Quoted 1 line · tip-bot for Eric Dumazet
printk: Use this_cpu_{read|write} api on
printk_pending
One of the patches in Tejun's percpu tree includes the same modifications.
Re: [tip:sched/core] printk: Use this_cpu_{read|write} api on printk_pending
Quoted 7 lines · Christoph Lameter
Quoted 1 line · tip-bot for Eric Dumazet
printk: Use this_cpu_{read|write} api on
printk_pendingOne of the patches in Tejun's percpu tree includes the same modifications.
Its a bit strange, I sent mine a long time ago when reviewing Heiko Carstens patch. You even commented on it.
https://patchwork.kernel.org/patch/359182/
What happened then, you included my patch on one of yours ?
Re: [tip:sched/core] printk: Use this_cpu_{read|write} api on printk_pending
Quoted 6 lines · Eric Dumazet
Its a bit strange, I sent mine a long time ago when reviewing Heiko Carstens patch. You even commented on it.
https://patchwork.kernel.org/patch/359182/
What happened then, you included my patch on one of yours ?
I just went through the tree and changed all __get_cpu_var. Out came a single core patch that also covered this earlier case.