Skip to contentSkip to conversation

three cpu hotplug fixes

PATCH In review Heiko Carstens 5 participants 32 messages Nov 26, 2010 – Dec 9, 2010 s390

lore
Share & tools
lore
#0 lore raw

cover three cpu hotplug fixes

Three fixes for some very rare to find cpu hotplug bugs.

#1 lore raw

PATCH 1/3 printk: fix wake_up_klogd() vs cpu hotplug

From: Heiko Carstens <[email protected]>

wake_up_klogd() may get called from preemtible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.

No idea why that behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.

Let's use get_cpu_var() instead which disables preemption and makes sure that the outlined scenario cannot happen.

---
 kernel/printk.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/printk.c +4 -2
@@ -1087,8 +1087,10 @@ int printk_needs_cpu(int cpu)
 
 void wake_up_klogd(void)
 {
-	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+	if (waitqueue_active(&log_wait)) {
+		get_cpu_var(printk_pending) = 1;
+		put_cpu_var(printk_pending);
+	}
 }
 
 /**
#2 lore raw

Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug

Quoted 27 lines · Heiko Carstens

plain text document attachment (001_printk_preempt.diff)
From: Heiko Carstens <[email protected]>

wake_up_klogd() may get called from preemtible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.

No idea why that behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.

Let's use get_cpu_var() instead which disables preemption and makes sure that the outlined scenario cannot happen.

---
 kernel/printk.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/printk.c +0 -0
@@ -1087,8 +1087,10 @@ int printk_needs_cpu(int cpu)
 
 void wake_up_klogd(void)

kernel/printk.c int printk_needs_cpu(int cpu) · line 1096

Quoted 10 lines
kernel/printk.c +4 -2
 {
-	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+	if (waitqueue_active(&log_wait)) {
+		get_cpu_var(printk_pending) = 1;
+		put_cpu_var(printk_pending);
+	}
 }
 
 /**

But but but, the cpu can still be offlined between writing this state and the next tick happening, right?

#3 lore raw

Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug

Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · Heiko Carstens

plain text document attachment (001_printk_preempt.diff)
From: Heiko Carstens <[email protected]>

wake_up_klogd() may get called from preemtible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.

No idea why that behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.

Let's use get_cpu_var() instead which disables preemption and makes sure that the outlined scenario cannot happen.

---
 kernel/printk.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/printk.c +4 -2
@@ -1087,8 +1087,10 @@ int printk_needs_cpu(int cpu)
 
 void wake_up_klogd(void)
 {
-	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+	if (waitqueue_active(&log_wait)) {
+		get_cpu_var(printk_pending) = 1;
+		put_cpu_var(printk_pending);
+	}
 }
 
Quoted 5 lines

kernel/printk.c int printk_needs_cpu(int cpu) · line 1096

Quoted 2 lines
kernel/printk.c +0 -0
 /**

But but but, the cpu can still be offlined between writing this state and the next tick happening, right?

Yes, that's what the second patch would fix as a side effect.

#4 lore raw

Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug

Quoted 4 lines · Heiko Carstens
Quoted 2 lines

But but but, the cpu can still be offlined between writing this state and the next tick happening, right?

Yes, that's what the second patch would fix as a side effect.

Ah, right, ok took them both.

#5 lore raw

Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug

In reply to Heiko Carstens
Quoted 26 lines · Heiko Carstens

pièce jointe document texte brut (001_printk_preempt.diff)
From: Heiko Carstens <[email protected]>

wake_up_klogd() may get called from preemtible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.

No idea why that behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.

Let's use get_cpu_var() instead which disables preemption and makes sure that the outlined scenario cannot happen.

---
 kernel/printk.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/printk.c +0 -0
@@ -1087,8 +1087,10 @@ int printk_needs_cpu(int cpu)
 
 void wake_up_klogd(void)

kernel/printk.c int printk_needs_cpu(int cpu) · line 1096

Quoted 10 lines
kernel/printk.c +4 -2
 {
-	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+	if (waitqueue_active(&log_wait)) {
+		get_cpu_var(printk_pending) = 1;
+		put_cpu_var(printk_pending);
+	}
 }
 
 

Please use :

this_cpu_write(printk_pending, 1);

It is faster on x86, and does the right thing too.

#6 lore raw

Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug

Quoted 13 lines · Eric Dumazet

Review

Quoted 9 lines · Heiko Carstens

void wake_up_klogd(void)
{

(quoted diff) +4 -2
-	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+	if (waitqueue_active(&log_wait)) {
+		get_cpu_var(printk_pending) = 1;
+		put_cpu_var(printk_pending);
+	}
 }

Please use :

Quoted 3 lines

this_cpu_write(printk_pending, 1);

It is faster on x86, and does the right thing too.

Ah, right. I wasn't aware that such a thing even exists. Updated patch below:

Subject: [PATCH] printk: fix wake_up_klogd() vs cpu hotplug

From: Heiko Carstens <[email protected]>

wake_up_klogd() may get called from preemtible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.

No idea why that behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.

Let's use this_cpu_write() instead which disables preemption and makes sure that the outlined scenario cannot happen.

---
 kernel/printk.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/printk.c +1 -1
@@ -1088,7 +1088,7 @@ int printk_needs_cpu(int cpu)
 void wake_up_klogd(void)
 {
 	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+		this_cpu_write(printk_pending, 1);
 }
 
 /**
Acked-by
#7 lore raw

Re: [patch 1/3] printk: fix wake_up_klogd() vs cpu hotplug

Quoted 48 lines · Heiko Carstens
Quoted 13 lines · Eric Dumazet

Review

Quoted 9 lines · Heiko Carstens

void wake_up_klogd(void)
{

(quoted diff) +4 -2
-	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+	if (waitqueue_active(&log_wait)) {
+		get_cpu_var(printk_pending) = 1;
+		put_cpu_var(printk_pending);
+	}
 }

Please use :

Quoted 3 lines

this_cpu_write(printk_pending, 1);

It is faster on x86, and does the right thing too.

Ah, right. I wasn't aware that such a thing even exists. Updated patch below:

Subject: [PATCH] printk: fix wake_up_klogd() vs cpu hotplug

From: Heiko Carstens <[email protected]>

wake_up_klogd() may get called from preemtible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.

No idea why that behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.

Let's use this_cpu_write() instead which disables preemption and makes sure that the outlined scenario cannot happen.

---
 kernel/printk.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/printk.c int printk_needs_cpu(int cpu) · line 1094

Quoted 8 lines
kernel/printk.c +1 -1
@@ -1088,7 +1088,7 @@ int printk_needs_cpu(int cpu)
 void wake_up_klogd(void)
 {
 	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+		this_cpu_write(printk_pending, 1);
 }
 
 /**
#8 lore raw

[tip:sched/urgent] printk: Fix wake_up_klogd() vs cpu hotplug

In reply to Heiko Carstens
Commit-ID:  49f4138346b3cec2706adff02658fe27ceb1e46f
Gitweb:     http://git.kernel.org/tip/49f4138346b3cec2706adff02658fe27ceb1e46f
Author:     Heiko Carstens <[email protected]>
AuthorDate: Fri, 26 Nov 2010 13:42:47 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 26 Nov 2010 15:03:11 +0100

printk: Fix wake_up_klogd() vs cpu hotplug

wake_up_klogd() may get called from preemptible context but uses __raw_get_cpu_var() to write to a per cpu variable. If it gets preempted between getting the address and writing to it, the cpu in question could be offline if the process gets scheduled back and hence writes to the per cpu data of an offline cpu.

This buggy behaviour was introduced with fa33507a "printk: robustify printk, fix #2" which was supposed to fix a "using smp_processor_id() in preemptible" warning.

Let's use this_cpu_write() instead which disables preemption and makes sure that the outlined scenario cannot happen.

LKML-Reference: <[email protected]>

---
 kernel/printk.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/printk.c +1 -1
index 9a2264f..cf7588e 100644
@@ -1088,7 +1088,7 @@ int printk_needs_cpu(int cpu)
 void wake_up_klogd(void)
 {
 	if (waitqueue_active(&log_wait))
-		__raw_get_cpu_var(printk_pending) = 1;
+		this_cpu_write(printk_pending, 1);
 }
 
 /**
#9 lore raw

PATCH 2/3 nohz: fix printk_needs_cpu() return value on offline cpus

In reply to Heiko Carstens

From: Heiko Carstens <[email protected]>

This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.

printk_needs_cpu() may return 1 if called on offline cpus. When a cpu gets offlined it schedules the idle process which, before killing its own cpu, will call tick_nohz_stop_sched_tick().
That function in turn will call printk_needs_cpu() in order to check if the local tick can be disabled. On offline cpus this function should naturally return 0 since regardless if the tick gets disabled or not the cpu will be dead short after. That is besides the fact that __cpu_disable() should already have made sure that no interrupts on the offlined cpu will be delivered anyway.

In this case it prevents tick_nohz_stop_sched_tick() to call select_nohz_load_balancer(). No idea if that really is a problem. However what made me debug this is that on 2.6.32 the function get_nohz_load_balancer() is used within __mod_timer() to select a cpu on which a timer gets enqueued. If printk_needs_cpu() returns 1 then the nohz_load_balancer cpu doesn't get updated when a cpu gets offlined. It may contain the cpu number of an offline cpu. In turn timers get enqueued on an offline cpu and not very surprisingly they never expire and cause system hangs.

This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.

Easiest way to fix this is just to test if the current cpu is offline and call printk_tick() directly which clears the condition.

Alternatively I tried a cpu hotplug notifier which would clear the condition, however between calling the notifier function and printk_needs_cpu() something could have called printk() again and the problem is back again. This seems to be the safest fix.

---
 kernel/printk.c |    2 ++
 1 file changed, 2 insertions(+)
kernel/printk.c +2 -0
@@ -1082,6 +1082,8 @@ void printk_tick(void)
 
 int printk_needs_cpu(int cpu)
 {
+	if (unlikely(cpu_is_offline(cpu)))
+		printk_tick();
 	return per_cpu(printk_pending, cpu);
 }
 
#10 lore raw

Re: [patch 2/3] nohz: fix printk_needs_cpu() return value on offline cpus

Quoted 46 lines · Heiko Carstens

plain text document attachment (002_printk_needs_cpu.diff)
From: Heiko Carstens <[email protected]>

This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.

printk_needs_cpu() may return 1 if called on offline cpus. When a cpu gets offlined it schedules the idle process which, before killing its own cpu, will call tick_nohz_stop_sched_tick().
That function in turn will call printk_needs_cpu() in order to check if the local tick can be disabled. On offline cpus this function should naturally return 0 since regardless if the tick gets disabled or not the cpu will be dead short after. That is besides the fact that __cpu_disable() should already have made sure that no interrupts on the offlined cpu will be delivered anyway.

In this case it prevents tick_nohz_stop_sched_tick() to call select_nohz_load_balancer(). No idea if that really is a problem. However what made me debug this is that on 2.6.32 the function get_nohz_load_balancer() is used within __mod_timer() to select a cpu on which a timer gets enqueued. If printk_needs_cpu() returns 1 then the nohz_load_balancer cpu doesn't get updated when a cpu gets offlined. It may contain the cpu number of an offline cpu. In turn timers get enqueued on an offline cpu and not very surprisingly they never expire and cause system hangs.

This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.

Easiest way to fix this is just to test if the current cpu is offline and call printk_tick() directly which clears the condition.

Alternatively I tried a cpu hotplug notifier which would clear the condition, however between calling the notifier function and printk_needs_cpu() something could have called printk() again and the problem is back again. This seems to be the safest fix.

---
 kernel/printk.c |    2 ++
 1 file changed, 2 insertions(+)

kernel/printk.c void printk_tick(void) · line 1089

Quoted 8 lines
kernel/printk.c +2 -0
@@ -1082,6 +1082,8 @@ void printk_tick(void)
 
 int printk_needs_cpu(int cpu)
 {
+	if (unlikely(cpu_is_offline(cpu)))
+		printk_tick();
 	return per_cpu(printk_pending, cpu);
 }
 

Nice,.. applied.

#11 lore raw

Re: [stable] [patch 2/3] nohz: fix printk_needs_cpu() return value on offline cpus

Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Heiko Carstens

plain text document attachment (002_printk_needs_cpu.diff)
From: Heiko Carstens <[email protected]>

This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.

printk_needs_cpu() may return 1 if called on offline cpus. When a cpu gets offlined it schedules the idle process which, before killing its own cpu, will call tick_nohz_stop_sched_tick().
That function in turn will call printk_needs_cpu() in order to check if the local tick can be disabled. On offline cpus this function should naturally return 0 since regardless if the tick gets disabled or not the cpu will be dead short after. That is besides the fact that __cpu_disable() should already have made sure that no interrupts on the offlined cpu will be delivered anyway.

In this case it prevents tick_nohz_stop_sched_tick() to call select_nohz_load_balancer(). No idea if that really is a problem. However what made me debug this is that on 2.6.32 the function get_nohz_load_balancer() is used within __mod_timer() to select a cpu on which a timer gets enqueued. If printk_needs_cpu() returns 1 then the nohz_load_balancer cpu doesn't get updated when a cpu gets offlined. It may contain the cpu number of an offline cpu. In turn timers get enqueued on an offline cpu and not very surprisingly they never expire and cause system hangs.

This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.

Easiest way to fix this is just to test if the current cpu is offline and call printk_tick() directly which clears the condition.

Alternatively I tried a cpu hotplug notifier which would clear the condition, however between calling the notifier function and printk_needs_cpu() something could have called printk() again and the problem is back again. This seems to be the safest fix.

---
 kernel/printk.c |    2 ++
 1 file changed, 2 insertions(+)
kernel/printk.c +0 -0
@@ -1082,6 +1082,8 @@ void printk_tick(void)
 
Quoted 9 lines

kernel/printk.c void printk_tick(void) · line 1089

Quoted 7 lines
kernel/printk.c +2 -0
 int printk_needs_cpu(int cpu)
 {
+	if (unlikely(cpu_is_offline(cpu)))
+		printk_tick();
 	return per_cpu(printk_pending, cpu);
 }
 

Nice,.. applied.

Is this going to make it into .37, or is it going to wait until .38?

thanks,

greg k-h

#12 lore raw

Re: [stable] [patch 2/3] nohz: fix printk_needs_cpu() return value on offline cpus

Quoted 7 lines · Greg KH
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Heiko Carstens

plain text document attachment (002_printk_needs_cpu.diff)
From: Heiko Carstens <[email protected]>

This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.

[...]

Quoted 2 lines

Is this going to make it into .37, or is it going to wait until .38?

I hope it will go into .37. The same should be true for the get_next_timer_interrupt() which addresses the same problem. Peter, Ingo?

#13 lore raw

Re: [stable] [patch 2/3] nohz: fix printk_needs_cpu() return value on offline cpus

Quoted 8 lines · Heiko Carstens
Quoted 7 lines · Greg KH
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Heiko Carstens

plain text document attachment (002_printk_needs_cpu.diff)
From: Heiko Carstens <[email protected]>

This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.

Quoted 7 lines

[...]

Quoted 2 lines

Is this going to make it into .37, or is it going to wait until .38?

I hope it will go into .37. The same should be true for the get_next_timer_interrupt() which addresses the same problem. Peter, Ingo?

Ingo just send the pull request to Linus for two of the three patches, I'll try and feed him the last one today.

#14 lore raw

[tip:sched/urgent] nohz: Fix printk_needs_cpu() return value on offline cpus

In reply to Heiko Carstens
Commit-ID:  61ab25447ad6334a74e32f60efb135a3467223f8
Gitweb:     http://git.kernel.org/tip/61ab25447ad6334a74e32f60efb135a3467223f8
Author:     Heiko Carstens <[email protected]>
AuthorDate: Fri, 26 Nov 2010 13:00:59 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 26 Nov 2010 15:03:12 +0100

nohz: Fix printk_needs_cpu() return value on offline cpus

This patch fixes a hang observed with 2.6.32 kernels where timers got enqueued on offline cpus.

printk_needs_cpu() may return 1 if called on offline cpus. When a cpu gets offlined it schedules the idle process which, before killing its own cpu, will call tick_nohz_stop_sched_tick(). That function in turn will call printk_needs_cpu() in order to check if the local tick can be disabled. On offline cpus this function should naturally return 0 since regardless if the tick gets disabled or not the cpu will be dead short after. That is besides the fact that __cpu_disable() should already have made sure that no interrupts on the offlined cpu will be delivered anyway.

In this case it prevents tick_nohz_stop_sched_tick() to call select_nohz_load_balancer(). No idea if that really is a problem. However what made me debug this is that on 2.6.32 the function get_nohz_load_balancer() is used within __mod_timer() to select a cpu on which a timer gets enqueued. If printk_needs_cpu() returns 1 then the nohz_load_balancer cpu doesn't get updated when a cpu gets offlined. It may contain the cpu number of an offline cpu. In turn timers get enqueued on an offline cpu and not very surprisingly they never expire and cause system hangs.

This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.

Easiest way to fix this is just to test if the current cpu is offline and call printk_tick() directly which clears the condition.

Alternatively I tried a cpu hotplug notifier which would clear the condition, however between calling the notifier function and printk_needs_cpu() something could have called printk() again and the problem is back again. This seems to be the safest fix.

LKML-Reference: <[email protected]>

---
 kernel/printk.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/printk.c +2 -0
index cf7588e..a23315d 100644
@@ -1082,6 +1082,8 @@ void printk_tick(void)
 
 int printk_needs_cpu(int cpu)
 {
+	if (unlikely(cpu_is_offline(cpu)))
+		printk_tick();
 	return per_cpu(printk_pending, cpu);
 }
 
#15 lore raw

PATCH 1/1 printk: use this_cpu_{read|write} api on printk_pending

Le vendredi 26 novembre 2010 à 15:02 +0000, tip-bot for Heiko Carstens a écrit :

Quoted 7 lines
Commit-ID:  61ab25447ad6334a74e32f60efb135a3467223f8
Gitweb:     http://git.kernel.org/tip/61ab25447ad6334a74e32f60efb135a3467223f8
Author:     Heiko Carstens <[email protected]>
AuthorDate: Fri, 26 Nov 2010 13:00:59 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 26 Nov 2010 15:03:12 +0100

...

kernel/printk.c void printk_tick(void) · line 1089

Quoted 13 lines
kernel/printk.c +2 -0
index cf7588e..a23315d 100644
@@ -1082,6 +1082,8 @@ void printk_tick(void)
 
 int printk_needs_cpu(int cpu)
 {
+	if (unlikely(cpu_is_offline(cpu)))
+		printk_tick();
 	return per_cpu(printk_pending, cpu);
 }
 

The unlikely() was not needed, since we already have :
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))

Anyway, this patch implies 'cpu' is always the current cpu.

If so, we can have a followup patch to use modern this_cpu api.

Thanks

[PATCH] printk: use this_cpu_{read|write} api on printk_pending

__get_cpu_var() is a bit inefficient, lets use __this_cpu_read() and __this_cpu_write() to manipulate printk_pending.

printk_needs_cpu(cpu) is called only for the current cpu :
Use faster __this_cpu_read().

remove the redondant unlikely on (cpu_is_offline(cpu)) test

# size kernel/printk.o*
   text	   data	    bss	    dec	    hex	filename
   9942	    756	 263488	 274186	  42f0a	kernel/printk.o.new
   9990	    756	 263488	 274234	  42f3a	kernel/printk.o.old
Cc Peter Zijlstra <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Christoph Lameter <[email protected]>
---
 kernel/printk.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/printk.c +4 -4
index a23315d..ab3ffc5 100644
@@ -1074,17 +1074,17 @@ static DEFINE_PER_CPU(int, printk_pending);
 
 void printk_tick(void)
 {
-	if (__get_cpu_var(printk_pending)) {
-		__get_cpu_var(printk_pending) = 0;
+	if (__this_cpu_read(printk_pending)) {
+		__this_cpu_write(printk_pending, 0);
 		wake_up_interruptible(&log_wait);
 	}
 }
 
 int printk_needs_cpu(int cpu)
 {
-	if (unlikely(cpu_is_offline(cpu)))
+	if (cpu_is_offline(cpu))
 		printk_tick();
-	return per_cpu(printk_pending, cpu);
+	return __this_cpu_read(printk_pending);
 }
 
 void wake_up_klogd(void)
#16 lore raw

Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending

Quoted 8 lines · Eric Dumazet

[PATCH] printk: use this_cpu_{read|write} api on printk_pending

__get_cpu_var() is a bit inefficient, lets use __this_cpu_read() and __this_cpu_write() to manipulate printk_pending.

printk_needs_cpu(cpu) is called only for the current cpu :
Use faster __this_cpu_read().

Quoted 6 lines

remove the redondant unlikely on (cpu_is_offline(cpu)) test

# size kernel/printk.o*
   text    data     bss     dec     hex filename
   9942     756  263488  274186   42f0a kernel/printk.o.new
   9990     756  263488  274234   42f3a kernel/printk.o.old

Thanks, queued!

#17 lore raw

Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending

In reply to Eric Dumazet

kernel/printk.c static DEFINE_PER_CPU(int, printk_pending); · line 1081

Quoted 13 lines · Eric Dumazet
kernel/printk.c +2 -2
@@ -1074,17 +1074,17 @@ static DEFINE_PER_CPU(int, printk_pending);
 
 void printk_tick(void)
 {
-	if (__get_cpu_var(printk_pending)) {
-		__get_cpu_var(printk_pending) = 0;
+	if (__this_cpu_read(printk_pending)) {
+		__this_cpu_write(printk_pending, 0);
 		wake_up_interruptible(&log_wait);
 	}
 }

The above looks ok.

kernel/printk.c static DEFINE_PER_CPU(int, printk_pending); · line 1088

Quoted 8 lines
kernel/printk.c +2 -2
 int printk_needs_cpu(int cpu)
 {
-	if (unlikely(cpu_is_offline(cpu)))
+	if (cpu_is_offline(cpu))
 		printk_tick();
-	return per_cpu(printk_pending, cpu);
+	return __this_cpu_read(printk_pending);
 }

This only works if printk_needs_cpu is always passed the current cpu. The check for cpu_is_offline suggests that the function is also used for other cpus. The __this_cpu_read() will then get the printk_pending stat for the current cpu and not for the specified cpu.

#18 lore raw

Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending

Quoted 6 lines · Christoph Lameter

This only works if printk_needs_cpu is always passed the current cpu. The check for cpu_is_offline suggests that the function is also used for other cpus. The __this_cpu_read() will then get the printk_pending stat for the current cpu and not for the specified cpu.

I guess you read the changelog too fast ;)

I posted this patch after noticing printk_needs_cpu() was always used for the current cpu.

We have other functions around, that always work for the current cpu.

We pass cpu as a parameter, mostly because smp_processor_id() was a bit expensive in old kernels, and is still expensive because of sanity tests.

Now we have this_cpu_ api, we probably can do a cleanup to avoid now useless parameter passing.

If we had this_cpu_is_online() function, we could avoid (int cpu) param in printk_needs_cpu()

#19 lore raw

Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending

Quoted 1 line · Eric Dumazet

I guess you read the changelog too fast ;)

No I only read the code.... ;-)

Quoted 2 lines

I posted this patch after noticing printk_needs_cpu() was always used for the current cpu.

Ok.

Quoted 1 line

We have other functions around, that always work for the current cpu.

Ok.

Quoted 3 lines

We pass cpu as a parameter, mostly because smp_processor_id() was a bit expensive in old kernels, and is still expensive because of sanity tests.

I thought sanity tests do not apply for performance testing scenarios?

smp_processor_id() is usually a simple per cpu data access and very fast.

#20 lore raw

Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending

Quoted 2 lines · Christoph Lameter

I thought sanity tests do not apply for performance testing scenarios?

CONFIG_DEBUG_PREEMPT

Quoted 2 lines

smp_processor_id() is usually a simple per cpu data access and very fast.

Sure, but when its already in a register, its better to use the register ;)

#21 lore raw

Re: [PATCH] printk: use this_cpu_{read|write} api on printk_pending

Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Christoph Lameter

I thought sanity tests do not apply for performance testing scenarios?

CONFIG_DEBUG_PREEMPT

You really enable that option for performance testing?

Quoted 5 lines
Quoted 2 lines

smp_processor_id() is usually a simple per cpu data access and very fast.

Sure, but when its already in a register, its better to use the register ;)

Well that increases register pressure in the caller. There may be no need to handle the current processor number in the caller.

#22 lore raw

[tip:sched/core] printk: Use this_cpu_{read|write} api on printk_pending

In reply to Eric Dumazet
Commit-ID:  40dc11ffb35e8c4e8fa71092048e0f8de9db758c
Gitweb:     http://git.kernel.org/tip/40dc11ffb35e8c4e8fa71092048e0f8de9db758c
Author:     Eric Dumazet <[email protected]>
AuthorDate: Fri, 26 Nov 2010 17:22:16 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 8 Dec 2010 20:16:01 +0100

printk: Use this_cpu_{read|write} api on printk_pending

__get_cpu_var() is a bit inefficient, lets use __this_cpu_read() and __this_cpu_write() to manipulate printk_pending.

printk_needs_cpu(cpu) is called only for the current cpu :
Use faster __this_cpu_read().

Remove the redundant unlikely on (cpu_is_offline(cpu)) test:

 # size kernel/printk.o*
   text	   data	    bss	    dec	    hex	filename
   9942	    756	 263488	 274186	  42f0a	kernel/printk.o.new
   9990	    756	 263488	 274234	  42f3a	kernel/printk.o.old
Cc Heiko Carstens <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Christoph Lameter <[email protected]>

LKML-Reference: <1290788536.2855.237.camel@edumazet-laptop>

---
 kernel/printk.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/printk.c +4 -4
index a23315d..ab3ffc5 100644
@@ -1074,17 +1074,17 @@ static DEFINE_PER_CPU(int, printk_pending);
 
 void printk_tick(void)
 {
-	if (__get_cpu_var(printk_pending)) {
-		__get_cpu_var(printk_pending) = 0;
+	if (__this_cpu_read(printk_pending)) {
+		__this_cpu_write(printk_pending, 0);
 		wake_up_interruptible(&log_wait);
 	}
 }
 
 int printk_needs_cpu(int cpu)
 {
-	if (unlikely(cpu_is_offline(cpu)))
+	if (cpu_is_offline(cpu))
 		printk_tick();
-	return per_cpu(printk_pending, cpu);
+	return __this_cpu_read(printk_pending);
 }
 
 void wake_up_klogd(void)
#23 lore raw

Re: [tip:sched/core] printk: Use this_cpu_{read|write} api on printk_pending

Quoted 1 line · tip-bot for Eric Dumazet

printk: Use this_cpu_{read|write} api on printk_pending

One of the patches in Tejun's percpu tree includes the same modifications.

#24 lore raw

Re: [tip:sched/core] printk: Use this_cpu_{read|write} api on printk_pending

Quoted 7 lines · Christoph Lameter
Quoted 1 line · tip-bot for Eric Dumazet

printk: Use this_cpu_{read|write} api on printk_pending

One of the patches in Tejun's percpu tree includes the same modifications.

Its a bit strange, I sent mine a long time ago when reviewing Heiko Carstens patch. You even commented on it.

https://patchwork.kernel.org/patch/359182/

What happened then, you included my patch on one of yours ?

#25 lore raw

Re: [tip:sched/core] printk: Use this_cpu_{read|write} api on printk_pending

Quoted 6 lines · Eric Dumazet

Its a bit strange, I sent mine a long time ago when reviewing Heiko Carstens patch. You even commented on it.

https://patchwork.kernel.org/patch/359182/

What happened then, you included my patch on one of yours ?

I just went through the tree and changed all __get_cpu_var. Out came a single core patch that also covered this earlier case.

#26 lore raw

PATCH 3/3 nohz/s390: fix arch_needs_cpu() return value on offline cpus

In reply to Heiko Carstens

From: Heiko Carstens <[email protected]>

This fixes the same problem as described in the patch "nohz: fix printk_needs_cpu() return value on offline cpus" for the arch_needs_cpu() primitive.
This specific bug was indrocuded with 3c5d92a0 "nohz: Introduce arch_needs_cpu".

In this case a cpu hotplug notifier is used to fix the issue in order to keep the normal/fast path small. All we need to do is to clear the condition that makes arch_needs_cpu() return 1 since it is just a performance improvement which is supposed to keep the local tick running for a short period if a cpu goes idle. Nothing special needs to be done except for clearing the condition.

---
 arch/s390/kernel/vtime.c |   19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
arch/s390/kernel/vtime.c +19 -0
@@ -19,6 +19,7 @@
 #include <linux/kernel_stat.h>
 #include <linux/rcupdate.h>
 #include <linux/posix-timers.h>
+#include <linux/cpu.h>
 
 #include <asm/s390_ext.h>
 #include <asm/timer.h>
@@ -566,6 +567,23 @@ void init_cpu_vtimer(void)
 	__ctl_set_bit(0,10);
 }
 
+static int __cpuinit s390_nohz_notify(struct notifier_block *self,
+				      unsigned long action, void *hcpu)
+{
+	struct s390_idle_data *idle;
+	long cpu = (long) hcpu;
+
+	idle = &per_cpu(s390_idle, cpu);
+	switch (action) {
+	case CPU_DYING:
+	case CPU_DYING_FROZEN:
+		idle->nohz_delay = 0;
+	default:
+		break;
+	}
+	return NOTIFY_OK;
+}
+
 void __init vtime_init(void)
 {
 	/* request the cpu timer external interrupt */
@@ -574,5 +592,6 @@ void __init vtime_init(void)
 
 	/* Enable cpu timer interrupts on the boot cpu. */
 	init_cpu_vtimer();
+	cpu_notifier(s390_nohz_notify, 0);
 }
 
Acked-by on v1 3/3
#27 lore raw

Re: [patch 3/3] nohz/s390: fix arch_needs_cpu() return value on offline cpus

Quoted 8 lines · Heiko Carstens

plain text document attachment (003_arch_needs_cpu.diff)
From: Heiko Carstens <[email protected]>

This fixes the same problem as described in the patch "nohz: fix printk_needs_cpu() return value on offline cpus" for the arch_needs_cpu() primitive.
This specific bug was indrocuded with 3c5d92a0 "nohz: Introduce arch_needs_cpu".

Quoted 8 lines

In this case a cpu hotplug notifier is used to fix the issue in order to keep the normal/fast path small. All we need to do is to clear the condition that makes arch_needs_cpu() return 1 since it is just a performance improvement which is supposed to keep the local tick running for a short period if a cpu goes idle. Nothing special needs to be done except for clearing the condition.

OK, and s390 seems to be the only architecture making use of that interface.

Did you audit all the other *_needs_cpu() interfaces for this same problem?

Anyway,

Will you take this through the s390?

Quoted 46 lines
---
 arch/s390/kernel/vtime.c |   19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
arch/s390/kernel/vtime.c +19 -0
@@ -19,6 +19,7 @@
 #include <linux/kernel_stat.h>
 #include <linux/rcupdate.h>
 #include <linux/posix-timers.h>
+#include <linux/cpu.h>
 
 #include <asm/s390_ext.h>
 #include <asm/timer.h>
@@ -566,6 +567,23 @@ void init_cpu_vtimer(void)
 	__ctl_set_bit(0,10);
 }
 
+static int __cpuinit s390_nohz_notify(struct notifier_block *self,
+				      unsigned long action, void *hcpu)
+{
+	struct s390_idle_data *idle;
+	long cpu = (long) hcpu;
+
+	idle = &per_cpu(s390_idle, cpu);
+	switch (action) {
+	case CPU_DYING:
+	case CPU_DYING_FROZEN:
+		idle->nohz_delay = 0;
+	default:
+		break;
+	}
+	return NOTIFY_OK;
+}
+
 void __init vtime_init(void)
 {
 	/* request the cpu timer external interrupt */
@@ -574,5 +592,6 @@ void __init vtime_init(void)
 
 	/* Enable cpu timer interrupts on the boot cpu. */
 	init_cpu_vtimer();
+	cpu_notifier(s390_nohz_notify, 0);
 }
 
#28 lore raw

Re: [patch 3/3] nohz/s390: fix arch_needs_cpu() return value on offline cpus

Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Heiko Carstens

plain text document attachment (003_arch_needs_cpu.diff)
From: Heiko Carstens <[email protected]>

This fixes the same problem as described in the patch "nohz: fix printk_needs_cpu() return value on offline cpus" for the arch_needs_cpu() primitive.
This specific bug was indrocuded with 3c5d92a0 "nohz: Introduce arch_needs_cpu".

In this case a cpu hotplug notifier is used to fix the issue in order to keep the normal/fast path small. All we need to do is to clear the condition that makes arch_needs_cpu() return 1 since it is just a performance improvement which is supposed to keep the local tick running for a short period if a cpu goes idle. Nothing special needs to be done except for clearing the condition.

Quoted 8 lines
Quoted 2 lines

OK, and s390 seems to be the only architecture making use of that interface.

Did you audit all the other *_needs_cpu() interfaces for this same problem?

There's only rcu_needs_cpu() left and rcu indeed has a cpu hotplug notifier. So there _shouldn't_ be any problems with that.

Quoted 3 lines

Will you take this through the s390?

Yes. Thanks!

#29 lore raw

Re: [patch 3/3] nohz/s390: fix arch_needs_cpu() return value on offline cpus

In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Heiko Carstens

plain text document attachment (003_arch_needs_cpu.diff)
From: Heiko Carstens <[email protected]>

This fixes the same problem as described in the patch "nohz: fix printk_needs_cpu() return value on offline cpus" for the arch_needs_cpu() primitive.
This specific bug was indrocuded with 3c5d92a0 "nohz: Introduce arch_needs_cpu".

In this case a cpu hotplug notifier is used to fix the issue in order to keep the normal/fast path small. All we need to do is to clear the condition that makes arch_needs_cpu() return 1 since it is just a performance improvement which is supposed to keep the local tick running for a short period if a cpu goes idle. Nothing special needs to be done except for clearing the condition.

Quoted 8 lines
Quoted 2 lines

OK, and s390 seems to be the only architecture making use of that interface.

Did you audit all the other *_needs_cpu() interfaces for this same problem?

Yes I did, but guess what.. get_next_timer_interrupt() bites us as well *sigh*.

Here is yet another nohz vs cpu hotplug patch which should also go into stable.

Subject: [PATCH] nohz: fix get_next_timer_interrupt() vs cpu hotplug

From: Heiko Carstens <[email protected]>

This fixes a bug as seen on 2.6.32 based kernels where timers got enqueued on offline cpus.

If a cpu goes offline it might still have pending timers. These will be migrated during CPU_DEAD handling after the cpu is offline. However while the cpu is going offline it will schedule the idle task which will then call tick_nohz_stop_sched_tick().
That function in turn will call get_next_timer_intterupt() to figure out if the tick of the cpu can be stopped or not. If it turns out that the next tick is just one jiffy off (delta_jiffies == 1) tick_nohz_stop_sched_tick() incorrectly assumes that the tick should not stop and takes an early exit and thus it won't update the load balancer cpu.
Just afterwards the cpu will be killed and the load balancer cpu could be the offline cpu.
On 2.6.32 based kernel get_nohz_load_balancer() gets called to decide on which cpu a timer should be enqueued (see __mod_timer()). Which leads to the possibility that timers get enqueued on an offline cpu. These will never expire and can cause a system hang.

This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.

The easiest and probably safest fix seems to be to let get_next_timer_interrupt() just lie and let it say there isn't any pending timer if the current cpu is offline.
I also thought of moving migrate_[hr]timers() from CPU_DEAD to CPU_DYING, but seeing that there already have been fixes at least in the hrtimer code in this area I'm afraid that this could add new subtle bugs.

---

 kernel/timer.c |    6 ++++++
 1 file changed, 6 insertions(+)
kernel/timer.c +6 -0
index 68a9ae7..f95277f 100644
@@ -1252,6 +1252,12 @@ unsigned long get_next_timer_interrupt(unsigned long now)
 	struct tvec_base *base = __get_cpu_var(tvec_bases);
 	unsigned long expires;
 
+	/*
+	 * Pretend that there is no timer pending if the cpu is offline.
+	 * Possible pending timers will be migrated later to an active cpu.
+	 */
+	if (cpu_is_offline(smp_processor_id()))
+		return now + NEXT_TIMER_MAX_DELTA;
 	spin_lock(&base->lock);
 	if (time_before_eq(base->next_timer, base->timer_jiffies))
 		base->next_timer = __next_timer_interrupt(base);
#30 lore raw

Re: [patch 3/3] nohz/s390: fix arch_needs_cpu() return value on offline cpus

Quoted 37 lines · Heiko Carstens

Subject: [PATCH] nohz: fix get_next_timer_interrupt() vs cpu hotplug

From: Heiko Carstens <[email protected]>

This fixes a bug as seen on 2.6.32 based kernels where timers got enqueued on offline cpus.

If a cpu goes offline it might still have pending timers. These will be migrated during CPU_DEAD handling after the cpu is offline. However while the cpu is going offline it will schedule the idle task which will then call tick_nohz_stop_sched_tick().
That function in turn will call get_next_timer_intterupt() to figure out if the tick of the cpu can be stopped or not. If it turns out that the next tick is just one jiffy off (delta_jiffies == 1) tick_nohz_stop_sched_tick() incorrectly assumes that the tick should not stop and takes an early exit and thus it won't update the load balancer cpu.
Just afterwards the cpu will be killed and the load balancer cpu could be the offline cpu.
On 2.6.32 based kernel get_nohz_load_balancer() gets called to decide on which cpu a timer should be enqueued (see __mod_timer()). Which leads to the possibility that timers get enqueued on an offline cpu. These will never expire and can cause a system hang.

This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.

The easiest and probably safest fix seems to be to let get_next_timer_interrupt() just lie and let it say there isn't any pending timer if the current cpu is offline.
I also thought of moving migrate_[hr]timers() from CPU_DEAD to CPU_DYING, but seeing that there already have been fixes at least in the hrtimer code in this area I'm afraid that this could add new subtle bugs.

Quoted 3 lines

---

Thanks Heiko, I queued this one as well.

#31 lore raw

[tip:sched/urgent] nohz: Fix get_next_timer_interrupt() vs cpu hotplug

In reply to Heiko Carstens
Commit-ID:  dbd87b5af055a0cc9bba17795c9a2b0d17795389
Gitweb:     http://git.kernel.org/tip/dbd87b5af055a0cc9bba17795c9a2b0d17795389
Author:     Heiko Carstens <[email protected]>
AuthorDate: Wed, 1 Dec 2010 10:11:09 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 8 Dec 2010 20:15:07 +0100

nohz: Fix get_next_timer_interrupt() vs cpu hotplug

This fixes a bug as seen on 2.6.32 based kernels where timers got enqueued on offline cpus.

If a cpu goes offline it might still have pending timers. These will be migrated during CPU_DEAD handling after the cpu is offline. However while the cpu is going offline it will schedule the idle task which will then call tick_nohz_stop_sched_tick().

That function in turn will call get_next_timer_intterupt() to figure out if the tick of the cpu can be stopped or not. If it turns out that the next tick is just one jiffy off (delta_jiffies == 1) tick_nohz_stop_sched_tick() incorrectly assumes that the tick should not stop and takes an early exit and thus it won't update the load balancer cpu.

Just afterwards the cpu will be killed and the load balancer cpu could be the offline cpu.

On 2.6.32 based kernel get_nohz_load_balancer() gets called to decide on which cpu a timer should be enqueued (see __mod_timer()). Which leads to the possibility that timers get enqueued on an offline cpu. These will never expire and can cause a system hang.

This has been observed 2.6.32 kernels. On current kernels __mod_timer() uses get_nohz_timer_target() which doesn't have that problem. However there might be other problems because of the too early exit tick_nohz_stop_sched_tick() in case a cpu goes offline.

The easiest and probably safest fix seems to be to let get_next_timer_interrupt() just lie and let it say there isn't any pending timer if the current cpu is offline.

I also thought of moving migrate_[hr]timers() from CPU_DEAD to CPU_DYING, but seeing that there already have been fixes at least in the hrtimer code in this area I'm afraid that this could add new subtle bugs.

LKML-Reference: <[email protected]>

---
 kernel/timer.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
kernel/timer.c +6 -0
index 7bd715f..353b922 100644
@@ -1252,6 +1252,12 @@ unsigned long get_next_timer_interrupt(unsigned long now)
 	struct tvec_base *base = __get_cpu_var(tvec_bases);
 	unsigned long expires;
 
+	/*
+	 * Pretend that there is no timer pending if the cpu is offline.
+	 * Possible pending timers will be migrated later to an active cpu.
+	 */
+	if (cpu_is_offline(smp_processor_id()))
+		return now + NEXT_TIMER_MAX_DELTA;
 	spin_lock(&base->lock);
 	if (time_before_eq(base->next_timer, base->timer_jiffies))
 		base->next_timer = __next_timer_interrupt(base);