Skip to contentSkip to conversation

FAILED: patch "[PATCH] tracepoints: Do not trace when cpu is offline" failed to apply to 3.14-stable tree

PATCH [email protected] 2 participants 11 messages Feb 24, 2016

Subsystems (2)
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

The patch below does not apply to the 3.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

Quoted 1 line

From f37755490fe9bf76f6ba1d8c6591745d3574a6a6 Mon Sep 17 00:00:00 2001

From: "Steven Rostedt (Red Hat)" <[email protected]>
Date: Mon, 15 Feb 2016 12:36:14 -0500
Subject: [PATCH] tracepoints: Do not trace when cpu is offline

The tracepoint infrastructure uses RCU sched protection to enable and disable tracepoints safely. There are some instances where tracepoints are used in infrastructure code (like kfree()) that get called after a CPU is going offline, and perhaps when it is coming back online but hasn't been registered yet.

This can probuce the following warning:

[ INFO: suspicious RCU usage. ]
4.4.0-00006-g0fe53e8-dirty #34 Tainted: G S
-------------------------------
include/trace/events/kmem.h:141 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

RCU used illegally from offline CPU! rcu_scheduler_active = 1, debug_locks = 1 no locks held by swapper/8/0.

 stack backtrace:
  CPU: 8 PID: 0 Comm: swapper/8 Tainted: G S              4.4.0-00006-g0fe53e8-dirty #34
  Call Trace:
  [c0000005b76c78d0] [c0000000008b9540] .dump_stack+0x98/0xd4 (unreliable)
  [c0000005b76c7950] [c00000000010c898] .lockdep_rcu_suspicious+0x108/0x170
  [c0000005b76c79e0] [c00000000029adc0] .kfree+0x390/0x440
  [c0000005b76c7a80] [c000000000055f74] .destroy_context+0x44/0x100
  [c0000005b76c7b00] [c0000000000934a0] .__mmdrop+0x60/0x150
  [c0000005b76c7b90] [c0000000000e3ff0] .idle_task_exit+0x130/0x140
  [c0000005b76c7c20] [c000000000075804] .pseries_mach_cpu_die+0x64/0x310
  [c0000005b76c7cd0] [c000000000043e7c] .cpu_die+0x3c/0x60
  [c0000005b76c7d40] [c0000000000188d8] .arch_cpu_idle_dead+0x28/0x40
  [c0000005b76c7db0] [c000000000101e6c] .cpu_startup_entry+0x50c/0x560
  [c0000005b76c7ed0] [c000000000043bd8] .start_secondary+0x328/0x360
  [c0000005b76c7f90] [c000000000008a6c] start_secondary_prolog+0x10/0x14

This warning is not a false positive either. RCU is not protecting code that is being executed while the CPU is offline.

Instead of playing "whack-a-mole(TM)" and adding conditional statements to the tracepoints we find that are used in this instance, simply add a cpu_online() test to the tracepoint code where the tracepoint will be ignored if the CPU is offline.

Use of raw_smp_processor_id() is fine, as there should never be a case where the tracepoint code goes from running on a CPU that is online and suddenly gets migrated to a CPU that is offline.

Fixes 97e1c18e8d17b ("tracing: Kernel Tracepoints")
Cc [email protected] # v2.6.28+
include/linux/tracepoint.h +5 -0
index acd522a91539..acfdbf353a0b 100644
@@ -14,8 +14,10 @@
  * See the file COPYING for more details.
  */
 
+#include <linux/smp.h>
 #include <linux/errno.h>
 #include <linux/types.h>
+#include <linux/cpumask.h>
 #include <linux/rcupdate.h>
 #include <linux/tracepoint-defs.h>
 
@@ -132,6 +134,9 @@ extern void syscall_unregfunc(void);
 		void *it_func;						\
 		void *__data;						\
 									\
+		if (!cpu_online(raw_smp_processor_id()))		\
+			return;						\
+									\
 		if (!(cond))						\
 			return;						\
 		prercu;							\
maintainer
#1 lore raw
Quoted 4 lines · On Tue, 23 Feb 2016 16:48:47 -0800 <[email protected]> wrote:

The patch below does not apply to the 3.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

Hmm, strange. I just checked out v3.14.60 and did:

 $ git cherry-pick f37755490fe9bf76f6ba1d8c6591745d3574a6a6

and it applied without issue.

Below is the result.

-- Steve

Quoted 8 lines

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

Quoted 1 line

From f37755490fe9bf76f6ba1d8c6591745d3574a6a6 Mon Sep 17 00:00:00 2001

Quoted 8 lines

From: "Steven Rostedt (Red Hat)" <[email protected]>
Date: Mon, 15 Feb 2016 12:36:14 -0500
Subject: [PATCH] tracepoints: Do not trace when cpu is offline

From 5921fea2b95c0795d88f7b82633242c341be2efc Mon Sep 17 00:00:00 2001

From: "Steven Rostedt (Red Hat)" <[email protected]>
Date: Mon, 15 Feb 2016 12:36:14 -0500
Subject: [PATCH] tracepoints: Do not trace when cpu is offline

The tracepoint infrastructure uses RCU sched protection to enable and disable tracepoints safely. There are some instances where tracepoints are used in infrastructure code (like kfree()) that get called after a CPU is going offline, and perhaps when it is coming back online but hasn't been registered yet.

This can probuce the following warning:

[ INFO: suspicious RCU usage. ]
4.4.0-00006-g0fe53e8-dirty #34 Tainted: G S
-------------------------------
include/trace/events/kmem.h:141 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

RCU used illegally from offline CPU! rcu_scheduler_active = 1, debug_locks = 1 no locks held by swapper/8/0.

 stack backtrace:
  CPU: 8 PID: 0 Comm: swapper/8 Tainted: G S              4.4.0-00006-g0fe53e8-dirty #34
  Call Trace:
  [c0000005b76c78d0] [c0000000008b9540] .dump_stack+0x98/0xd4 (unreliable)
  [c0000005b76c7950] [c00000000010c898] .lockdep_rcu_suspicious+0x108/0x170
  [c0000005b76c79e0] [c00000000029adc0] .kfree+0x390/0x440
  [c0000005b76c7a80] [c000000000055f74] .destroy_context+0x44/0x100
  [c0000005b76c7b00] [c0000000000934a0] .__mmdrop+0x60/0x150
  [c0000005b76c7b90] [c0000000000e3ff0] .idle_task_exit+0x130/0x140
  [c0000005b76c7c20] [c000000000075804] .pseries_mach_cpu_die+0x64/0x310
  [c0000005b76c7cd0] [c000000000043e7c] .cpu_die+0x3c/0x60
  [c0000005b76c7d40] [c0000000000188d8] .arch_cpu_idle_dead+0x28/0x40
  [c0000005b76c7db0] [c000000000101e6c] .cpu_startup_entry+0x50c/0x560
  [c0000005b76c7ed0] [c000000000043bd8] .start_secondary+0x328/0x360
  [c0000005b76c7f90] [c000000000008a6c] start_secondary_prolog+0x10/0x14

This warning is not a false positive either. RCU is not protecting code that is being executed while the CPU is offline.

Instead of playing "whack-a-mole(TM)" and adding conditional statements to the tracepoints we find that are used in this instance, simply add a cpu_online() test to the tracepoint code where the tracepoint will be ignored if the CPU is offline.

Use of raw_smp_processor_id() is fine, as there should never be a case where the tracepoint code goes from running on a CPU that is online and suddenly gets migrated to a CPU that is offline.

Fixes 97e1c18e8d17b ("tracing: Kernel Tracepoints")
Cc [email protected] # v2.6.28+
---
 include/linux/tracepoint.h | 5 +++++
 1 file changed, 5 insertions(+)
include/linux/tracepoint.h +5 -0
index 7159a0a..8e344f6 100644
@@ -14,8 +14,10 @@
  * See the file COPYING for more details.
  */
 
+#include <linux/smp.h>
 #include <linux/errno.h>
 #include <linux/types.h>
+#include <linux/cpumask.h>
 #include <linux/rcupdate.h>
 #include <linux/static_key.h>
 
@@ -126,6 +128,9 @@ static inline void tracepoint_synchronize_unregister(void)
 		void *it_func;						\
 		void *__data;						\
 									\
+		if (!cpu_online(raw_smp_processor_id()))		\
+			return;						\
+									\
 		if (!(cond))						\
 			return;						\
 		prercu;							\
Signature · 1 line
2.7.0
author maintainer
#2 lore raw
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Tue, 23 Feb 2016 16:48:47 -0800 <[email protected]> wrote:

The patch below does not apply to the 3.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

Quoted 5 lines

Hmm, strange. I just checked out v3.14.60 and did:

 $ git cherry-pick f37755490fe9bf76f6ba1d8c6591745d3574a6a6

and it applied without issue.

Yes, it applies, sorry, but it breaks the build :(

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

thanks,

greg k-h

maintainer
#3 lore raw
Quoted 13 lines · On Tue, 23 Feb 2016 19:37:04 -0800 Greg KH <[email protected]> wrote:
Quoted 12 lines · Steven Rostedt
Quoted 4 lines · On Tue, 23 Feb 2016 16:48:47 -0800 <[email protected]> wrote:

The patch below does not apply to the 3.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

Hmm, strange. I just checked out v3.14.60 and did:

 $ git cherry-pick f37755490fe9bf76f6ba1d8c6591745d3574a6a6
Quoted 7 lines
Quoted 1 line

and it applied without issue.

Yes, it applies, sorry, but it breaks the build :(

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

Yep I'll go work on that. But can you change your scripts to say it broke the build, and possibly give a config file with it. The current email just says "The patch below does not apply to the 3.14-stable tree."

Thanks!

-- Steve

author maintainer
#4 lore raw
Quoted 20 lines · Steven Rostedt
Quoted 17 lines · On Tue, 23 Feb 2016 19:37:04 -0800 Greg KH <[email protected]> wrote:
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Tue, 23 Feb 2016 16:48:47 -0800 <[email protected]> wrote:

The patch below does not apply to the 3.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

Quoted 5 lines

Hmm, strange. I just checked out v3.14.60 and did:

 $ git cherry-pick f37755490fe9bf76f6ba1d8c6591745d3574a6a6

and it applied without issue.

Yes, it applies, sorry, but it breaks the build :(

Quoted 8 lines
Quoted 3 lines

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

Yep I'll go work on that. But can you change your scripts to say it broke the build, and possibly give a config file with it. The current email just says "The patch below does not apply to the 3.14-stable tree."

The number of times that a patch applies but then breaks the build makes it pretty low for me to write a new script for that, sorry...

maintainer
#5 lore raw
Quoted 25 lines · On Wed, 24 Feb 2016 09:07:33 -0800 Greg KH <[email protected]> wrote:
Quoted 24 lines · Steven Rostedt
Quoted 20 lines · On Tue, 23 Feb 2016 19:37:04 -0800 Greg KH <[email protected]> wrote:
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Tue, 23 Feb 2016 16:48:47 -0800 <[email protected]> wrote:

The patch below does not apply to the 3.14-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

Quoted 5 lines

Hmm, strange. I just checked out v3.14.60 and did:

 $ git cherry-pick f37755490fe9bf76f6ba1d8c6591745d3574a6a6

and it applied without issue.

Yes, it applies, sorry, but it breaks the build :(

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

Quoted 7 lines
Quoted 4 lines

Yep I'll go work on that. But can you change your scripts to say it broke the build, and possibly give a config file with it. The current email just says "The patch below does not apply to the 3.14-stable tree."

The number of times that a patch applies but then breaks the build makes it pretty low for me to write a new script for that, sorry...

OK, then what about changing the wording to something like

"The patch below does not apply or breaks the build for 3.14-stable tree"

?

It would prevent people like me from bothering you about saying "but it does apply!"

-- Steve

maintainer
#6 lore raw
In reply to Greg KH
Quoted 6 lines · On Tue, 23 Feb 2016 19:37:04 -0800 Greg KH <[email protected]> wrote:

Yes, it applies, sorry, but it breaks the build :(

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

Bah. smp.h isn't enough for using raw_smp_processor_id(). One must also include percpu.h. That was fixed in mainline, but not in stable. But not a big deal. I'll included percpu.h in tracepoint.h as well.

Here's the update for 3.14, and I'm guessing it would work in 3.10 as well. I'll try it out.

-- Steve

Quoted 1 line

From c3d795582c64396ac163ed409cb9578d3cf3dbc6 Mon Sep 17 00:00:00 2001

From: "Steven Rostedt (Red Hat)" <[email protected]>
Date: Mon, 15 Feb 2016 12:36:14 -0500
Subject: [PATCH] tracepoints: Do not trace when cpu is offline

The tracepoint infrastructure uses RCU sched protection to enable and disable tracepoints safely. There are some instances where tracepoints are used in infrastructure code (like kfree()) that get called after a CPU is going offline, and perhaps when it is coming back online but hasn't been registered yet.

This can probuce the following warning:

[ INFO: suspicious RCU usage. ]
4.4.0-00006-g0fe53e8-dirty #34 Tainted: G S
-------------------------------
include/trace/events/kmem.h:141 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

RCU used illegally from offline CPU! rcu_scheduler_active = 1, debug_locks = 1 no locks held by swapper/8/0.

 stack backtrace:
  CPU: 8 PID: 0 Comm: swapper/8 Tainted: G S              4.4.0-00006-g0fe53e8-dirty #34
  Call Trace:
  [c0000005b76c78d0] [c0000000008b9540] .dump_stack+0x98/0xd4 (unreliable)
  [c0000005b76c7950] [c00000000010c898] .lockdep_rcu_suspicious+0x108/0x170
  [c0000005b76c79e0] [c00000000029adc0] .kfree+0x390/0x440
  [c0000005b76c7a80] [c000000000055f74] .destroy_context+0x44/0x100
  [c0000005b76c7b00] [c0000000000934a0] .__mmdrop+0x60/0x150
  [c0000005b76c7b90] [c0000000000e3ff0] .idle_task_exit+0x130/0x140
  [c0000005b76c7c20] [c000000000075804] .pseries_mach_cpu_die+0x64/0x310
  [c0000005b76c7cd0] [c000000000043e7c] .cpu_die+0x3c/0x60
  [c0000005b76c7d40] [c0000000000188d8] .arch_cpu_idle_dead+0x28/0x40
  [c0000005b76c7db0] [c000000000101e6c] .cpu_startup_entry+0x50c/0x560
  [c0000005b76c7ed0] [c000000000043bd8] .start_secondary+0x328/0x360
  [c0000005b76c7f90] [c000000000008a6c] start_secondary_prolog+0x10/0x14

This warning is not a false positive either. RCU is not protecting code that is being executed while the CPU is offline.

Instead of playing "whack-a-mole(TM)" and adding conditional statements to the tracepoints we find that are used in this instance, simply add a cpu_online() test to the tracepoint code where the tracepoint will be ignored if the CPU is offline.

Use of raw_smp_processor_id() is fine, as there should never be a case where the tracepoint code goes from running on a CPU that is online and suddenly gets migrated to a CPU that is offline.

Fixes 97e1c18e8d17b ("tracing: Kernel Tracepoints")
Cc [email protected] # v2.6.28+
---
 include/linux/tracepoint.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/linux/tracepoint.h +6 -0
index 7159a0a933df..97c8689c7e51 100644
@@ -14,8 +14,11 @@
  * See the file COPYING for more details.
  */
 
+#include <linux/smp.h>
 #include <linux/errno.h>
 #include <linux/types.h>
+#include <linux/percpu.h>
+#include <linux/cpumask.h>
 #include <linux/rcupdate.h>
 #include <linux/static_key.h>
 
@@ -126,6 +129,9 @@ static inline void tracepoint_synchronize_unregister(void)
 		void *it_func;						\
 		void *__data;						\
 									\
+		if (!cpu_online(raw_smp_processor_id()))		\
+			return;						\
+									\
 		if (!(cond))						\
 			return;						\
 		prercu;							\
Signature · 1 line
1.8.3.1
maintainer
#7 lore raw
Quoted 11 lines · On Wed, 24 Feb 2016 10:18:32 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 6 lines · On Tue, 23 Feb 2016 19:37:04 -0800 Greg KH <[email protected]> wrote:

Yes, it applies, sorry, but it breaks the build :(

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

Quoted 6 lines

Bah. smp.h isn't enough for using raw_smp_processor_id(). One must also include percpu.h. That was fixed in mainline, but not in stable. But not a big deal. I'll included percpu.h in tracepoint.h as well.

Here's the update for 3.14, and I'm guessing it would work in 3.10 as well. I'll try it out.

I tested this too on 3.10 (cherry picked it from my 3.14 branch). It builds and boots fine.

-- Steve

author maintainer
#8 lore raw
In reply to Steven Rostedt
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · On Tue, 23 Feb 2016 19:37:04 -0800 Greg KH <[email protected]> wrote:

Yes, it applies, sorry, but it breaks the build :(

Quoted 8 lines
Quoted 4 lines

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

Bah. smp.h isn't enough for using raw_smp_processor_id(). One must also include percpu.h. That was fixed in mainline, but not in stable. But not a big deal. I'll included percpu.h in tracepoint.h as well.

What was the commit id that did that change? Is it simple enough to just take that patch instead?

Quoted 3 lines

Here's the update for 3.14, and I'm guessing it would work in 3.10 as well. I'll try it out.

Thanks, I'll queue these up for the next round of stable releases after these.

greg k-h

maintainer
#9 lore raw
Quoted 12 lines · On Wed, 24 Feb 2016 09:14:18 -0800 Greg KH <[email protected]> wrote:
Quoted 11 lines · Steven Rostedt
Quoted 6 lines · On Tue, 23 Feb 2016 19:37:04 -0800 Greg KH <[email protected]> wrote:

Yes, it applies, sorry, but it breaks the build :(

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

Quoted 7 lines
Quoted 3 lines

Bah. smp.h isn't enough for using raw_smp_processor_id(). One must also include percpu.h. That was fixed in mainline, but not in stable. But not a big deal. I'll included percpu.h in tracepoint.h as well.

What was the commit id that did that change? Is it simple enough to just take that patch instead?

Hmm, I don't know where the include is done. Perhaps something else included it and it was just a luck that this commit works in mainline.

-- Steve

author maintainer
#10 lore raw
Quoted 19 lines · Steven Rostedt
Quoted 16 lines · On Wed, 24 Feb 2016 09:14:18 -0800 Greg KH <[email protected]> wrote:
Quoted 14 lines · Steven Rostedt
Quoted 6 lines · On Tue, 23 Feb 2016 19:37:04 -0800 Greg KH <[email protected]> wrote:

Yes, it applies, sorry, but it breaks the build :(

Same for 3.10-stable. It applies but can not compile.

can you fix that up?

Bah. smp.h isn't enough for using raw_smp_processor_id(). One must also include percpu.h. That was fixed in mainline, but not in stable. But not a big deal. I'll included percpu.h in tracepoint.h as well.

Quoted 6 lines
Quoted 3 lines

What was the commit id that did that change? Is it simple enough to just take that patch instead?

Hmm, I don't know where the include is done. Perhaps something else included it and it was just a luck that this commit works in mainline.

Ok, I'll stick with what you sent, thanks.

greg k-h