tracing/mm: don't trace kfree on offline CPUs
PATCH v2 1/2 tracing/mm: don't trace kfree on offline CPUs
Tracepoints use RCU for protection and they must not be called on offline CPUS. So make this tracepoint conditional.
INFO: suspicious RCU usage. ]
[ 99.749382] 4.4.0-00006-g0fe53e8-dirty #34 Tainted: G S [ 99.749383] ------------------------------- [ 99.749385] include/trace/events/kmem.h:141 suspicious rcu_dereference_check() usage! [ 99.749385]
other info that might help us debug this:
[ 99.749387] RCU used illegally from offline CPU! rcu_scheduler_active = 1, debug_locks = 1 [ 99.749388] no locks held by swapper/8/0. [ 99.749388] stack backtrace: [ 99.749390] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G S 4.4.0-00006-g0fe53e8-dirty #34 [ 99.749391] Call Trace: [ 99.749399] [c0000005b76c78d0] [c0000000008b9540] .dump_stack+0x98/0xd4 (unreliable) [ 99.749402] [c0000005b76c7950] [c00000000010c898] .lockdep_rcu_suspicious+0x108/0x170 [ 99.749405] [c0000005b76c79e0] [c00000000029adc0] .kfree+0x390/0x440 [ 99.749409] [c0000005b76c7a80] [c000000000055f74] .destroy_context+0x44/0x100 [ 99.749412] [c0000005b76c7b00] [c0000000000934a0] .__mmdrop+0x60/0x150 [ 99.749415] [c0000005b76c7b90] [c0000000000e3ff0] .idle_task_exit+0x130/0x140 [ 99.749417] [c0000005b76c7c20] [c000000000075804] .pseries_mach_cpu_die+0x64/0x310 [ 99.749420] [c0000005b76c7cd0] [c000000000043e7c] .cpu_die+0x3c/0x60 [ 99.749423] [c0000005b76c7d40] [c0000000000188d8] .arch_cpu_idle_dead+0x28/0x40 [ 99.749426] [c0000005b76c7db0] [c000000000101e6c] .cpu_startup_entry+0x50c/0x560 [ 99.749428] [c0000005b76c7ed0] [c000000000043bd8] .start_secondary+0x328/0x360 [ 99.749431] [c0000005b76c7f90] [c000000000008a6c] start_secondary_prolog+0x10/0x14
--- include/trace/events/kmem.h | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
include/trace/events/kmem.h +13 -2
@@ -133,11 +133,22 @@ DECLARE_EVENT_CLASS(kmem_free,TP_printk ("call_site=%lx ptr=%p" , __entry->call_site, __entry->ptr) ); -DEFINE_EVENT (kmem_free, kfree, +DEFINE_EVENT_CONDITION (kmem_free, kfree, TP_PROTO(unsigned long call_site, const void *ptr), -TP_ARGS (call_site, ptr) +TP_ARGS (call_site, ptr), + +/* +* This trace can be potentially called from an offlined cpu. +* Since trace points use RCU and RCU should not be used from +* offline cpus, filter such calls out. +* While this trace can be called from a preemptable section, +* it has no impact on the condition since tasks can migrate +* only from online cpus to other online cpus. Thus its safe +* to use raw_smp_processor_id. +*/ +TP_CONDITION (cpu_online (raw_smp_processor_id ())) ); DEFINE_EVENT_CONDITION(kmem_free, kmem_cache_free,
Signature · 1 line
2.4.0
PATCH v2 2/2 tracing/rcu: don't trace rcu_callback on offline CPUs
Tracepoints use RCU for protection and they must not be called on offline CPUS. So make this tracepoint conditional.
NFO: suspicious RCU usage. ]
[ 413.344670] 4.4.0-00006-g0fe53e8-dirty #33 Tainted: G S [ 413.344672] ------------------------------- [ 413.344673] include/trace/events/rcu.h:457 suspicious rcu_dereference_check() usage! [ 413.344674]
other info that might help us debug this:
[ 413.344676] RCU used illegally from offline CPU! rcu_scheduler_active = 1, debug_locks = 1 [ 413.344678] no locks held by swapper/4/0. [ 413.344679] stack backtrace: [ 413.344682] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G S 4.4.0-00006-g0fe53e8-dirty #33 [ 413.344683] Call Trace: [ 413.344692] [c0000005b76b7800] [c0000000008bb080] .dump_stack+0x98/0xd4 (unreliable) [ 413.344698] [c0000005b76b7880] [c00000000010c8b8] .lockdep_rcu_suspicious+0x108/0x170 [ 413.344703] [c0000005b76b7910] [c00000000013b9e4] .__call_rcu.constprop.60+0x264/0x600 [ 413.344708] [c0000005b76b79e0] [c0000000002bceec] .put_object+0x5c/0x80 [ 413.344712] [c0000005b76b7a60] [c00000000029a368] .kmem_cache_free+0x298/0x450 [ 413.344716] [c0000005b76b7b00] [c000000000093494] .__mmdrop+0x54/0x150 [ 413.344720] [c0000005b76b7b90] [c0000000000e4010] .idle_task_exit+0x130/0x140 [ 413.344725] [c0000005b76b7c20] [c000000000075804] .pseries_mach_cpu_die+0x64/0x310 [ 413.344730] [c0000005b76b7cd0] [c000000000043e7c] .cpu_die+0x3c/0x60 [ 413.344734] [c0000005b76b7d40] [c0000000000188d8] .arch_cpu_idle_dead+0x28/0x40 [ 413.344738] [c0000005b76b7db0] [c000000000101e8c] .cpu_startup_entry+0x50c/0x560 [ 413.344741] [c0000005b76b7ed0] [c000000000043bd8] .start_secondary+0x328/0x360 [ 413.344746] [c0000005b76b7f90] [c000000000008a6c] start_secondary_prolog+0x10/0x14
---
v2: Fix the build error that was made
while sending the pacthes from another machine
include/trace/events/rcu.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
include/trace/events/rcu.h +3 -1
@@ -428,13 +428,15 @@ TRACE_EVENT(rcu_prep_idle,* number of lazy callbacks queued, and the fourth element is the * total number of callbacks queued. */ -TRACE_EVENT(rcu_callback, +TRACE_EVENT_CONDITION(rcu_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen_lazy, long qlen), TP_ARGS(rcuname, rhp, qlen_lazy, qlen), + TP_CONDITION(cpu_online(raw_smp_processor_id())), + TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp)
Signature · 1 line
2.4.0
Re: [PATCH v2 2/2] tracing/rcu: don't trace rcu_callback on offline CPUs
Quoted 2 lines · Denis Kirjanov
Tracepoints use RCU for protection and they must not be called on offline CPUS. So make this tracepoint conditional.
Good catch! Queued for review and testing.
Thanx, Paul
Quoted 62 lines
NFO: suspicious RCU usage. ]
[ 413.344670] 4.4.0-00006-g0fe53e8-dirty #33 Tainted: G S [ 413.344672] ------------------------------- [ 413.344673] include/trace/events/rcu.h:457 suspicious rcu_dereference_check() usage! [ 413.344674]other info that might help us debug this:
[ 413.344676] RCU used illegally from offline CPU! rcu_scheduler_active = 1, debug_locks = 1 [ 413.344678] no locks held by swapper/4/0. [ 413.344679] stack backtrace: [ 413.344682] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G S 4.4.0-00006-g0fe53e8-dirty #33 [ 413.344683] Call Trace: [ 413.344692] [c0000005b76b7800] [c0000000008bb080] .dump_stack+0x98/0xd4 (unreliable) [ 413.344698] [c0000005b76b7880] [c00000000010c8b8] .lockdep_rcu_suspicious+0x108/0x170 [ 413.344703] [c0000005b76b7910] [c00000000013b9e4] .__call_rcu.constprop.60+0x264/0x600 [ 413.344708] [c0000005b76b79e0] [c0000000002bceec] .put_object+0x5c/0x80 [ 413.344712] [c0000005b76b7a60] [c00000000029a368] .kmem_cache_free+0x298/0x450 [ 413.344716] [c0000005b76b7b00] [c000000000093494] .__mmdrop+0x54/0x150 [ 413.344720] [c0000005b76b7b90] [c0000000000e4010] .idle_task_exit+0x130/0x140 [ 413.344725] [c0000005b76b7c20] [c000000000075804] .pseries_mach_cpu_die+0x64/0x310 [ 413.344730] [c0000005b76b7cd0] [c000000000043e7c] .cpu_die+0x3c/0x60 [ 413.344734] [c0000005b76b7d40] [c0000000000188d8] .arch_cpu_idle_dead+0x28/0x40 [ 413.344738] [c0000005b76b7db0] [c000000000101e8c] .cpu_startup_entry+0x50c/0x560 [ 413.344741] [c0000005b76b7ed0] [c000000000043bd8] .start_secondary+0x328/0x360 [ 413.344746] [c0000005b76b7f90] [c000000000008a6c] start_secondary_prolog+0x10/0x14Signed-off-by Denis Kirjanov <[email protected]>---
v2: Fix the build error that was made
while sending the pacthes from another machineinclude/trace/events/rcu.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)include/trace/events/rcu.h +3
-1@@ -428,13 +428,15 @@ TRACE_EVENT(rcu_prep_idle,* number of lazy callbacks queued, and the fourth element is the * total number of callbacks queued. */ -TRACE_EVENT(rcu_callback, +TRACE_EVENT_CONDITION(rcu_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen_lazy, long qlen), TP_ARGS(rcuname, rhp, qlen_lazy, qlen), + TP_CONDITION(cpu_online(raw_smp_processor_id())), + TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp)Signature · 1 line
2.4.0
Re: [PATCH v2 2/2] tracing/rcu: don't trace rcu_callback on offline CPUs
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines · Denis Kirjanov
Tracepoints use RCU for protection and they must not be called on offline CPUS. So make this tracepoint conditional.
Good catch! Queued for review and testing.
And I dequeued this in favor of Steven's more recent patch.
But there is one other hitch... It is currently not legal to invoke call_rcu() from an offline CPU. You can get away with it during a short window towards the end of the offline process, but shortly after the outgoing CPU hits the idle loop, call_rcu() will splat and leak the callback.
So what exactly is the purpose of invoking call_rcu() from an offline CPU? (Yes, I could probably make it work, but there needs to be a good reason.)
Thanx, Paul
Quoted 62 lines
Quoted 62 lines
NFO: suspicious RCU usage. ]
[ 413.344670] 4.4.0-00006-g0fe53e8-dirty #33 Tainted: G S [ 413.344672] ------------------------------- [ 413.344673] include/trace/events/rcu.h:457 suspicious rcu_dereference_check() usage! [ 413.344674]other info that might help us debug this:
[ 413.344676] RCU used illegally from offline CPU! rcu_scheduler_active = 1, debug_locks = 1 [ 413.344678] no locks held by swapper/4/0. [ 413.344679] stack backtrace: [ 413.344682] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G S 4.4.0-00006-g0fe53e8-dirty #33 [ 413.344683] Call Trace: [ 413.344692] [c0000005b76b7800] [c0000000008bb080] .dump_stack+0x98/0xd4 (unreliable) [ 413.344698] [c0000005b76b7880] [c00000000010c8b8] .lockdep_rcu_suspicious+0x108/0x170 [ 413.344703] [c0000005b76b7910] [c00000000013b9e4] .__call_rcu.constprop.60+0x264/0x600 [ 413.344708] [c0000005b76b79e0] [c0000000002bceec] .put_object+0x5c/0x80 [ 413.344712] [c0000005b76b7a60] [c00000000029a368] .kmem_cache_free+0x298/0x450 [ 413.344716] [c0000005b76b7b00] [c000000000093494] .__mmdrop+0x54/0x150 [ 413.344720] [c0000005b76b7b90] [c0000000000e4010] .idle_task_exit+0x130/0x140 [ 413.344725] [c0000005b76b7c20] [c000000000075804] .pseries_mach_cpu_die+0x64/0x310 [ 413.344730] [c0000005b76b7cd0] [c000000000043e7c] .cpu_die+0x3c/0x60 [ 413.344734] [c0000005b76b7d40] [c0000000000188d8] .arch_cpu_idle_dead+0x28/0x40 [ 413.344738] [c0000005b76b7db0] [c000000000101e8c] .cpu_startup_entry+0x50c/0x560 [ 413.344741] [c0000005b76b7ed0] [c000000000043bd8] .start_secondary+0x328/0x360 [ 413.344746] [c0000005b76b7f90] [c000000000008a6c] start_secondary_prolog+0x10/0x14Signed-off-by Denis Kirjanov <[email protected]>---
v2: Fix the build error that was made
while sending the pacthes from another machineinclude/trace/events/rcu.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)include/trace/events/rcu.h +3
-1@@ -428,13 +428,15 @@ TRACE_EVENT(rcu_prep_idle,* number of lazy callbacks queued, and the fourth element is the * total number of callbacks queued. */ -TRACE_EVENT(rcu_callback, +TRACE_EVENT_CONDITION(rcu_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen_lazy, long qlen), TP_ARGS(rcuname, rhp, qlen_lazy, qlen), + TP_CONDITION(cpu_online(raw_smp_processor_id())), + TP_STRUCT__entry( __field(const char *, rcuname) __field(void *, rhp)Signature · 1 line
2.4.0