Skip to contentSkip to conversation

INFO: suspicious RCU usage

BUG -tip/sched Yong Zhang 3 participants 5 messages Dec 7, 2011 – Dec 8, 2011 Scheduler

lore
Share & tools
lore

Yong Zhang

author
#0 lore raw

===============================
[ INFO: suspicious RCU usage. ]
-------------------------------
/build/linux/kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

rcu_scheduler_active = 1, debug_locks = 1
1 lock held by swapper/0/1:
#0: (&sb->s_type->i_mutex_key#2){+.+.+.}, at: [<ffffffff812a0e0c>] debugfs_create_file+0x8c/0x270

stack backtrace:
Pid: 1, comm: swapper/0 Not tainted 3.2.0-rc1-00339-gcd490c5 #14
Call Trace:
 <IRQ>  [<ffffffff815f9968>] ? _raw_spin_unlock_irqrestore+0x38/0x80
 [<ffffffff8109407f>] lockdep_rcu_suspicious+0xbf/0xe0
 [<ffffffff8107dd48>] trigger_load_balance+0x5b8/0x5c0
 [<ffffffff81070544>] scheduler_tick+0x104/0x160
 [<ffffffff8104a1ce>] update_process_times+0x6e/0x90
 [<ffffffff8108e32c>] tick_periodic+0x2c/0x70
 [<ffffffff8108e394>] tick_handle_periodic+0x24/0x80
 [<ffffffff812a07a7>] ? T.562+0xf7/0x170
 [<ffffffff815fce56>] smp_apic_timer_interrupt+0x66/0x98
 [<ffffffff815fb2f0>] apic_timer_interrupt+0x70/0x80
 <EOI>  [<ffffffff810984a4>] ? lock_release+0xe4/0x2b0
 [<ffffffff815f98f3>] _raw_spin_unlock+0x23/0x60
 [<ffffffff812a07a7>] T.562+0xf7/0x170
 [<ffffffff812a0e71>] debugfs_create_file+0xf1/0x270
 [<ffffffff810e96c9>] trace_create_file+0x19/0x50
 [<ffffffff81af7b3c>] tracer_init_debugfs+0x2bd/0x3c0
 [<ffffffff810e96c9>] ? trace_create_file+0x19/0x50
 [<ffffffff81af787f>] ? clear_boot_tracer+0x2d/0x2d
 [<ffffffff810001d3>] do_one_initcall+0x43/0x180
 [<ffffffff81adf637>] kernel_init+0xd0/0x156
 [<ffffffff815fc574>] kernel_thread_helper+0x4/0x10
 [<ffffffff8106fdb8>] ? finish_task_switch+0x88/0xf0
 [<ffffffff815f99eb>] ? _raw_spin_unlock_irq+0x3b/0x70
 [<ffffffff815f9f61>] ? retint_restore_args+0xe/0xe
 [<ffffffff81adf567>] ? parse_early_options+0x20/0x20
 [<ffffffff815fc570>] ? gs_change+0xb/0xb

Seems due to commit [0b005cf5: sched, nohz: Implement sched group, domain aware nohz idle load balancing]

Thanks,
Yong

#1 lore raw
Quoted 42 lines · Yong Zhang

===============================
[ INFO: suspicious RCU usage. ]
-------------------------------
/build/linux/kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!

other info that might help us debug this:

rcu_scheduler_active = 1, debug_locks = 1
1 lock held by swapper/0/1:
#0: (&sb->s_type->i_mutex_key#2){+.+.+.}, at: [<ffffffff812a0e0c>] debugfs_create_file+0x8c/0x270

stack backtrace:
Pid: 1, comm: swapper/0 Not tainted 3.2.0-rc1-00339-gcd490c5 #14
Call Trace:
 <IRQ>  [<ffffffff815f9968>] ? _raw_spin_unlock_irqrestore+0x38/0x80
 [<ffffffff8109407f>] lockdep_rcu_suspicious+0xbf/0xe0
 [<ffffffff8107dd48>] trigger_load_balance+0x5b8/0x5c0
 [<ffffffff81070544>] scheduler_tick+0x104/0x160
 [<ffffffff8104a1ce>] update_process_times+0x6e/0x90
 [<ffffffff8108e32c>] tick_periodic+0x2c/0x70
 [<ffffffff8108e394>] tick_handle_periodic+0x24/0x80
 [<ffffffff812a07a7>] ? T.562+0xf7/0x170
 [<ffffffff815fce56>] smp_apic_timer_interrupt+0x66/0x98
 [<ffffffff815fb2f0>] apic_timer_interrupt+0x70/0x80
 <EOI>  [<ffffffff810984a4>] ? lock_release+0xe4/0x2b0
 [<ffffffff815f98f3>] _raw_spin_unlock+0x23/0x60
 [<ffffffff812a07a7>] T.562+0xf7/0x170
 [<ffffffff812a0e71>] debugfs_create_file+0xf1/0x270
 [<ffffffff810e96c9>] trace_create_file+0x19/0x50
 [<ffffffff81af7b3c>] tracer_init_debugfs+0x2bd/0x3c0
 [<ffffffff810e96c9>] ? trace_create_file+0x19/0x50
 [<ffffffff81af787f>] ? clear_boot_tracer+0x2d/0x2d
 [<ffffffff810001d3>] do_one_initcall+0x43/0x180
 [<ffffffff81adf637>] kernel_init+0xd0/0x156
 [<ffffffff815fc574>] kernel_thread_helper+0x4/0x10
 [<ffffffff8106fdb8>] ? finish_task_switch+0x88/0xf0
 [<ffffffff815f99eb>] ? _raw_spin_unlock_irq+0x3b/0x70
 [<ffffffff815f9f61>] ? retint_restore_args+0xe/0xe
 [<ffffffff81adf567>] ? parse_early_options+0x20/0x20
 [<ffffffff815fc570>] ? gs_change+0xb/0xb
Quoted 2 lines

Seems due to commit [0b005cf5: sched, nohz: Implement sched group, domain aware nohz idle load balancing]

I think something like the below should suffice, the sd argument only lives inside the for_each_domain() so wrapping the whole block should suffice.

---
 kernel/sched/fair.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/sched/fair.c +7 -2
index 6482136..a4d2b7a 100644
@@ -5088,23 +5088,28 @@ static inline int nohz_kick_needed(struct rq *rq, int cpu)
 	if (rq->nr_running >= 2)
 		goto need_kick;
 
+	rcu_read_lock();
 	for_each_domain(cpu, sd) {
 		struct sched_group *sg = sd->groups;
 		struct sched_group_power *sgp = sg->sgp;
 		int nr_busy = atomic_read(&sgp->nr_busy_cpus);
 
 		if (sd->flags & SD_SHARE_PKG_RESOURCES && nr_busy > 1)
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (sd->flags & SD_ASYM_PACKING && nr_busy != sg->group_weight
 		    && (cpumask_first_and(nohz.idle_cpus_mask,
 					  sched_domain_span(sd)) < cpu))
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (!(sd->flags & (SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING)))
 			break;
 	}
+	rcu_read_unlock();
 	return 0;
+
+need_kick_unlock:
+	rcu_read_unlock();
 need_kick:
 	return 1;
 }
maintainer
#2 lore raw

I made that

---
Subject: sched, nohz: Fix missing RCU read lock
From: Peter Zijlstra <[email protected]>
Date: Wed Dec 07 14:28:44 CET 2011

Yong Zhang reported:

Quoted 2 lines

[ INFO: suspicious RCU usage. ] kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!

This is due to the sched_domain stuff being RCU protected and commit 0b005cf5 ("sched, nohz: Implement sched group, domain aware nohz idle load balancing") overlooking this fact.

The sd variable only lives inside the for_each_domain() block, so we only need to wrap that.

---

kernel/sched/fair.c +7 -2
Index: linux-2.6/kernel/sched/fair.c
===================================================================
@@ -5088,23 +5088,28 @@ static inline int nohz_kick_needed(struc
 	if (rq->nr_running >= 2)
 		goto need_kick;
 
+	rcu_read_lock();
 	for_each_domain(cpu, sd) {
 		struct sched_group *sg = sd->groups;
 		struct sched_group_power *sgp = sg->sgp;
 		int nr_busy = atomic_read(&sgp->nr_busy_cpus);
 
 		if (sd->flags & SD_SHARE_PKG_RESOURCES && nr_busy > 1)
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (sd->flags & SD_ASYM_PACKING && nr_busy != sg->group_weight
 		    && (cpumask_first_and(nohz.idle_cpus_mask,
 					  sched_domain_span(sd)) < cpu))
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (!(sd->flags & (SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING)))
 			break;
 	}
+	rcu_read_unlock();
 	return 0;
+
+need_kick_unlock:
+	rcu_read_unlock();
 need_kick:
 	return 1;
 }
author Tested-by
#3 lore raw
Quoted 12 lines · Peter Zijlstra

I made that

---
Subject: sched, nohz: Fix missing RCU read lock
From: Peter Zijlstra <[email protected]>
Date: Wed Dec 07 14:28:44 CET 2011

Yong Zhang reported:

Quoted 2 lines

[ INFO: suspicious RCU usage. ] kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!

Quoted 7 lines

This is due to the sched_domain stuff being RCU protected and commit 0b005cf5 ("sched, nohz: Implement sched group, domain aware nohz idle load balancing") overlooking this fact.

The sd variable only lives inside the for_each_domain() block, so we only need to wrap that.

Works for me.

Thanks,
Yong

Quoted 46 lines

---
Index: linux-2.6/kernel/sched/fair.c
===================================================================

kernel/sched/fair.c +7 -3
@@ -5088,23 +5088,28 @@ static inline int nohz_kick_needed(struc
 	if (rq->nr_running >= 2)
 		goto need_kick;
 
+	rcu_read_lock();
 	for_each_domain(cpu, sd) {
 		struct sched_group *sg = sd->groups;
 		struct sched_group_power *sgp = sg->sgp;
 		int nr_busy = atomic_read(&sgp->nr_busy_cpus);
 
 		if (sd->flags & SD_SHARE_PKG_RESOURCES && nr_busy > 1)
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (sd->flags & SD_ASYM_PACKING && nr_busy != sg->group_weight
 		    && (cpumask_first_and(nohz.idle_cpus_mask,
 					  sched_domain_span(sd)) < cpu))
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (!(sd->flags & (SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING)))
 			break;
 	}
+	rcu_read_unlock();
 	return 0;
+
+need_kick_unlock:
+	rcu_read_unlock();
 need_kick:
 	return 1;
 }
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 1 line
Only stand for myself
#4 lore raw

[tip:sched/core] sched, nohz: Fix missing RCU read lock

In reply to Peter Zijlstra
Commit-ID:  067491b7313c41f49607fce782d29344d1472587
Gitweb:     http://git.kernel.org/tip/067491b7313c41f49607fce782d29344d1472587
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Wed, 7 Dec 2011 14:32:08 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 8 Dec 2011 05:45:48 +0100

sched, nohz: Fix missing RCU read lock

Yong Zhang reported:

Quoted 2 lines

[ INFO: suspicious RCU usage. ] kernel/sched/fair.c:5091 suspicious rcu_dereference_check() usage!

This is due to the sched_domain stuff being RCU protected and commit 0b005cf5 ("sched, nohz: Implement sched group, domain aware nohz idle load balancing") overlooking this fact.

The sd variable only lives inside the for_each_domain() block, so we only need to wrap that.

---
 kernel/sched/fair.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
kernel/sched/fair.c +7 -2
index 6482136..a4d2b7a 100644
@@ -5088,23 +5088,28 @@ static inline int nohz_kick_needed(struct rq *rq, int cpu)
 	if (rq->nr_running >= 2)
 		goto need_kick;
 
+	rcu_read_lock();
 	for_each_domain(cpu, sd) {
 		struct sched_group *sg = sd->groups;
 		struct sched_group_power *sgp = sg->sgp;
 		int nr_busy = atomic_read(&sgp->nr_busy_cpus);
 
 		if (sd->flags & SD_SHARE_PKG_RESOURCES && nr_busy > 1)
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (sd->flags & SD_ASYM_PACKING && nr_busy != sg->group_weight
 		    && (cpumask_first_and(nohz.idle_cpus_mask,
 					  sched_domain_span(sd)) < cpu))
-			goto need_kick;
+			goto need_kick_unlock;
 
 		if (!(sd->flags & (SD_SHARE_PKG_RESOURCES | SD_ASYM_PACKING)))
 			break;
 	}
+	rcu_read_unlock();
 	return 0;
+
+need_kick_unlock:
+	rcu_read_unlock();
 need_kick:
 	return 1;
 }