Skip to contentSkip to conversation

sched: Teach might_sleep about preemptable rcu

PATCH In review Frederic Weisbecker 3 participants 13 messages Dec 14, 2009 – Dec 17, 2009 RCU

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched: Teach might_sleep about preemptable rcu

In practice, it is harmless to voluntarily sleep in a rcu_read_lock() section if we are running under preempt rcu, but it is illegal because if we build a kernel running non-preemptable rcu.

Currently, might_sleep() doesn't notice sleepable operations under rcu_read_lock() sections if we are running under preemptable rcu because preempt_count() is left untouched after rcu_read_lock() in this case. But we want developers who test their changes under such config to notice the "sleeping while atomic" issues.

Then we add rcu_read_lock_nesting to prempt_count() in might_sleep() checks.

Cc "Paul E. McKenney" <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 include/linux/rcutree.h |   11 +++++++++++
 kernel/sched.c          |    2 +-
 2 files changed, 12 insertions(+), 1 deletions(-)
include/linux/rcutree.h +11 -0
index c93eee5..8044b1b 100644
@@ -45,6 +45,12 @@ extern void __rcu_read_unlock(void);
 extern void synchronize_rcu(void);
 extern void exit_rcu(void);
 
+/*
+ * Defined as macro as it is a very low level header
+ * included from areas that don't even know about current
+ */
+#define rcu_preempt_depth() (current->rcu_read_lock_nesting)
+
 #else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock(void)
@@ -63,6 +69,11 @@ static inline void exit_rcu(void)
 {
 }
 
+static inline int rcu_preempt_depth(void)
+{
+	return 0;
+}
+
 #endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock_bh(void)
kernel/sched.c +1 -1
index ab42754..586c82c 100644
@@ -9658,7 +9658,7 @@ void __init sched_init(void)
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
 static inline int preempt_count_equals(int preempt_offset)
 {
-	int nested = preempt_count() & ~PREEMPT_ACTIVE;
+	int nested = (preempt_count() & ~PREEMPT_ACTIVE) + rcu_preempt_depth();
 
 	return (nested == PREEMPT_INATOMIC_BASE + preempt_offset);
 }
Signature · 1 line
1.6.2.3
Reviewed-by on v1 1/1
#1 lore raw
Quoted 4 lines · Frederic Weisbecker

In practice, it is harmless to voluntarily sleep in a rcu_read_lock() section if we are running under preempt rcu, but it is illegal because if we build a kernel running non-preemptable rcu.

Quoted 8 lines

Currently, might_sleep() doesn't notice sleepable operations under rcu_read_lock() sections if we are running under preemptable rcu because preempt_count() is left untouched after rcu_read_lock() in this case. But we want developers who test their changes under such config to notice the "sleeping while atomic" issues.

Then we add rcu_read_lock_nesting to prempt_count() in might_sleep() checks.

Cute!!!

Quoted 53 lines
Cc "Paul E. McKenney" <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 include/linux/rcutree.h |   11 +++++++++++
 kernel/sched.c          |    2 +-
 2 files changed, 12 insertions(+), 1 deletions(-)
include/linux/rcutree.h +11 -0
index c93eee5..8044b1b 100644
@@ -45,6 +45,12 @@ extern void __rcu_read_unlock(void);
 extern void synchronize_rcu(void);
 extern void exit_rcu(void);
 
+/*
+ * Defined as macro as it is a very low level header
+ * included from areas that don't even know about current
+ */
+#define rcu_preempt_depth() (current->rcu_read_lock_nesting)
+
 #else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock(void)
@@ -63,6 +69,11 @@ static inline void exit_rcu(void)
 {
 }
 
+static inline int rcu_preempt_depth(void)
+{
+	return 0;
+}
+
 #endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock_bh(void)
kernel/sched.c +1 -1
index ab42754..586c82c 100644
@@ -9658,7 +9658,7 @@ void __init sched_init(void)
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
 static inline int preempt_count_equals(int preempt_offset)
 {
-	int nested = preempt_count() & ~PREEMPT_ACTIVE;
+	int nested = (preempt_count() & ~PREEMPT_ACTIVE) + rcu_preempt_depth();
 
 	return (nested == PREEMPT_INATOMIC_BASE + preempt_offset);
 }
Signature · 1 line
1.6.2.3
#2 lore raw

[tip:sched/urgent] sched: Teach might_sleep() about preemptible RCU

In reply to Frederic Weisbecker
Commit-ID:  ba1b1cbcc9b458c4b9ebb28c9cf6a2ccf64ba1cd
Gitweb:     http://git.kernel.org/tip/ba1b1cbcc9b458c4b9ebb28c9cf6a2ccf64ba1cd
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Mon, 14 Dec 2009 23:44:32 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 16 Dec 2009 08:43:16 +0100

sched: Teach might_sleep() about preemptible RCU

In practice, it is harmless to voluntarily sleep in a rcu_read_lock() section if we are running under preempt rcu, but it is illegal if we build a kernel running non-preemptable rcu.

Currently, might_sleep() doesn't notice sleepable operations under rcu_read_lock() sections if we are running under preemptable rcu because preempt_count() is left untouched after rcu_read_lock() in this case. But we want developers who test their changes under such config to notice the "sleeping while atomic" issues.

So we add rcu_read_lock_nesting to prempt_count() in might_sleep() checks.

LKML-Reference: <[email protected]>

---
 include/linux/rcutree.h |   11 +++++++++++
 kernel/sched.c          |    2 +-
 2 files changed, 12 insertions(+), 1 deletions(-)
include/linux/rcutree.h +11 -0
index c93eee5..8044b1b 100644
@@ -45,6 +45,12 @@ extern void __rcu_read_unlock(void);
 extern void synchronize_rcu(void);
 extern void exit_rcu(void);
 
+/*
+ * Defined as macro as it is a very low level header
+ * included from areas that don't even know about current
+ */
+#define rcu_preempt_depth() (current->rcu_read_lock_nesting)
+
 #else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock(void)
@@ -63,6 +69,11 @@ static inline void exit_rcu(void)
 {
 }
 
+static inline int rcu_preempt_depth(void)
+{
+	return 0;
+}
+
 #endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock_bh(void)
kernel/sched.c +1 -1
index db5c266..d76c790 100644
@@ -9655,7 +9655,7 @@ void __init sched_init(void)
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
 static inline int preempt_count_equals(int preempt_offset)
 {
-	int nested = preempt_count() & ~PREEMPT_ACTIVE;
+	int nested = (preempt_count() & ~PREEMPT_ACTIVE) + rcu_preempt_depth();
 
 	return (nested == PREEMPT_INATOMIC_BASE + preempt_offset);
 }
#3 lore raw

Re: [tip:sched/urgent] sched: Teach might_sleep() about preemptible RCU

Quoted 8 lines · tip-bot for Frederic Weisbecker
Commit-ID:  ba1b1cbcc9b458c4b9ebb28c9cf6a2ccf64ba1cd
Gitweb:     http://git.kernel.org/tip/ba1b1cbcc9b458c4b9ebb28c9cf6a2ccf64ba1cd
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Mon, 14 Dec 2009 23:44:32 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 16 Dec 2009 08:43:16 +0100

sched: Teach might_sleep() about preemptible RCU

Causes a build failure, so skipped this patch for now:

kernel/sched.c:9661: error: implicit declaration of function 'rcu_preempt_depth'

Ingo

#4 lore raw

Re: [tip:sched/urgent] sched: Teach might_sleep() about preemptible RCU

Quoted 10 lines · Ingo Molnar
Quoted 7 lines · tip-bot for Frederic Weisbecker
Commit-ID:  ba1b1cbcc9b458c4b9ebb28c9cf6a2ccf64ba1cd
Gitweb:     http://git.kernel.org/tip/ba1b1cbcc9b458c4b9ebb28c9cf6a2ccf64ba1cd
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Mon, 14 Dec 2009 23:44:32 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 16 Dec 2009 08:43:16 +0100
Quoted 5 lines
Quoted 1 line

sched: Teach might_sleep() about preemptible RCU

Causes a build failure, so skipped this patch for now:

kernel/sched.c:9661: error: implicit declaration of function 'rcu_preempt_depth'

I guess my testing missed something. :-/ Could you please send the .config file that caused this?

Thanx, Paul

#5 lore raw

Re: [tip:sched/urgent] sched: Teach might_sleep() about preemptible RCU

Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Ingo Molnar
Quoted 8 lines · tip-bot for Frederic Weisbecker
Commit-ID:  ba1b1cbcc9b458c4b9ebb28c9cf6a2ccf64ba1cd
Gitweb:     http://git.kernel.org/tip/ba1b1cbcc9b458c4b9ebb28c9cf6a2ccf64ba1cd
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Mon, 14 Dec 2009 23:44:32 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 16 Dec 2009 08:43:16 +0100

sched: Teach might_sleep() about preemptible RCU

Quoted 8 lines
Quoted 3 lines

Causes a build failure, so skipped this patch for now:

kernel/sched.c:9661: error: implicit declaration of function 'rcu_preempt_depth'

I guess my testing missed something. :-/ Could you please send the .config file that caused this?

Thanx, Paul

Looks like the problem is here:

rcupdate.h:

	#if defined(CONFIG_TREE_RCU) || defined(CONFIG_TREE_PREEMPT_RCU)
	#include <linux/rcutree.h>
	#elif defined(CONFIG_TINY_RCU)
	#include <linux/rcutiny.h>

I guess it barfs in rcutiny configs.

I should define these helpers in rcupdate.

#6 lore raw

Re: [tip:sched/urgent] sched: Teach might_sleep() about preemptible RCU

Quoted 4 lines · Frederic Weisbecker

Looks like the problem is here:

rcupdate.h:

Quoted 9 lines
	#if defined(CONFIG_TREE_RCU) || defined(CONFIG_TREE_PREEMPT_RCU)
	#include <linux/rcutree.h>
	#elif defined(CONFIG_TINY_RCU)
	#include <linux/rcutiny.h>

I guess it barfs in rcutiny configs.

I should define these helpers in rcupdate.

Actually I'm going to define the rcu tiny off-case from rcutiny.h It looks like rcupdate.h should really keep only the generic bits.

#7 lore raw

PATCH 1/1 sched: Teach might_sleep() about preemptible RCU

In reply to Ingo Molnar

From: tip-bot for Frederic Weisbecker <[email protected]>

In practice, it is harmless to voluntarily sleep in a rcu_read_lock() section if we are running under preempt rcu, but it is illegal if we build a kernel running non-preemptable rcu.

Currently, might_sleep() doesn't notice sleepable operations under rcu_read_lock() sections if we are running under preemptable rcu because preempt_count() is left untouched after rcu_read_lock() in this case. But we want developers who test their changes under such config to notice the "sleeping while atomic" issues.

So we add rcu_read_lock_nesting to prempt_count() in might_sleep() checks.

v2: Handle rcu tiny

---
 include/linux/rcutiny.h |    5 +++++
 include/linux/rcutree.h |   11 +++++++++++
 kernel/sched.c          |    2 +-
 3 files changed, 17 insertions(+), 1 deletions(-)
include/linux/rcutiny.h +5 -0
index c4ba9a7..96cc307 100644
@@ -101,4 +101,9 @@ static inline void exit_rcu(void)
 {
 }
 
+static inline int rcu_preempt_depth(void)
+{
+	return 0;
+}
+
 #endif /* __LINUX_RCUTINY_H */
include/linux/rcutree.h +11 -0
index c93eee5..8044b1b 100644
@@ -45,6 +45,12 @@ extern void __rcu_read_unlock(void);
 extern void synchronize_rcu(void);
 extern void exit_rcu(void);
 
+/*
+ * Defined as macro as it is a very low level header
+ * included from areas that don't even know about current
+ */
+#define rcu_preempt_depth() (current->rcu_read_lock_nesting)
+
 #else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock(void)
@@ -63,6 +69,11 @@ static inline void exit_rcu(void)
 {
 }
 
+static inline int rcu_preempt_depth(void)
+{
+	return 0;
+}
+
 #endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock_bh(void)
kernel/sched.c +1 -1
index 8a2bfd3..418e638 100644
@@ -9682,7 +9682,7 @@ void __init sched_init(void)
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
 static inline int preempt_count_equals(int preempt_offset)
 {
-	int nested = preempt_count() & ~PREEMPT_ACTIVE;
+	int nested = (preempt_count() & ~PREEMPT_ACTIVE) + rcu_preempt_depth();
 
 	return (nested == PREEMPT_INATOMIC_BASE + preempt_offset);
 }
Signature · 1 line
1.6.2.3
#8 lore raw

Re: [PATCH] sched: Teach might_sleep() about preemptible RCU

Quoted 1 line · Frederic Weisbecker

From: tip-bot for Frederic Weisbecker <[email protected]>

Crap... I think I'm really becoming this cyborg...

Quoted 2 lines

Although I've happily put Paul's Reviewed-by tag in this v2, I'd feel more comfortable if he could confirm it :)

Thanks!

#9 lore raw

Re: [PATCH] sched: Teach might_sleep() about preemptible RCU

Quoted 10 lines · Frederic Weisbecker
Quoted 1 line · Frederic Weisbecker

From: tip-bot for Frederic Weisbecker <[email protected]>

Crap... I think I'm really becoming this cyborg...

Quoted 2 lines

Although I've happily put Paul's Reviewed-by tag in this v2, I'd feel more comfortable if he could confirm it :)

Given that I missed the TINY_RCU problem, I am not sure how much my Reviewed-by tag is worth, but I have no problem attaching it to this updated patch. ;-)

Thanx, Paul

#10 lore raw

Re: [PATCH] sched: Teach might_sleep() about preemptible RCU

Quoted 9 lines · Paul E. McKenney
Quoted 8 lines · Frederic Weisbecker
Quoted 1 line · Frederic Weisbecker

From: tip-bot for Frederic Weisbecker <[email protected]>

Crap... I think I'm really becoming this cyborg...

Quoted 2 lines
Quoted 6 lines
Quoted 2 lines

Although I've happily put Paul's Reviewed-by tag in this v2, I'd feel more comfortable if he could confirm it :)

Given that I missed the TINY_RCU problem, I am not sure how much my Reviewed-by tag is worth, but I have no problem attaching it to this updated patch. ;-)

Reviewed-by is valuable in terms of you both acking the patch and expressing that you thought it through and like it. It does not mean you have done a mental kernel build test of all 2^3000 kernel .config variants and have proven the patch correct under circumstances! :-)

So yes, i've added it - thanks Paul!

Ingo

#11 lore raw

Re: [PATCH] sched: Teach might_sleep() about preemptible RCU

Quoted 19 lines · Ingo Molnar
Quoted 9 lines · Paul E. McKenney
Quoted 8 lines · Frederic Weisbecker
Quoted 1 line · Frederic Weisbecker

From: tip-bot for Frederic Weisbecker <[email protected]>

Crap... I think I'm really becoming this cyborg...

Quoted 2 lines
Quoted 6 lines
Quoted 2 lines

Although I've happily put Paul's Reviewed-by tag in this v2, I'd feel more comfortable if he could confirm it :)

Given that I missed the TINY_RCU problem, I am not sure how much my Reviewed-by tag is worth, but I have no problem attaching it to this updated patch. ;-)

Quoted 6 lines

Reviewed-by is valuable in terms of you both acking the patch and expressing that you thought it through and like it. It does not mean you have done a mental kernel build test of all 2^3000 kernel .config variants and have proven the patch correct under circumstances! :-)

So yes, i've added it - thanks Paul!

The one misgiving I have is that addition of threaded interrupts and sleeping spinlocks might break this, but those changes will be large enough that this is the least of the worries. We might then need to split might_sleep() into might_sleep() and might_sleep_rt(), but hard to say before the fact. And this check will be very worthwhile as-is in the meantime! ;-)

Thanx, Paul

#12 lore raw

[tip:sched/urgent] sched: Teach might_sleep() about preemptible RCU

In reply to Frederic Weisbecker
Commit-ID:  234da7bcdc7aaa935846534c3b726dbc79a9cdd5
Gitweb:     http://git.kernel.org/tip/234da7bcdc7aaa935846534c3b726dbc79a9cdd5
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Wed, 16 Dec 2009 20:21:05 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 17 Dec 2009 09:46:44 +0100

sched: Teach might_sleep() about preemptible RCU

In practice, it is harmless to voluntarily sleep in a rcu_read_lock() section if we are running under preempt rcu, but it is illegal if we build a kernel running non-preemptable rcu.

Currently, might_sleep() doesn't notice sleepable operations under rcu_read_lock() sections if we are running under preemptable rcu because preempt_count() is left untouched after rcu_read_lock() in this case. But we want developers who test their changes under such config to notice the "sleeping while atomic" issues.

So we add rcu_read_lock_nesting to prempt_count() in might_sleep() checks.

[ v2: Handle rcu-tiny ]
Cc Peter Zijlstra <[email protected]>

LKML-Reference: <[email protected]>

---
 include/linux/rcutiny.h |    5 +++++
 include/linux/rcutree.h |   11 +++++++++++
 kernel/sched.c          |    2 +-
 3 files changed, 17 insertions(+), 1 deletions(-)
include/linux/rcutiny.h +5 -0
index c4ba9a7..96cc307 100644
@@ -101,4 +101,9 @@ static inline void exit_rcu(void)
 {
 }
 
+static inline int rcu_preempt_depth(void)
+{
+	return 0;
+}
+
 #endif /* __LINUX_RCUTINY_H */
include/linux/rcutree.h +11 -0
index c93eee5..8044b1b 100644
@@ -45,6 +45,12 @@ extern void __rcu_read_unlock(void);
 extern void synchronize_rcu(void);
 extern void exit_rcu(void);
 
+/*
+ * Defined as macro as it is a very low level header
+ * included from areas that don't even know about current
+ */
+#define rcu_preempt_depth() (current->rcu_read_lock_nesting)
+
 #else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock(void)
@@ -63,6 +69,11 @@ static inline void exit_rcu(void)
 {
 }
 
+static inline int rcu_preempt_depth(void)
+{
+	return 0;
+}
+
 #endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */
 
 static inline void __rcu_read_lock_bh(void)
kernel/sched.c +1 -1
index af7dfa7..7be88a7 100644
@@ -9682,7 +9682,7 @@ void __init sched_init(void)
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
 static inline int preempt_count_equals(int preempt_offset)
 {
-	int nested = preempt_count() & ~PREEMPT_ACTIVE;
+	int nested = (preempt_count() & ~PREEMPT_ACTIVE) + rcu_preempt_depth();
 
 	return (nested == PREEMPT_INATOMIC_BASE + preempt_offset);
 }