Skip to contentSkip to conversation

Fix hibernate/resume in presence of PREEMPT_RCU and hotplug

PATCH In review Paul E. McKenney 3 participants 7 messages Feb 28, 2008

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix hibernate/resume in presence of PREEMPT_RCU and hotplug

Hello!

This fixes a oops encountered when doing hibernate/resume in presence of PREEMPT_RCU. The problem was that the code failed to disable preemption when accessing a per-CPU variable. This is OK when called from code that already has preemption disabled, but such is not the case from the suspend/resume code path.

---

 rcupreempt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcupreempt.c +2 -1
diff -urpNa -X dontdiff linux-2.6.25-rc3/kernel/rcupreempt.c linux-2.6.25-rc3-rcuoffline/kernel/rcupreempt.c
@@ -702,8 +702,9 @@ void rcu_offline_cpu(int cpu)
 	 * fix.
 	 */
 
+	local_irq_save(flags);
 	rdp = RCU_DATA_ME();
-	spin_lock_irqsave(&rdp->lock, flags);
+	spin_lock(&rdp->lock);
 	*rdp->nexttail = list;
 	if (list)
 		rdp->nexttail = tail;
Reviewed-by on v1 1/1
#1 lore raw
Quoted 7 lines · Paul E. McKenney

Hello!

This fixes a oops encountered when doing hibernate/resume in presence of PREEMPT_RCU. The problem was that the code failed to disable preemption when accessing a per-CPU variable. This is OK when called from code that already has preemption disabled, but such is not the case from the suspend/resume code path.

Well, rcu_offline_cpu() is called from the cpu-offline callback path, and AFAICS, it doesn't disable preemption.

I wonder how we're seeing this only now! Indeed a good catch!

Quoted 23 lines

---

 rcupreempt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff -urpNa -X dontdiff linux-2.6.25-rc3/kernel/rcupreempt.c linux-2.6.25-rc3-rcuoffline/kernel/rcupreempt.c

kernel/rcupreempt.c +2 -1
@@ -702,8 +702,9 @@ void rcu_offline_cpu(int cpu)
 	 * fix.
 	 */
 
+	local_irq_save(flags);
 	rdp = RCU_DATA_ME();
-	spin_lock_irqsave(&rdp->lock, flags);
+	spin_lock(&rdp->lock);
 	*rdp->nexttail = list;
 	if (list)
 		rdp->nexttail = tail;
Signature · 2 lines
Thanks and Regards
gautham
#2 lore raw
Quoted 9 lines · Gautham R Shenoy
Quoted 7 lines · Paul E. McKenney

Hello!

This fixes a oops encountered when doing hibernate/resume in presence of PREEMPT_RCU. The problem was that the code failed to disable preemption when accessing a per-CPU variable. This is OK when called from code that already has preemption disabled, but such is not the case from the suspend/resume code path.

Quoted 4 lines

Well, rcu_offline_cpu() is called from the cpu-offline callback path, and AFAICS, it doesn't disable preemption.

I wonder how we're seeing this only now! Indeed a good catch!

I wonder as well. In fact there is (at least) one other such bug, submitting patch separately. :-/

Thanx, Paul

Quoted 29 lines
Quoted 23 lines

---

 rcupreempt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff -urpNa -X dontdiff linux-2.6.25-rc3/kernel/rcupreempt.c linux-2.6.25-rc3-rcuoffline/kernel/rcupreempt.c

kernel/rcupreempt.c +2 -1
@@ -702,8 +702,9 @@ void rcu_offline_cpu(int cpu)
 	 * fix.
 	 */
 
+	local_irq_save(flags);
 	rdp = RCU_DATA_ME();
-	spin_lock_irqsave(&rdp->lock, flags);
+	spin_lock(&rdp->lock);
 	*rdp->nexttail = list;
 	if (list)
 		rdp->nexttail = tail;
Signature · 2 lines
Thanks and Regards
gautham
#3 lore raw

PATCH 1/1 Remove never-migrates assumption from rcu_process_callbacks()

In reply to Paul E. McKenney

Hello again!

This patch fixes a potentially invalid access to a per-CPU variable in rcu_process_callbacks(). This per-CPU access needs to be done in such a way as to guarantee that the code using it cannot move to some other CPU before all uses of the value accessed have completed. Even though this code is currently only invoked from softirq context, which currrently cannot migrate to some other CPU, life would be better if this code did not silently make such an assumption.

---

 rcupreempt.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/rcupreempt.c +4 -2
diff -urpNa -X dontdiff linux-2.6.25-rc3-rcuoffline/kernel/rcupreempt.c linux-2.6.25-rc3-rcuoffline-fp/kernel/rcupreempt.c
@@ -736,9 +736,11 @@ static void rcu_process_callbacks(struct
 {
 	unsigned long flags;
 	struct rcu_head *next, *list;
-	struct rcu_data *rdp = RCU_DATA_ME();
+	struct rcu_data *rdp;
 
-	spin_lock_irqsave(&rdp->lock, flags);
+	local_irq_save(flags);
+	rdp = RCU_DATA_ME();
+	spin_lock(&rdp->lock);
 	list = rdp->donelist;
 	if (list == NULL) {
 		spin_unlock_irqrestore(&rdp->lock, flags);
Tested-by Dave Young on v1 1/1
#4 lore raw

Re: [PATCH] Remove never-migrates assumption from rcu_process_callbacks()

Quoted 9 lines · Paul E. McKenney

Hello again!

This patch fixes a potentially invalid access to a per-CPU variable in rcu_process_callbacks(). This per-CPU access needs to be done in such a way as to guarantee that the code using it cannot move to some other CPU before all uses of the value accessed have completed. Even though this code is currently only invoked from softirq context, which currrently cannot migrate to some other CPU, life would be better if this code did not silently make such an assumption.

i've got the patch below queued up already - same thing, right?

Ingo

----------->
Subject: rcupreempt: fix hibernate/resume in presence of PREEMPT_RCU and hotplug
From: "Paul E. McKenney" <[email protected]>
Date: Wed, 27 Feb 2008 16:21:10 -0800

This fixes a oops encountered when doing hibernate/resume in presence of PREEMPT_RCU. The problem was that the code failed to disable preemption when accessing a per-CPU variable. This is OK when called from code that already has preemption disabled, but such is not the case from the suspend/resume code path.

---
 kernel/rcupreempt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcupreempt.c +2 -1
Index: linux/kernel/rcupreempt.c
===================================================================
@@ -918,8 +918,9 @@ void rcu_offline_cpu(int cpu)
 	 * fix.
 	 */
 
+	local_irq_save(flags);
 	rdp = RCU_DATA_ME();
-	spin_lock_irqsave(&rdp->lock, flags);
+	spin_lock(&rdp->lock);
 	*rdp->nexttail = list;
 	if (list)
 		rdp->nexttail = tail;
#5 lore raw

Re: [PATCH] Remove never-migrates assumption from rcu_process_callbacks()

Quoted 1 line · Ingo Molnar

i've got the patch below queued up already - same thing, right?

oh, not the same thing - just looking very similar.

queued up the second patch too :-)

Ingo

#6 lore raw

Re: [PATCH] Remove never-migrates assumption from rcu_process_callbacks()

Quoted 8 lines · Ingo Molnar
Quoted 1 line · Ingo Molnar

i've got the patch below queued up already - same thing, right?

oh, not the same thing - just looking very similar.

queued up the second patch too :-)

Thank you, Ingo!

/me goes back to analyzing the rcu-preempt code...

Thanx, Paul