race of RCU vs NOHU
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.
The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.
Now if I look at the RCU data structures I find this:
rcu_ctrlblk
Quoted 1 line
Quoted 1 line
px *(struct
rcu_ctrlblk*)0x810000
struct rcu_ctrlblk {
cur = 0xffffffffffffff99
completed = 0xffffffffffffff98
pending = 0xffffffffffffff99
signaled = 0x0
lock = spinlock_t {
raw_lock = raw_spinlock_t {
owner_cpu = 0x0
}
break_lock = 0x0
magic = 0xdead4ead
owner_cpu = 0xffffffff
owner = 0xffffffffffffffff
dep_map = struct lockdep_map {
key = 0x810118
class_cache = 0xcbcff0
name = 0x63e944
cpu = 0x0
ip = 0x1a7f64
}
}
cpumask = {
[0] 0x2
}
}rcu_data cpu #0
Quoted 1 line
Quoted 1 line
px *(struct
rcu_data*)0x872f8430
struct rcu_data {
quiescbatch = 0xffffffffffffff99
passed_quiesc = 0x1
qs_pending = 0x0
batch = 0xffffffffffffff97
nxtlist = (nil)
nxttail = {
[0] 0x872f8448
[1] 0x872f8448
[2] 0x872f8448
}
qlen = 0x0
donelist = (nil)
donetail = 0x872f8470
blimit = 0xa
cpu = 0x0
barrier = struct rcu_head {
next = (nil)
func = 0x0
}
}rcu_data cpu #1
Quoted 1 line
Quoted 1 line
px *(struct
rcu_data*)0x874be430
struct rcu_data {
quiescbatch = 0xffffffffffffff98
passed_quiesc = 0x1
qs_pending = 0x0
batch = 0xffffffffffffff97
nxtlist = (nil)
nxttail = {
[0] 0x874be448
[1] 0x874be448
[2] 0x874be448
}
qlen = 0x0
donelist = (nil)
donetail = 0x874be470
blimit = 0xa
cpu = 0x1
barrier = struct rcu_head {
next = (nil)
func = 0x0
}
}rcu_data cpu #2
Quoted 1 line
Quoted 1 line
px *(struct
rcu_data*)0x87684430
struct rcu_data {
quiescbatch = 0xffffffffffffff99
passed_quiesc = 0x1
qs_pending = 0x0
batch = 0xffffffffffffff99
nxtlist = 0xffc1fc18
nxttail = {
[0] 0x87684448
[1] 0x87684448
[2] 0xffc1fc18
}
qlen = 0x1
donelist = (nil)
donetail = 0x87684470
blimit = 0xa
cpu = 0x2
barrier = struct rcu_head {
next = (nil)
func = 0x0
}
}rcu_data cpu #3
Quoted 1 line
Quoted 1 line
px *(struct
rcu_data*)0x8784a430
struct rcu_data {
quiescbatch = 0xffffffffffffff99
passed_quiesc = 0x1
qs_pending = 0x0
batch = 0xffffffffffffff63
nxtlist = (nil)
nxttail = {
[0] 0x8784a448
[1] 0x8784a448
[2] 0x8784a448
}
qlen = 0x0
donelist = (nil)
donetail = 0x8784a470
blimit = 0xa
cpu = 0x3
barrier = struct rcu_head {
next = (nil)
func = 0x0
}
}At the time cpu #1 went to sleep rcu_needs_cpu must have answered false, otherwise a 1 tick delay would have been programmed. rcu_pending compares rcu_ctrlblk.cur with rcu_data.quiescbatch for cpu #1. So these two must have been equal otherwise rcu_needs_cpu would have answered true. That means that the rcu_needs_cpu check has been completed before rcu_start_batch for batch 0xffffffffffffff99. The bit for cpu #1 is still set in the rcu_ctrlblk.cpumask, therefore the bit for cpu #1 in nohz_cpu_mask can not have been set at the time rcu_start_batch has completed. That gives the following race (cpu 0 is starting the batch, cpu 1 is going to sleep):
cpu 1: tick_nohz_stop_sched_tick: rcu_needs_cpu();
cpu 0: rcu_start_batch: rcp->cur++;
cpu 0: rcu_start_batch: cpumask_andnot(to_cpumask(rcp->cpumask),
cpu_online_mask, nonz_cpu_mask);
cpu 1: tick_nohz_stop_schedk_tick: cpumask_set_cpu(1, nohz_cpu_mask);
The order of i) setting the bit in nohz_cpu_mask and ii) the rcu_needs_cpu() check in tick_nohz_stop_sched_tick is wrong, no? Or did I miss some suble check that comes afterwards ?
Signature · 4 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin.
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.
Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.
Thanx, Paul
Quoted 153 lines
Now if I look at the RCU data structures I find this:
rcu_ctrlblkQuoted 1 line
Quoted 1 line
px *(struct
rcu_ctrlblk*)0x810000struct rcu_ctrlblk { cur = 0xffffffffffffff99 completed = 0xffffffffffffff98 pending = 0xffffffffffffff99 signaled = 0x0 lock = spinlock_t { raw_lock = raw_spinlock_t { owner_cpu = 0x0 } break_lock = 0x0 magic = 0xdead4ead owner_cpu = 0xffffffff owner = 0xffffffffffffffff dep_map = struct lockdep_map { key = 0x810118 class_cache = 0xcbcff0 name = 0x63e944 cpu = 0x0 ip = 0x1a7f64 } } cpumask = { [0] 0x2 } }
rcu_datacpu #0Quoted 1 line
Quoted 1 line
px *(struct
rcu_data*)0x872f8430struct rcu_data { quiescbatch = 0xffffffffffffff99 passed_quiesc = 0x1 qs_pending = 0x0 batch = 0xffffffffffffff97 nxtlist = (nil) nxttail = { [0] 0x872f8448 [1] 0x872f8448 [2] 0x872f8448 } qlen = 0x0 donelist = (nil) donetail = 0x872f8470 blimit = 0xa cpu = 0x0 barrier = struct rcu_head { next = (nil) func = 0x0 } }
rcu_datacpu #1Quoted 1 line
Quoted 1 line
px *(struct
rcu_data*)0x874be430struct rcu_data { quiescbatch = 0xffffffffffffff98 passed_quiesc = 0x1 qs_pending = 0x0 batch = 0xffffffffffffff97 nxtlist = (nil) nxttail = { [0] 0x874be448 [1] 0x874be448 [2] 0x874be448 } qlen = 0x0 donelist = (nil) donetail = 0x874be470 blimit = 0xa cpu = 0x1 barrier = struct rcu_head { next = (nil) func = 0x0 } }
rcu_datacpu #2Quoted 1 line
Quoted 1 line
px *(struct
rcu_data*)0x87684430struct rcu_data { quiescbatch = 0xffffffffffffff99 passed_quiesc = 0x1 qs_pending = 0x0 batch = 0xffffffffffffff99 nxtlist = 0xffc1fc18 nxttail = { [0] 0x87684448 [1] 0x87684448 [2] 0xffc1fc18 } qlen = 0x1 donelist = (nil) donetail = 0x87684470 blimit = 0xa cpu = 0x2 barrier = struct rcu_head { next = (nil) func = 0x0 } }
rcu_datacpu #3Quoted 1 line
Quoted 1 line
px *(struct
rcu_data*)0x8784a430struct rcu_data { quiescbatch = 0xffffffffffffff99 passed_quiesc = 0x1 qs_pending = 0x0 batch = 0xffffffffffffff63 nxtlist = (nil) nxttail = { [0] 0x8784a448 [1] 0x8784a448 [2] 0x8784a448 } qlen = 0x0 donelist = (nil) donetail = 0x8784a470 blimit = 0xa cpu = 0x3 barrier = struct rcu_head { next = (nil) func = 0x0 } }At the time cpu #1 went to sleep
rcu_needs_cpumust have answered false, otherwise a 1 tick delay would have been programmed.rcu_pendingcompares rcu_ctrlblk.cur with rcu_data.quiescbatch for cpu #1. So these two must have been equal otherwisercu_needs_cpuwould have answered true. That means that thercu_needs_cpucheck has been completed beforercu_start_batchfor batch0xffffffffffffff99. The bit for cpu #1 is still set in the rcu_ctrlblk.cpumask, therefore the bit for cpu #1 innohz_cpu_maskcan not have been set at the timercu_start_batchhas completed. That gives the following race (cpu 0 is starting the batch, cpu 1 is going to sleep):cpu 1:
tick_nohz_stop_sched_tick:rcu_needs_cpu();
cpu 0:rcu_start_batch: rcp->cur++;
cpu 0:rcu_start_batch: cpumask_andnot(to_cpumask(rcp->cpumask),cpu_online_mask, nonz_cpu_mask);cpu 1:
tick_nohz_stop_schedk_tick:cpumask_set_cpu(1, nohz_cpu_mask);The order of i) setting the bit in
nohz_cpu_maskand ii) thercu_needs_cpu()check intick_nohz_stop_sched_tickis wrong, no? Or did I miss some suble check that comes afterwards ?Signature · 4 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin.
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.
Quoted 4 lines
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.
SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
Signature · 4 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin.
Quoted 19 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.
Quoted 7 lines
Quoted 4 lines
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?
Thanx, Paul
Quoted 25 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 24 lines · Martin Schwidefsky
Quoted 20 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.
Quoted 8 lines
Quoted 2 lines
SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and
CONFIG_NO_HZ, would backportingCONFIG_TREE_RCUmake more sense than playing whack-a-mole on Classic RCU bugs?
Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.
Signature · 4 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin.
Quoted 31 lines · Martin Schwidefsky
Quoted 28 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 26 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Quoted 4 lines
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
Quoted 8 lines
Quoted 5 lines
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and
CONFIG_NO_HZ, would backportingCONFIG_TREE_RCUmake more sense than playing whack-a-mole on Classic RCU bugs?Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.
Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?
Thanx, Paul
Quoted 38 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 37 lines · Martin Schwidefsky
Quoted 33 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 19 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Quoted 7 lines
Quoted 4 lines
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and
CONFIG_NO_HZ, would backportingCONFIG_TREE_RCUmake more sense than playing whack-a-mole on Classic RCU bugs?
Quoted 7 lines
Quoted 2 lines
Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.
Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting
CONFIG_TREE_RCU, if I was to send you a patch backportingCONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?
We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.
Signature · 4 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin.
Quoted 31 lines · Martin Schwidefsky
Quoted 28 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 26 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Quoted 4 lines
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
Quoted 8 lines
Quoted 5 lines
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and
CONFIG_NO_HZ, would backportingCONFIG_TREE_RCUmake more sense than playing whack-a-mole on Classic RCU bugs?Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.
File a bug in bugzilla.novell.com with the problem and the proper people at Novell will evaluate it there.
thanks,
greg k-h
Quoted 44 lines · Martin Schwidefsky
Quoted 41 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 39 lines · Martin Schwidefsky
Quoted 25 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 24 lines · Martin Schwidefsky
Quoted 20 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.Quoted 8 lines
Quoted 2 lines
SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and
CONFIG_NO_HZ, would backportingCONFIG_TREE_RCUmake more sense than playing whack-a-mole on Classic RCU bugs?Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.
Quoted 8 lines
Quoted 4 lines
Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting
CONFIG_TREE_RCU, if I was to send you a patch backportingCONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.
Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.
Thanx, Paul
Hi!
Quoted 14 lines
Quoted 14 lines
Quoted 8 lines
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.
The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.
Quoted 7 lines
Quoted 4 lines
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
Plus... if config_rcu_classic is known buggy (and you are not willing to fix it), it should be disabled (or made depend on !CONFIG_NOHZ, or maybe made depend on CONFIG_BROKEN).
Ugh. rcu_classic does not seem to have help text, and I'm running
known bad code here :-(. Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 21 lines · Pavel Machek
Hi!
Quoted 19 lines
Quoted 18 lines
Quoted 8 lines
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.
The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.
Quoted 6 lines
Quoted 2 lines
SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
Plus... if
config_rcu_classicis known buggy (and you are not willing to fix it), it should be disabled (or made depend on !CONFIG_NOHZ, or maybe made depend onCONFIG_BROKEN).
Already done in -tip. ;-)
paulmck@paulmck-laptop:/home/git/linux-2.6-tip$ ls kernel/rcuclassic.c ls: kernel/rcuclassic.c: No such file or directory
Thanx, Paul
Quoted 6 lines
Ugh.
rcu_classicdoes not seem to have help text, and I'm runningknown bad code here :-(.
Quoted 50 lines · On Tue, 11 Aug 2009 11:04:07 -0700 "Paul E. McKenney"…
Quoted 49 lines · Martin Schwidefsky
Quoted 45 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 31 lines · Martin Schwidefsky
Quoted 28 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 26 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Quoted 4 lines
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
Quoted 8 lines
Quoted 5 lines
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and
CONFIG_NO_HZ, would backportingCONFIG_TREE_RCUmake more sense than playing whack-a-mole on Classic RCU bugs?Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.
Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting
CONFIG_TREE_RCU, if I was to send you a patch backportingCONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?
Quoted 6 lines
Quoted 3 lines
We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.
Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.
There is no specific test, just a regular system boot. The boot did not finish and our tester took a dump. This boot failure seems to happen from time to time.
Signature · 4 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin.
Quoted 57 lines · Martin Schwidefsky
Quoted 54 lines · On Tue, 11 Aug 2009 11:04:07 -0700 "Paul E. McKenney"…
Quoted 52 lines · Martin Schwidefsky
Quoted 38 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 37 lines · Martin Schwidefsky
Quoted 33 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 19 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Quoted 7 lines
Quoted 4 lines
Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and
CONFIG_NO_HZ, would backportingCONFIG_TREE_RCUmake more sense than playing whack-a-mole on Classic RCU bugs?Quoted 7 lines
Quoted 2 lines
Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.
Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting
CONFIG_TREE_RCU, if I was to send you a patch backportingCONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.
Quoted 6 lines
Quoted 2 lines
Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.
There is no specific test, just a regular system boot. The boot did not finish and our tester took a dump. This boot failure seems to happen from time to time.
OK. Has CONFIG_TREE_RCU been working for you? If so, which variant of 2.6.27 do you need a backport to?
Thanx, Paul
Quoted 61 lines · On Fri, 21 Aug 2009 08:54:18 -0700 "Paul E. McKenney"…
Quoted 60 lines · Martin Schwidefsky
Quoted 56 lines · On Tue, 11 Aug 2009 11:04:07 -0700 "Paul E. McKenney"…
Quoted 44 lines · Martin Schwidefsky
Quoted 41 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 39 lines · Martin Schwidefsky
Quoted 25 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 24 lines · Martin Schwidefsky
Quoted 20 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky
Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for
NEXT_TIMER_MAX_DELTAticks.Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from
CONFIG_RCU_CLASSICtoCONFIG_RCU_TREE, which was added in 2.6.29.Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on
CONFIG_RCU_TREErather than backwards onCONFIG_RCU_CLASSIC.Quoted 8 lines
Quoted 2 lines
SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/
I was afraid of that. ;-)
Given that there are some other theoretical bugs in Classic RCU involving interrupts and
CONFIG_NO_HZ, would backportingCONFIG_TREE_RCUmake more sense than playing whack-a-mole on Classic RCU bugs?Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.
Quoted 8 lines
Quoted 4 lines
Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting
CONFIG_TREE_RCU, if I was to send you a patch backportingCONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.
Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.
Quoted 6 lines
Quoted 3 lines
There is no specific test, just a regular system boot. The boot did not finish and our tester took a dump. This boot failure seems to happen from time to time.
OK. Has
CONFIG_TREE_RCUbeen working for you? If so, which variant of 2.6.27 do you need a backport to?
We changed the configuration of our test kernels to CONFIG_TREE_RCU. So far the problem has not shown up again. As we a dealing with a rare race here this has to be taken with a grain of salt.
Signature · 4 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin.
Quoted 7 lines · Martin Schwidefsky
Quoted 5 lines · On Fri, 21 Aug 2009 08:54:18 -0700 "Paul E. McKenney"…
Quoted 4 lines · Martin Schwidefsky
Quoted 2 lines · On Tue, 11 Aug 2009 11:04:07 -0700 "Paul E. McKenney"…
Quoted 1 line · Martin Schwidefsky
On Tue, 11 Aug 2009 07:52:22 -0700
[ . . . ]
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.
Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.
There is no specific test, just a regular system boot. The boot did not finish and our tester took a dump. This boot failure seems to happen from time to time.
Quoted 6 lines
Quoted 2 lines
OK. Has
CONFIG_TREE_RCUbeen working for you? If so, which variant of 2.6.27 do you need a backport to?We changed the configuration of our test kernels to
CONFIG_TREE_RCU. So far the problem has not shown up again. As we a dealing with a rare race here this has to be taken with a grain of salt.
Thank you for trying it out!
Did you by any chance record the success and failure statistic? Perhaps something like number of failures per unit time, time to first failure, number of successful vs. failed reboots, or whatever? This would allow calculation of confidence statistics.
Thanx, Paul