Skip to contentSkip to conversation

race of RCU vs NOHU

BUG Martin Schwidefsky 4 participants 15 messages Aug 7, 2009 – Aug 31, 2009

lore
Share & tools
lore
#0 lore raw

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Now if I look at the RCU data structures I find this:

rcu_ctrlblk

Quoted 1 line
Quoted 1 line

px *(struct rcu_ctrlblk *) 0x810000

struct rcu_ctrlblk {
        cur = 0xffffffffffffff99
        completed = 0xffffffffffffff98
        pending = 0xffffffffffffff99
        signaled = 0x0
        lock = spinlock_t {
                raw_lock = raw_spinlock_t {
                        owner_cpu = 0x0
                }
                break_lock = 0x0
                magic = 0xdead4ead
                owner_cpu = 0xffffffff
                owner = 0xffffffffffffffff
                dep_map = struct lockdep_map {
                        key = 0x810118
                        class_cache = 0xcbcff0
                        name = 0x63e944
                        cpu = 0x0
                        ip = 0x1a7f64
                }
        }
        cpumask = {
                [0] 0x2
        }
}

rcu_data cpu #0

Quoted 1 line
Quoted 1 line

px *(struct rcu_data *) 0x872f8430

struct rcu_data {
        quiescbatch = 0xffffffffffffff99
        passed_quiesc = 0x1
        qs_pending = 0x0
        batch = 0xffffffffffffff97
        nxtlist = (nil)
        nxttail = {
                [0]         0x872f8448
                [1]         0x872f8448
                [2]         0x872f8448
        }
        qlen = 0x0
        donelist = (nil)
        donetail = 0x872f8470
        blimit = 0xa
        cpu = 0x0
        barrier = struct rcu_head {
                next = (nil)
                func = 0x0
        }
}

rcu_data cpu #1

Quoted 1 line
Quoted 1 line

px *(struct rcu_data *) 0x874be430

struct rcu_data {
        quiescbatch = 0xffffffffffffff98
        passed_quiesc = 0x1
        qs_pending = 0x0
        batch = 0xffffffffffffff97
        nxtlist = (nil)
        nxttail = {
                [0]         0x874be448
                [1]         0x874be448
                [2]         0x874be448
        }
        qlen = 0x0
        donelist = (nil)
        donetail = 0x874be470
        blimit = 0xa
        cpu = 0x1
        barrier = struct rcu_head {
                next = (nil)
                func = 0x0
        }
}

rcu_data cpu #2

Quoted 1 line
Quoted 1 line

px *(struct rcu_data *) 0x87684430

struct rcu_data {
        quiescbatch = 0xffffffffffffff99
        passed_quiesc = 0x1
        qs_pending = 0x0
        batch = 0xffffffffffffff99
        nxtlist = 0xffc1fc18
        nxttail = {
                [0]         0x87684448
                [1]         0x87684448
                [2]         0xffc1fc18
        }
        qlen = 0x1
        donelist = (nil)
        donetail = 0x87684470
        blimit = 0xa
        cpu = 0x2
        barrier = struct rcu_head {
                next = (nil)
                func = 0x0
        }
}

rcu_data cpu #3

Quoted 1 line
Quoted 1 line

px *(struct rcu_data *) 0x8784a430

struct rcu_data {
        quiescbatch = 0xffffffffffffff99
        passed_quiesc = 0x1
        qs_pending = 0x0
        batch = 0xffffffffffffff63
        nxtlist = (nil)
        nxttail = {
                [0]         0x8784a448
                [1]         0x8784a448
                [2]         0x8784a448
        }
        qlen = 0x0
        donelist = (nil)
        donetail = 0x8784a470
        blimit = 0xa
        cpu = 0x3
        barrier = struct rcu_head {
                next = (nil)
                func = 0x0
        }
}

At the time cpu #1 went to sleep rcu_needs_cpu must have answered false, otherwise a 1 tick delay would have been programmed. rcu_pending compares rcu_ctrlblk.cur with rcu_data.quiescbatch for cpu #1. So these two must have been equal otherwise rcu_needs_cpu would have answered true. That means that the rcu_needs_cpu check has been completed before rcu_start_batch for batch 0xffffffffffffff99. The bit for cpu #1 is still set in the rcu_ctrlblk.cpumask, therefore the bit for cpu #1 in nohz_cpu_mask can not have been set at the time rcu_start_batch has completed. That gives the following race (cpu 0 is starting the batch, cpu 1 is going to sleep):

cpu 1: tick_nohz_stop_sched_tick: rcu_needs_cpu();
cpu 0: rcu_start_batch: rcp->cur++;
cpu 0: rcu_start_batch: cpumask_andnot(to_cpumask(rcp->cpumask),

					cpu_online_mask, nonz_cpu_mask);

cpu 1: tick_nohz_stop_schedk_tick: cpumask_set_cpu(1, nohz_cpu_mask);

The order of i) setting the bit in nohz_cpu_mask and ii) the rcu_needs_cpu() check in tick_nohz_stop_sched_tick is wrong, no? Or did I miss some suble check that comes afterwards ?

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#1 lore raw
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

Thanx, Paul

Quoted 153 lines

Now if I look at the RCU data structures I find this:

rcu_ctrlblk

Quoted 1 line
Quoted 1 line

px *(struct rcu_ctrlblk *) 0x810000

struct rcu_ctrlblk {
        cur = 0xffffffffffffff99
        completed = 0xffffffffffffff98
        pending = 0xffffffffffffff99
        signaled = 0x0
        lock = spinlock_t {
                raw_lock = raw_spinlock_t {
                        owner_cpu = 0x0
                }
                break_lock = 0x0
                magic = 0xdead4ead
                owner_cpu = 0xffffffff
                owner = 0xffffffffffffffff
                dep_map = struct lockdep_map {
                        key = 0x810118
                        class_cache = 0xcbcff0
                        name = 0x63e944
                        cpu = 0x0
                        ip = 0x1a7f64
                }
        }
        cpumask = {
                [0] 0x2
        }
}

rcu_data cpu #0

Quoted 1 line
Quoted 1 line

px *(struct rcu_data *) 0x872f8430

struct rcu_data {
        quiescbatch = 0xffffffffffffff99
        passed_quiesc = 0x1
        qs_pending = 0x0
        batch = 0xffffffffffffff97
        nxtlist = (nil)
        nxttail = {
                [0]         0x872f8448
                [1]         0x872f8448
                [2]         0x872f8448
        }
        qlen = 0x0
        donelist = (nil)
        donetail = 0x872f8470
        blimit = 0xa
        cpu = 0x0
        barrier = struct rcu_head {
                next = (nil)
                func = 0x0
        }
}

rcu_data cpu #1

Quoted 1 line
Quoted 1 line

px *(struct rcu_data *) 0x874be430

struct rcu_data {
        quiescbatch = 0xffffffffffffff98
        passed_quiesc = 0x1
        qs_pending = 0x0
        batch = 0xffffffffffffff97
        nxtlist = (nil)
        nxttail = {
                [0]         0x874be448
                [1]         0x874be448
                [2]         0x874be448
        }
        qlen = 0x0
        donelist = (nil)
        donetail = 0x874be470
        blimit = 0xa
        cpu = 0x1
        barrier = struct rcu_head {
                next = (nil)
                func = 0x0
        }
}

rcu_data cpu #2

Quoted 1 line
Quoted 1 line

px *(struct rcu_data *) 0x87684430

struct rcu_data {
        quiescbatch = 0xffffffffffffff99
        passed_quiesc = 0x1
        qs_pending = 0x0
        batch = 0xffffffffffffff99
        nxtlist = 0xffc1fc18
        nxttail = {
                [0]         0x87684448
                [1]         0x87684448
                [2]         0xffc1fc18
        }
        qlen = 0x1
        donelist = (nil)
        donetail = 0x87684470
        blimit = 0xa
        cpu = 0x2
        barrier = struct rcu_head {
                next = (nil)
                func = 0x0
        }
}

rcu_data cpu #3

Quoted 1 line
Quoted 1 line

px *(struct rcu_data *) 0x8784a430

struct rcu_data {
        quiescbatch = 0xffffffffffffff99
        passed_quiesc = 0x1
        qs_pending = 0x0
        batch = 0xffffffffffffff63
        nxtlist = (nil)
        nxttail = {
                [0]         0x8784a448
                [1]         0x8784a448
                [2]         0x8784a448
        }
        qlen = 0x0
        donelist = (nil)
        donetail = 0x8784a470
        blimit = 0xa
        cpu = 0x3
        barrier = struct rcu_head {
                next = (nil)
                func = 0x0
        }
}

At the time cpu #1 went to sleep rcu_needs_cpu must have answered false, otherwise a 1 tick delay would have been programmed. rcu_pending compares rcu_ctrlblk.cur with rcu_data.quiescbatch for cpu #1. So these two must have been equal otherwise rcu_needs_cpu would have answered true. That means that the rcu_needs_cpu check has been completed before rcu_start_batch for batch 0xffffffffffffff99. The bit for cpu #1 is still set in the rcu_ctrlblk.cpumask, therefore the bit for cpu #1 in nohz_cpu_mask can not have been set at the time rcu_start_batch has completed. That gives the following race (cpu 0 is starting the batch, cpu 1 is going to sleep):

cpu 1: tick_nohz_stop_sched_tick: rcu_needs_cpu();
cpu 0: rcu_start_batch: rcp->cur++;
cpu 0: rcu_start_batch: cpumask_andnot(to_cpumask(rcp->cpumask),

					cpu_online_mask, nonz_cpu_mask);

cpu 1: tick_nohz_stop_schedk_tick: cpumask_set_cpu(1, nohz_cpu_mask);

The order of i) setting the bit in nohz_cpu_mask and ii) the rcu_needs_cpu() check in tick_nohz_stop_sched_tick is wrong, no? Or did I miss some suble check that comes afterwards ?

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#2 lore raw
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Quoted 4 lines

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#3 lore raw
Quoted 19 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Quoted 7 lines
Quoted 4 lines

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Thanx, Paul

#4 lore raw
Quoted 25 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 24 lines · Martin Schwidefsky
Quoted 20 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

Quoted 8 lines
Quoted 2 lines

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#5 lore raw
Quoted 31 lines · Martin Schwidefsky
Quoted 28 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 26 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Quoted 4 lines

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

Quoted 8 lines
Quoted 5 lines

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.

Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?

Thanx, Paul

#6 lore raw
Quoted 38 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 37 lines · Martin Schwidefsky
Quoted 33 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 19 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Quoted 7 lines
Quoted 4 lines

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Quoted 7 lines
Quoted 2 lines

Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.

Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?

We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#12 lore raw
In reply to Martin Schwidefsky
Quoted 31 lines · Martin Schwidefsky
Quoted 28 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 26 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Quoted 4 lines

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

Quoted 8 lines
Quoted 5 lines

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.

File a bug in bugzilla.novell.com with the problem and the proper people at Novell will evaluate it there.

thanks,

greg k-h

#7 lore raw
In reply to Martin Schwidefsky
Quoted 44 lines · Martin Schwidefsky
Quoted 41 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 39 lines · Martin Schwidefsky
Quoted 25 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 24 lines · Martin Schwidefsky
Quoted 20 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

Quoted 8 lines
Quoted 2 lines

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.

Quoted 8 lines
Quoted 4 lines

Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?

We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.

Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.

Thanx, Paul

#13 lore raw
In reply to Martin Schwidefsky

Hi!

Quoted 14 lines
Quoted 14 lines
Quoted 8 lines

I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Quoted 7 lines
Quoted 4 lines

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

Plus... if config_rcu_classic is known buggy (and you are not willing to fix it), it should be disabled (or made depend on !CONFIG_NOHZ, or maybe made depend on CONFIG_BROKEN).

Ugh. rcu_classic does not seem to have help text, and I'm running

								known
								bad
								code
								here
								:-(.


								Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#14 lore raw
Quoted 21 lines · Pavel Machek

Hi!

Quoted 19 lines
Quoted 18 lines
Quoted 8 lines

I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

Quoted 6 lines
Quoted 2 lines

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

Plus... if config_rcu_classic is known buggy (and you are not willing to fix it), it should be disabled (or made depend on !CONFIG_NOHZ, or maybe made depend on CONFIG_BROKEN).

Already done in -tip. ;-)

	paulmck@paulmck-laptop:/home/git/linux-2.6-tip$ ls kernel/rcuclassic.c
	ls: kernel/rcuclassic.c: No such file or directory

Thanx, Paul

Quoted 6 lines

Ugh. rcu_classic does not seem to have help text, and I'm running

								known
								bad
								code
								here
								:-(.
#8 lore raw
In reply to Paul E. McKenney
Quoted 50 lines · On Tue, 11 Aug 2009 11:04:07 -0700 "Paul E. McKenney"…
Quoted 49 lines · Martin Schwidefsky
Quoted 45 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 31 lines · Martin Schwidefsky
Quoted 28 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 26 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Quoted 4 lines

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

Quoted 8 lines
Quoted 5 lines

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.

Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?

Quoted 6 lines
Quoted 3 lines

We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.

Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.

There is no specific test, just a regular system boot. The boot did not finish and our tester took a dump. This boot failure seems to happen from time to time.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#9 lore raw
Quoted 57 lines · Martin Schwidefsky
Quoted 54 lines · On Tue, 11 Aug 2009 11:04:07 -0700 "Paul E. McKenney"…
Quoted 52 lines · Martin Schwidefsky
Quoted 38 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 37 lines · Martin Schwidefsky
Quoted 33 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 19 lines · Martin Schwidefsky
Quoted 16 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Quoted 7 lines
Quoted 4 lines

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Quoted 7 lines
Quoted 2 lines

Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.

Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?

We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.

Quoted 6 lines
Quoted 2 lines

Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.

There is no specific test, just a regular system boot. The boot did not finish and our tester took a dump. This boot failure seems to happen from time to time.

OK. Has CONFIG_TREE_RCU been working for you? If so, which variant of 2.6.27 do you need a backport to?

Thanx, Paul

#10 lore raw
Quoted 61 lines · On Fri, 21 Aug 2009 08:54:18 -0700 "Paul E. McKenney"…
Quoted 60 lines · Martin Schwidefsky
Quoted 56 lines · On Tue, 11 Aug 2009 11:04:07 -0700 "Paul E. McKenney"…
Quoted 44 lines · Martin Schwidefsky
Quoted 41 lines · On Tue, 11 Aug 2009 07:52:22 -0700 "Paul E. McKenney"…
Quoted 39 lines · Martin Schwidefsky
Quoted 25 lines · On Mon, 10 Aug 2009 08:08:07 -0700 "Paul E. McKenney"…
Quoted 24 lines · Martin Schwidefsky
Quoted 20 lines · On Fri, 7 Aug 2009 07:29:57 -0700 "Paul E. McKenney"…
Quoted 9 lines · Martin Schwidefsky

Hi Paul,
I analysed a dump of a hanging 2.6.30 system and found what I think is a bug of RCU vs NOHZ. There are a number of patches ontop of that kernel but they should be independent of the bug.

The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up recently, cpu #1 has been sleeping for 5 minutes, but there is a pending rcu batch. The timer wheel for cpu #1 is empty, it will continue to sleep for NEXT_TIMER_MAX_DELTA ticks.

Congratulations, Martin! You have exercised what to date has been a theoretical bug identified last year by Manfred Spraul. The fix is to switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in 2.6.29.

Of course, if you need to work with an old kernel version, you might still need a patch, perhaps for the various -stable versions. If so, please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE rather than backwards on CONFIG_RCU_CLASSIC.

Quoted 8 lines
Quoted 2 lines

SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is present there and I think it needs to be fixed :-/

I was afraid of that. ;-)

Given that there are some other theoretical bugs in Classic RCU involving interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more sense than playing whack-a-mole on Classic RCU bugs?

Fine with me but I don't know if SuSE/Novell is willing to accept such a big change for an existing distribution. I've put Ihno and Greg on Cc.

Quoted 8 lines
Quoted 4 lines

Good point! While they are thinking about the tradeoff between whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel version(s) should I backport it to?

We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.

Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.

Quoted 6 lines
Quoted 3 lines

There is no specific test, just a regular system boot. The boot did not finish and our tester took a dump. This boot failure seems to happen from time to time.

OK. Has CONFIG_TREE_RCU been working for you? If so, which variant of 2.6.27 do you need a backport to?

We changed the configuration of our test kernels to CONFIG_TREE_RCU. So far the problem has not shown up again. As we a dealing with a rare race here this has to be taken with a grain of salt.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#11 lore raw
Quoted 7 lines · Martin Schwidefsky
Quoted 5 lines · On Fri, 21 Aug 2009 08:54:18 -0700 "Paul E. McKenney"…
Quoted 4 lines · Martin Schwidefsky
Quoted 2 lines · On Tue, 11 Aug 2009 11:04:07 -0700 "Paul E. McKenney"…
Quoted 1 line · Martin Schwidefsky

On Tue, 11 Aug 2009 07:52:22 -0700

[ . . . ]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

We found the bug with kernel version 2.6.30 - the kernel on our test systems still use classic RCU. For us it is easy to switch to tree-RCU, no patch required.

Ah! Could you please send me the test you use? My tests were insufficient to force this problem to happen.

There is no specific test, just a regular system boot. The boot did not finish and our tester took a dump. This boot failure seems to happen from time to time.

Quoted 6 lines
Quoted 2 lines

OK. Has CONFIG_TREE_RCU been working for you? If so, which variant of 2.6.27 do you need a backport to?

We changed the configuration of our test kernels to CONFIG_TREE_RCU. So far the problem has not shown up again. As we a dealing with a rare race here this has to be taken with a grain of salt.

Thank you for trying it out!

Did you by any chance record the success and failure statistic? Perhaps something like number of failures per unit time, time to first failure, number of successful vs. failed reboots, or whatever? This would allow calculation of confidence statistics.

Thanx, Paul