Skip to contentSkip to conversation

preempt rcu bug on s390

DISCUSSION Heiko Carstens 3 participants 7 messages Feb 9, 2008 – Feb 11, 2008 s390

lore
Share & tools
lore
#0 lore raw

Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always gets stuck when running with more than one cpu.
When booting with four cpus I get all four cpus caught withing cpu_idle and not advancing anymore. However there is the init process which is waitung for synchronize_rcu() to complete (lcrash output):

STACK TRACE FOR TASK: 0xf84d968 (swapper)

STACK:
0 schedule+842 [0x36c956]
1 schedule_timeout+172 [0x36d0e4]
2 wait_for_common+204 [0x36c398]
3 synchronize_rcu+76 [0x567bc]
4 netlink_change_ngroups+150 [0x2b4302]
5 genl_register_mc_group+256 [0x2b6174]
6 genl_init+188 [0x534e44]
7 kernel_init+444 [0x518334]
8 kernel_thread_starter+6 [0x192a6]

If I change the code so that timer ticks won't be disabled everything runs fine. So my guess is that rcu_needs_cpu() doesn't do the right thing for the rcu preemptible case.

Kernel version is git head of today.

Any ideas?

#1 lore raw
Quoted 19 lines · Heiko Carstens

Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always gets stuck when running with more than one cpu.
When booting with four cpus I get all four cpus caught withing cpu_idle and not advancing anymore. However there is the init process which is waitung for synchronize_rcu() to complete (lcrash output):

STACK TRACE FOR TASK: 0xf84d968 (swapper)

STACK:
0 schedule+842 [0x36c956]
1 schedule_timeout+172 [0x36d0e4]
2 wait_for_common+204 [0x36c398]
3 synchronize_rcu+76 [0x567bc]
4 netlink_change_ngroups+150 [0x2b4302]
5 genl_register_mc_group+256 [0x2b6174]
6 genl_init+188 [0x534e44]
7 kernel_init+444 [0x518334]
8 kernel_thread_starter+6 [0x192a6]

Quoted 7 lines

If I change the code so that timer ticks won't be disabled everything runs fine. So my guess is that rcu_needs_cpu() doesn't do the right thing for the rcu preemptible case.

Kernel version is git head of today.

Any ideas?

Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?

If not, could you please check it out?

Thanx, Paul

#2 lore raw
Quoted 24 lines · Paul E. McKenney
Quoted 23 lines · Heiko Carstens

Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always gets stuck when running with more than one cpu.
When booting with four cpus I get all four cpus caught withing cpu_idle and not advancing anymore. However there is the init process which is waitung for synchronize_rcu() to complete (lcrash output):

STACK TRACE FOR TASK: 0xf84d968 (swapper)

STACK:
0 schedule+842 [0x36c956]
1 schedule_timeout+172 [0x36d0e4]
2 wait_for_common+204 [0x36c398]
3 synchronize_rcu+76 [0x567bc]
4 netlink_change_ngroups+150 [0x2b4302]
5 genl_register_mc_group+256 [0x2b6174]
6 genl_init+188 [0x534e44]
7 kernel_init+444 [0x518334]
8 kernel_thread_starter+6 [0x192a6]

If I change the code so that timer ticks won't be disabled everything runs fine. So my guess is that rcu_needs_cpu() doesn't do the right thing for the rcu preemptible case.

Quoted 7 lines
Quoted 3 lines

Kernel version is git head of today.

Any ideas?

Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?

If not, could you please check it out?

It's not applied, however it doesn't change anything. Also the patch is tied to the dynticks implementation which is differently from s390's nohz implementation.
I had to add the patch below so it would make at least some sense. But it doesn't fix the problem.

---
 arch/s390/kernel/time.c |    2 ++
 include/linux/hardirq.h |    2 +-
 kernel/rcupreempt.c     |    2 +-
 3 files changed, 4 insertions(+), 2 deletions(-)
kernel/rcupreempt.c +1 -1
Index: linux-2.6/kernel/rcupreempt.c
===================================================================
@@ -413,7 +413,7 @@ static void __rcu_advance_callbacks(stru
 	}
 }
 
-#ifdef CONFIG_NO_HZ
+#if defined(CONFIG_NO_HZ) || defined(CONFIG_NO_IDLE_HZ)
 
 DEFINE_PER_CPU(long, dynticks_progress_counter) = 1;
 static DEFINE_PER_CPU(long, rcu_dyntick_snapshot);
arch/s390/kernel/time.c +2 -0
Index: linux-2.6/arch/s390/kernel/time.c
===================================================================
@@ -200,6 +200,7 @@ static void stop_hz_timer(void)
 		if (timer >= jiffies_timer_cc)
 			todval = timer;
 	}
+	rcu_enter_nohz();
 	set_clock_comparator(todval);
 }
 
@@ -213,6 +214,7 @@ static void start_hz_timer(void)
 
 	if (!cpu_isset(smp_processor_id(), nohz_cpu_mask))
 		return;
+	rcu_exit_nohz();
 	account_ticks(get_clock());
 	set_clock_comparator(S390_lowcore.jiffy_timer + CPU_DEVIATION);
 	cpu_clear(smp_processor_id(), nohz_cpu_mask);
include/linux/hardirq.h +1 -1
Index: linux-2.6/include/linux/hardirq.h
===================================================================
@@ -109,7 +109,7 @@ static inline void account_system_vtime(
 }
 #endif
 
-#if defined(CONFIG_PREEMPT_RCU) && defined(CONFIG_NO_HZ)
+#if defined(CONFIG_PREEMPT_RCU) && (defined(CONFIG_NO_HZ) || defined(CONFIG_NO_IDLE_HZ))
 extern void rcu_irq_enter(void);
 extern void rcu_irq_exit(void);
 #else
#3 lore raw
Quoted 31 lines · Heiko Carstens
Quoted 30 lines · Paul E. McKenney
Quoted 19 lines · Heiko Carstens

Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always gets stuck when running with more than one cpu.
When booting with four cpus I get all four cpus caught withing cpu_idle and not advancing anymore. However there is the init process which is waitung for synchronize_rcu() to complete (lcrash output):

STACK TRACE FOR TASK: 0xf84d968 (swapper)

STACK:
0 schedule+842 [0x36c956]
1 schedule_timeout+172 [0x36d0e4]
2 wait_for_common+204 [0x36c398]
3 synchronize_rcu+76 [0x567bc]
4 netlink_change_ngroups+150 [0x2b4302]
5 genl_register_mc_group+256 [0x2b6174]
6 genl_init+188 [0x534e44]
7 kernel_init+444 [0x518334]
8 kernel_thread_starter+6 [0x192a6]

Quoted 7 lines

If I change the code so that timer ticks won't be disabled everything runs fine. So my guess is that rcu_needs_cpu() doesn't do the right thing for the rcu preemptible case.

Kernel version is git head of today.

Any ideas?

Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?

Quoted 7 lines
Quoted 1 line

If not, could you please check it out?

It's not applied, however it doesn't change anything. Also the patch is tied to the dynticks implementation which is differently from s390's nohz implementation.
I had to add the patch below so it would make at least some sense. But it doesn't fix the problem.

OK, I was afraid of that. ;-)

Does s390 start out in nohz mode? The reason I ask is that it feels like an off-by-one error for the dynticks_progress_counter.

Thanx, Paul

Quoted 52 lines
---
 arch/s390/kernel/time.c |    2 ++
 include/linux/hardirq.h |    2 +-
 kernel/rcupreempt.c     |    2 +-
 3 files changed, 4 insertions(+), 2 deletions(-)

Index: linux-2.6/kernel/rcupreempt.c
===================================================================

kernel/rcupreempt.c +1 -1
@@ -413,7 +413,7 @@ static void __rcu_advance_callbacks(stru
 	}
 }
 
-#ifdef CONFIG_NO_HZ
+#if defined(CONFIG_NO_HZ) || defined(CONFIG_NO_IDLE_HZ)
 
 DEFINE_PER_CPU(long, dynticks_progress_counter) = 1;
 static DEFINE_PER_CPU(long, rcu_dyntick_snapshot);

Index: linux-2.6/arch/s390/kernel/time.c
===================================================================

arch/s390/kernel/time.c +2 -0
@@ -200,6 +200,7 @@ static void stop_hz_timer(void)
 		if (timer >= jiffies_timer_cc)
 			todval = timer;
 	}
+	rcu_enter_nohz();
 	set_clock_comparator(todval);
 }
 
@@ -213,6 +214,7 @@ static void start_hz_timer(void)
 
 	if (!cpu_isset(smp_processor_id(), nohz_cpu_mask))
 		return;
+	rcu_exit_nohz();
 	account_ticks(get_clock());
 	set_clock_comparator(S390_lowcore.jiffy_timer + CPU_DEVIATION);
 	cpu_clear(smp_processor_id(), nohz_cpu_mask);

Index: linux-2.6/include/linux/hardirq.h
===================================================================

include/linux/hardirq.h +1 -1
@@ -109,7 +109,7 @@ static inline void account_system_vtime(
 }
 #endif
 
-#if defined(CONFIG_PREEMPT_RCU) && defined(CONFIG_NO_HZ)
+#if defined(CONFIG_PREEMPT_RCU) && (defined(CONFIG_NO_HZ) || defined(CONFIG_NO_IDLE_HZ))
 extern void rcu_irq_enter(void);
 extern void rcu_irq_exit(void);
 #else
#4 lore raw
Quoted 24 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines

Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always gets stuck when running with more than one cpu.
When booting with four cpus I get all four cpus caught withing cpu_idle and not advancing anymore. However there is the init process which is waitung for synchronize_rcu() to complete (lcrash output):

Quoted 7 lines

If I change the code so that timer ticks won't be disabled everything runs fine. So my guess is that rcu_needs_cpu() doesn't do the right thing for the rcu preemptible case.

Kernel version is git head of today.

Any ideas?

Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?

Quoted 7 lines
Quoted 1 line

If not, could you please check it out?

It's not applied, however it doesn't change anything. Also the patch is tied to the dynticks implementation which is differently from s390's nohz implementation.
I had to add the patch below so it would make at least some sense. But it doesn't fix the problem.

Quoted 4 lines

OK, I was afraid of that. ;-)

Does s390 start out in nohz mode? The reason I ask is that it feels like an off-by-one error for the dynticks_progress_counter.

Actually I forgot to add a few ifdefs to make the code do something :) That just reveals that we have a conflict with the dynticks implementation and s390's nohz that shows up in what rcu_irq_enter/exit assume. I didn't patch s390 and common code so it will work, but I think the patch you mentionened will fix the problem I reported.
So I guess we should either convert s390 to use the generic dynticks implementation or disable preemptible rcu on s390 until we converted our code.

Thanks for helping debugging this!

#5 lore raw
Quoted 29 lines · Heiko Carstens
Quoted 24 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines

Using CONFIG_PREEMPT_RCU and CONFIG_NO_IDLE_HZ on s390 my system always gets stuck when running with more than one cpu.
When booting with four cpus I get all four cpus caught withing cpu_idle and not advancing anymore. However there is the init process which is waitung for synchronize_rcu() to complete (lcrash output):

Quoted 7 lines

If I change the code so that timer ticks won't be disabled everything runs fine. So my guess is that rcu_needs_cpu() doesn't do the right thing for the rcu preemptible case.

Kernel version is git head of today.

Any ideas?

Does this tree have http://lkml.org/lkml/2008/1/29/208 applied?

Quoted 7 lines
Quoted 1 line

If not, could you please check it out?

It's not applied, however it doesn't change anything. Also the patch is tied to the dynticks implementation which is differently from s390's nohz implementation.
I had to add the patch below so it would make at least some sense. But it doesn't fix the problem.

Quoted 4 lines

OK, I was afraid of that. ;-)

Does s390 start out in nohz mode? The reason I ask is that it feels like an off-by-one error for the dynticks_progress_counter.

Quoted 8 lines

Actually I forgot to add a few ifdefs to make the code do something :) That just reveals that we have a conflict with the dynticks implementation and s390's nohz that shows up in what rcu_irq_enter/exit assume. I didn't patch s390 and common code so it will work, but I think the patch you mentionened will fix the problem I reported.
So I guess we should either convert s390 to use the generic dynticks implementation or disable preemptible rcu on s390 until we converted our code.

Sounds good to me!!! (Especially converting s390 to generic algorithm.)

I believe that the generic implementation will do what you need, but I am sure you will let me know of any problems that arise.

Quoted 1 line

Thanks for helping debugging this!

Thank you for tracking it down!

Thanx, Paul

#6 lore raw
In reply to Heiko Carstens
Quoted 12 lines · Heiko Carstens
Quoted 2 lines

Does s390 start out in nohz mode? The reason I ask is that it feels like an off-by-one error for the dynticks_progress_counter.

Actually I forgot to add a few ifdefs to make the code do something :) That just reveals that we have a conflict with the dynticks implementation and s390's nohz that shows up in what rcu_irq_enter/exit assume. I didn't patch s390 and common code so it will work, but I think the patch you mentionened will fix the problem I reported.
So I guess we should either convert s390 to use the generic dynticks implementation or disable preemptible rcu on s390 until we converted our code.

Quoted 1 line

Thanks for helping debugging this!

Heiko, thanks for reporting this.

This patch still didn't make it into -rc1, and it really should. Because without this patch, PREEMPT_RCU and NO_HZ together is broken, on all boxes.

The patch is in Ingo's sched-devel git tree, as 9460545f81ea48b07dbb20456a8ede776d8ebc1b (last I checked) and titled:

rcu: add support for dynamic ticks and preempt rcu

-- Steve