INFO: task rcu_kthread:6 blocked for more than 120 seconds.

DISCUSSION Russell King 3 participants 6 messages Jan 12, 2011 – Jan 13, 2011

Share & tools

Russell King

author

2011-01-12 17:43 UTC #0 lore raw

I'm seeing rcu_kthread get stuck from time to time on ARM Versatile PB (ARM926). Kernel is based upon commit 9e9bc973 (immediately prior to the merge of the OMAP tree.)

~ # ps aux|grep rcu

root         6  0.0  0.0      0     0 ?        S    02:31   0:01 [rcu_kthread]
root       502  9.0  0.3   3740   472 ttyAMA0  S+   02:59   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       504  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       506  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       508  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu

~ # cat /proc/6/wchan
rcu_kthread
~ # INFO: task rcu_kthread:6 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rcu_kthread D c02c6684 0 6 2 0x00000000

Backtrace:
[<c02c6388>] (schedule+0x0/0x354) from [<c00836a0>] (rcu_kthread+0x78/0xf4)
[<c0083628>] (rcu_kthread+0x0/0xf4) from [<c00633ac>] (kthread+0x88/0x90)
[<c0063324>] (kthread+0x0/0x90) from [<c004d12c>] (do_exit+0x0/0x5f8)
 r7:00000013 r6:c004d12c r5:c0063324 r4:c181df28
no locks held by rcu_kthread/6.

Signature · 3 lines

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Paul E. McKenney

2011-01-12 22:22 UTC #1 lore raw

Quoted 21 lines · Russell King

I'm seeing rcu_kthread get stuck from time to time on ARM Versatile PB (ARM926). Kernel is based upon commit 9e9bc973 (immediately prior to the merge of the OMAP tree.)
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        S    02:31   0:01 [rcu_kthread]
root       502  9.0  0.3   3740   472 ttyAMA0  S+   02:59   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       504  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       506  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       508  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # cat /proc/6/wchan
rcu_kthread
~ # INFO: task rcu_kthread:6 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rcu_kthread D c02c6684 0 6 2 0x00000000

Quoted 6 lines

Backtrace:
[<c02c6388>] (schedule+0x0/0x354) from [<c00836a0>] (rcu_kthread+0x78/0xf4)
[<c0083628>] (rcu_kthread+0x0/0xf4) from [<c00633ac>] (kthread+0x88/0x90)
[<c0063324>] (kthread+0x0/0x90) from [<c004d12c>] (do_exit+0x0/0x5f8)
 r7:00000013 r6:c004d12c r5:c0063324 r4:c181df28
no locks held by rcu_kthread/6.

Does the following patch help?

Thanx, Paul

------------------------------------------------------------------------

commit 70ce43253a4465211a9c51e33d56a417c759a718

Author: Paul E. McKenney <[email protected]>
Date:   Wed Jan 12 14:18:11 2011 -0800

rcu: avoid pointless blocked-task warnings

If the RCU callback-processing kthread has nothing to do, it parks in a wait_event(). If RCU remains idle for more than two minutes, the kernel complains about this. This commit changes from wait_event() to wait_event_interruptible() to prevent the kernel from complaining just because RCU is idle.

Reported-by Russell King <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

kernel/rcutiny.c +2 -1

index 0344937..0c343b9 100644
@@ -189,7 +189,8 @@ static int rcu_kthread(void *arg)
 	unsigned long flags;
 
 	for (;;) {
-		wait_event(rcu_kthread_wq, have_rcu_kthread_work != 0);
+		wait_event_interruptible(rcu_kthread_wq,
+					 have_rcu_kthread_work != 0);
 		morework = rcu_boost();
 		local_irq_save(flags);
 		work = have_rcu_kthread_work;

Thomas Weber

Tested-by

2011-01-13 15:17 UTC #2 lore raw

Am 12.01.2011 23:22, schrieb Paul E. McKenney:

Quoted 65 lines · Russell King

Quoted 21 lines
I'm seeing rcu_kthread get stuck from time to time on ARM Versatile PB (ARM926). Kernel is based upon commit 9e9bc973 (immediately prior to the merge of the OMAP tree.)
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        S    02:31   0:01 [rcu_kthread]
root       502  9.0  0.3   3740   472 ttyAMA0  S+   02:59   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       504  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       506  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       508  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # cat /proc/6/wchan
rcu_kthread
~ # INFO: task rcu_kthread:6 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rcu_kthread D c02c6684 0 6 2 0x00000000
Quoted 6 lines
Backtrace:
[<c02c6388>] (schedule+0x0/0x354) from [<c00836a0>] (rcu_kthread+0x78/0xf4)
[<c0083628>] (rcu_kthread+0x0/0xf4) from [<c00633ac>] (kthread+0x88/0x90)
[<c0063324>] (kthread+0x0/0x90) from [<c004d12c>] (do_exit+0x0/0x5f8)
 r7:00000013 r6:c004d12c r5:c0063324 r4:c181df28
no locks held by rcu_kthread/6.
Does the following patch help?
Thanx, Paul
------------------------------------------------------------------------
commit 70ce43253a4465211a9c51e33d56a417c759a718
Author: Paul E. McKenney <[email protected]>
Date:   Wed Jan 12 14:18:11 2011 -0800
rcu: avoid pointless blocked-task warnings
If the RCU callback-processing kthread has nothing to do, it parks in a wait_event(). If RCU remains idle for more than two minutes, the kernel complains about this. This commit changes from wait_event() to wait_event_interruptible() to prevent the kernel from complaining just because RCU is idle.
Reported-by Russell King <[email protected]>
Signed-off-by Paul E. McKenney <[email protected]>
kernel/rcutiny.c +2 -2
index 0344937..0c343b9 100644
@@ -189,7 +189,8 @@ static int rcu_kthread(void *arg)
 	unsigned long flags;
 
 	for (;;) {
-		wait_event(rcu_kthread_wq, have_rcu_kthread_work != 0);
+		wait_event_interruptible(rcu_kthread_wq,
+					 have_rcu_kthread_work != 0);
 		morework = rcu_boost();
 		local_irq_save(flags);
 		work = have_rcu_kthread_work;
--

Quoted 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hello Paul,
here it helps. Tested on Devkit8000 (omap3, beagleboard clone). Without this patch I had permanent crashes. With this patch board is running.

Tested-by Thomas Weber <[email protected]>

Paul E. McKenney

2011-01-13 15:44 UTC #3 lore raw

Quoted 75 lines · Thomas Weber

Am 12.01.2011 23:22, schrieb Paul E. McKenney:
Quoted 65 lines · Russell King
Quoted 21 lines
I'm seeing rcu_kthread get stuck from time to time on ARM Versatile PB (ARM926). Kernel is based upon commit 9e9bc973 (immediately prior to the merge of the OMAP tree.)
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        S    02:31   0:01 [rcu_kthread]
root       502  9.0  0.3   3740   472 ttyAMA0  S+   02:59   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       504  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       506  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # ps aux|grep rcu
root         6  0.0  0.0      0     0 ?        D    02:31   0:01 [rcu_kthread]
root       508  0.0  0.3   3740   472 ttyAMA0  S+   03:00   0:00 grep rcu
~ # cat /proc/6/wchan
rcu_kthread
~ # INFO: task rcu_kthread:6 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rcu_kthread D c02c6684 0 6 2 0x00000000
Quoted 6 lines
Backtrace:
[<c02c6388>] (schedule+0x0/0x354) from [<c00836a0>] (rcu_kthread+0x78/0xf4)
[<c0083628>] (rcu_kthread+0x0/0xf4) from [<c00633ac>] (kthread+0x88/0x90)
[<c0063324>] (kthread+0x0/0x90) from [<c004d12c>] (do_exit+0x0/0x5f8)
 r7:00000013 r6:c004d12c r5:c0063324 r4:c181df28
no locks held by rcu_kthread/6.
Does the following patch help?
Thanx, Paul
------------------------------------------------------------------------
commit 70ce43253a4465211a9c51e33d56a417c759a718
Author: Paul E. McKenney <[email protected]>
Date:   Wed Jan 12 14:18:11 2011 -0800
rcu: avoid pointless blocked-task warnings
If the RCU callback-processing kthread has nothing to do, it parks in a wait_event(). If RCU remains idle for more than two minutes, the kernel complains about this. This commit changes from wait_event() to wait_event_interruptible() to prevent the kernel from complaining just because RCU is idle.
Reported-by Russell King <[email protected]>
Signed-off-by Paul E. McKenney <[email protected]>
kernel/rcutiny.c +2 -2
index 0344937..0c343b9 100644
@@ -189,7 +189,8 @@ static int rcu_kthread(void *arg)
 	unsigned long flags;
 
 	for (;;) {
-		wait_event(rcu_kthread_wq, have_rcu_kthread_work != 0);
+		wait_event_interruptible(rcu_kthread_wq,
+					 have_rcu_kthread_work != 0);
 		morework = rcu_boost();
 		local_irq_save(flags);
 		work = have_rcu_kthread_work;
--
Quoted 4 lines
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hello Paul,
here it helps. Tested on Devkit8000 (omap3, beagleboard clone). Without this patch I had permanent crashes. With this patch board is running.

Quoted 1 line

Tested-by Thomas Weber <[email protected]>

Thank you, Thomas!!! I have added your Tested-by.

Thanx, Paul

Russell King

author Tested-by

2011-01-13 20:06 UTC #4 lore raw

In reply to Paul E. McKenney

Quoted 1 line · Paul E. McKenney

Does the following patch help?

Yes it does, thanks.

Tested-by Russell King <[email protected]>

Quoted 6 lines

commit 70ce43253a4465211a9c51e33d56a417c759a718
Author: Paul E. McKenney <[email protected]>
Date:   Wed Jan 12 14:18:11 2011 -0800
rcu: avoid pointless blocked-task warnings

Quoted 7 lines

If the RCU callback-processing kthread has nothing to do, it parks in a wait_event(). If RCU remains idle for more than two minutes, the kernel complains about this. This commit changes from wait_event() to wait_event_interruptible() to prevent the kernel from complaining just because RCU is idle.
Reported-by Russell King <[email protected]>

Please add '+kernel' in there, thanks.

Signature · 3 lines

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Paul E. McKenney

2011-01-13 20:41 UTC #5 lore raw

Quoted 19 lines · Russell King

Quoted 1 line · Paul E. McKenney
Does the following patch help?
Yes it does, thanks.
Tested-by Russell King <[email protected]>
Quoted 12 lines
commit 70ce43253a4465211a9c51e33d56a417c759a718
Author: Paul E. McKenney <[email protected]>
Date:   Wed Jan 12 14:18:11 2011 -0800
rcu: avoid pointless blocked-task warnings
If the RCU callback-processing kthread has nothing to do, it parks in a wait_event(). If RCU remains idle for more than two minutes, the kernel complains about this. This commit changes from wait_event() to wait_event_interruptible() to prevent the kernel from complaining just because RCU is idle.

Quoted 3 lines

Quoted 1 line
Reported-by Russell King <[email protected]>
Please add '+kernel' in there, thanks.

Done! FWIW, I copied the original from my email agent.

Thanx, Paul