Bug#746232: 3.2.57-rt: rcu_preempt detected stalls at boot
Hello.
One of my computers fails to boot with 3.2.57-rt kernel, i686. The kernel is the Debian one: linux-image-3.2.0-4-rt-686-pae, version 3.2.57-3. Same non-rt kernel boots fine. All newer -rt kernels (I've tried kernels starting from 3.6) boot fine.
You can find some details of the hardware in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746232,
including lspci output and full kernel output.
Here is the end of the serial log from the failed boot:
[ 2.868692] ERST: Failed to get Error Log Address Range.
[ 3.705427] Refined TSC clocksource calibration: 3599.999 MHz.
[ 3.711340] Switching to clocksource tsc
[ 62.804918] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 3, t=15002 jiffies)
[ 62.804923] INFO: Stall ended before state dump start
[ 242.351898] INFO: task swapper/0:1 blocked for more than 120 seconds.
[ 242.372354] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 242.380260] swapper/0 D 00000001 0 1 0 0x00000000
[ 242.386812] e8cd56f0 00000046 c12c30e0 00000001 e8cd6000 00000000 2c2abdd0 c14cb680
[ 242.400292] e8cd58c0 c14cb680 00000001 c12c5c47 c154bfdc c12c2f02 00003520 c10283fb
[ 242.408607] 00000001 c12c5bb3 e8c18800 c12c3002 c118f8e7 c140007c c140007c e8c18800
[ 242.416931] Call Trace:
[ 242.419453] [<c12c30e0>] ? need_resched+0x17/0x1b
[ 242.424296] [<c12c5c47>] ? add_preempt_count+0x88/0x97
[ 242.429577] [<c12c2f02>] ? _raw_spin_lock_irqsave+0x1b/0x37
[ 242.435308] [<c10283fb>] ? get_parent_ip+0x8/0x19
[ 242.440158] [<c12c5bb3>] ? sub_preempt_count+0x74/0x80
[ 242.445439] [<c12c3002>] ? _raw_spin_unlock_irqrestore+0x21/0x2b
[ 242.451602] [<c118f8e7>] ? vgacon_scroll+0x175/0x18f
[ 242.456703] [<c12c5ccd>] ? __atomic_notifier_call_chain+0x33/0x3c
[ 242.462942] [<c12c20ac>] ? schedule+0x64/0x7d
[ 242.467441] [<c12c22d5>] ? schedule_timeout+0x1f/0xac
[ 242.472641] [<c11d530a>] ? notify_update+0x1f/0x23
[ 242.477574] [<c10283fb>] ? get_parent_ip+0x8/0x19
[ 242.482421] [<c12c5bb3>] ? sub_preempt_count+0x74/0x80
[ 242.487700] [<c10283fb>] ? get_parent_ip+0x8/0x19
[ 242.492543] [<c12c5bb3>] ? sub_preempt_count+0x74/0x80
[ 242.497831] [<c102b2de>] ? migrate_enable+0x120/0x132
[ 242.503024] [<c12c1f7f>] ? wait_for_common+0x7f/0xe1
[ 242.508130] [<c102b0d6>] ? try_to_wake_up+0x189/0x189
[ 242.513336] [<c1077820>] ? __call_rcu+0xe6/0xe6
[ 242.518025] [<c1044fcc>] ? wait_rcu_gp+0x2e/0x33
[ 242.522787] [<c1044fd1>] ? wait_rcu_gp+0x33/0x33
[ 242.527553] [<c1020000>] ? huge_pte_alloc+0x1cc/0x216
[ 242.532750] [<c102b2de>] ? migrate_enable+0x120/0x132
[ 242.537953] [<c119bbcb>] ? acpi_post_unmap_gar+0x78/0x91
[ 242.543424] [<c11ba24a>] ? post_unmap_gar_callback+0x15/0x18
[ 242.549224] [<c11b9c97>] ? apei_exec_for_each_entry+0x64/0x78
[ 242.555110] [<c11ba235>] ? apei_resources_request+0x17f/0x17f
[ 242.561008] [<c143aa36>] ? setup_erst_disable+0xd/0xd
[ 242.566203] [<c11b9cc9>] ? apei_exec_post_unmap_gars+0xe/0x10
[ 242.572088] [<c143ac67>] ? erst_init+0x231/0x288
[ 242.576851] [<c11ef04a>] ? bus_add_driver+0x180/0x1b2
[ 242.582054] [<c100115c>] ? do_one_initcall+0x66/0x10e
[ 242.587246] [<c14177d8>] ? kernel_init+0xfc/0x173
[ 242.592093] [<c14176dc>] ? start_kernel+0x31a/0x31a
[ 242.597110] [<c12c76fe>] ? kernel_thread_helper+0x6/0xdSignature · 2 lines
Alexandra N. Kossovsky OKTET Labs (http://www.oktetlabs.ru/)
Re: 3.2.57-rt: rcu_preempt detected stalls at boot
Quoted 9 lines · Alexandra N. Kossovsky
Hello.
One of my computers fails to boot with 3.2.57-rt kernel, i686. The kernel is the Debian one: linux-image-3.2.0-4-rt-686-pae, version 3.2.57-3. Same non-rt kernel boots fine. All newer -rt kernels (I've tried kernels starting from 3.6) boot fine.
You can find some details of the hardware in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746232,
including lspci output and full kernel output.
One of the more recent additions was to ensure RCU_BOOST was auto selected via Kconfig dependencies. It might not have been present in 3.2.57-rt; not sure w/o checking what debian has in there. But you might want to check the .config you have to see whether it is on or off, and try enabling it.
Paul.
--
Quoted 51 lines
Here is the end of the serial log from the failed boot:
[ 2.868692] ERST: Failed to get Error Log Address Range. [ 3.705427] Refined TSC clocksource calibration: 3599.999 MHz. [ 3.711340] Switching to clocksource tsc [ 62.804918] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 3, t=15002 jiffies) [ 62.804923] INFO: Stall ended before state dump start [ 242.351898] INFO: task swapper/0:1 blocked for more than 120 seconds. [ 242.372354] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.380260] swapper/0 D 00000001 0 1 0 0x00000000 [ 242.386812] e8cd56f0 00000046 c12c30e0 00000001 e8cd6000 00000000 2c2abdd0 c14cb680 [ 242.400292] e8cd58c0 c14cb680 00000001 c12c5c47 c154bfdc c12c2f02 00003520 c10283fb [ 242.408607] 00000001 c12c5bb3 e8c18800 c12c3002 c118f8e7 c140007c c140007c e8c18800 [ 242.416931] Call Trace: [ 242.419453] [<c12c30e0>] ? need_resched+0x17/0x1b [ 242.424296] [<c12c5c47>] ? add_preempt_count+0x88/0x97 [ 242.429577] [<c12c2f02>] ? _raw_spin_lock_irqsave+0x1b/0x37 [ 242.435308] [<c10283fb>] ? get_parent_ip+0x8/0x19 [ 242.440158] [<c12c5bb3>] ? sub_preempt_count+0x74/0x80 [ 242.445439] [<c12c3002>] ? _raw_spin_unlock_irqrestore+0x21/0x2b [ 242.451602] [<c118f8e7>] ? vgacon_scroll+0x175/0x18f [ 242.456703] [<c12c5ccd>] ? __atomic_notifier_call_chain+0x33/0x3c [ 242.462942] [<c12c20ac>] ? schedule+0x64/0x7d [ 242.467441] [<c12c22d5>] ? schedule_timeout+0x1f/0xac [ 242.472641] [<c11d530a>] ? notify_update+0x1f/0x23 [ 242.477574] [<c10283fb>] ? get_parent_ip+0x8/0x19 [ 242.482421] [<c12c5bb3>] ? sub_preempt_count+0x74/0x80 [ 242.487700] [<c10283fb>] ? get_parent_ip+0x8/0x19 [ 242.492543] [<c12c5bb3>] ? sub_preempt_count+0x74/0x80 [ 242.497831] [<c102b2de>] ? migrate_enable+0x120/0x132 [ 242.503024] [<c12c1f7f>] ? wait_for_common+0x7f/0xe1 [ 242.508130] [<c102b0d6>] ? try_to_wake_up+0x189/0x189 [ 242.513336] [<c1077820>] ? __call_rcu+0xe6/0xe6 [ 242.518025] [<c1044fcc>] ? wait_rcu_gp+0x2e/0x33 [ 242.522787] [<c1044fd1>] ? wait_rcu_gp+0x33/0x33 [ 242.527553] [<c1020000>] ? huge_pte_alloc+0x1cc/0x216 [ 242.532750] [<c102b2de>] ? migrate_enable+0x120/0x132 [ 242.537953] [<c119bbcb>] ? acpi_post_unmap_gar+0x78/0x91 [ 242.543424] [<c11ba24a>] ? post_unmap_gar_callback+0x15/0x18 [ 242.549224] [<c11b9c97>] ? apei_exec_for_each_entry+0x64/0x78 [ 242.555110] [<c11ba235>] ? apei_resources_request+0x17f/0x17f [ 242.561008] [<c143aa36>] ? setup_erst_disable+0xd/0xd [ 242.566203] [<c11b9cc9>] ? apei_exec_post_unmap_gars+0xe/0x10 [ 242.572088] [<c143ac67>] ? erst_init+0x231/0x288 [ 242.576851] [<c11ef04a>] ? bus_add_driver+0x180/0x1b2 [ 242.582054] [<c100115c>] ? do_one_initcall+0x66/0x10e [ 242.587246] [<c14177d8>] ? kernel_init+0xfc/0x173 [ 242.592093] [<c14176dc>] ? start_kernel+0x31a/0x31a [ 242.597110] [<c12c76fe>] ? kernel_thread_helper+0x6/0xd
Quoted 16 lines · Paul Gortmaker
Quoted 9 lines · Alexandra N. Kossovsky
Hello.
One of my computers fails to boot with 3.2.57-rt kernel, i686. The kernel is the Debian one: linux-image-3.2.0-4-rt-686-pae, version 3.2.57-3. Same non-rt kernel boots fine. All newer -rt kernels (I've tried kernels starting from 3.6) boot fine.
You can find some details of the hardware in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746232,
including lspci output and full kernel output.One of the more recent additions was to ensure
RCU_BOOSTwas auto selected viaKconfigdependencies. It might not have been present in 3.2.57-rt; not sure w/o checking what debian has in there. But you might want to check the .config you have to see whether it is on or off, and try enabling it.
[...]
$ grep RCU_BOOST /usr/src/linux-headers-3.2.0-4-rt-686-pae/.config # CONFIG_RCU_BOOST is not set
Sasha, could you try enabling this? (Instructions for rebuilding the
package are at
<http://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official>, or you could take the source and rt patch from linux-source-3.2 and then use 'make deb-pkg'.)
Ben.
Signature · 3 lines
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
- Albert CamusQuoted 17 lines · Ben Hutchings
Quoted 11 lines · Paul Gortmaker
Quoted 9 lines · Alexandra N. Kossovsky
Hello.
One of my computers fails to boot with 3.2.57-rt kernel, i686. The kernel is the Debian one: linux-image-3.2.0-4-rt-686-pae, version 3.2.57-3. Same non-rt kernel boots fine. All newer -rt kernels (I've tried kernels starting from 3.6) boot fine.
You can find some details of the hardware in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746232,
including lspci output and full kernel output.Quoted 5 lines
One of the more recent additions was to ensure
RCU_BOOSTwas auto selected viaKconfigdependencies. It might not have been present in 3.2.57-rt; not sure w/o checking what debian has in there. But you might want to check the .config you have to see whether it is on or off, and try enabling it.
Quoted 4 lines
[...]
$ grep RCU_BOOST /usr/src/linux-headers-3.2.0-4-rt-686-pae/.config # CONFIG_RCU_BOOST is not set
$ grep BOOST .config CONFIG_RCU_BOOST=y CONFIG_RCU_BOOST_PRIO=1 CONFIG_RCU_BOOST_DELAY=500
Does not help.
Should I play with PRIO and DELAY numbers?
Log:
[ 5.414059] Switching to clocksource tsc
[ 64.503649] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 6, t=15002 jiffies)
[ 64.503655] INFO: Stall ended before state dump start
[ 244.054622] INFO: task swapper/0:1 blocked for more than 120 seconds.
[ 244.075069] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 244.088152] swapper/0 D e9f40680 0 1 0 0x00000000
[ 244.094720] e28d56f0 00000046 00000df4 e9f40680 e28d58c4 c102815b c14c7680 c14c7680
[ 244.103035] c1547fdc c12c35f2 000035c0 c102815b 00000001 c12c6149 e2818800 c12c357c
[ 244.111342] c119012e 00000004 c13fc07c e2818800 00000000 c13fc07c c12c6276 ffffffff
[ 244.119649] Call Trace:
[ 244.122163] [<c102815b>] ? get_parent_ip+0x8/0x19
[ 244.127012] [<c12c35f2>] ? _raw_spin_lock_irqsave+0x1b/0x39
[ 244.132726] [<c102815b>] ? get_parent_ip+0x8/0x19
[ 244.137576] [<c12c6149>] ? sub_preempt_count+0x68/0x84
[ 244.142859] [<c12c357c>] ? _raw_spin_unlock_irqrestore+0x1f/0x29
[ 244.149031] [<c119012e>] ? vgacon_scroll+0x109/0x19a
[ 244.154142] [<c12c6276>] ? __atomic_notifier_call_chain+0x33/0x3c
[ 244.160374] [<c12c26ca>] ? schedule+0x65/0x7e
[ 244.164871] [<c12c28f2>] ? schedule_timeout+0x1f/0xaf
[ 244.170064] [<c102815b>] ? get_parent_ip+0x8/0x19
[ 244.174909] [<c12c6149>] ? sub_preempt_count+0x68/0x84
[ 244.180188] [<c102815b>] ? get_parent_ip+0x8/0x19
[ 244.185032] [<c102815b>] ? get_parent_ip+0x8/0x19
[ 244.189880] [<c12c6149>] ? sub_preempt_count+0x68/0x84
[ 244.195166] [<c102b005>] ? migrate_enable+0xb3/0x135
[ 244.200272] [<c12c259c>] ? wait_for_common+0x7f/0xe1
[ 244.205377] [<c102aed0>] ? try_to_wake_up+0x183/0x183
[ 244.210577] [<c10781d5>] ? __call_rcu+0xe7/0xe7
[ 244.215257] [<c1044f64>] ? wait_rcu_gp+0x2e/0x33
[ 244.220019] [<c1044f69>] ? wait_rcu_gp+0x33/0x33
[ 244.224790] [<c1020000>] ? kmap_atomic_prot+0x7b/0xd9
[ 244.229979] [<c102b005>] ? migrate_enable+0xb3/0x135
[ 244.235097] [<c119bd45>] ? acpi_post_unmap_gar+0x78/0x93
[ 244.240562] [<c11b9ff2>] ? apei_res_sub+0xd2/0xd2
[ 244.245407] [<c11ba007>] ? post_unmap_gar_callback+0x15/0x18
[ 244.251208] [<c11b9e14>] ? apei_exec_for_each_entry+0x56/0x68
[ 244.257105] [<c1436be8>] ? setup_erst_disable+0xd/0xd
[ 244.262302] [<c11b9e44>] ? apei_exec_post_unmap_gars+0xe/0x10
[ 244.268183] [<c1436e18>] ? erst_init+0x230/0x29d
[ 244.272956] [<c11ef1ca>] ? bus_add_driver+0x17d/0x1ce
[ 244.278158] [<c1436a00>] ? acpi_initialize_subsystem+0x83/0x83
[ 244.284140] [<c11efbb8>] ? driver_register+0x75/0xd6
[ 244.289252] [<c100115c>] ? do_one_initcall+0x66/0x10e
[ 244.294445] [<c14137d7>] ? kernel_init+0xfc/0x173
[ 244.299291] [<c14136db>] ? start_kernel+0x314/0x314
[ 244.304310] [<c12c7d3e>] ? kernel_thread_helper+0x6/0xd
[ 244.309676] INFO: task rcub/1:13 blocked for more than 120 seconds.
[ 244.315993] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 244.323885] rcub/1 D e9f40680 0 13 2 0x00000000
[ 244.330414] e28eec10 00000046 6572756c e9f40680 30056543 00000001 c14c7680 c14c7680
[ 244.338732] c104ba42 c1471680 c10061df e9f406c4 ff48e500 ffffffff e28d56f0 c12cc3d8
[ 244.347049] e28d56f0 c12cc3d8 c1020f7b
[ 244.350142] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=60034 jiffies)
[ 244.350143] INFO: Stall ended before state dump start
[ 244.365081] e28d56f0 e9f40680 00000062 c102a24f 00000078
[ 244.370853] Call Trace:
[ 364.546439] [<c104ba42>] ? sched_clock_cpu+0x10/0x146
[ 364.551634] [<c10061df>] ? __cycles_2_ns+0x19/0x7e
[ 364.556567] [<c1020f7b>] ? check_class_changed+0x25/0x33
[ 364.562020] [<c102a24f>] ? rt_mutex_setprio+0x10d/0x134
[ 364.567384] [<c12c26ca>] ? schedule+0x65/0x7e
[ 364.571882] [<c12c2f87>] ? __rt_mutex_slowlock+0x79/0xb7
[ 364.577336] [<c12c3086>] ? rt_mutex_slowlock+0xc1/0x116
[ 364.582701] [<c1024abb>] ? finish_task_switch+0x66/0x9f
[ 364.588066] [<c10579a7>] ? rt_mutex_fastlock.constprop.14+0x12/0x2a
[ 364.594472] [<c1076fba>] ? rcu_boost_kthread+0xdf/0x125
[ 364.599843] [<c1020101>] ? __kunmap_atomic+0x2f/0x6a
[ 364.604949] [<c1076edb>] ? trace_rcu_utilization+0x50/0x50
[ 364.610573] [<c1046c54>] ? kthread+0x64/0x69
[ 364.614986] [<c1046bf0>] ? flush_kthread_worker+0x7d/0x7d
[ 364.620525] [<c12c7d3e>] ? kernel_thread_helper+0x6/0xdSasha.
Signature · 2 lines
Alexandra N. Kossovsky OKTET Labs (http://www.oktetlabs.ru/)
* Alexandra N. Kossovsky | 2014-05-15 12:58:25 [+0400]:
you seem to have have Xeon / i7. Can't it run x8664 / amd64? I have no idea what is going wrong here except that something is preventing rcu from running.
Sebastian
cover Lock problem when setting cpufreq variables
Hi Sebastian,
your 3.18.7-rt1 kernel compiled without problem. First quick test shows good latency values (i7 at rack #4, slot #6) and looks promising. Will continue testing on more machines and architectures.
The cpufreq problem that first appeared in 3.14 - obviously - is still there. Here comes a workaround that I needed to conduct tests at various clock frequencies. It may also help to demonstrate the situation.
Thanks,
-Carsten.
PATCH 1/1 Workaround to access sysfs cpufreq variables
Any attempt to write to one of the sysfs cpufreq variables results in a write error such as
# echo performance >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor -bash: echo: write error: Invalid argument
that is caused by lock contention. This patch is not a solution but a workaround to demonstrate the situation.
--- drivers/cpufreq/cpufreq.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/cpufreq/cpufreq.c +10 -2
@@ -205,8 +205,16 @@ struct cpufreq_policy *cpufreq_cpu_get(uif (cpufreq_disabled () || (cpu >= nr_cpu_ids))return NULL ; -if (!down_read_trylock (&cpufreq_rwsem)) -return NULL ; +if (!down_read_trylock (&cpufreq_rwsem)) { +struct rt_mutex *rtm = &cpufreq_rwsem.lock; +struct task_struct *owner = rtm->owner; +if (owner->pid != current->pid) { +printk (KERN_INFO +"%s: owner->pid %d (%s), current->pid %d\n" , + __func__, owner->pid, owner->comm, current->pid); +return NULL ; + } + }/* get the cpufreq driver */ read_lock_irqsave (&cpufreq_driver_lock, flags);