2.6.39: raid1 check blocks jbd on other md more than 120 seconds
The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.
/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]
1885338488 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sda1[0] sdb1[1]
33555384 blocks super 1.2 [2/2] [UU]
kernel messages:
(/etc/cron.weekly/99-raid-check kicks in)
Jun 2 04:04:00 janus md: data-check of RAID array md3 Jun 2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units) Jun 2 04:04:00 janus md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Jun 2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jun 2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks. Jun 2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds. Jun 2 04:55:54 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 04:55:54 janus jbd2/md1-8 D Jun 2 04:55:54 f5739e68 Jun 2 04:55:54 janus 0 1188 2 0x00000000 Jun 2 04:55:54 f5739e78 Jun 2 04:55:54 00000046 Jun 2 04:55:54 00000002 Jun 2 04:55:54 f5739e68 Jun 2 04:55:54 00000000 Jun 2 04:55:54 01000000 Jun 2 04:55:54 f5739e08 Jun 2 04:55:54 f6124f00 Jun 2 04:55:54 janus Jun 2 04:55:54 c1999c00 Jun 2 04:55:54 00000001 Jun 2 04:55:54 c1999680 Jun 2 04:55:54 c1999680 Jun 2 04:55:54 00000003 Jun 2 04:55:54 f6126940 Jun 2 04:55:54 f6126ac0 Jun 2 04:55:54 f6786680 Jun 2 04:55:54 janus Jun 2 04:55:54 000045d1 Jun 2 04:55:54 00000000 Jun 2 04:55:54 00000000 Jun 2 04:55:54 57087b88 Jun 2 04:55:54 000045d1 Jun 2 04:55:54 000045d1 Jun 2 04:55:54 f6686c00 Jun 2 04:55:54 f6686c10 Jun 2 04:55:54 janus Jun 2 04:55:54 Call Trace: Jun 2 04:55:54 [<c103822b>] ? get_parent_ip+0xb/0x40 Jun 2 04:55:54 [<c11698a2>] jbd2_journal_commit_transaction+0x182/0x1500 Jun 2 04:55:54 [<c100249f>] ? __switch_to+0xcf/0x1c0 Jun 2 04:55:54 [<c103822b>] ? get_parent_ip+0xb/0x40 Jun 2 04:55:54 [<c162f05c>] ? schedule+0x2ac/0x730 Jun 2 04:55:54 [<c1056c90>] ? wake_up_bit+0x80/0x80 (reboots)
Signature · 1 line
Frank
Quoted 60 lines · Frank van Maarseveen
The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.
/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]
1885338488 blocks super 1.2 [2/2] [UU]md1 : active raid1 sda1[0] sdb1[1]
33555384 blocks super 1.2 [2/2] [UU]kernel messages:
(/etc/cron.weekly/99-raid-check kicks in)
Jun 2 04:04:00 janus md: data-check of RAID array md3
Jun 2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units) Jun 2 04:04:00 janus md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jun 2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jun 2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks.
Jun 2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds.
Jun 2 04:55:54 "echo 0 >/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 2 04:55:54 janus jbd2/md1-8 D
Jun 2 04:55:54 f5739e68
Jun 2 04:55:54 janus 0 1188 20x00000000
Jun 2 04:55:54 f5739e78
Jun 2 04:55:54 00000046
Jun 2 04:55:54 00000002
Jun 2 04:55:54 f5739e68
Jun 2 04:55:54 00000000
Jun 2 04:55:54 01000000
Jun 2 04:55:54 f5739e08
Jun 2 04:55:54 f6124f00
Jun 2 04:55:54 janus
Jun 2 04:55:54 c1999c00
Jun 2 04:55:54 00000001
Jun 2 04:55:54 c1999680
Jun 2 04:55:54 c1999680
Jun 2 04:55:54 00000003
Jun 2 04:55:54 f6126940
Jun 2 04:55:54 f6126ac0
Jun 2 04:55:54 f6786680
Jun 2 04:55:54 janus
Jun 2 04:55:54 000045d1
Jun 2 04:55:54 00000000
Jun 2 04:55:54 00000000
Jun 2 04:55:54 57087b88
Jun 2 04:55:54 000045d1
Jun 2 04:55:54 000045d1
Jun 2 04:55:54 f6686c00
Jun 2 04:55:54 f6686c10
Jun 2 04:55:54 janus
Jun 2 04:55:54 Call Trace:
Jun 2 04:55:54 [<c103822b>] ?get_parent_ip+0xb/0x40
Jun 2 04:55:54 [<c11698a2>]jbd2_journal_commit_transaction+0x182/0x1500
Jun 2 04:55:54 [<c100249f>] ?__switch_to+0xcf/0x1c0
Jun 2 04:55:54 [<c103822b>] ?get_parent_ip+0xb/0x40
Jun 2 04:55:54 [<c162f05c>] ?schedule+0x2ac/0x730
Jun 2 04:55:54 [<c1056c90>] ?wake_up_bit+0x80/0x80
(reboots)
Quoted 7 lines
--
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Same behavior if you lower this?
Jun 2 04:04:00 janus md: using maximum available idle IO bandwidth
(but not more than 200000 KB/sec) for data-check.
/M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 21 lines · Mathias Burén
Quoted 12 lines · Frank van Maarseveen
The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.
/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]
1885338488 blocks super 1.2 [2/2] [UU]md1 : active raid1 sda1[0] sdb1[1]
33555384 blocks super 1.2 [2/2] [UU]Quoted 8 lines
kernel messages:
(/etc/cron.weekly/99-raid-check kicks in)
Jun 2 04:04:00 janus md: data-check of RAID array md3
Jun 2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units) Jun 2 04:04:00 janus md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jun 2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jun 2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks.
Jun 2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds.
Quoted 6 lines
[...]
Same behavior if you lower this?
Jun 2 04:04:00 janus md: using maximum available idle IO bandwidth(but not more than 200000 KB/sec) for data-check.
Practical bandwidth is usually slightly more than 100MB/s at start to approximately 60MB/s at the end of the disk. I tried setting sync_speed_max at 70000kB/s. The problem seems to correlate with the max. practical bandwidth because at the end of the data-check there were a couple of hung task messages again, referring to postfix- and other daemons this time. Timeline:
Jun 2 11:52:30 janus kernel: md: data-check of RAID array md3 Jun 2 11:52:30 janus kernel: md: using maximum available idle IO bandwidth (but not more than 70000 KB/sec) for data-check. Jun 2 18:48:44 hung task Jun 2 18:48:44 hung task Jun 2 18:50:44 hung task Jun 2 18:50:45 hung task Jun 2 19:28:45 hung task Jun 2 19:28:45 hung task Jun 2 19:34:45 hung task Jun 2 19:34:45 hung task Jun 2 19:34:45 hung task Jun 2 19:34:45 hung task Jun 2 19:53:29 janus kernel: md: md3: data-check done.
Kernel has been booted with hung_task_panic=0.
Signature · 5 lines
Frank -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · Frank van Maarseveen
The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.
/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]1885338488 blocks super 1.2 [2/2] [UU]md1 : active raid1 sda1[0] sdb1[1]
33555384 blocks super 1.2 [2/2] [UU]kernel messages:
(/etc/cron.weekly/99-raid-checkkicks in)
Quoted 8 lines
Jun 2 04:04:00 janus md: data-check of RAID array md3 Jun 2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units) Jun 2 04:04:00 janus md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Jun 2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jun 2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks. Jun 2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds. Jun 2 04:55:54 "echo 0> /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 04:55:54 janus jbd2/md1-8 D
That's a bug that you'll see in CentOS/RHEL in cases where there are multiple arrays to be checked, that use the same set of disks. I first saw it in CentOS 5.5 (or maybe 5.6).
https://bugzilla.redhat.com/show_bug.cgi?id=573106
It's an annoying message, but the weekly raid sync runs fine.
Quoted 24 lines · Thomas Harold
Quoted 14 lines · Frank van Maarseveen
The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.
/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]1885338488 blocks super 1.2 [2/2] [UU]md1 : active raid1 sda1[0] sdb1[1]
33555384 blocks super 1.2 [2/2] [UU]kernel messages:
(/etc/cron.weekly/99-raid-checkkicks in)Quoted 8 lines
Jun 2 04:04:00 janus md: data-check of RAID array md3 Jun 2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units) Jun 2 04:04:00 janus md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Jun 2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jun 2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks. Jun 2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds. Jun 2 04:55:54 "echo 0> /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 2 04:55:54 janus jbd2/md1-8 D
Quoted 7 lines
That's a bug that you'll see in CentOS/RHEL in cases where there are multiple arrays to be checked, that use the same set of disks. I first saw it in CentOS 5.5 (or maybe 5.6).
https://bugzilla.redhat.com/show_bug.cgi?id=573106
It's an annoying message, but the weekly raid sync runs fine.
According to the bugzilla report it was the resync itself which got stuck, unlike what I am seeing where any random program may get stuck. Depending on kernel configuration it may trigger a kernel panic. Last time:
Jun 2 18:48:44 janus kernel: INFO: task master:2705 blocked for more than 120 seconds. Jun 2 18:48:44 janus kernel: INFO: task pickup:19276 blocked for more than 120 seconds. Jun 2 18:50:44 janus kernel: INFO: task jbd2/md1-8:1187 blocked for more than 120 seconds. Jun 2 18:50:45 janus kernel: INFO: task python:1890 blocked for more than 120 seconds. Jun 2 19:28:45 janus kernel: INFO: task master:2705 blocked for more than 120 seconds. Jun 2 19:28:45 janus kernel: INFO: task pickup:20589 blocked for more than 120 seconds. Jun 2 19:34:45 janus kernel: INFO: task jbd2/md1-8:1187 blocked for more than 120 seconds. Jun 2 19:34:45 janus kernel: INFO: task master:2705 blocked for more than 120 seconds. Jun 2 19:34:45 janus kernel: INFO: task qmgr:2718 blocked for more than 120 seconds. Jun 2 19:34:45 janus kernel: INFO: task pickup:20589 blocked for more than 120 seconds.
Signature · 1 line
Frank
2.6.39[.1]: raid1 check blocks jbd on other md more than 120 seconds -- workaround
In reply to Frank van MaarseveenI did some more testing and apparently the issue can be avoided by choosing the deadline scheduler instead of cfq. Either configure the kernel with
CONFIG_DEFAULT_IOSCHED="deadline"
or select the deadline I/O scheduler per device. In my case:
echo deadline >/sys/block/sda/queue/scheduler echo deadline >/sys/block/sdb/queue/scheduler
Signature · 1 line
Frank