Skip to contentSkip to conversation

2.6.39: raid1 check blocks jbd on other md more than 120 seconds

DISCUSSION Frank van Maarseveen 3 participants 6 messages Jun 2, 2011 – Jun 7, 2011 MD RAID

lore
Share & tools
lore
#0 lore raw

The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.

/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]

      1885338488 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]

      33555384 blocks super 1.2 [2/2] [UU]

kernel messages:
(/etc/cron.weekly/99-raid-check kicks in)

Jun  2 04:04:00 janus md: data-check of RAID array md3
Jun  2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units)
Jun  2 04:04:00 janus md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jun  2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Jun  2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks.
Jun  2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds.
Jun  2 04:55:54 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun  2 04:55:54 janus jbd2/md1-8     D
Jun  2 04:55:54  f5739e68
Jun  2 04:55:54 janus    0  1188      2 0x00000000
Jun  2 04:55:54  f5739e78
Jun  2 04:55:54  00000046
Jun  2 04:55:54  00000002
Jun  2 04:55:54  f5739e68
Jun  2 04:55:54  00000000
Jun  2 04:55:54  01000000
Jun  2 04:55:54  f5739e08
Jun  2 04:55:54  f6124f00
Jun  2 04:55:54 janus
Jun  2 04:55:54  c1999c00
Jun  2 04:55:54  00000001
Jun  2 04:55:54  c1999680
Jun  2 04:55:54  c1999680
Jun  2 04:55:54  00000003
Jun  2 04:55:54  f6126940
Jun  2 04:55:54  f6126ac0
Jun  2 04:55:54  f6786680
Jun  2 04:55:54 janus
Jun  2 04:55:54  000045d1
Jun  2 04:55:54  00000000
Jun  2 04:55:54  00000000
Jun  2 04:55:54  57087b88
Jun  2 04:55:54  000045d1
Jun  2 04:55:54  000045d1
Jun  2 04:55:54  f6686c00
Jun  2 04:55:54  f6686c10
Jun  2 04:55:54 janus
Jun  2 04:55:54 Call Trace:
Jun  2 04:55:54  [<c103822b>] ? get_parent_ip+0xb/0x40
Jun  2 04:55:54  [<c11698a2>] jbd2_journal_commit_transaction+0x182/0x1500
Jun  2 04:55:54  [<c100249f>] ? __switch_to+0xcf/0x1c0
Jun  2 04:55:54  [<c103822b>] ? get_parent_ip+0xb/0x40
Jun  2 04:55:54  [<c162f05c>] ? schedule+0x2ac/0x730
Jun  2 04:55:54  [<c1056c90>] ? wake_up_bit+0x80/0x80
	(reboots)
Signature · 1 line
Frank
#1 lore raw
Quoted 60 lines · Frank van Maarseveen

The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.

/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]
     1885338488 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
     33555384 blocks super 1.2 [2/2] [UU]

kernel messages:
       (/etc/cron.weekly/99-raid-check kicks in)
Jun  2 04:04:00 janus md: data-check of RAID array md3
Jun  2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units) Jun  2 04:04:00 janus md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jun  2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jun  2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks.
Jun  2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds.
Jun  2 04:55:54 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun  2 04:55:54 janus jbd2/md1-8     D
Jun  2 04:55:54  f5739e68
Jun  2 04:55:54 janus    0  1188      2 0x00000000
Jun  2 04:55:54  f5739e78
Jun  2 04:55:54  00000046
Jun  2 04:55:54  00000002
Jun  2 04:55:54  f5739e68
Jun  2 04:55:54  00000000
Jun  2 04:55:54  01000000
Jun  2 04:55:54  f5739e08
Jun  2 04:55:54  f6124f00
Jun  2 04:55:54 janus
Jun  2 04:55:54  c1999c00
Jun  2 04:55:54  00000001
Jun  2 04:55:54  c1999680
Jun  2 04:55:54  c1999680
Jun  2 04:55:54  00000003
Jun  2 04:55:54  f6126940
Jun  2 04:55:54  f6126ac0
Jun  2 04:55:54  f6786680
Jun  2 04:55:54 janus
Jun  2 04:55:54  000045d1
Jun  2 04:55:54  00000000
Jun  2 04:55:54  00000000
Jun  2 04:55:54  57087b88
Jun  2 04:55:54  000045d1
Jun  2 04:55:54  000045d1
Jun  2 04:55:54  f6686c00
Jun  2 04:55:54  f6686c10
Jun  2 04:55:54 janus
Jun  2 04:55:54 Call Trace:
Jun  2 04:55:54  [<c103822b>] ? get_parent_ip+0xb/0x40
Jun  2 04:55:54  [<c11698a2>] jbd2_journal_commit_transaction+0x182/0x1500
Jun  2 04:55:54  [<c100249f>] ? __switch_to+0xcf/0x1c0
Jun  2 04:55:54  [<c103822b>] ? get_parent_ip+0xb/0x40
Jun  2 04:55:54  [<c162f05c>] ? schedule+0x2ac/0x730
Jun  2 04:55:54  [<c1056c90>] ? wake_up_bit+0x80/0x80
       (reboots)

Quoted 7 lines

--
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Same behavior if you lower this?

Jun  2 04:04:00 janus md: using maximum available idle IO bandwidth

(but not more than 200000 KB/sec) for data-check.

/M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 21 lines · Mathias Burén
Quoted 12 lines · Frank van Maarseveen

The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.

/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]
     1885338488 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
     33555384 blocks super 1.2 [2/2] [UU]

Quoted 8 lines

kernel messages:
       (/etc/cron.weekly/99-raid-check kicks in)
Jun  2 04:04:00 janus md: data-check of RAID array md3
Jun  2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units) Jun  2 04:04:00 janus md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jun  2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jun  2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks.
Jun  2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds.

Quoted 6 lines

[...]

Same behavior if you lower this?

Jun  2 04:04:00 janus md: using maximum available idle IO bandwidth

(but not more than 200000 KB/sec) for data-check.

Practical bandwidth is usually slightly more than 100MB/s at start to approximately 60MB/s at the end of the disk. I tried setting sync_speed_max at 70000kB/s. The problem seems to correlate with the max. practical bandwidth because at the end of the data-check there were a couple of hung task messages again, referring to postfix- and other daemons this time. Timeline:

Jun  2 11:52:30 janus kernel: md: data-check of RAID array md3
Jun  2 11:52:30 janus kernel: md: using maximum available idle IO bandwidth (but not more than 70000 KB/sec) for data-check.
Jun  2 18:48:44 hung task
Jun  2 18:48:44 hung task
Jun  2 18:50:44 hung task
Jun  2 18:50:45 hung task
Jun  2 19:28:45 hung task
Jun  2 19:28:45 hung task
Jun  2 19:34:45 hung task
Jun  2 19:34:45 hung task
Jun  2 19:34:45 hung task
Jun  2 19:34:45 hung task
Jun  2 19:53:29 janus kernel: md: md3: data-check done.

Kernel has been booted with hung_task_panic=0.

Signature · 5 lines
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw
In reply to Frank van Maarseveen
Quoted 14 lines · Frank van Maarseveen

The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.

/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]

       1885338488 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]

       33555384 blocks super 1.2 [2/2] [UU]

kernel messages:
(/etc/cron.weekly/99-raid-check kicks in)

Quoted 8 lines
Jun  2 04:04:00 janus md: data-check of RAID array md3
Jun  2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units)
Jun  2 04:04:00 janus md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jun  2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Jun  2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks.
Jun  2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds.
Jun  2 04:55:54 "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun  2 04:55:54 janus jbd2/md1-8     D

That's a bug that you'll see in CentOS/RHEL in cases where there are multiple arrays to be checked, that use the same set of disks. I first saw it in CentOS 5.5 (or maybe 5.6).

https://bugzilla.redhat.com/show_bug.cgi?id=573106

It's an annoying message, but the weekly raid sync runs fine.

#4 lore raw
Quoted 24 lines · Thomas Harold
Quoted 14 lines · Frank van Maarseveen

The system runs FC14 with an (almost) stock 2.6.39 kernel, configured to panic if it seems to hang. That's exactly what started to happen without anything being logged in the normal way except over netconsole.

/proc/mdstat:
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] md3 : active raid1 sda3[0] sdb3[1]

      1885338488 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]

      33555384 blocks super 1.2 [2/2] [UU]

kernel messages:
(/etc/cron.weekly/99-raid-check kicks in)

Quoted 8 lines
Jun  2 04:04:00 janus md: data-check of RAID array md3
Jun  2 04:04:00 janus md: delaying data-check of md1 until md3 has finished (they share one or more physical units)
Jun  2 04:04:00 janus md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jun  2 04:04:00 janus md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Jun  2 04:04:00 janus md: using 128k window, over a total of 1885338488 blocks.
Jun  2 04:55:54 janus INFO: task jbd2/md1-8:1188 blocked for more than 120 seconds.
Jun  2 04:55:54 "echo 0>  /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun  2 04:55:54 janus jbd2/md1-8     D
Quoted 7 lines

That's a bug that you'll see in CentOS/RHEL in cases where there are multiple arrays to be checked, that use the same set of disks. I first saw it in CentOS 5.5 (or maybe 5.6).

https://bugzilla.redhat.com/show_bug.cgi?id=573106

It's an annoying message, but the weekly raid sync runs fine.

According to the bugzilla report it was the resync itself which got stuck, unlike what I am seeing where any random program may get stuck. Depending on kernel configuration it may trigger a kernel panic. Last time:

Jun  2 18:48:44 janus kernel: INFO: task master:2705 blocked for more than 120 seconds.
Jun  2 18:48:44 janus kernel: INFO: task pickup:19276 blocked for more than 120 seconds.
Jun  2 18:50:44 janus kernel: INFO: task jbd2/md1-8:1187 blocked for more than 120 seconds.
Jun  2 18:50:45 janus kernel: INFO: task python:1890 blocked for more than 120 seconds.
Jun  2 19:28:45 janus kernel: INFO: task master:2705 blocked for more than 120 seconds.
Jun  2 19:28:45 janus kernel: INFO: task pickup:20589 blocked for more than 120 seconds.
Jun  2 19:34:45 janus kernel: INFO: task jbd2/md1-8:1187 blocked for more than 120 seconds.
Jun  2 19:34:45 janus kernel: INFO: task master:2705 blocked for more than 120 seconds.
Jun  2 19:34:45 janus kernel: INFO: task qmgr:2718 blocked for more than 120 seconds.
Jun  2 19:34:45 janus kernel: INFO: task pickup:20589 blocked for more than 120 seconds.
Signature · 1 line
Frank
#5 lore raw

2.6.39[.1]: raid1 check blocks jbd on other md more than 120 seconds -- workaround

In reply to Frank van Maarseveen

I did some more testing and apparently the issue can be avoided by choosing the deadline scheduler instead of cfq. Either configure the kernel with

CONFIG_DEFAULT_IOSCHED="deadline"

or select the deadline I/O scheduler per device. In my case:

echo deadline >/sys/block/sda/queue/scheduler echo deadline >/sys/block/sdb/queue/scheduler

Signature · 1 line
Frank