MD RAID1 deadlock on failed disk

DISCUSSION Hubert Tonneau 2 participants 2 messages Oct 27, 2010 SCSI

Share & tools

Hubert Tonneau

author

2010-10-27 09:38 UTC #0 lore raw

Hi,

The configuration is:
Perc H200 controller configured with no RAID (mpt2sas driver), 2 SATA disks (sda and sdb),
Linux MD Sofware RAID1 (md0),
stock Linux 2.6.35.7 kernel.

I hotunplug the second (sdb) disk, and the result is:
. as expected, I can read sda device,
. as expected, any read to sdb device fails, . unexpectedly, any read to md0 never returns.

No oops or thing like that in the kernel log. I did not try the same with other kernel releases.

2.6.32.24 kernel worked fine.

Neil Brown asked for /proc/sysrq-trigger ouput, and concluded that the problem is related to 'fw_event0'. See his answer bellow.

Regards,
Hubert Tonneau

Quoted 17 lines · Neil Brown

The fw_event0 process is interesting.
It seems to be hung trying to 'sync' the drive that has just been pulled. If that is somehow causing some IO request from the md/raid1 to be delayed then that would certainly hang the array.
There is a section in the middle of the trace which is missing - presumably the sysrq-trigger output overflowed a buffer - that isn't uncommon.
So I cannot see all the timing clearly.
How long after pulling the drive was this trace taken?
I suspect that you need to post this to [email protected] and ask about that fw_event0 thread - whether that should happen, whether it has been fixed, and whether it could delay pending IO requests.
NeilBrown

Neil Brown

2010-10-27 09:52 UTC #1 lore raw

Quoted 22 lines · On Wed, 27 Oct 2010 10:44:02 GMT Hubert Tonneau…

Hi,
The configuration is:
Perc H200 controller configured with no RAID (mpt2sas driver), 2 SATA disks (sda and sdb),
Linux MD Sofware RAID1 (md0),
stock Linux 2.6.35.7 kernel.
I hotunplug the second (sdb) disk, and the result is:
. as expected, I can read sda device,
. as expected, any read to sdb device fails, . unexpectedly, any read to md0 never returns.
No oops or thing like that in the kernel log. I did not try the same with other kernel releases.
2.6.32.24 kernel worked fine.
Neil Brown asked for /proc/sysrq-trigger ouput, and concluded that the problem is related to 'fw_event0'. See his answer bellow.

Quoted 22 lines

Regards,
Hubert Tonneau
Quoted 17 lines · Neil Brown
The fw_event0 process is interesting.
It seems to be hung trying to 'sync' the drive that has just been pulled. If that is somehow causing some IO request from the md/raid1 to be delayed then that would certainly hang the array.
There is a section in the middle of the trace which is missing - presumably the sysrq-trigger output overflowed a buffer - that isn't uncommon.
So I cannot see all the timing clearly.
How long after pulling the drive was this trace taken?
I suspect that you need to post this to [email protected] and ask about that fw_event0 thread - whether that should happen, whether it has been fixed, and whether it could delay pending IO requests.
NeilBrown

It probably would help to have included the sysrq-T output so the scsi people could see why I pointed the finger at fw_event0.

Here is that part of the trace

<6>[  318.881486] fw_event0     D 0000000000000000     0   244      2 0x00000000
<4>[  318.881493]  ffff88081d191570 0000000000000046 ffff880800000000 00000000000158c0
<4>[  318.881500]  ffff88081d191fd8 00000000000158c0 ffff88081d191fd8 ffff88081d188000
<4>[  318.881507]  00000000000158c0 00000000000158c0 ffff88081d191fd8 00000000000158c0
<4>[  318.881514] Call Trace:
<4>[  318.881520]  [<ffffffff815a296d>] schedule_timeout+0x22d/0x310
<4>[  318.881526]  [<ffffffff813a21f0>] ? __scsi_queue_insert+0xb0/0x130
<4>[  318.881533]  [<ffffffff815a252b>] wait_for_common+0xdb/0x1a0
<4>[  318.881540]  [<ffffffff81051910>] ? default_wake_function+0x0/0x20
<4>[  318.881546]  [<ffffffff81294093>] ? __generic_unplug_device+0x33/0x40
<4>[  318.881553]  [<ffffffff815a26cd>] wait_for_completion+0x1d/0x20
<4>[  318.881560]  [<ffffffff8129a9fe>] blk_execute_rq+0x8e/0xf0
<4>[  318.881567]  [<ffffffff8129666c>] ? blk_get_request+0x6c/0xa0
<4>[  318.881573]  [<ffffffff813a129c>] scsi_execute+0xfc/0x160
<4>[  318.881580]  [<ffffffff813a2cec>] scsi_execute_req+0xac/0x180
<4>[  318.881589]  [<ffffffff813c5fd0>] sd_sync_cache+0xd0/0x120
<4>[  318.881598]  [<ffffffff815a187a>] ? printk+0x68/0x6e
<4>[  318.881604]  [<ffffffff813c6283>] sd_shutdown+0x83/0x1b0
<4>[  318.881610]  [<ffffffff813c6562>] sd_remove+0x62/0xa0
<4>[  318.881618]  [<ffffffff81377555>] __device_release_driver+0x75/0xe0
<4>[  318.881624]  [<ffffffff81377acd>] device_release_driver+0x2d/0x40
<4>[  318.881631]  [<ffffffff81376532>] bus_remove_device+0xb2/0xf0
<4>[  318.881637]  [<ffffffff81374237>] device_del+0x127/0x1b0
<4>[  318.881644]  [<ffffffff813a74d5>] __scsi_remove_device+0xb5/0xc0
<4>[  318.881650]  [<ffffffff813a7510>] scsi_remove_device+0x30/0x50
<4>[  318.881656]  [<ffffffff813a7601>] __scsi_remove_target+0xb1/0xe0
<4>[  318.881662]  [<ffffffff813a76a0>] ? __remove_child+0x0/0x30
<4>[  318.881667]  [<ffffffff813a76c3>] __remove_child+0x23/0x30
<4>[  318.881673]  [<ffffffff8137399c>] device_for_each_child+0x4c/0x80
<4>[  318.881679]  [<ffffffff813a766e>] scsi_remove_target+0x3e/0x70
<4>[  318.881686]  [<ffffffff813abcc5>] sas_rphy_remove+0x75/0x80
<4>[  318.881692]  [<ffffffff813ac266>] sas_rphy_delete+0x16/0x30
<4>[  318.881698]  [<ffffffff813ac2aa>] sas_port_delete+0x2a/0x130
<4>[  318.881704]  [<ffffffff813bf3ca>] mpt2sas_transport_port_remove+0x15a/0x240
<4>[  318.881711]  [<ffffffff813ba9ed>] _scsih_remove_device+0xcd/0x120
<4>[  318.881720]  [<ffffffff81035d09>] ? default_spin_lock_flags+0x9/0x10
<4>[  318.881726]  [<ffffffff813bea00>] ? mpt2sas_transport_update_links+0x80/0x1a0
<4>[  318.881733]  [<ffffffff813be0ee>] _firmware_event_work+0x155e/0x1af0
<4>[  318.881742]  [<ffffffff8100860b>] ? __switch_to+0xcb/0x350
<4>[  318.881749]  [<ffffffff8104de5a>] ? finish_task_switch+0x4a/0xd0
<4>[  318.881756]  [<ffffffff813bcb90>] ? _firmware_event_work+0x0/0x1af0
<4>[  318.881762]  [<ffffffff810792cf>] worker_thread+0x17f/0x2b0
<4>[  318.881769]  [<ffffffff8107d9c0>] ? autoremove_wake_function+0x0/0x40
<4>[  318.881775]  [<ffffffff81079150>] ? worker_thread+0x0/0x2b0
<4>[  318.881781]  [<ffffffff8107d466>] kthread+0x96/0xa0
<4>[  318.881787]  [<ffffffff8100ae64>] kernel_thread_helper+0x4/0x10
<4>[  318.881794]  [<ffffffff8107d3d0>] ? kthread+0x0/0xa0
<4>[  318.881799]  [<ffffffff8100ae60>] ? kernel_thread_helper+0x0/0x10

It seems to hang here, and while it hangs old IO requests don't complete so md/raid1 cannot proceed.

NeilBrown