How to online remove an error scsi disk from the system?

DISCUSSION Tao Ma 6 participants 14 messages Feb 1, 2013 – Oct 16, 2013 SCSI

Share & tools

Tao Ma

author

2013-02-01 06:13 UTC #0 lore raw

Hi All,
In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang forever and we will get something in /var/log/messages like below. It seems to me that the io sent to the scsi layer is never returned back with -EIO which is a little bit surprised for me(It should be a timeout somewhere, right?). We have tried echo "offline" > /sys/block/sdl/device/state, but it doesn't work. So is there any way for us to let the scsi device returns all the io requests back with EIO so that all the end_io can be called accordingly? Am I missing something here?

Thanks,
Tao

sd 0:0:11:0: attempting task abort! scmd(ffff88180e900580) sd 0:0:11:0: [sdl] CDB: Write(10): 2a 00 0d ca e0 3f 00 04 00 00 target0:0:11: handle(0x0015), sas_address(0x500e004aaaaaaa0b), phy(11) target0:0:11: enclosure_logical_id(0x500e004aaaaaaa00), slot(11)

INFO: task jbd2/sdl1-8:4629 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/sdl1-8 D 0000000000000000 0 4629 2 0x00000000
ffff88180aa79ae0 0000000000000046 ffff88180aa79aa8 0000000000000000 ffff88007ce0fe40 0000000000015f40 ffff8818102c0638 ffff8818102c0080 ffff880a9184a100 ffff8818102c0638 0000000105006028 0000000100000000

Call Trace:
 [<ffffffff81236a15>] ? cpumask_next_and+0x25/0x40
 [<ffffffff810122b6>] ? read_tsc+0x16/0x40
 [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810122b6>] ? read_tsc+0x16/0x40
 [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff814a8a53>] io_schedule+0x73/0xc0
 [<ffffffff811036a8>] sync_page+0x38/0x50
 [<ffffffff814a927e>] __wait_on_bit+0x5e/0x90
 [<ffffffff81103670>] ? sync_page+0x0/0x50
 [<ffffffff81103845>] wait_on_page_bit+0x75/0x80
 [<ffffffff81089320>] ? wake_bit_function+0x0/0x40
 [<ffffffff811197c7>] ? pagevec_lookup_tag+0x27/0x40
 [<ffffffff81118b55>] write_cache_pages+0x1d5/0x440
 [<ffffffff811172f0>] ? __writepage+0x0/0x40
 [<ffffffff81118de4>] generic_writepages+0x24/0x30
 [<ffffffffa02dc719>] jbd2_journal_commit_transaction+0x3e9/0x1490 [jbd2]
 [<ffffffff81074299>] ? try_to_del_timer_sync+0x49/0xe0
 [<ffffffffa02e2734>] kjournald2+0xb4/0x220 [jbd2]
 [<ffffffff810892e0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02e2680>] ? kjournald2+0x0/0x220 [jbd2]
 [<ffffffff81089166>] kthread+0x96/0xa0
 [<ffffffff8100c08a>] child_rip+0xa/0x20
 [<ffffffff810890d0>] ? kthread+0x0/0xa0
 [<ffffffff8100c080>] ? child_rip+0x0/0x20

Bart Van Assche

2013-02-01 07:54 UTC #1 lore raw

Quoted 10 lines · Tao Ma

In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang forever and we will get something in /var/log/messages like below. It seems to me that the io sent to the scsi layer is never returned back with -EIO which is a little bit surprised for me(It should be a timeout somewhere, right?). We have tried echo "offline" > /sys/block/sdl/device/state, but it doesn't work. So is there any way for us to let the scsi device returns all the io requests back with EIO so that all the end_io can be called accordingly? Am I missing something here?

Please note that I'm not familiar with SAS. But I found this in drivers/scsi/scsi_proc.c:

* proc_scsi_write - handle writes to /proc/scsi/scsi
* @file: not used
* @buf: buffer to write
* @length: length of buf, at most PAGE_SIZE
* @ppos: not used
*
* Description: this provides a legacy mechanism to add or remove
* devices by Host, Channel, ID, and Lun. To use,
* "echo 'scsi add-single-device 0 1 2 3' > /proc/scsi/scsi" or
* "echo 'scsi remove-single-device 0 1 2 3' > /proc/scsi/scsi" with
* "0 1 2 3" replaced by the Host, Channel, Id, and Lun.

Bart.

Tao Ma

author

2013-02-01 09:07 UTC #2 lore raw

Quoted 15 lines · Bart Van Assche

Quoted 10 lines · Tao Ma
In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang forever and we will get something in /var/log/messages like below. It seems to me that the io sent to the scsi layer is never returned back with -EIO which is a little bit surprised for me(It should be a timeout somewhere, right?). We have tried echo "offline" > /sys/block/sdl/device/state, but it doesn't work. So is there any way for us to let the scsi device returns all the io requests back with EIO so that all the end_io can be called accordingly? Am I missing something here?
Please note that I'm not familiar with SAS. But I found this in drivers/scsi/scsi_proc.c:

Quoted 11 lines

* proc_scsi_write - handle writes to /proc/scsi/scsi
* @file: not used
* @buf: buffer to write
* @length: length of buf, at most PAGE_SIZE
* @ppos: not used
*
* Description: this provides a legacy mechanism to add or remove
* devices by Host, Channel, ID, and Lun. To use,
* "echo 'scsi add-single-device 0 1 2 3' > /proc/scsi/scsi" or
* "echo 'scsi remove-single-device 0 1 2 3' > /proc/scsi/scsi" with
* "0 1 2 3" replaced by the Host, Channel, Id, and Lun.

Sorry, it doesn't work since it will also send some IOs to the scsi. And it hangs...

bash D 0000000000000000 0 57479 57477 0x00000000
ffff8817fee2dba0 0000000000000086 0000000000000000 0000000000000002 ffffffff817c4ed5 0000000000015f40 ffff88180c7e45f8 ffff88180c7e4040 ffffffff81a2d020 ffff88180c7e45f8 000000010fa4af09 0000000000000004

Call Trace:
 [<ffffffff8123eecf>] ? string+0x3f/0xd0
 [<ffffffff8123fdc2>] ? vsnprintf+0x242/0x580
 [<ffffffff811a0b14>] ? fsnotify_clear_marks_by_inode+0x34/0xf0
 [<ffffffff811d33c0>] ? sysfs_delete_inode+0x0/0x60
 [<ffffffff814aa5c5>] rwsem_down_failed_common+0x95/0x1c0
 [<ffffffff814aa746>] rwsem_down_read_failed+0x26/0x30
 [<ffffffff81241814>] call_rwsem_down_read_failed+0x14/0x30
 [<ffffffff812385a0>] ? kobject_release+0x0/0x1f0
 [<ffffffff814a9cd4>] ? down_read+0x24/0x30
 [<ffffffff81167794>] get_super+0x74/0xc0
 [<ffffffff8119aa9e>] fsync_bdev+0x1e/0x60
 [<ffffffff812253ce>] invalidate_partition+0x2e/0x60
 [<ffffffff811d0bfe>] del_gendisk+0x3e/0x130
 [<ffffffff813070da>] ? device_del+0x16a/0x1a0
 [<ffffffff8132f437>] sd_remove+0x67/0xb0
 [<ffffffff8130adcf>] __device_release_driver+0x6f/0xe0
 [<ffffffff8130ae6d>] device_release_driver+0x2d/0x40
 [<ffffffff8130a723>] bus_remove_device+0x83/0xe0
 [<ffffffff8130709f>] device_del+0x12f/0x1a0
 [<ffffffff8132a7f5>] __scsi_remove_device+0xa5/0xb0
 [<ffffffff8132a830>] scsi_remove_device+0x30/0x50
 [<ffffffff8132cc3f>] proc_scsi_write+0x23f/0x280
 [<ffffffff81182869>] ? mntput_no_expire+0x39/0xd0
 [<ffffffff811c482f>] proc_reg_write+0x7f/0xc0
 [<ffffffff81165c6c>] vfs_write+0xcc/0x1a0
 [<ffffffff81165e25>] sys_write+0x55/0x90
 [<ffffffff8100b032>] system_call_fastpath+0x16/0x1b

Thanks,
Tao

Quoted 7 lines

Bart.

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Bryn M. Reeves

2013-02-01 09:52 UTC #3 lore raw

In reply to Bart Van Assche

Quoted 11 lines · Bart Van Assche

* proc_scsi_write - handle writes to /proc/scsi/scsi
* @file: not used
* @buf: buffer to write
* @length: length of buf, at most PAGE_SIZE
* @ppos: not used
*
* Description: this provides a legacy mechanism to add or remove
* devices by Host, Channel, ID, and Lun. To use,
* "echo 'scsi add-single-device 0 1 2 3' > /proc/scsi/scsi" or
* "echo 'scsi remove-single-device 0 1 2 3' > /proc/scsi/scsi" with
* "0 1 2 3" replaced by the Host, Channel, Id, and Lun.

The proc interface is deprecated; this can all be done via sysfs today, e.g.:

echo 1 > /sys/block/sdc/device/delete

Is equivalent to issuing scsi remove-single-device to proc.

Regards,
Bryn.

Tao Ma

author

2013-02-01 09:59 UTC #4 lore raw

Quoted 13 lines · Bryn M. Reeves

Quoted 11 lines · Bart Van Assche
* proc_scsi_write - handle writes to /proc/scsi/scsi
* @file: not used
* @buf: buffer to write
* @length: length of buf, at most PAGE_SIZE
* @ppos: not used
*
* Description: this provides a legacy mechanism to add or remove
* devices by Host, Channel, ID, and Lun. To use,
* "echo 'scsi add-single-device 0 1 2 3' > /proc/scsi/scsi" or
* "echo 'scsi remove-single-device 0 1 2 3' > /proc/scsi/scsi" with
* "0 1 2 3" replaced by the Host, Channel, Id, and Lun.

Quoted 6 lines

The proc interface is deprecated; this can all be done via sysfs today, e.g.:
echo 1 > /sys/block/sdc/device/delete
Is equivalent to issuing scsi remove-single-device to proc.

yes, but the result is the same. It will do some IO first which will cause this command hang.

Thanks,
Tao

Quoted 10 lines

Regards,
Bryn.

Signature · 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Bryn M. Reeves

2013-02-01 10:07 UTC #5 lore raw

Quoted 2 lines · Tao Ma

yes, but the result is the same. It will do some IO first which will cause this command hang.

You seem to have a problem with either the device/adapter or in the driver. The backtrace you posted shows that jbd2 (ext4) is still waiting on IO that's been submitted to an mpt2sas or mpt3sas adapter (I only know that because I recognise their log messages - you should try to include relevant details like this when seeking assistance).

The adapter/driver hasn't completed the IO and it looks like the SCSI layer is trying to abort it. Depending on the state of the driver and hardware your only option might be to reboot (or physically hot remove the device if your hardware allows it).

You don't mention the versions of the kernel and driver you're using - if the system is in production I would suggest contacting who ever normally provides support for the kernel and distribution that you are running.

Regards,
Bryn.

Tao Ma

author

2013-02-01 11:13 UTC #6 lore raw

Quoted 9 lines · Bryn M. Reeves

Quoted 2 lines · Tao Ma
yes, but the result is the same. It will do some IO first which will cause this command hang.
You seem to have a problem with either the device/adapter or in the driver. The backtrace you posted shows that jbd2 (ext4) is still waiting on IO that's been submitted to an mpt2sas or mpt3sas adapter (I only know that because I recognise their log messages - you should try to include relevant details like this when seeking assistance).

This should be a mpt2sas adapter
#lsmod|grep mpt

mptctl                 96789  0
mptbase                97052  1 mptctl
mpt2sas               164962  18
scsi_transport_sas     35232  3 isci,libsas,mpt2sas
raid_class              4746  1 mpt2sas

The system has 12 sata disks. What else do you need? I am willing to provide any details you want.

Quoted 5 lines

The adapter/driver hasn't completed the IO and it looks like the SCSI layer is trying to abort it. Depending on the state of the driver and hardware your only option might be to reboot (or physically hot remove the device if your hardware allows it).

OK, so let me describe the situation here. This is one of our storage system. So 12 2TB sata disk in one box, normally when one disk fails, we just want to remove it from the system by *software*, and then continue to use the 11 disks left. We have found that sometimes an unsuccessful umount or some actions against this disk can lead to some bad situation(Say some very high load because many processes are 'D'ed). So ideally if we can remove this device successfully, all the ios to this disk will fail and there will be no 'D' processes and the loadavg will also be low.

Quoted 5 lines

You don't mention the versions of the kernel and driver you're using - if the system is in production I would suggest contacting who ever normally provides support for the kernel and distribution that you are running.

We use CentOS6.2 and the kernel version is 2.6.32-220.23.1.

Thanks,
Tao

Bryn M. Reeves

2013-02-01 12:08 UTC #7 lore raw

Quoted 5 lines · Tao Ma

Quoted 4 lines
You don't mention the versions of the kernel and driver you're using - if the system is in production I would suggest contacting who ever normally provides support for the kernel and distribution that you are running.
We use CentOS6.2 and the kernel version is 2.6.32-220.23.1.

This is ancient, even by CentOS or RHEL standards. There are thousands of patches in more recent kernels (either at kernel.org or in the updates in CentOS repositories).

Nobody on linux-kernel or the other lists you copied is going to want to investigate problems on such an old kernel - you'll need to either reproduce with something current or seek assistance from the CentOS community (who will probably tell you to update your kernel first anyway).

Regards,
Bryn.

Jack Wang

2013-02-01 09:01 UTC #8 lore raw

In reply to Tao Ma

Thanks,
Tao
[Jack Wang]
Hi Tao,

Have you tried:
echo 1 > /sys/block/sdv/device/delete
echo "- - -" > /sys/class/scsi_host/host

another way is :
find out which phy the disk attached to and:
echo 1 > /sys/class/sas_phy/phy-x:x:x/link_reset

Jack

Tao Ma

author

2013-02-01 09:18 UTC #9 lore raw

Quoted 11 lines · Jack Wang

Hi All,
In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang forever and we will get something in /var/log/messages like below. It seems to me that the io sent to the scsi layer is never returned back with -EIO which is a little bit surprised for me(It should be a timeout somewhere, right?). We have tried echo "offline" > /sys/block/sdl/device/state, but it doesn't work. So is there any way for us to let the scsi device returns all the io requests back with EIO so that all the end_io can be called accordingly? Am I missing something here?

Quoted 7 lines

Thanks,
Tao
[Jack Wang]
Hi Tao,
Have you tried:
echo 1 > /sys/block/sdv/device/delete

It will do some IO first so it will hang doing IO.

Quoted 1 line

echo "- - -" > /sys/class/scsi_host/host

What do you mean for this line?

Quoted 4 lines

another way is :
find out which phy the disk attached to and:
echo 1 > /sys/class/sas_phy/phy-x:x:x/link_reset

sorry, I have done it, but there is no response.

Thanks,
Tao

Quoted 9 lines

Jack
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Jack Wang

2013-02-01 09:24 UTC #10 lore raw

Quoted 4 lines · Jack Wang

Hi All,
In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang forever and we will get something in /var/log/messages like

below.

Quoted 6 lines

It seems to me that the io sent to the scsi layer is never returned back with -EIO which is a little bit surprised for me(It should be a timeout somewhere, right?). We have tried echo "offline" > /sys/block/sdl/device/state, but it doesn't work. So is there any way for us to let the scsi device returns all the io requests back with EIO so that all the end_io can be called accordingly? Am I missing

something here?

Quoted 8 lines

Thanks,
Tao
[Jack Wang]
Hi Tao,
Have you tried:
echo 1 > /sys/block/sdv/device/delete

It will do some IO first so it will hang doing IO.

Quoted 1 line

echo "- - -" > /sys/class/scsi_host/host

What do you mean for this line?

[Jack Wang] Sorry I mean to let the driver rescan to get the disk back. The line should be :
echo "- - -" > /sys/class/scsi_host/hostx/scan.

Per above delete does not work , so no need to run this.

Quoted 4 lines

another way is :
find out which phy the disk attached to and:
echo 1 > /sys/class/sas_phy/phy-x:x:x/link_reset

sorry, I have done it, but there is no response.

[Jack Wang]
What about
echo 1 > /sys/class/sas_phy/phy-x:x:x/hard_reset

?
Thanks,
Tao

Quoted 9 lines

Jack
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Tao Ma

author

2013-02-01 09:48 UTC #11 lore raw

Quoted 32 lines · Jack Wang

Quoted 4 lines · Jack Wang
Hi All,
In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang forever and we will get something in /var/log/messages like
below.
Quoted 6 lines
It seems to me that the io sent to the scsi layer is never returned back with -EIO which is a little bit surprised for me(It should be a timeout somewhere, right?). We have tried echo "offline" > /sys/block/sdl/device/state, but it doesn't work. So is there any way for us to let the scsi device returns all the io requests back with EIO so that all the end_io can be called accordingly? Am I missing
something here?
Quoted 8 lines
Thanks,
Tao
[Jack Wang]
Hi Tao,
Have you tried:
echo 1 > /sys/block/sdv/device/delete
It will do some IO first so it will hang doing IO.
Quoted 1 line
echo "- - -" > /sys/class/scsi_host/host
What do you mean for this line?
[Jack Wang] Sorry I mean to let the driver rescan to get the disk back. The line should be :
echo "- - -" > /sys/class/scsi_host/hostx/scan.
Per above delete does not work , so no need to run this.
Quoted 1 line

Quoted 8 lines

Quoted 3 lines
another way is :
find out which phy the disk attached to and:
echo 1 > /sys/class/sas_phy/phy-x:x:x/link_reset
sorry, I have done it, but there is no response.
[Jack Wang]
What about
echo 1 > /sys/class/sas_phy/phy-x:x:x/hard_reset

sorry, no response either.

Thanks,
Tao

Quoted 25 lines

?
Thanks,
Tao
Quoted 9 lines
Jack
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hillf Danton

2013-02-01 14:41 UTC #12 lore raw

In reply to Tao Ma

Quoted 43 lines · Tao Ma

Hi All,
In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang forever and we will get something in /var/log/messages like below. It seems to me that the io sent to the scsi layer is never returned back with -EIO which is a little bit surprised for me(It should be a timeout somewhere, right?). We have tried echo "offline" > /sys/block/sdl/device/state, but it doesn't work. So is there any way for us to let the scsi device returns all the io requests back with EIO so that all the end_io can be called accordingly? Am I missing something here?
Thanks,
Tao
sd 0:0:11:0: attempting task abort! scmd(ffff88180e900580) sd 0:0:11:0: [sdl] CDB: Write(10): 2a 00 0d ca e0 3f 00 04 00 00 target0:0:11: handle(0x0015), sas_address(0x500e004aaaaaaa0b), phy(11) target0:0:11: enclosure_logical_id(0x500e004aaaaaaa00), slot(11)
INFO: task jbd2/sdl1-8:4629 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/sdl1-8 D 0000000000000000 0 4629 2 0x00000000
ffff88180aa79ae0 0000000000000046 ffff88180aa79aa8 0000000000000000 ffff88007ce0fe40 0000000000015f40 ffff8818102c0638 ffff8818102c0080 ffff880a9184a100 ffff8818102c0638 0000000105006028 0000000100000000
Call Trace:
 [<ffffffff81236a15>] ? cpumask_next_and+0x25/0x40
 [<ffffffff810122b6>] ? read_tsc+0x16/0x40
 [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810122b6>] ? read_tsc+0x16/0x40
 [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff814a8a53>] io_schedule+0x73/0xc0
 [<ffffffff811036a8>] sync_page+0x38/0x50
 [<ffffffff814a927e>] __wait_on_bit+0x5e/0x90
 [<ffffffff81103670>] ? sync_page+0x0/0x50
 [<ffffffff81103845>] wait_on_page_bit+0x75/0x80
 [<ffffffff81089320>] ? wake_bit_function+0x0/0x40
 [<ffffffff811197c7>] ? pagevec_lookup_tag+0x27/0x40
 [<ffffffff81118b55>] write_cache_pages+0x1d5/0x440
 [<ffffffff811172f0>] ? __writepage+0x0/0x40
 [<ffffffff81118de4>] generic_writepages+0x24/0x30
 [<ffffffffa02dc719>] jbd2_journal_commit_transaction+0x3e9/0x1490 [jbd2]

Quoted 9 lines

 [<ffffffff81074299>] ? try_to_del_timer_sync+0x49/0xe0
 [<ffffffffa02e2734>] kjournald2+0xb4/0x220 [jbd2]
 [<ffffffff810892e0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02e2680>] ? kjournald2+0x0/0x220 [jbd2]
 [<ffffffff81089166>] kthread+0x96/0xa0
 [<ffffffff8100c08a>] child_rip+0xa/0x20
 [<ffffffff810890d0>] ? kthread+0x0/0xa0
 [<ffffffff8100c080>] ? child_rip+0x0/0x20

Can you try upstream?

taco

2013-10-16 16:22 UTC #13 lore raw

In reply to Tao Ma

Quoted 12 lines · Tao Ma

Hi All,
In our product system, we have several sata disks attached to one
machine. So when one of the disk fails, the jbd2(yes, we use ext4) will hang forever and we will get something in /var/log/messages like below. It seems to me that the io sent to the scsi layer is never returned back with -EIO which is a little bit surprised for me(It should be a timeout somewhere, right?). We have tried echo "offline" > /sys/block/sdl/device/state, but it doesn't work. So is there any way for us to let the scsi device returns all the io requests back with EIO so that all the end_io can be called accordingly? Am I missing something here?

Quoted 5 lines

Thanks,
Tao
sd 0:0:11:0: attempting task abort! scmd(ffff88180e900580)

It seems that IO timeout cause HBA's driver to abort scmd, the aborted IO came back with scmd->result = DID_RESET << 16; with this result code the Middle layer of scsi will retry this IO. IO timeout again due to Bad disk so, this IO loop forever and never come back.

might it is a bug of mpt2sas driver.

Quoted 39 lines

sd 0:0:11:0: [sdl] CDB: Write(10): 2a 00 0d ca e0 3f 00 04 00 00 target0:0:11: handle(0x0015), sas_address(0x500e004aaaaaaa0b), phy(11) target0:0:11: enclosure_logical_id(0x500e004aaaaaaa00), slot(11)
INFO: task jbd2/sdl1-8:4629 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/sdl1-8 D 0000000000000000 0 4629 2 0x00000000
ffff88180aa79ae0 0000000000000046 ffff88180aa79aa8 0000000000000000 ffff88007ce0fe40 0000000000015f40 ffff8818102c0638 ffff8818102c0080 ffff880a9184a100 ffff8818102c0638 0000000105006028 0000000100000000
Call Trace:
 [<ffffffff81236a15>] ? cpumask_next_and+0x25/0x40
 [<ffffffff810122b6>] ? read_tsc+0x16/0x40
 [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810122b6>] ? read_tsc+0x16/0x40
 [<ffffffff81093cd9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff814a8a53>] io_schedule+0x73/0xc0
 [<ffffffff811036a8>] sync_page+0x38/0x50
 [<ffffffff814a927e>] __wait_on_bit+0x5e/0x90
 [<ffffffff81103670>] ? sync_page+0x0/0x50
 [<ffffffff81103845>] wait_on_page_bit+0x75/0x80
 [<ffffffff81089320>] ? wake_bit_function+0x0/0x40
 [<ffffffff811197c7>] ? pagevec_lookup_tag+0x27/0x40
 [<ffffffff81118b55>] write_cache_pages+0x1d5/0x440
 [<ffffffff811172f0>] ? __writepage+0x0/0x40
 [<ffffffff81118de4>] generic_writepages+0x24/0x30
 [<ffffffffa02dc719>] jbd2_journal_commit_transaction+0x3e9/0x1490 [jbd2]
 [<ffffffff81074299>] ? try_to_del_timer_sync+0x49/0xe0
 [<ffffffffa02e2734>] kjournald2+0xb4/0x220 [jbd2]
 [<ffffffff810892e0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02e2680>] ? kjournald2+0x0/0x220 [jbd2]
 [<ffffffff81089166>] kthread+0x96/0xa0
 [<ffffffff8100c08a>] child_rip+0xa/0x20
 [<ffffffff810890d0>] ? kthread+0x0/0xa0
 [<ffffffff8100c080>] ? child_rip+0x0/0x20
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html