New: iSCSI initiator lockup during login

https://bugzilla.kernel.org/show_bug.cgi?id=85751

            Bug ID: 85751
           Summary: iSCSI initiator lockup during login
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 3.16.4
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 152781
--> https://bugzilla.kernel.org/attachment.cgi?id=152781&action=edit
Output generated by echo w > /proc/sysrq

If I interpret the sysrq output correctly this lockup is caused by the thread with PID 15219 waiting inside async_synchronize_cookie_domain() while the thread with PID 15221 is waiting for the ihost mutex held by the first thread ?

You are receiving this mail because:
You are the assignee for the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=85751

Sagi Grimberg <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #1 from Sagi Grimberg <[email protected]> --- Hey Bart,

Any chance you provide on the scenario you used to reproduce this issue? did you simply login and got it?
is it a firm reproduce?

You are receiving this mail because:
You are the assignee for the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=85751

Bart Van Assche <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

            Summary|iSCSI initiator lockup      |iSCSI initiator lockup
                   |during login                |during logout

You are receiving this mail because:
You are the assignee for the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=85751

--- Comment #2 from Bart Van Assche <[email protected]> ---
This is not easy to reproduce and happened while testing a modification of iSCSI target software. The following commands were issued at the initiator side:
* iscsiadm -m discovery -p 192.168.1.124 -t st
* iscsiadm -m node -p 192.168.1.124 -l
(something went wrong at the target side and the iSCSI initiator tried to relogin)
* iscsiadm -m node -p 192.168.1.124 -u
Even after I powered off the virtual machine that was running the iSCSI target software that last logout command did not finish. The system load raised to a high value due to several kernel threads at the initiator side that got stuck in an uninterruptible wait. It is only after the logout command had been issued that I ran "echo w > /proc/sysrq-trigger".

You are receiving this mail because:
You are the assignee for the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=85751

Bart Van Assche <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

 Attachment #152781|Output generated by echo w  |Output generated by echo w
        description|> /proc/sysrq               |> /proc/sysrq-trigger

You are receiving this mail because:
You are the assignee for the bug.

Did this login command ever complete normally? Did you ctrl-c it or did you run the logout command below in another terminal?

The login command looks like it is stuck waiting for scan/setup related commands to complete:

 [<ffffffff811f54f5>] blk_execute_rq+0x75/0xf0
 [<ffffffff811f16fa>] ? blk_rq_bio_prep+0x6a/0x90
 [<ffffffff811f52b9>] ? blk_rq_map_kern+0xb9/0x160
 [<ffffffff811eef0a>] ? blk_get_request+0x7a/0xc0
 [<ffffffffa0008e3b>] scsi_execute+0xdb/0x160 [scsi_mod]
 [<ffffffffa000a52e>] scsi_execute_req_flags+0x8e/0x100 [scsi_mod]
 [<ffffffffa000c8ed>] scsi_probe_and_add_lun+0x20d/0xc40 [scsi_mod]
 [<ffffffffa0008e69>] ? scsi_execute+0x109/0x160 [scsi_mod]
 [<ffffffffa000db02>] __scsi_scan_target+0x522/0x6c0 [scsi_mod]
 [<ffffffff812e040d>] ? __pm_runtime_resume+0x4d/0x60
 [<ffffffffa000dd90>] scsi_scan_target+0xf0/0x110 [scsi_mod]
 [<ffffffffa0852d6f>] iscsi_user_scan_session.part.13+0x10f/0x150

[scsi_transport_iscsi]
[<ffffffffa0852db0>] ? iscsi_user_scan_session.part.13+0x150/0x150
[scsi_transport_iscsi]
[<ffffffffa0852dd1>] iscsi_user_scan_session+0x21/0x30
[scsi_transport_iscsi]

And so above it should have the ihost->mutex held while stuck.

But here it looks the logout command has been run and we have passed the part where we take the mutex in there and already started to delete the target.

 [<ffffffff812d41c9>] device_del+0x129/0x1c0
 [<ffffffffa0010005>] __scsi_remove_device+0xd5/0xe0 [scsi_mod]
 [<ffffffffa0010036>] scsi_remove_device+0x26/0x40 [scsi_mod]
 [<ffffffffa00101f0>] scsi_remove_target+0x170/0x230 [scsi_mod]
 [<ffffffffa0851f84>] __iscsi_unbind_session+0xc4/0x1a0

[scsi_transport_iscsi]

Before you ran the logout command, did the login/scan hang? Did we ever log back in?

Then did you run the logout command to try and clean things up.

Hello Mike,

Thanks for jumping in. I had indeed interrupted login via Ctrl-C because login took longer than expected.

Bart.

https://bugzilla.kernel.org/show_bug.cgi?id=85751

--- Comment #3 from Bart Van Assche <[email protected]> ---

Hello Mike,

Thanks for jumping in. I had indeed interrupted login via Ctrl-C because login took longer than expected.

Bart.

You are receiving this mail because:
You are the assignee for the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=85751

Jaden <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #4 from Jaden <[email protected]> ---
I also encountered the similar issue, but not in logout stage. If the links are down could also meet this issue occasionally. Below are my reproduce steps:

1.while :; do dd if=/dev/sdc of=/dev/null bs=1K count=1 iflag=direct; done 2.kill -SIGSTOP pidof iscsid
3.iptables -A OUTPUT -p tcp --dport 3260 -j DROP

I think it is cause by a status conflict between the waitting for lost IO request and iscsi device remove procedure. Any new thoughs?

You are receiving this mail because:
You are the assignee for the bug.