Skip to contentSkip to conversation

New: iscsi repeatedly times out

Bug 12119 [email protected] 2 participants 9 messages Nov 29, 2008 – Dec 1, 2008

Subsystems (2)
SCSI iSCSI
lore
Share & tools
lore

[email protected]

author
#0 lore raw

http://bugzilla.kernel.org/show_bug.cgi?id=12119

           Summary: iscsi repeatedly times out
           Product: IO/Storage
           Version: 2.5
     KernelVersion: 2.6.27.7
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Latest working kernel version:2.6.24.4-64
Earliest failing kernel version:2.6.27.5
Distribution:fc8
Hardware Environment:diskless i386, swap on iscsi device Software Environment:open-iscsi v 6.2
Problem Description:As soon as the iscsi device is activated, the following messages are logged:

Nov 29 21:24:51 dhcp-0-20-34-211-135-63 iscsid: Nop-out timedout after 15
seconds on connection 1:0 state (3). Dropping session.
Nov 29 21:24:55 dhcp-0-20-34-211-135-63 iscsid: connection1:0 is operational

after recovery (2 attempts)

These repeat precisely every 30 seconds essentially forever, seemingly independent of activity on the iscsi device.

IO to the device seems to actually work OK (eg dd if=/dev/sdb of=dev/null bs=64k count=100)

The messages are obviously from the iscsid userland daemon, but booting with an older kernel 2.6.24.4-64 does not cause these messages.

Steps to reproduce:

Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
author
#1 lore raw

[Bug 12119] iscsi repeatedly times out

http://bugzilla.kernel.org/show_bug.cgi?id=12119

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------

         Regression|0                           |1
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
#3 lore raw
Quoted 7 lines · Mike Christie
Quoted 2 lines · [email protected] wrote:

Hardware Environment:diskless i386, swap on iscsi device Software Environment:open-iscsi v 6.2

What is the rest of this version string? With fedora run

rpm -q iscsi-initiator-utils

Ignore that request.

Try the attached patch made over 2.6.27 which was the base for the kernel I thought you were using. This should also work with 2.6.28-rc*.

Oh yeah for 2.6.27 are you using a fedora kernel? If you are building your own kernel and you are using F8 then make sure you compile the kernel with

CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y

because a lot of tools in F8 may not work right without it. The older iscsi tools in there will not and you will run into lots of other problems, because there were some sysfs changes which affected iscsi and we were not ready for them back then. Newer tools in F9 and F10 and open-iscsi.org will handle any sysfs layout so if you can you might want to grab the iscsi-initiator-utils rpm from the F9 repository.

drivers/scsi/libiscsi.c +5 -6
@@ -494,12 +494,6 @@ __iscsi_conn_send_pdu(struct iscsi_conn 
 		if (!__kfifo_get(session->cmdpool.queue,
 				 (void*)&task, sizeof(void*)))
 			return NULL;
-
-		if ((hdr->opcode == (ISCSI_OP_NOOP_OUT | ISCSI_OP_IMMEDIATE)) &&
-		     hdr->ttt == RESERVED_ITT) {
-			conn->ping_task = task;
-			conn->last_ping = jiffies;
-		}
 	}
 	/*
 	 * released in complete pdu for task we expect a response for, and

@@ -674,6 +668,11 @@ static void iscsi_send_nopout(struct isc
 	task = __iscsi_conn_send_pdu(conn, (struct iscsi_hdr *)&hdr, NULL, 0);
 	if (!task)
 		iscsi_conn_printk(KERN_ERR, conn, "Could not send nopout\n");
+	else if (!rhdr) {
+		/* only track our nops */
+		conn->ping_task = task;
+		conn->last_ping = jiffies;
+	}
 }
 
 static int iscsi_handle_reject(struct iscsi_conn *conn, struct iscsi_hdr *hdr,
author
#4 lore raw

[Bug 12119] iscsi repeatedly times out

In reply to [email protected]

http://bugzilla.kernel.org/show_bug.cgi?id=12119

------- Comment #1 from [email protected] 2008-11-30 17:39 -------
Reply-To: [email protected]

Quoted 2 lines · [email protected] wrote:

Hardware Environment:diskless i386, swap on iscsi device Software Environment:open-iscsi v 6.2

What is the rest of this version string? With fedora run

rpm -q iscsi-initiator-utils

Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
author
#5 lore raw

[Bug 12119] iscsi repeatedly times out

In reply to [email protected]

http://bugzilla.kernel.org/show_bug.cgi?id=12119

------- Comment #2 from [email protected] 2008-11-30 20:11 -------
Reply-To: [email protected]

Quoted 7 lines · Mike Christie
Quoted 2 lines · [email protected] wrote:

Hardware Environment:diskless i386, swap on iscsi device Software Environment:open-iscsi v 6.2

What is the rest of this version string? With fedora run

rpm -q iscsi-initiator-utils

Ignore that request.

Try the attached patch made over 2.6.27 which was the base for the kernel I thought you were using. This should also work with 2.6.28-rc*.

Oh yeah for 2.6.27 are you using a fedora kernel? If you are building your own kernel and you are using F8 then make sure you compile the kernel with

CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y

because a lot of tools in F8 may not work right without it. The older iscsi tools in there will not and you will run into lots of other problems, because there were some sysfs changes which affected iscsi and we were not ready for them back then. Newer tools in F9 and F10 and open-iscsi.org will handle any sysfs layout so if you can you might want to grab the iscsi-initiator-utils rpm from the F9 repository.
--- linux-2.6.27-rc5.work/drivers/scsi/libiscsi.c 2008-08-28
17:52:02.000000000 -0500
+++ linux-2.6.27-rc5/drivers/scsi/libiscsi.c 2008-11-30 21:46:55.000000000
-0600
@@ -494,12 +494,6 @@ __iscsi_conn_send_pdu(struct iscsi_conn

                if (!__kfifo_get(session->cmdpool.queue,
                                 (void*)&task, sizeof(void*)))
                        return NULL;

-
- if ((hdr->opcode == (ISCSI_OP_NOOP_OUT | ISCSI_OP_IMMEDIATE))
&&
- hdr->ttt == RESERVED_ITT) {
- conn->ping_task = task;
- conn->last_ping = jiffies;
- }

        }
        /*
         * released in complete pdu for task we expect a response for, and
@@ -674,6 +668,11 @@ static void iscsi_send_nopout(struct isc
        task = __iscsi_conn_send_pdu(conn, (struct iscsi_hdr *)&hdr, NULL, 0);
        if (!task)
                iscsi_conn_printk(KERN_ERR, conn, "Could not send nopout\n");
+       else if (!rhdr) {
+               /* only track our nops */
+               conn->ping_task = task;
+               conn->last_ping = jiffies;

+ }
}

static int iscsi_handle_reject(struct iscsi_conn *conn, struct iscsi_hdr *hdr,

Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
author
#6 lore raw

[Bug 12119] iscsi repeatedly times out

In reply to [email protected]

http://bugzilla.kernel.org/show_bug.cgi?id=12119

------- Comment #3 from [email protected] 2008-12-01 05:39 ------- Thanks. The patch wouldn't apply but I applied it by hand. It works fine.

I didn't know about the CONFIG_SYSFS_DEPRECIATED options. Unfortunately I can't upgrade to FC9 or FC10 because of a kernel nfs bug(s) [11061,11154]. I'm not sure if I could run FC10 with an older kernel, but things like the changes to sysfs shows it is best not to assume so!

Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
#7 lore raw

Re: [Bug 12119] iscsi repeatedly times out

Quoted 8 lines · [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=12119

------- Comment #3 from [email protected] 2008-12-01 05:39 ------- Thanks. The patch wouldn't apply but I applied it by hand. It works fine.

Ok. I will submit a patch. Thanks for the report and testing.

Quoted 6 lines

I didn't know about the CONFIG_SYSFS_DEPRECIATED options. Unfortunately I can't upgrade to FC9 or FC10 because of a kernel nfs bug(s) [11061,11154]. I'm not sure if I could run FC10 with an older kernel, but things like the changes to sysfs shows it is best not to assume so!

I did not mean to upgrade to all F10 or F9 - just the iscsi package in there. The code across them is the same more or less. F9 and F8 were the same for iscsi, so if you use:

http://mirrors.kernel.org/fedora/updates/9/x86_64.newkey/iscsi-initiator-utils-6.2.0.870-1.0.fc9.x86_64.rpm

(for i386 or ppc then swap the x86_64 refs for your arch) on F8 then you should be ok.

author
#8 lore raw

[Bug 12119] iscsi repeatedly times out

In reply to [email protected]

http://bugzilla.kernel.org/show_bug.cgi?id=12119

------- Comment #4 from [email protected] 2008-12-01 08:20 -------
Reply-To: [email protected]

Quoted 8 lines · [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=12119

------- Comment #3 from [email protected] 2008-12-01 05:39 ------- Thanks. The patch wouldn't apply but I applied it by hand. It works fine.

Ok. I will submit a patch. Thanks for the report and testing.

Quoted 6 lines

I didn't know about the CONFIG_SYSFS_DEPRECIATED options. Unfortunately I can't upgrade to FC9 or FC10 because of a kernel nfs bug(s) [11061,11154]. I'm not sure if I could run FC10 with an older kernel, but things like the changes to sysfs shows it is best not to assume so!

I did not mean to upgrade to all F10 or F9 - just the iscsi package in there. The code across them is the same more or less. F9 and F8 were the same for iscsi, so if you use:

http://mirrors.kernel.org/fedora/updates/9/x86_64.newkey/iscsi-initiator-utils-6.2.0.870-1.0.fc9.x86_64.rpm

(for i386 or ppc then swap the x86_64 refs for your arch) on F8 then you should be ok.

Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.