Skip to contentSkip to conversation

Issue with qla2xxx_probe_one

DISCUSSION Alan D. Brunelle 4 participants 6 messages Apr 29, 2008 – Apr 30, 2008 SCSI

lore
Share & tools
lore
#0 lore raw

I /think/ that there is an issue with this routine /if/ the firmware images are not loaded properly - on a 16-way ia64 box I am starting to see this with an up-stream kernel (Jens Axboe's origin/io-cpu-affinity branch). In any event, it looks to me that :

        if (qla2x00_initialize_adapter(ha)) {
                qla_printk(KERN_WARNING, ha,
                    "Failed to initialize adapter\n");

                DEBUG2(printk("scsi(%ld): Failed to initialize adapter - "
                    "Adapter flags %x.\n",
                    ha->host_no, ha->device_flags));

                ret = -ENODEV;
                goto probe_failed;
        }

skips around:

        ret = scsi_add_host(host, &pdev->dev);

which is needed to properly initialize the freelist (via:
scsi_setup_command_freelist).

When qla2xxx_probe_one ends up calling scsi_host_put in this error path it eventually gets to scsi_destroy_command_freelist and we get the error below.

There's a lot of code here to go through for me, but perhaps someone out there has a quicker way of figuring out what is really wrong and/or being able to provide a fix.

BTW: I have had the issue with firmware for a while, just never gotten around to fixing it - typically just:

modprobe -r qla2xxx
modprobe qla2xxx

has gotten it to work in the past, but now with the NaT issue I can't unload and reload the module.

Alan D. Brunelle
HP

=========================================================

qla2xxx 0000:2a:01.0: Found an ISP2312, irq 100, iobase 0xc0000f4010040000 qla2xxx 0000:2a:01.0: Configuring PCI space...
qla2xxx 0000:2a:01.0: Configure NVRAM parameters...
qla2xxx 0000:2a:01.0: Verifying loaded RISC code...
qla2xxx 0000:2a:01.0: Firmware image unavailable.
qla2xxx 0000:2a:01.0: Firmware images can be retrieved from:
ftp://ftp.qlogic.com/outgoing/linux/firmware/.
qla2xxx 0000:2a:01.0: Failed to initialize adapter insmod[1828]: NaT consumption 17179869216 [1]

Modules linked in: qla2xxx(+) firmware_class ehci_hcd ohci_hcd uhci_hcd

usbcore

Pid: 1828, CPU 0, comm: insmod
psr : 0000101008526010 ifs : 8000000000000206 ip : [<a00000010054b1b0>]
Not tainted (2.6.25io-cpu-affinity)
ip is at scsi_destroy_command_freelist+0x10/0xe0
unat: 0000000000000000 pfs : 0000000000000307 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr : 0000000000005559
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70433f csd : 0000000000000000 ssd : 0000000000000000
b0 : a00000010054c820 b6 : a000000100012bc0 b7 : a000000100009a60 f6 : 000000000000000000000 f7 : 1003e0000000000400000
f8 : 1003e0000000028000000 f9 : 1003e0000002d98c5cede
f10 : 1003e1fdee852b0000000 f11 : 1003e0000000000000007
r1 : a000000100c26610 r2 : e000076381418030 r3 : e000076381418150 r8 : 0000000000000000 r9 : a0000001009fb698 r10 : 00000000000001d0 r11 : a0000001009fb680 r12 : e000070384057d10 r13 : e000070384050000 r14 : 0000000000000000 r15 : e000070384050c10 r16 : a0000001009fb6a8 r17 : 0000000000000000 r18 : 0000000000000008 r19 : 0009804c8a70433f r20 : e000070384057d10 r21 : a0000001009fb688 r22 : 0000000000000001 r23 : a0000001009fb690 r24 : 0000000000004000 r25 : 0000000000004000 r26 : a0000001009fb698 r27 : 0000000000000000 r28 : 0000000000006659 r29 : e000070384050c10 r30 : e000070384050c10 r31 : a0000001009fb698

Call Trace:
 [<a000000100012020>] show_stack+0x40/0xa0
                                sp=e000070384057760 bsp=e000070384051380
 [<a000000100012930>] show_regs+0x850/0x8a0
                                sp=e000070384057930 bsp=e000070384051328
 [<a000000100035a70>] die+0x1b0/0x2c0
                                sp=e000070384057930 bsp=e0000703840512e0
 [<a000000100035bd0>] die_if_kernel+0x50/0x80
                                sp=e000070384057930 bsp=e0000703840512b0
 [<a00000010071be20>] __kprobes_text_start+0x11a0/0x12c0
                                sp=e000070384057930 bsp=e000070384051258
 [<a00000010000a260>] ia64_leave_kernel+0x0/0x270
                                sp=e000070384057b40 bsp=e000070384051258
 [<a00000010054b1b0>] scsi_destroy_command_freelist+0x10/0xe0
                                sp=e000070384057d10 bsp=e000070384051228
 [<a00000010054c820>] scsi_host_dev_release+0x140/0x1e0
                                sp=e000070384057d10 bsp=e0000703840511f0
 [<a0000001004be5a0>] device_release+0xc0/0x160
                                sp=e000070384057d10 bsp=e0000703840511d0
 [<a0000001003e7c30>] kobject_release+0xd0/0x120
                                sp=e000070384057d10 bsp=e000070384051198
 [<a0000001003e9b50>] kref_put+0xb0/0xe0
                                sp=e000070384057d10 bsp=e000070384051170
 [<a0000001003e7950>] kobject_put+0x90/0xc0
                                sp=e000070384057d10 bsp=e000070384051150
 [<a0000001004be970>] put_device+0x30/0x60
                                sp=e000070384057d10 bsp=e000070384051130
 [<a00000010054c620>] scsi_host_put+0x20/0x40
                                sp=e000070384057d10 bsp=e000070384051110
 [<a000000207aa7ff0>] qla2x00_probe_one+0x2170/0x4110 [qla2xxx]
                                sp=e000070384057d10 bsp=e0000703840510a0
 [<a000000100408750>] pci_device_probe+0x170/0x240
                                sp=e000070384057d90 bsp=e000070384051058
 [<a0000001004c4dc0>] driver_probe_device+0x220/0x360
                                sp=e000070384057da0 bsp=e000070384051020
 [<a0000001004c4f80>] __driver_attach+0x80/0xe0
                                sp=e000070384057da0 bsp=e000070384050fe0
 [<a0000001004c3870>] bus_for_each_dev+0x90/0x100
                                sp=e000070384057da0 bsp=e000070384050fa8
 [<a0000001004c4960>] driver_attach+0x40/0x60
                                sp=e000070384057dc0 bsp=e000070384050f88
 [<a0000001004c4300>] bus_add_driver+0x160/0x4a0
                                sp=e000070384057dc0 bsp=e000070384050f40
 [<a0000001004c5580>] driver_register+0x140/0x2a0
                                sp=e000070384057dc0 bsp=e000070384050ef8
 [<a000000100408cf0>] __pci_register_driver+0xb0/0x140
                                sp=e000070384057dc0 bsp=e000070384050ec0
 [<a000000207968260>] qla2x00_module_init+0x260/0x400 [qla2xxx]
                                sp=e000070384057dd0 bsp=e000070384050e88
 [<a0000001000db000>] sys_init_module+0x35a0/0x38c0
                                sp=e000070384057dd0 bsp=e000070384050d08
 [<a00000010000a0c0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e000070384057e30 bsp=e000070384050d08
 [<a000000000010720>] __kernel_syscall_via_break+0x0/0x20
                                sp=e000070384058000 bsp=e000070384050d08
#1 lore raw
Quoted 17 lines · Alan D. Brunelle

I /think/ that there is an issue with this routine /if/ the firmware images are not loaded properly - on a 16-way ia64 box I am starting to see this with an up-stream kernel (Jens Axboe's origin/io-cpu-affinity branch). In any event, it looks to me that :

        if (qla2x00_initialize_adapter(ha)) {
                qla_printk(KERN_WARNING, ha,
                    "Failed to initialize adapter\n");

                DEBUG2(printk("scsi(%ld): Failed to initialize adapter - "
                    "Adapter flags %x.\n",
                    ha->host_no, ha->device_flags));

                ret = -ENODEV;
                goto probe_failed;
        }
Quoted 6 lines

skips around:

        ret = scsi_add_host(host, &pdev->dev);

which is needed to properly initialize the freelist (via:
scsi_setup_command_freelist).

Wasn't something like this posted recently to linux-scsi:

http://lkml.org/lkml/2008/4/27/333

this is sitting in scsi-misc-2.6.git:

[SCSI] bug fix for free list handling
http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=a79cbe1aa5dd695f0ee012ecde1ff88b1192e326

which I gather will be pushed soon...

Quoted 3 lines

When qla2xxx_probe_one ends up calling scsi_host_put in this error path it eventually gets to scsi_destroy_command_freelist and we get the error below.

#2 lore raw
Quoted 7 lines · Andrew Vasquez
Quoted 5 lines · Alan D. Brunelle

I /think/ that there is an issue with this routine /if/ the firmware images are not loaded properly - on a 16-way ia64 box I am starting to see this with an up-stream kernel (Jens Axboe's origin/io-cpu-affinity branch). In any event, it looks to me that :

Quoted 5 lines
Quoted 5 lines
        if (qla2x00_initialize_adapter(ha)) {
                qla_printk(KERN_WARNING, ha,
                    "Failed to initialize adapter\n");

                DEBUG2(printk("scsi(%ld): Failed to initialize

adapter - "

Quoted 14 lines
Quoted 7 lines
                    "Adapter flags %x.\n",
                    ha->host_no, ha->device_flags));

                ret = -ENODEV;
                goto probe_failed;
        }
Quoted 6 lines

skips around:

        ret = scsi_add_host(host, &pdev->dev);

which is needed to properly initialize the freelist (via:
scsi_setup_command_freelist).

Quoted 8 lines

Wasn't something like this posted recently to linux-scsi:

http://lkml.org/lkml/2008/4/27/333

this is sitting in scsi-misc-2.6.git:

[SCSI] bug fix for free list handling

http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=a79cbe1aa5dd695f0ee012ecde1ff88b1192e326

Quoted 2 lines

which I gather will be pushed soon...

My apologies for not having seeing that.

But after looking at it, doesn't it still have a hole?

o scsi_setup_command_freelist initializes the free_list list.

o It then invokes scsi_get_host_cmd_pool, if this fails there is no need to invoke scsi_put_host_cmd_pool (it wasn't gotten).

o If scsi_get_host_cmd_pool succeeds but scsi_pool_alloc_command fails, it will (correctly) invoke scsi_put_host_cmd_pool.

However, if either of scsi_get_host_cmd_pool or scsi_put_host_cmd_pool happens to fail, we'll end up in scsi_destroy_command_freelist - and since the free_list was initialized, the while loop will be bypassed, but scsi_put_host_cmd_pool will be invoked an extra time. And this is badness, right?

Wouldn't the attached patch [boot tested on my previously failing system] be correct (and perhaps cleaner - you're not looking at the innards of the list data structure to determine things)?

Alan

Quoted 1 line

From 344f31749fe26fe8b56fcd6ff3f3902cedb8144c Mon Sep 17 00:00:00 2001

From: Alan D. Brunelle <[email protected]>
Date: Tue, 29 Apr 2008 15:46:36 -0400
Subject: [PATCH] Ensure proper handling of the scsi free_list handling upon errors

Only release resources in scsi_destroy_command_freelist that have been correctly initialized.

---
 drivers/scsi/scsi.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
drivers/scsi/scsi.c +8 -0
index 12d69d7..749c9c7 100644
@@ -469,6 +469,7 @@ int scsi_setup_command_freelist(struct Scsi_Host *shost)
 	cmd = scsi_pool_alloc_command(shost->cmd_pool, gfp_mask);
 	if (!cmd) {
 		scsi_put_host_cmd_pool(gfp_mask);
+		shost->cmd_pool = NULL;
 		return -ENOMEM;
 	}
 	list_add(&cmd->list, &shost->free_list);
@@ -481,6 +482,13 @@ int scsi_setup_command_freelist(struct Scsi_Host *shost)
  */
 void scsi_destroy_command_freelist(struct Scsi_Host *shost)
 {
+	/*
+	 * If cmd_pool is NULL the free list was not initialized, so
+	 * do not attempt to release resources.
+	 */
+	if (!shost->cmd_pool)
+		return;
+
 	while (!list_empty(&shost->free_list)) {
 		struct scsi_cmnd *cmd;
 
Signature · 1 line
1.5.4.3
#3 lore raw
Quoted 44 lines · Alan D. Brunelle
Quoted 7 lines · Andrew Vasquez
Quoted 5 lines · Alan D. Brunelle

I /think/ that there is an issue with this routine /if/ the firmware images are not loaded properly - on a 16-way ia64 box I am starting to see this with an up-stream kernel (Jens Axboe's origin/io-cpu-affinity branch). In any event, it looks to me that :

Quoted 5 lines
Quoted 5 lines
        if (qla2x00_initialize_adapter(ha)) {
                qla_printk(KERN_WARNING, ha,
                    "Failed to initialize adapter\n");

                DEBUG2(printk("scsi(%ld): Failed to initialize

adapter - "

Quoted 14 lines
Quoted 7 lines
                    "Adapter flags %x.\n",
                    ha->host_no, ha->device_flags));

                ret = -ENODEV;
                goto probe_failed;
        }
Quoted 6 lines

skips around:

        ret = scsi_add_host(host, &pdev->dev);

which is needed to properly initialize the freelist (via:
scsi_setup_command_freelist).

Quoted 8 lines

Wasn't something like this posted recently to linux-scsi:

http://lkml.org/lkml/2008/4/27/333

this is sitting in scsi-misc-2.6.git:

[SCSI] bug fix for free list handling

http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=a79cbe1aa5dd695f0ee012ecde1ff88b1192e326

Quoted 2 lines

which I gather will be pushed soon...

My apologies for not having seeing that.

But after looking at it, doesn't it still have a hole?

Quoted 7 lines

o scsi_setup_command_freelist initializes the free_list list.

o It then invokes scsi_get_host_cmd_pool, if this fails there is no need to invoke scsi_put_host_cmd_pool (it wasn't gotten).

o If scsi_get_host_cmd_pool succeeds but scsi_pool_alloc_command fails, it will (correctly) invoke scsi_put_host_cmd_pool.

...
<snip>

Hmm, I'll defer to James B. on that...

--
av

#4 lore raw
In reply to Alan D. Brunelle
Quoted 54 lines · On Tue, 29 Apr 2008 16:12:51 -0400 "Alan D. Brunelle" <[email protected]>…

This is a multi-part message in MIME format.
--------------050404010600090601000605
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Quoted 7 lines · Andrew Vasquez
Quoted 5 lines · Alan D. Brunelle

I /think/ that there is an issue with this routine /if/ the firmware images are not loaded properly - on a 16-way ia64 box I am starting to see this with an up-stream kernel (Jens Axboe's origin/io-cpu-affinity branch). In any event, it looks to me that :

Quoted 5 lines
Quoted 5 lines
        if (qla2x00_initialize_adapter(ha)) {
                qla_printk(KERN_WARNING, ha,
                    "Failed to initialize adapter\n");

                DEBUG2(printk("scsi(%ld): Failed to initialize

adapter - "

Quoted 14 lines
Quoted 7 lines
                    "Adapter flags %x.\n",
                    ha->host_no, ha->device_flags));

                ret = -ENODEV;
                goto probe_failed;
        }
Quoted 6 lines

skips around:

        ret = scsi_add_host(host, &pdev->dev);

which is needed to properly initialize the freelist (via:
scsi_setup_command_freelist).

Quoted 8 lines

Wasn't something like this posted recently to linux-scsi:

http://lkml.org/lkml/2008/4/27/333

this is sitting in scsi-misc-2.6.git:

[SCSI] bug fix for free list handling

http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=a79cbe1aa5dd695f0ee012ecde1ff88b1192e326

Quoted 2 lines

which I gather will be pushed soon...

My apologies for not having seeing that.

But after looking at it, doesn't it still have a hole?

o scsi_setup_command_freelist initializes the free_list list.

o It then invokes scsi_get_host_cmd_pool, if this fails there is no need to invoke scsi_put_host_cmd_pool (it wasn't gotten).

Quoted 8 lines

o If scsi_get_host_cmd_pool succeeds but scsi_pool_alloc_command fails, it will (correctly) invoke scsi_put_host_cmd_pool.

However, if either of scsi_get_host_cmd_pool or scsi_put_host_cmd_pool happens to fail, we'll end up in scsi_destroy_command_freelist - and since the free_list was initialized, the while loop will be bypassed, but scsi_put_host_cmd_pool will be invoked an extra time. And this is badness, right?

scsi_put_host_cmd_pool doesn't fail but I think that you are right. If scsi_get_host_cmd_pool or scsi_pool_alloc_command in scsi_setup_command_freelist fails, we hit the problem.

Quoted 3 lines

Wouldn't the attached patch [boot tested on my previously failing system] be correct (and perhaps cleaner - you're not looking at the innards of the list data structure to determine things)?

Looks correct to me.

maintainer
#5 lore raw
In reply to Alan D. Brunelle
Quoted 58 lines · Alan D. Brunelle
Quoted 7 lines · Andrew Vasquez
Quoted 5 lines · Alan D. Brunelle

I /think/ that there is an issue with this routine /if/ the firmware images are not loaded properly - on a 16-way ia64 box I am starting to see this with an up-stream kernel (Jens Axboe's origin/io-cpu-affinity branch). In any event, it looks to me that :

Quoted 5 lines
Quoted 5 lines
        if (qla2x00_initialize_adapter(ha)) {
                qla_printk(KERN_WARNING, ha,
                    "Failed to initialize adapter\n");

                DEBUG2(printk("scsi(%ld): Failed to initialize

adapter - "

Quoted 14 lines
Quoted 7 lines
                    "Adapter flags %x.\n",
                    ha->host_no, ha->device_flags));

                ret = -ENODEV;
                goto probe_failed;
        }
Quoted 6 lines

skips around:

        ret = scsi_add_host(host, &pdev->dev);

which is needed to properly initialize the freelist (via:
scsi_setup_command_freelist).

Quoted 8 lines

Wasn't something like this posted recently to linux-scsi:

http://lkml.org/lkml/2008/4/27/333

this is sitting in scsi-misc-2.6.git:

[SCSI] bug fix for free list handling

http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=a79cbe1aa5dd695f0ee012ecde1ff88b1192e326

Quoted 2 lines

which I gather will be pushed soon...

My apologies for not having seeing that.

But after looking at it, doesn't it still have a hole?

o scsi_setup_command_freelist initializes the free_list list.

o It then invokes scsi_get_host_cmd_pool, if this fails there is no need to invoke scsi_put_host_cmd_pool (it wasn't gotten).

o If scsi_get_host_cmd_pool succeeds but scsi_pool_alloc_command fails, it will (correctly) invoke scsi_put_host_cmd_pool.

However, if either of scsi_get_host_cmd_pool or scsi_put_host_cmd_pool happens to fail, we'll end up in scsi_destroy_command_freelist - and since the free_list was initialized, the while loop will be bypassed, but scsi_put_host_cmd_pool will be invoked an extra time. And this is badness, right?

Quoted 3 lines

Wouldn't the attached patch [boot tested on my previously failing system] be correct (and perhaps cleaner - you're not looking at the innards of the list data structure to determine things)?

Yes, that looks like a better fix. I tidied up your change log, because it's helpful to identify the original problem commit, but otherwise applied it unchanged.

Thanks,

James