New: starfire causes kernel BUG when interface goes up

Bugme-new In review Andrew Morton 3 participants 11 messages Jan 26, 2010 1 message not in the archive

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author

2010-01-26 01:09 UTC #1 lore raw

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 30 lines · On Wed, 20 Jan 2010 04:29:20 GMT [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=15091
           Summary: starfire causes kernel BUG when interface goes up
           Product: Drivers
           Version: 2.5
    Kernel Version: 2.6.32
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: No
Created an attachment (id=24651)
--> (http://bugzilla.kernel.org/attachment.cgi?id=24651)
ls -l /dev (before crash)
I formerly used 2.6.20 and 2.6.24 with a couple of starfire 4 port ethernet cards. On 2.6.32 the interfaces don't start on boot and when I issue "ifconfig ethX up" (where X is a starfire port).
Sometimes the exception causes the whole kernel to freeze. Sometimes the kernel keeps going. On the occasion that the kernel kept going I was able to retrieve syslog, which has the full kernel information.

Quoted 9 lines

Note that in syslog, you can see that I inserted a USB memory stick in order to copy off the attached files. The kernel oops happens without the USB memory stick inserted.
I can reproduce this at will. At the moment I simply can't use my two four port starfire network cards.
This PC is a root-over-NFS system.

Starfire is triggering the BUG_ON(!test_bit(NAPI_STATE_SCHED, &n->state)); in napi_enable().

This is a regression somewhere between 2.6.24 and 2.6.32(!).

Michael

2010-01-26 02:00 UTC #2 lore raw

Hi Andrew,

I believe that this is a regression, yes.

I will attempt to compile up some kernels this week and provide more info. Should I start at 26 and go up or at 31 and go down?

I can't use anything lower than 26 according to udev. I was running 24 but compiled 32 when I upgraded udev.

Regards,
Michael.

Quoted 55 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 30 lines · On Wed, 20 Jan 2010 04:29:20 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=15091
           Summary: starfire causes kernel BUG when interface goes up
           Product: Drivers
           Version: 2.5
    Kernel Version: 2.6.32
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: No
Created an attachment (id=24651)
--> (http://bugzilla.kernel.org/attachment.cgi?id=24651)
ls -l /dev (before crash)
I formerly used 2.6.20 and 2.6.24 with a couple of starfire 4 port ethernet cards. On 2.6.32 the interfaces don't start on boot and when I issue "ifconfig ethX up" (where X is a starfire port).
Sometimes the exception causes the whole kernel to freeze. Sometimes the kernel keeps going. On the occasion that the kernel kept going I was able to retrieve syslog, which has the full kernel information.
Quoted 10 lines
Note that in syslog, you can see that I inserted a USB memory stick in order to copy off the attached files. The kernel oops happens without the USB memory stick inserted.
I can reproduce this at will. At the moment I simply can't use my two four port starfire network cards.
This PC is a root-over-NFS system.
Starfire is triggering the BUG_ON(!test_bit(NAPI_STATE_SCHED, &n->state)); in napi_enable().
This is a regression somewhere between 2.6.24 and 2.6.32(!).

Andrew Morton

author

2010-01-26 01:51 UTC #3 lore raw

Quoted 10 lines · On Tue, 26 Jan 2010 14:44:31 +1300 Michael <[email protected]> wrote:

Hi Andrew,
I believe that this is a regression, yes.
I will attempt to compile up some kernels this week and provide more info. Should I start at 26 and go up or at 31 and go down?
I can't use anything lower than 26 according to udev. I was running 24 but compiled 32 when I upgraded udev.

Thanks.

Starfire is a pretty rarely-used driver, I suspect. Hopefully someone who understands the NAPI stuff can look at the code and go "ah-hah", and save you all that work.

But if that doesn't happen then yup, a bisection would be good, thanks.
The best way to do it really is with git.
http://landley.net/writing/git-quick.html has an explanation.

Ben Hutchings

2010-01-26 02:02 UTC #4 lore raw

PATCH 1/1 starfire: Clean up properly if firmware loading fails

In reply to Andrew Morton

netdev_open() will return without cleaning up net device or hardware state if firmware loading fails. This results in a BUG() on a second attempt to bring the interface up, reported in
<http://bugzilla.kernel.org/show_bug.cgi?id=15117>, and probably has even worse effects if the driver is removed afterwards.

Call netdev_close() to clean up on failure.
---

Quoted 9 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 3 lines · On Wed, 20 Jan 2010 04:29:20 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=15091
Summary: starfire causes kernel BUG when interface goes up

[...]

Quoted 3 lines

Quoted 3 lines
I formerly used 2.6.20 and 2.6.24 with a couple of starfire 4 port ethernet cards. On 2.6.32 the interfaces don't start on boot and when I issue "ifconfig ethX up" (where X is a starfire port).

[...]

Quoted 4 lines

Starfire is triggering the BUG_ON(!test_bit(NAPI_STATE_SCHED, &n->state)); in napi_enable().
This is a regression somewhere between 2.6.24 and 2.6.32(!).

This driver now attempts to load firmware when an interface is brought up, *after* calling napi_enable(). If that fails, it will return without calling napi_disable(). On the second attempt to bring the interface it calls napi_enable() a second time and triggers this assertion.

As a workaround, try installing the necessary firmware. :-)

Ben.

 drivers/net/starfire.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

drivers/net/starfire.c +4 -1

index 95db60a..f952113 100644
@@ -1063,7 +1063,7 @@ static int netdev_open(struct net_device *dev)
 	if (retval) {
 		printk(KERN_ERR "starfire: Failed to load firmware \"%s\"\n",
 		       FIRMWARE_RX);
-		return retval;
+		goto out_init;
 	}
 	if (fw_rx->size % 4) {
 		printk(KERN_ERR "starfire: bogus length %zu in \"%s\"\n",

@@ -1108,6 +1108,9 @@ out_tx:
 	release_firmware(fw_tx);
 out_rx:
 	release_firmware(fw_rx);
+out_init:
+	if (retval)
+		netdev_close(dev);
 	return retval;
 }

Signature · 6 lines

1.6.6

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Andrew Morton

author

2010-01-26 02:16 UTC #5 lore raw

Re: [PATCH] starfire: Clean up properly if firmware loading fails

Quoted 5 lines · On Tue, 26 Jan 2010 02:02:12 +0000 Ben Hutchings <[email protected]>…

netdev_open() will return without cleaning up net device or hardware state if firmware loading fails. This results in a BUG() on a second attempt to bring the interface up, reported in
<http://bugzilla.kernel.org/show_bug.cgi?id=15117>, and probably has even worse effects if the driver is removed afterwards.

(That's the wrong bugzilla URL)

Quoted 2 lines

Call netdev_close() to clean up on failure.

OK, thanks.

Quoted 21 lines

---
Quoted 9 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 3 lines · On Wed, 20 Jan 2010 04:29:20 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=15091
Summary: starfire causes kernel BUG when interface goes up
[...]
Quoted 3 lines
Quoted 3 lines
I formerly used 2.6.20 and 2.6.24 with a couple of starfire 4 port ethernet cards. On 2.6.32 the interfaces don't start on boot and when I issue "ifconfig ethX up" (where X is a starfire port).
[...]
Quoted 4 lines
Starfire is triggering the BUG_ON(!test_bit(NAPI_STATE_SCHED, &n->state)); in napi_enable().
This is a regression somewhere between 2.6.24 and 2.6.32(!).

Quoted 8 lines

This driver now attempts to load firmware when an interface is brought up, *after* calling napi_enable(). If that fails, it will return without calling napi_disable(). On the second attempt to bring the interface it calls napi_enable() a second time and triggers this assertion.
As a workaround, try installing the necessary firmware. :-)

Missing signed-off-by. I added it, OK? Also added a Cc:stable.

From: Ben Hutchings <[email protected]>

Call netdev_close() to clean up on failure.

Addresses http://bugzilla.kernel.org/show_bug.cgi?id=15091

Signed-off-by Ben Hutchings <[email protected]>

Reported-by Michael Moffatt <[email protected]>

Cc "David S. Miller" <[email protected]>

Cc <[email protected]>

Signed-off-by Andrew Morton <[email protected]>

---

 drivers/net/starfire.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

drivers/net/starfire.c renamed from drivers/net/starfire.c~starfire-clean-up-properly-if-firmware-loading-fails +4 -1

diff -puN drivers/net/starfire.c~starfire-clean-up-properly-if-firmware-loading-fails drivers/net/starfire.c
@@ -1063,7 +1063,7 @@ static int netdev_open(struct net_device
 	if (retval) {
 		printk(KERN_ERR "starfire: Failed to load firmware \"%s\"\n",
 		       FIRMWARE_RX);
-		return retval;
+		goto out_init;
 	}
 	if (fw_rx->size % 4) {
 		printk(KERN_ERR "starfire: bogus length %zu in \"%s\"\n",

@@ -1108,6 +1108,9 @@ out_tx:
 	release_firmware(fw_tx);
 out_rx:
 	release_firmware(fw_rx);
+out_init:
+	if (retval)
+		netdev_close(dev);
 	return retval;
 }

Ben Hutchings

2010-01-26 02:32 UTC #6 lore raw

Re: [PATCH] starfire: Clean up properly if firmware loading fails

Quoted 9 lines · Andrew Morton

Quoted 5 lines · On Tue, 26 Jan 2010 02:02:12 +0000 Ben Hutchings <[email protected]>…
netdev_open() will return without cleaning up net device or hardware state if firmware loading fails. This results in a BUG() on a second attempt to bring the interface up, reported in
<http://bugzilla.kernel.org/show_bug.cgi?id=15117>, and probably has even worse effects if the driver is removed afterwards.
(That's the wrong bugzilla URL)

Hah, well spotted. Must have copied from the wrong tab.

Quoted 31 lines

Quoted 1 line
Call netdev_close() to clean up on failure.
OK, thanks.
Quoted 27 lines
---
Quoted 9 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 3 lines · On Wed, 20 Jan 2010 04:29:20 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=15091
Summary: starfire causes kernel BUG when interface goes up
[...]
Quoted 3 lines
Quoted 3 lines
I formerly used 2.6.20 and 2.6.24 with a couple of starfire 4 port ethernet cards. On 2.6.32 the interfaces don't start on boot and when I issue "ifconfig ethX up" (where X is a starfire port).
[...]
Quoted 4 lines
Starfire is triggering the BUG_ON(!test_bit(NAPI_STATE_SCHED, &n->state)); in napi_enable().
This is a regression somewhere between 2.6.24 and 2.6.32(!).
This driver now attempts to load firmware when an interface is brought up, *after* calling napi_enable(). If that fails, it will return without calling napi_disable(). On the second attempt to bring the interface it calls napi_enable() a second time and triggers this assertion.

Quoted 4 lines

Quoted 2 lines
As a workaround, try installing the necessary firmware. :-)
Missing signed-off-by. I added it, OK? Also added a Cc:stable.

[...]

Well I was going to wait and see what the submitter says first, but I'm happy for you to add that in your patch set if it's a positive result.

Ben.

Signature · 3 lines

Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Michael

2010-01-26 02:58 UTC #7 lore raw

Re: [PATCH] starfire: Clean up properly if firmware loading fails

Hi guys,

I think I'm the submitter that Ben is referring to.

So that could be the answer to the kernel BUG I have reported, but I don't think that it will answer why the interface doesn't come up... or does it?

Cheers,
Michael.

Quoted 68 lines · Ben Hutchings

Quoted 11 lines · Andrew Morton
Quoted 6 lines · On Tue, 26 Jan 2010 02:02:12 +0000 Ben Hutchings <[email protected]>…
netdev_open() will return without cleaning up net device or hardware state if firmware loading fails. This results in a BUG() on a second attempt to bring the interface up, reported in
<http://bugzilla.kernel.org/show_bug.cgi?id=15117>, and probably has even worse effects if the driver is removed afterwards.
(That's the wrong bugzilla URL)
Hah, well spotted. Must have copied from the wrong tab.
Quoted 38 lines
Quoted 2 lines
Call netdev_close() to clean up on failure.
OK, thanks.
Quoted 33 lines
---
Quoted 11 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 4 lines · On Wed, 20 Jan 2010 04:29:20 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=15091
Summary: starfire causes kernel BUG when interface goes up
[...]
Quoted 4 lines
Quoted 4 lines
I formerly used 2.6.20 and 2.6.24 with a couple of starfire 4 port ethernet cards. On 2.6.32 the interfaces don't start on boot and when I issue "ifconfig ethX up" (where X is a starfire port).
[...]
Quoted 5 lines
Starfire is triggering the BUG_ON(!test_bit(NAPI_STATE_SCHED, &n->state)); in napi_enable().
This is a regression somewhere between 2.6.24 and 2.6.32(!).
This driver now attempts to load firmware when an interface is brought up, *after* calling napi_enable(). If that fails, it will return without calling napi_disable(). On the second attempt to bring the interface it calls napi_enable() a second time and triggers this assertion.
Quoted 5 lines
Quoted 3 lines
As a workaround, try installing the necessary firmware. :-)
Missing signed-off-by. I added it, OK? Also added a Cc:stable.
[...]
Well I was going to wait and see what the submitter says first, but I'm happy for you to add that in your patch set if it's a positive result.
Ben.

Andrew Morton

author

2010-01-26 03:29 UTC #8 lore raw

Re: [PATCH] starfire: Clean up properly if firmware loading fails

Quoted 9 lines · On Tue, 26 Jan 2010 15:58:39 +1300 Michael <[email protected]> wrote:

Hi guys,
I think I'm the submitter that Ben is referring to.
So that could be the answer to the kernel BUG I have reported, but I don't think that it will answer why the interface doesn't come up... or does it?
From this:

Jan 21 05:08:26 172 kernel: starfire: Failed to load firmware "adaptec/starfire_rx.bin"
Jan 21 05:08:26 172 kernel: device eth4 entered promiscuous mode
Jan 21 05:08:26 172 kernel: starfire 0000:03:06.0: firmware: requesting adaptec/starfire_rx.bin
Jan 21 05:08:26 172 kernel: starfire: Failed to load firmware "adaptec/starfire_rx.bin"
Jan 21 05:08:26 172 kernel: device eth5 entered promiscuous mode
Jan 21 05:08:26 172 kernel: starfire 0000:03:07.0: firmware: requesting adaptec/starfire_rx.bin
Jan 21 05:08:26 172 kernel: starfire: Failed to load firmware "adaptec/starfire_rx.bin"
Jan 21 05:08:26 172 kernel: device eth6 entered promiscuous mode
Jan 21 05:08:26 172 kernel: starfire 0000:04:04.0: firmware: requesting adaptec/starfire_rx.bin

I assume that it can't find the firmware?

Michael

2010-01-26 05:51 UTC #9 lore raw

Re: [PATCH] starfire: Clean up properly if firmware loading fails

Hi Andrew,

Yep, OK, I hadn't seen that log in dmesg.

That driver is new to me, it's never turned up (or been required) before so that must be new between 2.6.24 and 2.6.32.

As this is a root over nfs system, the kernel is compiled elsewhere and then installed manually. What I was missing was that there is a new adaptec directory that needed to be copied from /usr/src/linux-2.6.32/firmware/ to /lib/2.6.32. Actually, I am not sure that this is the right place for it (rather than say /lib/firmware/2.6.32|, but it seems to work anyway.

Quite a gotchya. After compiling a kernel on a separate compiling system, I don't actually run the 'make install' on the nfs system. Previously I used to run the 'make install' on a dedicated compiling server and then just copy the modules from that system into the root over nfs exported /lib/modules directory. That'd worked fine up until now.

I will have to find a cleverer way to copy over the new firmware libs for future compiles. The 'make install' seems to copy firmware objects into the compiling system's /lib/firmware/ directory without distinguishing the kernel version. So I can't easily tell which ones I'm supposed to be copying into the nfs export.

Many thanks for all your help. The interfaces are up and apparently you understand where the kernel BUG came from.

So does that complete the story now? In other words, is there anything further you need from me.

Many thanks,
Michael.
|

Quoted 24 lines · Andrew Morton

Quoted 8 lines · On Tue, 26 Jan 2010 15:58:39 +1300 Michael <[email protected]> wrote:
Hi guys,
I think I'm the submitter that Ben is referring to.
So that could be the answer to the kernel BUG I have reported, but I don't think that it will answer why the interface doesn't come up... or does it?
From this:
Jan 21 05:08:26 172 kernel: starfire: Failed to load firmware "adaptec/starfire_rx.bin"
Jan 21 05:08:26 172 kernel: device eth4 entered promiscuous mode
Jan 21 05:08:26 172 kernel: starfire 0000:03:06.0: firmware: requesting adaptec/starfire_rx.bin
Jan 21 05:08:26 172 kernel: starfire: Failed to load firmware "adaptec/starfire_rx.bin"
Jan 21 05:08:26 172 kernel: device eth5 entered promiscuous mode
Jan 21 05:08:26 172 kernel: starfire 0000:03:07.0: firmware: requesting adaptec/starfire_rx.bin
Jan 21 05:08:26 172 kernel: starfire: Failed to load firmware "adaptec/starfire_rx.bin"
Jan 21 05:08:26 172 kernel: device eth6 entered promiscuous mode
Jan 21 05:08:26 172 kernel: starfire 0000:04:04.0: firmware: requesting adaptec/starfire_rx.bin
I assume that it can't find the firmware?

Andrew Morton

author

2010-01-26 05:58 UTC #10 lore raw

Re: [PATCH] starfire: Clean up properly if firmware loading fails

Quoted 24 lines · On Tue, 26 Jan 2010 18:51:07 +1300 Michael <[email protected]> wrote:

Yep, OK, I hadn't seen that log in dmesg.
That driver is new to me, it's never turned up (or been required) before so that must be new between 2.6.24 and 2.6.32.
As this is a root over nfs system, the kernel is compiled elsewhere and then installed manually. What I was missing was that there is a new adaptec directory that needed to be copied from /usr/src/linux-2.6.32/firmware/ to /lib/2.6.32. Actually, I am not sure that this is the right place for it (rather than say /lib/firmware/2.6.32|, but it seems to work anyway.
Quite a gotchya. After compiling a kernel on a separate compiling system, I don't actually run the 'make install' on the nfs system. Previously I used to run the 'make install' on a dedicated compiling server and then just copy the modules from that system into the root over nfs exported /lib/modules directory. That'd worked fine up until now.
I will have to find a cleverer way to copy over the new firmware libs for future compiles. The 'make install' seems to copy firmware objects into the compiling system's /lib/firmware/ directory without distinguishing the kernel version. So I can't easily tell which ones I'm supposed to be copying into the nfs export.

Quoted 5 lines

Many thanks for all your help. The interfaces are up and apparently you understand where the kernel BUG came from.
So does that complete the story now? In other words, is there anything further you need from me.

Well, it'd be great if you could test Ben's patch with the firmware file inaccessible, and confirm that the kernel doesn't crash/freeze/etc.

Thanks.

Ben Hutchings

2010-01-26 14:40 UTC #11 lore raw

Re: [PATCH] starfire: Clean up properly if firmware loading fails

In reply to Michael

On Tue, 2010-01-26 at 18:51 +1300, Michael wrote:
[...]

Quoted 5 lines

I will have to find a cleverer way to copy over the new firmware libs for future compiles. The 'make install' seems to copy firmware objects into the compiling system's /lib/firmware/ directory without distinguishing the kernel version. So I can't easily tell which ones I'm supposed to be copying into the nfs export.

[...]

You should be setting INSTALL_MOD_PATH.

Ben.

Signature · 3 lines

Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.