sym53c8xx: weird behavior with udev
http://bugzilla.kernel.org/show_bug.cgi?id=10374
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|drivers_other@kernel- |scsi_drivers-
|bugs.osdl.org |sym53c8xx@kernel-
| |bugs.osdl.org
Component|Other |sym53c8xx
Product|Drivers |SCSI Drivers
Regression|0 |1Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #1 from [email protected] 2008-04-01 01:15 -------
Reply-To: [email protected]
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 48 lines · On Tue, 1 Apr 2008 01:01:22 -0700 (PDT) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=10374
Summary: sym53c8xx: weird behavior with udev Product: Drivers Version: 2.5 KernelVersion: 2.6.24.4 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: Other AssignedTo: [email protected] ReportedBy: [email protected]Latest working kernel version: 2.6.22.9
Earliest failing kernel version: 2.6.23
Distribution: Gentoo
Hardware Environment: sparc64 (Sun Blade 100)
Software Environment:
Problem Description:Since kernel 2.6.23, I have been having problems getting the sungem network device working on one of my two Blade 100s, see bug #10273.
After debugging this, I found that this seems to be somehow related to sym53c8xx and udev.
If I allow udev to load sym53c8xx during boot, the attached disks work fine but the network does not work at all. While the device is up and mii-diag says there is link beat, no packets can be sent or received, and attempts to use the network result in "network unreachable" errors.
If I blacklist sym53c8xx in
/etc/modprobe.d/blacklist, let the machine boot normally and then manually load sym53c8xx after everything is settled, both the scsi disks and the network appear to work fine.None of this happened in kernel versions 2.6.22.9 and earlier, 2.6.23 was the first to start showing this behavior, and it still persists in 2.6.24.
I've tried playing with the option
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODEas that is what seems to have changed between kernel versions 2.6.22 and 2.6.23. The recommended value for my machine is 0, but I have tried the other possible values without result. I have also tried togglingCONFIG_SCSI_SYM53C8XX_MMIO, also without result.
Quoted 3 lines
Steps to reproduce:
Let udev load sym53c8xx in kernel 2.6.23 or newer.
urgh. Perhaps it's related to platform IRQ routing or something.
I'd suggest that the next step would be to send us the `dmesg -s 1000000' output for both good and bad kernels. A comparison might show where things went bad.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #2 from [email protected] 2008-04-01 01:58 -------
Quoted 8 lines · On Tue, 1 Apr 2008 01:15:18 -0700 Andrew Morton <[email protected]>…
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
urgh. Perhaps it's related to platform IRQ routing or something.
I'd suggest that the next step would be to send us the `dmesg -s 1000000' output for both good and bad kernels. A comparison might show where things went bad.
Alright, here goes. Attached three files.
dmesg-2.6.22.9-good.txt is the dmesg output from 2.6.22.9 where everything works as it should.
dmesg-2.6.24.4-bad.txt is the dmesg output from 2.6.24.4 with sym53cxx loaded by udev and the network not responding.
dmesg-2.6.24.4-good.txt is the dmesg output from 2.6.24.4 with sym53cxx blacklisted and the network working fine.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #3 from [email protected] 2008-04-01 07:12 -------
Reply-To: [email protected]
Quoted 11 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 3 lines
Steps to reproduce:
Let udev load sym53c8xx in kernel 2.6.23 or newer.
urgh. Perhaps it's related to platform IRQ routing or something.
I'd suggest that the next step would be to send us the `dmesg -s 1000000' output for both good and bad kernels. A comparison might show where things went bad.
Yes, that would be my guess too ... although I don't see anything amiss
in the dmesg I note you have two ethernet interfaces:
eth0: Sun GEM (PCI) 10/100/1000BaseT Ethernet 00:03:ba:08:61:7c
eth1394: eth1: IPv4 over IEEE 1394 (fw-host0)
I'm assuming eth0 is the problem?
Could you also send us the output of /proc/interrupts, /proc/iomem and /proc/ioports just to see if we have a problem. Also, if eth0 is on its own interrupt line, does the interrupt count rise even while the interface is non functional?
Thanks,
James
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #4 from [email protected] 2008-04-01 07:48 -------
Quoted 6 lines · On Tue, 01 Apr 2008 09:11:55 -0500 James Bottomley…
Quoted 4 lines · Andrew Morton
I'd suggest that the next step would be to send us the `dmesg -s 1000000' output for both good and bad kernels. A comparison might show where things went bad.
Quoted 7 lines
Yes, that would be my guess too ... although I don't see anything amiss
in the dmesg I note you have two ethernet interfaces:eth0: Sun GEM (PCI) 10/100/1000BaseT Ethernet 00:03:ba:08:61:7c
eth1394: eth1: IPv4 over IEEE 1394 (fw-host0)I'm assuming eth0 is the problem?
Yes, the Sun GEM interface is the problem. Ethernet over FireWire (a direct link between the webserver and database server) works just fine. In a minimal config with no devices enabled except the SU serial line (needed for console), the dual SCSI adapter and the Sun GEM nic, the problem still remained the same.
Quoted 4 lines
Could you also send us the output of
/proc/interrupts,/proc/iomemand/proc/ioportsjust to see if we have a problem. Also, if eth0 is on its own interrupt line, does the interrupt count rise even while the interface is non functional?
Here goes:
seraphim ~ # cat /proc/interrupts
CPU0
0: 7259 <NULL> timer 8: 0 sun4u power 9: 405 sun4u su(serial) 11: 1 sun4u eth0 12: 95 sun4u ohci1394 13: 0 sun4u ohci_hcd:usb1 14: 0 sun4u ALI 5451 15: 2265 sun4u ide0 16: 293 sun4u sym53c8xx 17: 69 sun4u sym53c8xx
seraphim ~ # cat /proc/iomem
1fe020002e8-1fe020002ef : su
1fe020003f8-1fe020003ff : su
1fe02000800-1fe02000803 : power
1ff00000000-1ffffffffff : /pci@1f,0
1ff000a0000-1ff000bffff : Video RAM area
1ff000c0000-1ff000c7fff : Video ROM
1ff000f0000-1ff000fffff : System ROM
1ff00400000-1ff0041ffff : sungem
1ff00420000-1ff004207ff : ohci1394
1ff00424000-1ff00425fff : ALI 5451
1ff02000000-1ff02ffffff : ohci_hcd
1ff03000000-1ff03001fff : sym53c8xx
1ff03002000-1ff03003fff : sym53c8xx
1ff03004000-1ff03005fff : sym53c8xx
1ff03006000-1ff03007fff : sym53c8xx
1ffc0000000-1ffdfffffff : IOMMU
1fff1000000-1fff1001fff : clock
seraphim ~ # cat /proc/ioports
00000600-0000061f : ali1535_smbus
1fe02000000-1fe02ffffff : /pci@1f,0
1fe02000600-1fe0200061f : 0000:00:03.0
1fe02000800-1fe0200083f : 0000:00:03.0
1fe02000900-1fe020009ff : ALI 5451
1fe02000a00-1fe02000a07 : ide0
1fe02000a1a-1fe02000a1a : ide0
1fe02000a20-1fe02000a27 : ide0
1fe02000a28-1fe02000a2f : ide1
1fe02001000-1fe020010ff : sym53c8xx
1fe02001100-1fe020011ff : sym53c8xx
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #5 from [email protected] 2008-04-01 12:05 ------- Hello all,
I did a bit more testing, and I think this may be related to the order in which modules are loaded.
If I let udev load sungem, and load sym53c8xx manually, everything works.
If I let udev load sym53c8xx, and load sungem manually, I get the non-functional network.
If I let udev load both modules, I also get the non-functional network. While udev loads sungem first and sym53c8xx later, I don't suppose it waits for one module to 'settle' before loading the next. :-)
So to sum it up, the bug is triggered if sym53c8xx is loaded before sungem is.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #6 from [email protected] 2008-04-01 13:20 -------
Reply-To: [email protected]
Quoted 10 lines · Jos van der Ende
Hello all,
I did a bit more testing, and I think this may be related to the order in which modules are loaded.
If I let udev load sungem, and load sym53c8xx manually, everything works.
If I let udev load sym53c8xx, and load sungem manually, I get the non-functional network.
If I let udev load both modules, I also get the non-functional network. While udev loads sungem first and sym53c8xx later, I don't suppose it waits for one module to 'settle' before loading the next. :-)
That's odd ... it's behaving like a resource conflict. However, the ports and interrupt trace didn't betray anything. What does lspci -vv say for each of the devices? Also, if you remove the sym2 module in the problem case, does the sungem come back to life?
I'm afraid I can't see anything relevant looking over the sym2 changes, so you might need to bisect this to identify the culprit.
James
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #7 from [email protected] 2008-04-01 13:57 -------
Quoted 3 lines · On Tue, 01 Apr 2008 15:19:29 -0500 James Bottomley…
That's odd ... it's behaving like a resource conflict. However, the ports and interrupt trace didn't betray anything. What does lspci -vv say for each of the devices?
Output from lspci -vv attached.
Quoted 2 lines
Also, if you remove the sym2 module in the
problem case, does the sungem come back to life?
No, once it is hosed it stays hosed until the next boot. Fiddling with the wrong ioports maybe?
Quoted 2 lines
I'm afraid I can't see anything relevant looking over the sym2 changes, so you might need to bisect this to identify the culprit.
Working on that, but it is a hassle as this bitty-box needs some time to compile a kernel. 2.6.23-rc1 didn't boot, for starters.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #8 from [email protected] 2008-04-01 14:14 -------
Reply-To: [email protected]
Quoted 8 lines · Jos van der Ende
Quoted 3 lines · On Tue, 01 Apr 2008 15:19:29 -0500 James Bottomley…
That's odd ... it's behaving like a resource conflict. However, the ports and interrupt trace didn't betray anything. What does lspci -vv say for each of the devices?
Output from lspci -vv attached.
Thanks ... unfortunately looks normal too. The gem has a single memory region; the sym2 has 2 mem and one IO region, all of which show up in the /proc/iomem|ports.
Quoted 4 lines
Quoted 2 lines
Also, if you remove the sym2 module in the
problem case, does the sungem come back to life?No, once it is hosed it stays hosed until the next boot. Fiddling with the wrong ioports maybe?
Yes ... that's what I guess. Just as one last grasp at a straw, is there any difference in /proc/iomem or /proc/ioports for the working case (sungem loaded first followed by sym2)?
Quoted 4 lines
Quoted 2 lines
I'm afraid I can't see anything relevant looking over the sym2 changes, so you might need to bisect this to identify the culprit.
Working on that, but it is a hassle as this bitty-box needs some time to compile a kernel. 2.6.23-rc1 didn't boot, for starters.
Sorry ... can't think of much else that will help.
James
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #9 from [email protected] 2008-04-01 15:30 -------
Quoted 3 lines · On Tue, 01 Apr 2008 16:14:14 -0500 James Bottomley…
Yes ... that's what I guess. Just as one last grasp at a straw, is there any difference in
/proc/iomemor/proc/ioportsfor the working case (sungem loaded first followed by sym2)?
Nope, exactly the same.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'm afraid I can't see anything relevant looking over the sym2 changes, so you might need to bisect this to identify the culprit.
First results are in: 2.6.23-rc1 could not boot, 2.6.23-rc2 already had the problem.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #10 from [email protected] 2008-04-02 03:29 ------- Maybe it's nothing, but I did notice that sym53c8xx compiles with a warning of a possibly uninitialized variable:
CC [M] drivers/scsi/sym53c8xx_2/sym_glue.odrivers/scsi/sym53c8xx_2/sym_glue.c: In function 'sym_eh_handler':drivers/scsi/sym53c8xx_2/sym_glue.c:612: warning: 'io_reset' may be used uninitialized in this function
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #11 from [email protected] 2008-04-02 05:07 -------
Quoted 5 lines · Jos van der Ende
Maybe it's nothing, but I did notice that sym53c8xx compiles with a warning of a possibly uninitialized variable:
CC [M]
drivers/scsi/sym53c8xx_2/sym_glue.odrivers/scsi/sym53c8xx_2/sym_glue.c: In function 'sym_eh_handler':drivers/scsi/sym53c8xx_2/sym_glue.c:612: warning: 'io_reset' may be used uninitialized in this function
Yeah, that's nothing. It's actually a bug in GCC that produces that warning (and it's code that'll never be executed on your platform anyway).
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #12 from [email protected] 2008-04-02 07:09 ------- Finally, bisecting is done. :-)
Well, it took more reboots than a typical Windows XP installation (and thank the heavens for my Sparc64 cross compiler on my Core 2 Duo), but this seems to be the culprit:
5a606b72a4309a656cd1a19ad137dc5557c4b8ea is first bad commit commit 5a606b72a4309a656cd1a19ad137dc5557c4b8ea
Author: David S. Miller <[email protected]> Date: Mon Jul 9 22:40:36 2007 -0700
[SPARC64]: Do not ACK an INO if it is disabled or inprogress.
This is also a partial workaround for a bug in the LDOM firmware which double-transmits RX inos during high load. Without this, such an event causes the kernel to loop forever in the interrupt call chain ACK'ing but never actually running the IRQ handler (and thus clearing the interrupt condition in the device).
There is still a bad potential effect when double INOs occur, not covered by this changeset. Namely, if the INO is already on the per-cpu INO vector list, we still blindly re-insert it and thus we can end up losing interrupts already linked in after it.
We could deal with that by traversing the list before insertion, but that's too expensive for this edge case.
:040000 040000 7e65c9b16e6c37f2c3f83195c5a57b4d2b8f0a7ce7a7bedcc88d33793a6525e9337a1a51982bc513 M arch
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #13 from [email protected] 2008-04-02 08:50 -------
Reply-To: [email protected]
Quoted 1 line · Jos van der Ende
Finally, bisecting is done. :-)
Thanks for doing this ... we'd never have found it by looking at the driver code ...
Quoted 5 lines
Well, it took more reboots than a typical Windows XP installation (and thank the heavens for my Sparc64 cross compiler on my Core 2 Duo), but this seems to be the culprit:
5a606b72a4309a656cd1a19ad137dc5557c4b8eais first bad commit
Reading the code for this, it seems that something fiddled with the IRQ_DISABLED or IRQ_PENDING flags when it came time for the ->eoi() so the gem interrupt is always held pending (because it's never ended).
Since the sym2 is on interrupts 16 and 17 and gem on 11 (and the descriptors are separate entities in the irq_desc array) I can't really see how sym2 would be doing this.
James
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
------- Comment #14 from [email protected] 2008-04-02 09:07 -------
Quoted 3 lines · On Wed, 02 Apr 2008 10:49:19 -0500 James Bottomley…
Reading the code for this, it seems that something fiddled with the
IRQ_DISABLEDorIRQ_PENDINGflags when it came time for the ->eoi() so the gem interrupt is always held pending (because it's never ended).
So Andrew's first hunch that interrupts were somehow involved is right.
Quoted 3 lines
Since the sym2 is on interrupts 16 and 17 and gem on 11 (and the descriptors are separate entities in the
irq_descarray) I can't really see how sym2 would be doing this.
Yeah, that has me baffled too. Still, the fact is that I can only trigger the bug by loading sym53c8xx before sungem. I have yet to find any other conditions that trigger it. Loading sungem before sym53c8xx on an affected kernel gives no trouble at all.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=10374
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |CLOSED
Resolution| |CODE_FIX------- Comment #15 from [email protected] 2008-05-17 08:52 ------- This is fixed by commit a5f56179c861a23a2f08409711926ff812f30d38 in 2.6.25.4.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.