QUESTION: Network hangs with BP6 and 2.4.x kernels, hardware related?

DISCUSSION Frank de Lange 5 participants 18 messages Jan 10, 2001 – Jan 12, 2001 Char and misc drivers

Share & tools

Frank de Lange

author

2001-01-10 21:31 UTC #0 lore raw

Hi'all,

Ever since I put two ethernet-cards (cheap Winbond W89C940 based PCI NE2K clones) in my BP-6 system, I've been experiencing intermittent network hangs. A hang manifests itself as a total failure to communicate through either network card, and can only be solved by rebooting. Removing and reloading the modules does not fix the problem, only a reboot works.

I have searched high and low for possible causes, but I found no definite answer. I suspect the problem may be hardware-related (the BP-6 can be tricky sometimes), but I want to rule out software issues before I take the system apart. So, my question: does anyone else experience these network-related problems with a BP-6 based system? Or maybe other similar problems, where a specific subsystem hangs an can only be revived by rebooting the box? The network cards are currently in PCI4 and PCI5 (which should work, they are NON-busmastering cards after all...), but relocating the cards does not solve the problem. This problem has been nagging me ever since I moved to 2.3.x (somewhere around 2.3.30). As it is intermittent, it is very difficult to pin down. I suspect the APIC in not completely sane, or some timing on the bus is out of spec, but that's no more than a suspicion. And since I do not have access to a logic analyzer it is somewhat hard to prove...

Here's a lspci for the box:

00:00.0 Host bridge: Intel Corporation 440BX/ZX - 82443BX/ZX Host bridge (rev 03) 00:01.0 PCI bridge: Intel Corporation 440BX/ZX - 82443BX/ZX AGP bridge (rev 03) 00:07.0 ISA bridge: Intel Corporation 82371AB PIIX4 ISA (rev 02) 00:07.1 IDE interface: Intel Corporation 82371AB PIIX4 IDE (rev 01) 00:07.2 USB Controller: Intel Corporation 82371AB PIIX4 USB (rev 01) 00:07.3 Bridge: Intel Corporation 82371AB PIIX4 ACPI (rev 02)
00:09.0 Ethernet controller: Winbond Electronics Corp W89C940
00:0b.0 Multimedia video controller: Brooktree Corporation Bt878 (rev 02) 00:0b.1 Multimedia controller: Brooktree Corporation Bt878 (rev 02) 00:0d.0 SCSI storage controller: Symbios Logic Inc. (formerly NCR) 53c875 (rev 26) 00:0f.0 Multimedia audio controller: Ensoniq ES1371 [AudioPCI-97] (rev 06) 00:11.0 Ethernet controller: Winbond Electronics Corp W89C940 (rev 0b) 00:13.0 Unknown mass storage controller: Triones Technologies, Inc. HPT366 (rev 01) 00:13.1 Unknown mass storage controller: Triones Technologies, Inc. HPT366 (rev 01) 01:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G400 AGP (rev 04)

and here's a cat /proc/<interesting_file>

/proc/cpuinfo shows this box contains dual Celeron 466's (non-overclocked)

/proc/interrupts:
CPU0 CPU1

  0:   10003353    9483961    IO-APIC-edge  timer
  1:      85449      84279    IO-APIC-edge  keyboard
  2:          0          0          XT-PIC  cascade
  3:        167        249    IO-APIC-edge  serial
  4:     380807     381140    IO-APIC-edge  serial
 14:     136991     132077    IO-APIC-edge  ide0
 15:      25836      24605    IO-APIC-edge  ide1
 16:      42510      42482   IO-APIC-level  es1371, mga@PCI:1:0:0
 17:         26         26   IO-APIC-level  sym53c8xx
 18:       9287       8837   IO-APIC-level  bttv
 19:     205294     205191   IO-APIC-level  eth0, eth1, usb-uhci
NMI:   19487238   19487238 
LOC:   19488621   19488620 
ERR:          0

/proc/meminfo:
total: used: free: shared: buffers: cached:

Mem:  261984256 260354048  1630208        0 12873728 99012608
Swap: 511926272 14245888 497680384

...
...

# network cards share IRQ with USB, which hosts...
/proc/bus/usb/devices:
T:  Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=12  MxCh= 2
B:  Alloc= 11/900 us ( 1%), #Int=  1, #Iso=  0
D:  Ver= 1.00 Cls=09(hub  ) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=0000 ProdID=0000 Rev= 0.00
S:  Product=USB UHCI Root Hub
S:  SerialNumber=c000
C:* #Ifs= 1 Cfg#= 1 Atr=40 MxPwr=  0mA
I:  If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   8 Ivl=255ms
T:  Bus=01 Lev=01 Prnt=01 Port=01 Cnt=01 Dev#=  2 Spd=12  MxCh= 0
D:  Ver= 1.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=03f0 ProdID=0401 Rev= 1.00
S:  Product=HP ScanJet 5200C
S:  SerialNumber=SG95D1720ZHT
C:* #Ifs= 1 Cfg#= 1 Atr=60 MxPwr=  0mA
I:  If#= 0 Alt= 0 #EPs= 3 Cls=00(>ifc ) Sub=00 Prot=00 Driver=usbscanner
E:  Ad=81(I) Atr=02(Bulk) MxPS=  64 Ivl=  0ms
E:  Ad=02(O) Atr=02(Bulk) MxPS=  16 Ivl=  0ms
E:  Ad=83(I) Atr=03(Int.) MxPS=   1 Ivl=250ms

uname -a:
Linux behemoth.localnet 2.4.0 #1 SMP Fri Jan 5 15:41:39 CET 2001 i686 unknown

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Manfred Spraul

2001-01-10 22:26 UTC #1 lore raw

Quoted 6 lines · Frank de Lange

Hi'all,
Ever since I put two ethernet-cards (cheap Winbond W89C940 based PCI NE2K clones) in my BP-6 system, I've been experiencing intermittent network hangs.

Which driver do you use? The driver in 2.4.0 contains several bugfixes. If that driver still hangs then I'll double check the documentation.

Quoted 5 lines

A
hang manifests itself as a total failure to communicate through either network card, and can only be solved by rebooting. Removing and reloading the modules does not fix the problem, only a reboot works.

That's different from my problems:
unload+reload always fixed my problems with the unpatch winbond-840 driver.

Quoted 2 lines

which should work, they are
NON-busmastering cards after all...),

third line in w840_probe1():

	pci_set_master().

And the documentation begins with
W89C840F
PCI Bus Master Fast Ethernet LAN Controller.

--
Manfred
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-10 22:30 UTC #2 lore raw

In reply to Frank de Lange

Quoted 2 lines · Manfred Spraul

Which driver do you use? The driver in 2.4.0 contains several bugfixes. If that driver still hangs then I'll double check the documentation.

The NE2K PCI one... I'll try to fiddle around with the driver, who knows...

Quoted 3 lines

And the documentation begins with
W89C840F
PCI Bus Master Fast Ethernet LAN Controller.

That is the 'F' (Fast) version. My cards are just plain and simple 10base2/T, humble and non-busmastering (AFAIK).

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-10 22:41 UTC #3 lore raw

In reply to Frank de Lange

Quoted 9 lines · Manfred Spraul

Quoted 2 lines
which should work, they are
NON-busmastering cards after all...),
third line in w840_probe1():
	pci_set_master().
And the documentation begins with
W89C840F
PCI Bus Master Fast Ethernet LAN Controller.

...in addition to my previous reply, your cards use the Winbond 840 series, while my cards use the 940 series. Higher number, but a less capabpe chipset or so it seems...

Hm, but that reminds me not to get 840's to solve my problems :-)

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Andrew Morton

2001-01-11 11:42 UTC #4 lore raw

In reply to Frank de Lange

Quoted 9 lines · Frank de Lange

Hi'all,
Ever since I put two ethernet-cards (cheap Winbond W89C940 based PCI NE2K clones) in my BP-6 system, I've been experiencing intermittent network hangs. A hang manifests itself as a total failure to communicate through either network card, and can only be solved by rebooting. Removing and reloading the modules does not fix the problem, only a reboot works.

Losing both NICs at the same time could be the elusive "APIC stops generating interrupts" problem.

Do you get any transmit timeout messages in the logs? If so, send them.

Does it happen with a uniprocessor build?

Are you able to boot with the `noapic' LILO option? If so, does that make it stop?

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-11 15:22 UTC #5 lore raw

In reply to Frank de Lange

Quoted 2 lines · Andrew Morton

Losing both NICs at the same time could be the elusive "APIC stops generating interrupts" problem.

Yup, that's what I thought... But the real question is, is this a software/configuration problem or a hardware problem which can only be fixed by physically changing something on the board?... As it is, as you call it, 'elusive', it is a b*tch to pinpoint the source of these problems...

Quoted 2 lines

Do you get any transmit timeout messages in the logs? If so, send them.

Here they are (marked with ***):

grep -B2 -A2 transmit /var/log/messages:

    Jan 10 22:24:47 behemoth kernel: usb_control/bulk_msg: timeout 
    Jan 10 22:24:50 behemoth kernel: usb_control/bulk_msg: timeout 
*** Jan 10 22:56:51 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
    Jan 10 22:57:03 behemoth last message repeated 7 times
    Jan 10 22:57:03 behemoth kernel: SysRq: Emergency Sync 
    --
    Jan 10 22:57:09 behemoth kernel: Syncing device 16:07 ... OK 
    Jan 10 22:57:09 behemoth kernel: Done. 
*** Jan 10 22:57:09 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
    Jan 10 22:57:09 behemoth kernel: SysRq: Emergency Sync 
    Jan 10 22:57:09 behemoth kernel: Syncing device 03:01 ... OK

Quoted 1 line

Does it happen with a uniprocessor build?

Not tried yet, since I wanna use both CPU's :-).

Quoted 1 line

Are you able to boot with the `noapic' LILO option?

I am, and did it a while ago. As far as I remember, it did not make it stop... I'll try again (even though it is not a real solution, since that APIC is there for a reason...)

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-11 16:56 UTC #6 lore raw

In reply to Frank de Lange

Quoted 2 lines

Do you get any transmit timeout messages in the logs? If so, send them.

In addition to my previous message, here's what I get from the debug log facility:

Jan 10 22:56:51 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
Jan 10 22:56:51 behemoth kernel: eth0: Tx timed out, lost interrupt? TSR=0x3, ISR=0x3, t=33. 
Jan 10 22:56:52 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
Jan 10 22:56:52 behemoth kernel: eth0: Tx timed out, lost interrupt? TSR=0x3, ISR=0x3, t=26. 
Jan 10 22:56:53 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
Jan 10 22:56:53 behemoth kernel: eth0: Tx timed out, lost interrupt? TSR=0x3, ISR=0x3, t=30. 
Jan 10 22:56:56 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
Jan 10 22:56:56 behemoth kernel: eth0: Tx timed out, lost interrupt? TSR=0x3, ISR=0x3, t=78. 
Jan 10 22:56:56 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
Jan 10 22:56:56 behemoth kernel: eth0: Tx timed out, lost interrupt? TSR=0x3, ISR=0x3, t=32. 
Jan 10 22:56:58 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
Jan 10 22:56:58 behemoth kernel: eth0: Tx timed out, lost interrupt? TSR=0x3, ISR=0x3, t=89. 
Jan 10 22:57:00 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
Jan 10 22:57:00 behemoth kernel: eth0: Tx timed out, lost interrupt? TSR=0x3, ISR=0x3, t=77. 
Jan 10 22:57:03 behemoth kernel: NETDEV WATCHDOG: eth0: transmit timed out 
Jan 10 22:57:03 behemoth kernel: eth0: Tx timed out, lost interrupt? TSR=0x3, ISR=0x3, t=171.

So yeah, I get timeouts allright...

Currently running NOAPIC, pity to see CPU1 receiving no interrupts at all... In the same debug log I now just saw this:

Jan 11 17:37:05 behemoth kernel: spurious 8259A interrupt: IRQ7

That's weird, since there's nothing there...:

cat /proc/interrupts
CPU0 CPU1

  0:     232967          0          XT-PIC  timer
  1:       6424          0          XT-PIC  keyboard
  2:          0          0          XT-PIC  cascade
  3:        138          0          XT-PIC  serial
  4:      46201          0          XT-PIC  serial
  9:         52          0          XT-PIC  sym53c8xx
 10:     744329          0          XT-PIC  eth0, eth1, usb-uhci
 11:          0          0          XT-PIC  bttv
 12:          0          0          XT-PIC  es1371, mga@PCI:1:0:0
 14:      19778          0          XT-PIC  ide0
 15:       4520          0          XT-PIC  ide1
NMI:          0          0 
LOC:     232916     232914 
ERR:          1

See? Nothing on 7... This is with NOAPIC (as you can see from the XT-PIC's in the above dump). BP6 again?

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-11 19:19 UTC #7 lore raw

In reply to Frank de Lange

Here's another posting to the list which mentions problems with NE2K and BP6:

http://web.gnu.walfield.org/mail-archive/linux-kernel/2000-August/0132.html

"...In another machine, a dual celeron abit-bp6, recent 2.3.x kernels seem to dislike my realtek 8029 NIC. (I know, it's garbage plugged in to garbage...) The network card will die randomly, usually when I'm sending large amounts of data. When it dies, there are no kernel messages, and the interrupt count in /proc/interrupts for the card stop changing. Minor (painful) experimentation has shown that if the card is sharing the interrupt with anything else (say, ide2), it takes that with it. This only happens in "newer" kernels, it's fine in 2.2.16, and in some earlier 2.3.x kernels. It goes away if I boot with the noapic=1 kernel parameter, and seems to be replaced with harmless "spurious 8259A interrupt: IRQ7." messages. (I haven't configured any hardware at all to be on IRQ7 - though I'm lead to believe IRQ7 has some sort of special purpose) ..."

So I'm not the only one...

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-11 19:50 UTC #8 lore raw

In reply to Frank de Lange

Another observation wrt. behaviour with 'noapic'...

When streaming time-critical data over the network (running esound to another server, etc), sometimes there are hiccups in the stream. These hiccups seem to be much less frequent, if at all present, when running with 'noapic'. I'm currently running sound over a heavily loaded ethernet, no hiccups at all... Weird, since the apic ought to spread the load of handling the interrupts over all available CPU's.

Whatever is causing this, there seems to be something fishy in the way interrupts are handled when the apic(s) is/are enabled...

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-11 20:25 UTC #9 lore raw

In reply to Frank de Lange

Hm, the noapic option seems to help, as I'm currently beating the network to death but it won't die... As the problem is elusive, it is hard to tell, and it would not surprise me if the net dropped dead the moment this mail went through, but current indication is that noapic makes the sudden net-death disappear.

So.... we're still left with the question 'is this hardware-related, or is it a software/configuration problem'? Other people seem to have similar problems with dissimilar hardware (tulip cards instead of Winbond, etc), on 2.2.x as well as 2.3/4.x. As I do not run Windows (NT or 2K), I can not tell if this problem also occurs there. And my FreeBSD-box is uniprocessor... So... has anyone seen anything like this on other 'true' (SMP) OS's? If so, that would indicate a hardware problem...

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-11 21:10 UTC #10 lore raw

In reply to Frank de Lange

OK, just one last addition to what has nearly become my own thread...

I now am fairly certain that the problem (network stalls on multiprocessor systems) is not BP6 or NE2K-PCI specific. I found several postings which relate to similar problems on dissimilar hardware. Another interesting one is:

Re: PROBLEM : Networking stops working with kernel 2.4.0-test11
(http://www.mail-archive.com/[email protected]/msg18722.html)

"...I have an almos identical system as you, 2x200MMX motherboard (Gigabyte
586DX) also Voodoo3 (2000 pci) the same nic Realtek 8029AS, also a bt848 tv card, also SCSI (Aic-7880 onboard, but not used).

I have reported it some time ago, and now all I get with 2.4.0-test11-pre4 and I think a additional patch is NETDEV WATCHDOG:
eth0: transmit timed out, and something in the console about lost irq?

I can't reproduce it with a uniprocesor kernel, and I have a 3c503 card wich uses the 8390 module, so I suppose that the problem it's not in the 8390, and it seems to be smp related...."

ne2k-pci freezes with APIC error on 2.4.0-testX SMP
(http://www.mail-archive.com/[email protected]/msg14468.html)

"...

When doing massive NFS transfers (2.4 machine as the client) on my SMP box
(Abit BP6 2x celeronA 533mhz (non-overclocked) 64Mb ram, latest apt-get-ed
debian woody) my ne2k-pci card (Realtek Semiconductor Co., Ltd. RTL-8029(AS)
(rev 0)) suddenly stops working. test5 spits that in syslog:..."

More to be found when searching the archives. This problem has been around for a long, long time (probably since the current level of apic-support was added, somewhere around 2.3.1x?). It has been reported by several people, several times. I feel like rigging every apic-related piece of code with a zillion bells and printk's but that would surely only create more mayhem as this whole thing seems to be timing-related...

Anyone got any idea's on how to tackle this? Anyone who is 'intimate with' the apic-related code? It'll take me some time to dive into that part, so if there is anyone who already has taken the plunge, do tell...

Cheers//Frank

[ who is still running apic-less, without problems [

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Jeff Garzik

2001-01-11 21:48 UTC #11 lore raw

In reply to Frank de Lange

Quoted 8 lines · Frank de Lange

OK, just one last addition to what has nearly become my own thread...
I now am fairly certain that the problem (network stalls on multiprocessor systems) is not BP6 or NE2K-PCI specific. I found several postings which relate to similar problems on dissimilar hardware. Another interesting one is:
I have reported it some time ago, and now all I get with 2.4.0-test11-pre4 and I think a additional patch is NETDEV WATCHDOG:
eth0: transmit timed out, and something in the console about lost irq?

Are you judging based on the error message? The 'netdev watchdog ...' message is a generic error message that could have any number of causes. It's just saying, well, what it says :) The kernel was unable to transmit a packet in a certain amount of time. You might get these messages if you unplug a cable suddenly, or if your hardware isn't delivering interrupts, or many other things...

Jeff

Signature · 7 lines

Jeff Garzik       | "You see, in this world there's two kinds of
Building 1024     |  people, my friend: Those with loaded guns
MandrakeSoft      |  and those who dig. You dig."  --Blondie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-11 21:53 UTC #12 lore raw

In reply to Frank de Lange

Quoted 6 lines · Jeff Garzik

Are you judging based on the error message? The 'netdev watchdog ...' message is a generic error message that could have any number of causes. It's just saying, well, what it says :) The kernel was unable to transmit a packet in a certain amount of time. You might get these messages if you unplug a cable suddenly, or if your hardware isn't delivering interrupts, or many other things...

No, I'm judging based on the fact that I found reports from people using NE2K-PCI with several cards as well as tulip-based cards (different driver) on abit BP6 as well as Gigabyte motherboards, mostly on 2.3.x/2.4.x kernels. I found some postings with these problems on 2.2.x kernels.

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

David Woodhouse

2001-01-12 14:36 UTC #13 lore raw

Quoted 5 lines · [email protected] said:

No, I'm judging based on the fact that I found reports from people
using NE2K-PCI with several cards as well as tulip-based cards (different driver) on abit BP6 as well as Gigabyte motherboards, mostly on 2.3.x/2.4.x kernels. I found some postings with these problems on 2.2.x kernels.

IRQ 19 on my BP6 stopped arriving a few days ago.

19: 90373 90473 IO-APIC-level usb-uhci

Removing and reloading the usb-uhci driver didn't help. Loading the uhci driver just oopsed, which seems to be its normal behaviour on the occasions on which I try it.

Rebooting fixed it. I was half tempted to code a 'kick APIC because I think it broke' function, but then decided not to bother.

It might be nice in 2.5 to give drivers some way of kicking the APIC when they think they've missed an interrupt, much like the network code kicks the driver. And to deal more gracefully with IRQ storms.

Once a driver has a way of saying "Oi! Why isn't IRQ x working?" it would be feasible to just disable the damn thing if we receive a million of them in rapid succession.

--
dwmw2

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-12 00:29 UTC #14 lore raw

In reply to Frank de Lange

Quoted 2 lines · Jeff Garzik

Just out of curiosity, if you boot a Linux 2.4.0 kernel with the "noapic" command line option, does behavior improve?

For the curious, here's a summary of some tests I did:

apic, 2 cpu's, no smp affinity -> network hangs under load
apic, maxcpus=1, no smp affinity -> network hangs under load apic, 2 cpu's, smp affinity for all irq's on CPU1 -> network hangs under load noapic, 2 cpu's, no smp affinity -> NO HANG, WORKSFORME

Quick and dirty conclusion: as soon as the apic comes in to play, things get messy...

ps. load == 2 simultaneous nfs cp -rd <big_directory> sessions and streaming esd audio over the network

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Andrew Morton

2001-01-12 11:36 UTC #15 lore raw

In reply to Frank de Lange

Quoted 3 lines · Frank de Lange

Quick and dirty conclusion: as soon as the apic comes in to play, things get messy...

Yup.

Frank, for over a year there have been sporadic reports of APIC's forgetting how to deliver interrupts. Not only on BP6's. Often with 3com NICs, so I've never been 100% sure that it's not a failure in the NIC.

In your case, you have three devices on the same IRQ and they all go to lunch at the same time. That's pretty convincing.

Nobody has been able to repeat this frequently enough for any useful debugging to be done. Don't go away!

Here is a debugging patch. Could you please apply this, rebuild and:

1: Type ALT-SYSRQ-A when everything is good
2: Type ALT-SYSRQ-A when everything is bad
3: send the resulting logs.

I've Cc'ed Maciej, who understands this stuff.

arch/i386/kernel/io_apic.c +1 -1

@@ -692,7 +692,7 @@ void __init UNEXPECTED_IO_APIC(void)
 	printk(KERN_WARNING "          to [email protected]\n");
 }
 
-void __init print_IO_APIC(void)
+void /*__init*/ print_IO_APIC(void)
 {
 	int apic, i;
 	struct IO_APIC_reg_00 reg_00;

drivers/char/sysrq.c +9 -0

diff -up --recursive --new-file linux-2.4.0-test11.macro/drivers/char/sysrq.c linux-2.4.0-test11/drivers/char/sysrq.c
@@ -72,6 +72,15 @@ void handle_sysrq(int key, struct pt_reg
 	console_loglevel = 7;
 	printk(KERN_INFO "SysRq: ");
 	switch (key) {
+	case 'a':
+		printk("\n");
+		printk("print_PIC()\n");
+		print_PIC();
+		printk("print_IO_APIC()\n");
+		print_IO_APIC();
+		printk("print_all_local_APICs()\n");
+		print_all_local_APICs();
+		break;
 	case 'r':					    /* R -- Reset raw mode */
 		if (kbd) {
 			kbd->kbdmode = VC_XLATE;

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-12 15:09 UTC #16 lore raw

In reply to Frank de Lange

Quoted 10 lines · Andrew Morton

Quoted 3 lines · Frank de Lange
Quick and dirty conclusion: as soon as the apic comes in to play, things get messy...
Here is a debugging patch. Could you please apply this, rebuild and:
1: Type ALT-SYSRQ-A when everything is good
2: Type ALT-SYSRQ-A when everything is bad
3: send the resulting logs.

WillCo...

Now rebuilding...

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Frank de Lange

author

2001-01-12 15:40 UTC #17 lore raw

In reply to Frank de Lange

Quoted 6 lines · Andrew Morton

Here is a debugging patch. Could you please apply this, rebuild and:
1: Type ALT-SYSRQ-A when everything is good
2: Type ALT-SYSRQ-A when everything is bad
3: send the resulting logs.

OK, here's the results I get...

Before network hang
===================

print_PIC()
printing PIC contents
print_IO_APIC()
testing the IO APIC.......................

.................................... done.
print_all_local_APICs()

... APIC ID: 01000000 (1)
... APIC VERSION: 00040011
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000001000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000010000000000000000

... APIC ID: 00000000 (0)
... APIC VERSION: 00040011
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000010000000000000001000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000001000

NOTICE: results differ every time I hit ALT-SYSRQ-A.
The '1' bit at 'row 11, col. 26' stays '1' no matter how many times I use the magic keys. The other '1' bits jump around a bit, or disappear alltogether. Also, the sequence in which the APICs appear in the dump sometimes differs (this example shows 1 first, then 0, other times you'd see 0 first, then 1)

After network hang
==================

print_PIC()
printing PIC contents
print_IO_APIC()
testing the IO APIC.......................

.................................... done.
print_all_local_APICs()

... APIC ID: 00000000 (0)
... APIC VERSION: 00040011
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000010000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000010000000000000000

NOTICE: hmmm... see, now that '1' bit at row 11,
col. 26 for APIC 0 which was '1' before has turned to '0'. It will stay '0' no matter how many times I hit the magic keys... It seems to have been replaced by the '1' bit at row 11, col. 10, since that bit stays '1' no matter how many magic I throw at it...

Hope this helps... If you need more, let me know...

Cheers//Frank

Signature · 13 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/