WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e() with tg3 network

DISCUSSION Roger Heflin 5 participants 26 messages Nov 11, 2008 – Dec 2, 2008

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

Roger Heflin

author

2008-11-11 09:48 UTC #0 lore raw

I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.

So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.

Unloading all of the network modules and reloading them did not correct the problem.

Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?

http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238

The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).

Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.

I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.

The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.

I am first turning off all offload capabilities on tg3 and going to see if that changes anything.

The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.

I also have a second tg3 port that is slightly different, so I may try that eventually.

What else can I look at?

Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 [uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5

Peter Zijlstra

2008-11-11 11:31 UTC #1 lore raw

(netdev CC'ed)

Quoted 3 lines · Roger Heflin

I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.

How do more recent kernels do?

Quoted 100 lines

So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.
Unloading all of the network modules and reloading them did not correct the problem.
Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?
http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238
The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).
Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.
I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.
The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.
I am first turning off all offload capabilities on tg3 and going to see if that changes anything.
The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.
I also have a second tg3 port that is slightly different, so I may try that eventually.
What else can I look at?
Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs 
w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 
[uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Roger Heflin

author

2008-11-15 04:01 UTC #2 lore raw

Quoted 8 lines · Peter Zijlstra

(netdev CC'ed)
Quoted 3 lines · Roger Heflin
I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.
How do more recent kernels do?

I did not try more recent kernels, more testing seems to indicate that at the very least it the bug depends on a certain version of either tg3 and/or firmware to happen, as my second tg3 port does not
have it happen. More about this below.

Quoted 33 lines

Quoted 33 lines
So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.
Unloading all of the network modules and reloading them did not correct the problem.
Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?
http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238
The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).
Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.
I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.

Quoted 5 lines

Quoted 5 lines
The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.
I am first turning off all offload capabilities on tg3 and going to see if that changes anything.

This made no difference in the error.

Quoted 3 lines

Quoted 3 lines
The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.

Did not try.

Quoted 3 lines

Quoted 3 lines
I also have a second tg3 port that is slightly different, so I may try that eventually.

I tried this, and with the second port I don't appear to be getting
the error. The first port is a 5789-v3.29a and the second port is a
5788-v3.04, I know the first port is faster (pcie-x1) than the second
port (pci bus-built-in, unknown exact connection). The second port
will sustain about 50MB/second, were as the first port will get

Quoted 1 line

90MB/second.

It seems to me to likely be the firmware on the tg3, and it would seem unlikely that the driver could do anything more than work around the issue that is in the firmware, and currently my system works on the second port, and the second port is fast enough for my needs.

If someone else runs into this issue, since I have 2 ports I would be able to do some testing on it, right now my first port is locked up, and the machine is running fine on the second port.

lspci -vvv for the first (bad) port:

02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit Ethernet PCI Express (rev 11)
Subsystem: Foxconn International, Inc. Unknown device 0cc1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
Expansion ROM at <ignored> [disabled]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D3 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: 0101b8102a0f7b0c Data: f21e
Capabilities: [d0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <4us, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s <2us, L1 <64us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Virtual Channel

Quoted 37 lines

Quoted 36 lines

Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 [uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Willy Tarreau

2008-11-18 06:50 UTC #3 lore raw

Hello,

Quoted 8 lines · Roger Heflin

Quoted 7 lines · Peter Zijlstra
(netdev CC'ed)
Quoted 3 lines · Roger Heflin
I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.

Quoted 6 lines

Quoted 1 line
How do more recent kernels do?
I did not try more recent kernels, more testing seems to indicate that at the very least it the bug depends on a certain version of either tg3 and/or firmware to happen, as my second tg3 port does not
have it happen. More about this below.

I got the exact same one yesterday and wanted to fire a bug report, so yours is saving me some time :-)

It's on my notebook (HP nc8000, Pentium-M, no SMP, no HT). It was connected to a 100 Mbps switch and was sending nearly 100 Mbps of traffic. It had auto-negociated 100-full. I noticed long pauses on the other side, finally realizing that my notebook was ill. It's on 2.6.27.7-rc1. I have never had any such problem on this notebook with any earlier kernel up to 2.6.25.X. I've never run 2.6.26 on it.

My tg3 is just PCI-based, no PCIe in this beast. I can send more info when I turn it on. I don't think that the tg3 driver changes often, so most likely digging through the changes between 2.6.25 and 2.6.27 should not take much time. I just don't know if I can reliably reproduce the issue right now.

Quoted 93 lines

Quoted 32 lines
Quoted 32 lines
So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.
Unloading all of the network modules and reloading them did not correct the problem.
Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?
http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238
The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).
Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.
I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.
Quoted 5 lines
Quoted 5 lines
The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.
I am first turning off all offload capabilities on tg3 and going to see if that changes anything.
This made no difference in the error.
Quoted 3 lines
Quoted 3 lines
The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.
Did not try.
Quoted 3 lines
Quoted 3 lines
I also have a second tg3 port that is slightly different, so I may try that eventually.
I tried this, and with the second port I don't appear to be getting
the error. The first port is a 5789-v3.29a and the second port is a
5788-v3.04, I know the first port is faster (pcie-x1) than the second
port (pci bus-built-in, unknown exact connection). The second port
will sustain about 50MB/second, were as the first port will get
Quoted 1 line
90MB/second.
It seems to me to likely be the firmware on the tg3, and it would seem unlikely that the driver could do anything more than work around the issue that is in the firmware, and currently my system works on the second port, and the second port is fast enough for my needs.
If someone else runs into this issue, since I have 2 ports I would be able to do some testing on it, right now my first port is locked up, and the machine is running fine on the second port.
lspci -vvv for the first (bad) port:
02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit Ethernet PCI Express (rev 11)
Subsystem: Foxconn International, Inc. Unknown device 0cc1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K] Expansion ROM at <ignored> [disabled]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D3 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable-
Address: 0101b8102a0f7b0c Data: f21e
Capabilities: [d0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <4us, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

Quoted 68 lines

Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s <2us, L1 <64us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Virtual Channel
Quoted 58 lines
Quoted 58 lines
Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit 
timed out
Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss 
exportfs w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last
unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 
2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] 
usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] 
find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] 
clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] 
update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] 
uhci_scan_schedule+0x11b/0x6b0 [uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] 
usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] 
tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c2b4>] 
run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] 
smp_apic_timer_interrupt+0x6e/0x79
Nov 11 00:44:39 computer kernel:  [<c040431c>] 
apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5

Willy

Matt Carlson

2008-11-20 03:11 UTC #4 lore raw

Quoted 29 lines · Willy Tarreau

Hello,
Quoted 8 lines · Roger Heflin
Quoted 7 lines · Peter Zijlstra
(netdev CC'ed)
Quoted 3 lines · Roger Heflin
I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.
Quoted 6 lines
Quoted 1 line
How do more recent kernels do?
I did not try more recent kernels, more testing seems to indicate that at the very least it the bug depends on a certain version of either tg3 and/or firmware to happen, as my second tg3 port does not
have it happen. More about this below.
I got the exact same one yesterday and wanted to fire a bug report, so yours is saving me some time :-)
It's on my notebook (HP nc8000, Pentium-M, no SMP, no HT). It was connected to a 100 Mbps switch and was sending nearly 100 Mbps of traffic. It had auto-negociated 100-full. I noticed long pauses on the other side, finally realizing that my notebook was ill. It's on 2.6.27.7-rc1. I have never had any such problem on this notebook with any earlier kernel up to 2.6.25.X. I've never run 2.6.26 on it.

Quoted 5 lines

My tg3 is just PCI-based, no PCIe in this beast. I can send more info when I turn it on. I don't think that the tg3 driver changes often, so most likely digging through the changes between 2.6.25 and 2.6.27 should not take much time. I just don't know if I can reliably reproduce the issue right now.

Willy, this problem description sounds a little different than the original report. There was a bug where the driver would wait 2.5 seconds for a firmware event that would never get serviced. That fix has already landed in the 2.6.27 tree though.

I glanced over the changes between 2.6.25 and 2.6.27.6. There are quite a few changes related to phylib support for an upcoming device, but not so many changes that affect older devices. What device are you using?

Quoted 170 lines

Quoted 93 lines
Quoted 32 lines
Quoted 32 lines
So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.
Unloading all of the network modules and reloading them did not correct the problem.
Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?
http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238
The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).
Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.
I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.
Quoted 5 lines
Quoted 5 lines
The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.
I am first turning off all offload capabilities on tg3 and going to see if that changes anything.
This made no difference in the error.
Quoted 3 lines
Quoted 3 lines
The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.
Did not try.
Quoted 3 lines
Quoted 3 lines
I also have a second tg3 port that is slightly different, so I may try that eventually.
I tried this, and with the second port I don't appear to be getting
the error. The first port is a 5789-v3.29a and the second port is a
5788-v3.04, I know the first port is faster (pcie-x1) than the second
port (pci bus-built-in, unknown exact connection). The second port
will sustain about 50MB/second, were as the first port will get
Quoted 1 line
90MB/second.
It seems to me to likely be the firmware on the tg3, and it would seem unlikely that the driver could do anything more than work around the issue that is in the firmware, and currently my system works on the second port, and the second port is fast enough for my needs.
If someone else runs into this issue, since I have 2 ports I would be able to do some testing on it, right now my first port is locked up, and the machine is running fine on the second port.
lspci -vvv for the first (bad) port:
02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit Ethernet PCI Express (rev 11)
Subsystem: Foxconn International, Inc. Unknown device 0cc1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K] Expansion ROM at <ignored> [disabled]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D3 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3 Enable-
Address: 0101b8102a0f7b0c Data: f21e
Capabilities: [d0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <4us, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Quoted 68 lines
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s <2us, L1 <64us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Virtual Channel
Quoted 58 lines
Quoted 58 lines
Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit 
timed out
Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss 
exportfs w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last
unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 
2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] 
usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] 
find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] 
clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] 
update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] 
uhci_scan_schedule+0x11b/0x6b0 [uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] 
usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] 
tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c2b4>] 
run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] 
smp_apic_timer_interrupt+0x6e/0x79
Nov 11 00:44:39 computer kernel:  [<c040431c>] 
apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
Willy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Willy Tarreau

2008-11-20 05:37 UTC #5 lore raw

Hello Matt,

Quoted 6 lines · Matt Carlson

Quoted 5 lines
My tg3 is just PCI-based, no PCIe in this beast. I can send more info when I turn it on. I don't think that the tg3 driver changes often, so most likely digging through the changes between 2.6.25 and 2.6.27 should not take much time. I just don't know if I can reliably reproduce the issue right now.

Quoted 8 lines

Willy, this problem description sounds a little different than the original report. There was a bug where the driver would wait 2.5 seconds for a firmware event that would never get serviced. That fix has already landed in the 2.6.27 tree though.
I glanced over the changes between 2.6.25 and 2.6.27.6. There are quite a few changes related to phylib support for an upcoming device, but not so many changes that affect older devices. What device are you using?

I think it's a 5704, but I will check this this morning when I'm at work. I also want to try to reliably reproduce the problem. After that, I see only 29 patches which differ from the two kernels, it should be pretty easy to spot the culprit.

If you think it's a different bug than original report (though I really thought it was the same), I'll post my findings in a separate thread not to mix investigations.

Thanks,
Willy

Matt Carlson

2008-11-20 18:43 UTC #6 lore raw

Quoted 14 lines · Willy Tarreau

Hello Matt,
Quoted 11 lines · Matt Carlson
Quoted 5 lines
My tg3 is just PCI-based, no PCIe in this beast. I can send more info when I turn it on. I don't think that the tg3 driver changes often, so most likely digging through the changes between 2.6.25 and 2.6.27 should not take much time. I just don't know if I can reliably reproduce the issue right now.
Willy, this problem description sounds a little different than the original report. There was a bug where the driver would wait 2.5 seconds for a firmware event that would never get serviced. That fix has already landed in the 2.6.27 tree though.

Quoted 8 lines

Quoted 3 lines
I glanced over the changes between 2.6.25 and 2.6.27.6. There are quite a few changes related to phylib support for an upcoming device, but not so many changes that affect older devices. What device are you using?
I think it's a 5704, but I will check this this morning when I'm at work. I also want to try to reliably reproduce the problem. After that, I see only 29 patches which differ from the two kernels, it should be pretty easy to spot the culprit.

O.K. Let me know how it goes.

Could we clarify something though? In your previous email, you said you didn't have any problems on pre-2.6.25 kernels. I'm wondering if the problem goes back further than 2.6.25. From 2.6.24 to 2.6.25, there was a significant set of flow control changes that took place. I suspect that might have something to do with Roger's problem, and it may have something to do with your problem too. So, is it true that 2.6.25 works for you? If not, can you try disabling flow control and see if that helps?

Quoted 3 lines

If you think it's a different bug than original report (though I really thought it was the same), I'll post my findings in a separate thread not to mix investigations.

Right now, I think it is premature to say, so let's continue as if they were the same problem. We can always break it out into a separate discussion later.

Willy Tarreau

2008-11-20 21:27 UTC #7 lore raw

Quoted 19 lines · Matt Carlson

Quoted 18 lines · Willy Tarreau
Hello Matt,
Quoted 14 lines · Matt Carlson
Quoted 5 lines
My tg3 is just PCI-based, no PCIe in this beast. I can send more info when I turn it on. I don't think that the tg3 driver changes often, so most likely digging through the changes between 2.6.25 and 2.6.27 should not take much time. I just don't know if I can reliably reproduce the issue right now.
Willy, this problem description sounds a little different than the original report. There was a bug where the driver would wait 2.5 seconds for a firmware event that would never get serviced. That fix has already landed in the 2.6.27 tree though.
I glanced over the changes between 2.6.25 and 2.6.27.6. There are quite a few changes related to phylib support for an upcoming device, but not so many changes that affect older devices. What device are you using?

Quoted 6 lines

Quoted 4 lines
I think it's a 5704, but I will check this this morning when I'm at work. I also want to try to reliably reproduce the problem. After that, I see only 29 patches which differ from the two kernels, it should be pretty easy to spot the culprit.
O.K. Let me know how it goes.

Today, with the notebook connected to a gig switch, I could not reproduce the problem, even after one hour of approximately the same workload. I'll retry with the original 100 Mbps switch on monday.

Quoted 8 lines

Could we clarify something though? In your previous email, you said you didn't have any problems on pre-2.6.25 kernels. I'm wondering if the problem goes back further than 2.6.25. From 2.6.24 to 2.6.25, there was a significant set of flow control changes that took place. I suspect that might have something to do with Roger's problem, and it may have something to do with your problem too. So, is it true that 2.6.25 works for you? If not, can you try disabling flow control and see if that helps?

It works fine till 2.6.25.18 included. I have not tried any 2.6.26 on this machine. Just 2.6.27.7-rc1 (with a few patches, none of which affect tg3). My goal is 1) to reproduce with the exact same kernel on the original switch,
2) confirm that 2.6.25.18 does not exhibit the problem on the same switch,
3) to switch to plain 2.6.27.x to ease troubleshooting, and 4) to find which of the 29 tg3 patches between 2.6.25.18 and 2.6.27.x brings the issue.

Quoted 7 lines

Quoted 3 lines
If you think it's a different bug than original report (though I really thought it was the same), I'll post my findings in a separate thread not to mix investigations.
Right now, I think it is premature to say, so let's continue as if they were the same problem. We can always break it out into a separate discussion later.

OK that's fine for me.

Thanks Matt!
Willy

Matt Carlson

2008-11-20 21:53 UTC #8 lore raw

Quoted 25 lines · Willy Tarreau

Quoted 24 lines · Matt Carlson
Quoted 22 lines · Willy Tarreau
Hello Matt,
Quoted 6 lines · Matt Carlson
Quoted 5 lines
My tg3 is just PCI-based, no PCIe in this beast. I can send more info when I turn it on. I don't think that the tg3 driver changes often, so most likely digging through the changes between 2.6.25 and 2.6.27 should not take much time. I just don't know if I can reliably reproduce the issue right now.
Quoted 8 lines
Willy, this problem description sounds a little different than the original report. There was a bug where the driver would wait 2.5 seconds for a firmware event that would never get serviced. That fix has already landed in the 2.6.27 tree though.
I glanced over the changes between 2.6.25 and 2.6.27.6. There are quite a few changes related to phylib support for an upcoming device, but not so many changes that affect older devices. What device are you using?
I think it's a 5704, but I will check this this morning when I'm at work. I also want to try to reliably reproduce the problem. After that, I see only 29 patches which differ from the two kernels, it should be pretty easy to spot the culprit.

Quoted 5 lines

Quoted 1 line
O.K. Let me know how it goes.
Today, with the notebook connected to a gig switch, I could not reproduce the problem, even after one hour of approximately the same workload. I'll retry with the original 100 Mbps switch on monday.

Was it really a 5704 that you were using?

Willy Tarreau

2008-11-21 17:55 UTC #9 lore raw

Hi Matt,

Quoted 5 lines · Matt Carlson

Quoted 3 lines
Today, with the notebook connected to a gig switch, I could not reproduce the problem, even after one hour of approximately the same workload. I'll retry with the original 100 Mbps switch on monday.
Was it really a 5704 that you were using?

No, I was wrong, finally it's a 5705 :

tg3.c:v3.94 (August 14, 2008)
ACPI: PCI Interrupt Link [C0C5] enabled at IRQ 11
tg3 0000:02:0e.0: PCI INT A -> Link[C0C5] -> GSI 11 (level, low) -> IRQ 11
eth0: Tigon3 [partno(BMC5705mA3) rev 3003 PHY(5705)] (PCI:33MHz:32-bit) 10/100/1000Base-T Ethernet 00:0d:9d:91:ef:24
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1]
eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

Here's the lspci :

02:0e.0 Class 0200: 14e4:165e (rev 03)
Subsystem: 103c:088c
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (16000ns min), cache line size 10
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 90000000 (64-bit, non-prefetchable) [size=64K]
Expansion ROM at 5c000000 [disabled] [size=64K]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
Address: fbff71ffffbfebfc Data: edff
00: e4 14 5e 16 06 00 b0 02 03 00 00 02 10 40 00 00
10: 04 00 00 90 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 07 00 00 00 3c 10 8c 08
30: 00 00 00 00 48 00 00 00 00 00 00 00 0b 01 40 00

Regards,
Willy

Willy Tarreau

2008-11-24 13:27 UTC #10 lore raw

In reply to Matt Carlson

Hi Matt,

Quoted 3 lines · Matt Carlson

Quoted 3 lines
Today, with the notebook connected to a gig switch, I could not reproduce the problem, even after one hour of approximately the same workload. I'll retry with the original 100 Mbps switch on monday.

fairly easier now with the same switch. I just have to transfer 100k objects over HTTP via this switch to see the problem happen :

tg3: eth0: The system may be re-ordering memory-mapped I/O cycles to the network device, attempting to recover. Please report the problem to the driver maintainer and include system chipset information.
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

The switch is an el-cheapo D-Link 10/100. Note that this time I did not see any warning. Maybe I did not wait long enough though. Here are some stats and information I could collect :

root@wtap:/# ethtool -i eth0
driver: tg3
version: 3.94
firmware-version: 5705-v3.14
bus-info: 0000:02:0e.0

root@wtap:/# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
        Link detected: yes

root@wtap:/# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off

root@wtap:/# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             on
TX:             on

root@wtap:/# netstat -ni
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   1500   0   61665      0      0      0   65133      0      0      0 BMRU
lo    16436   0  945938      0      0      0  945938      0      0      0 LRU

root@wtap:/# ethtool -S eth0
NIC statistics:
     rx_octets: 52461785
     rx_fragments: 0
     rx_ucast_packets: 61018
     rx_mcast_packets: 0
     rx_bcast_packets: 476
     rx_fcs_errors: 0
     rx_align_errors: 0
     rx_xon_pause_rcvd: 0
     rx_xoff_pause_rcvd: 0
     rx_mac_ctrl_rcvd: 0
     rx_xoff_entered: 0
     rx_frame_too_long_errors: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_in_length_errors: 0
     rx_out_length_errors: 0
     rx_64_or_less_octet_packets: 0
     rx_65_to_127_octet_packets: 0
     rx_128_to_255_octet_packets: 0
     rx_256_to_511_octet_packets: 0
     rx_512_to_1023_octet_packets: 0
     rx_1024_to_1522_octet_packets: 0
     rx_1523_to_2047_octet_packets: 0
     rx_2048_to_4095_octet_packets: 0
     rx_4096_to_8191_octet_packets: 0
     rx_8192_to_9022_octet_packets: 0
     tx_octets: 54658258
     tx_collisions: 0
     tx_xon_sent: 0
     tx_xoff_sent: 0
     tx_flow_control: 0
     tx_mac_errors: 0
     tx_single_collisions: 0
     tx_mult_collisions: 0
     tx_deferred: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     tx_collide_2times: 0
     tx_collide_3times: 0
     tx_collide_4times: 0
     tx_collide_5times: 0
     tx_collide_6times: 0
     tx_collide_7times: 0
     tx_collide_8times: 0
     tx_collide_9times: 0
     tx_collide_10times: 0
     tx_collide_11times: 0
     tx_collide_12times: 0
     tx_collide_13times: 0
     tx_collide_14times: 0
     tx_collide_15times: 0
     tx_ucast_packets: 65033
     tx_mcast_packets: 0
     tx_bcast_packets: 0
     tx_carrier_sense_errors: 0
     tx_discards: 0
     tx_errors: 0
     dma_writeq_full: 0
     dma_write_prioq_full: 0
     rxbds_empty: 0
     rx_discards: 0
     rx_errors: 0
     rx_threshold_hit: 0
     dma_readq_full: 0
     dma_read_prioq_full: 0
     tx_comp_queue_full: 0
     ring_set_send_prod_index: 0
     ring_status_update: 0
     nic_irqs: 0
     nic_avoided_irqs: 0
     nic_tx_threshold_hit: 0

I will try to go back to older kernels then bisect tg3 changes. I cannot do this right now since I'm using my laptop for work ;-)

Regards,
Willy

Willy Tarreau

2008-11-24 21:53 UTC #11 lore raw

Hi Matt,

just a follow-up.

Quoted 17 lines · Willy Tarreau

Hi Matt,
Quoted 3 lines · Matt Carlson
Quoted 3 lines
Today, with the notebook connected to a gig switch, I could not reproduce the problem, even after one hour of approximately the same workload. I'll retry with the original 100 Mbps switch on monday.
fairly easier now with the same switch. I just have to transfer 100k objects over HTTP via this switch to see the problem happen :
tg3: eth0: The system may be re-ordering memory-mapped I/O cycles to the network device, attempting to recover. Please report the problem to the driver maintainer and include system chipset information.
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

Quoted 2 lines

The switch is an el-cheapo D-Link 10/100. Note that this time I did not see any warning. Maybe I did not wait long enough though.

Got it again, just had to be patient to fire a second test :

WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0x1a4/0x1b0()
NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Modules linked in: nfs lockd sunrpc mtdblock mtd_blkdevs slram mtd xt_tcpudp x_tables usbhid usb_storage ehci_hcd uhci_hcd usbcore snd_pcm_oss snd_mixer_oss snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc tg3 libphy ide_cs yenta_socket rsrc_nonstatic [last unloaded: ip_tables]
Pid: 0, comm: swapper Not tainted 2.6.27-wt2-wtap #1
 [<b01254a7>] warn_slowpath+0x67/0x90
 [<b01741a9>] ? get_slab+0x9/0x70
 [<b03d21af>] ? pskb_copy+0x2f/0x160
 [<b03aa332>] ? input_defuzz_abs_event+0x12/0xa0
 [<b03aa574>] ? input_handle_event+0x14/0x2a0
 [<b03b3d76>] ? synaptics_process_packet+0x2b6/0x3d0
 [<b0108a48>] ? native_io_delay+0x8/0x40
 [<b02ab4c9>] ? strlen+0x9/0x20
 [<b02a961e>] ? strlcpy+0x1e/0x60
 [<b03dbfbc>] ? netdev_drivername+0x3c/0x40
 [<b03e7c84>] dev_watchdog+0x1a4/0x1b0
 [<b013a27e>] ? run_hrtimer_pending+0xe/0xb0
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b012d548>] ? timer_stats_account_timer+0x38/0x40
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b012dbbc>] run_timer_softirq+0xac/0x170
 [<b013f863>] ? tick_periodic+0x33/0x70
 [<b013f8b7>] ? tick_handle_periodic+0x17/0x70
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b0129ae4>] __do_softirq+0x84/0xa0
 [<b0129b35>] do_softirq+0x35/0x40
 [<b0129bf6>] irq_exit+0x66/0x70
 [<b0105869>] do_IRQ+0x49/0x90
 [<b013bc30>] ? sched_clock_cpu+0xb0/0x100
 [<b010449b>] common_interrupt+0x23/0x28
 [<b0305158>] ? acpi_safe_halt+0x1b/0x29
 [<b0305b07>] acpi_idle_enter_c1+0xa6/0x117
 [<b03c096b>] cpuidle_idle_call+0x6b/0xa0
 [<b010206f>] cpu_idle+0x4f/0x70
 [<b04458dd>] rest_init+0x4d/0x50
 =======================
---[ end trace 1cc3b74458d87dab ]---

tg3: eth0: transmit timed out, resetting
tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

The ease with which I reproduce it here clearly indicates that this is related to the switch, probably just the fact that it is at 100 Mbps. Unfortunately this evening I must go, but I still have one 100 Mbps switch somewhere at home, I'll reproduce the same test ASAP in order to bisect the issue.

Regards,
Willy

Matt Carlson

2008-11-25 01:52 UTC #12 lore raw

Quoted 73 lines · Willy Tarreau

Hi Matt,
just a follow-up.
Quoted 17 lines · Willy Tarreau
Hi Matt,
Quoted 3 lines · Matt Carlson
Quoted 3 lines
Today, with the notebook connected to a gig switch, I could not reproduce the problem, even after one hour of approximately the same workload. I'll retry with the original 100 Mbps switch on monday.
fairly easier now with the same switch. I just have to transfer 100k objects over HTTP via this switch to see the problem happen :
tg3: eth0: The system may be re-ordering memory-mapped I/O cycles to the network device, attempting to recover. Please report the problem to the driver maintainer and include system chipset information.
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
Quoted 2 lines
The switch is an el-cheapo D-Link 10/100. Note that this time I did not see any warning. Maybe I did not wait long enough though.
Got it again, just had to be patient to fire a second test :
WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0x1a4/0x1b0()
NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Modules linked in: nfs lockd sunrpc mtdblock mtd_blkdevs slram mtd xt_tcpudp x_tables usbhid usb_storage ehci_hcd uhci_hcd usbcore snd_pcm_oss snd_mixer_oss snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc tg3 libphy ide_cs yenta_socket rsrc_nonstatic [last unloaded: ip_tables]
Pid: 0, comm: swapper Not tainted 2.6.27-wt2-wtap #1
 [<b01254a7>] warn_slowpath+0x67/0x90
 [<b01741a9>] ? get_slab+0x9/0x70
 [<b03d21af>] ? pskb_copy+0x2f/0x160
 [<b03aa332>] ? input_defuzz_abs_event+0x12/0xa0
 [<b03aa574>] ? input_handle_event+0x14/0x2a0
 [<b03b3d76>] ? synaptics_process_packet+0x2b6/0x3d0
 [<b0108a48>] ? native_io_delay+0x8/0x40
 [<b02ab4c9>] ? strlen+0x9/0x20
 [<b02a961e>] ? strlcpy+0x1e/0x60
 [<b03dbfbc>] ? netdev_drivername+0x3c/0x40
 [<b03e7c84>] dev_watchdog+0x1a4/0x1b0
 [<b013a27e>] ? run_hrtimer_pending+0xe/0xb0
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b012d548>] ? timer_stats_account_timer+0x38/0x40
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b012dbbc>] run_timer_softirq+0xac/0x170
 [<b013f863>] ? tick_periodic+0x33/0x70
 [<b013f8b7>] ? tick_handle_periodic+0x17/0x70
 [<b03e7ae0>] ? dev_watchdog+0x0/0x1b0
 [<b0129ae4>] __do_softirq+0x84/0xa0
 [<b0129b35>] do_softirq+0x35/0x40
 [<b0129bf6>] irq_exit+0x66/0x70
 [<b0105869>] do_IRQ+0x49/0x90
 [<b013bc30>] ? sched_clock_cpu+0xb0/0x100
 [<b010449b>] common_interrupt+0x23/0x28
 [<b0305158>] ? acpi_safe_halt+0x1b/0x29
 [<b0305b07>] acpi_idle_enter_c1+0xa6/0x117
 [<b03c096b>] cpuidle_idle_call+0x6b/0xa0
 [<b010206f>] cpu_idle+0x4f/0x70
 [<b04458dd>] rest_init+0x4d/0x50
 =======================
---[ end trace 1cc3b74458d87dab ]---
tg3: eth0: transmit timed out, resetting
tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

Quoted 8 lines

The ease with which I reproduce it here clearly indicates that this is related to the switch, probably just the fact that it is at 100 Mbps. Unfortunately this evening I must go, but I still have one 100 Mbps switch somewhere at home, I'll reproduce the same test ASAP in order to bisect the issue.
Regards,
Willy

Does turning off flow control help at all?

Willy Tarreau

2008-11-25 05:31 UTC #13 lore raw

On Mon, Nov 24, 2008 at 05:52:23PM -0800, Matt Carlson wrote:
(...)

Quoted 16 lines

Quoted 16 lines
tg3: eth0: transmit timed out, resetting
tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
The ease with which I reproduce it here clearly indicates that this is related to the switch, probably just the fact that it is at 100 Mbps. Unfortunately this evening I must go, but I still have one 100 Mbps switch somewhere at home, I'll reproduce the same test ASAP in order to bisect the issue.

Quoted 4 lines

Quoted 2 lines
Regards,
Willy
Does turning off flow control help at all?

I have not tested but I will. I hope to be able to trigger the problem on other similar switches, because I'm only once a week connected to the culprit...

Regards,
Willy

Matt Carlson

2008-11-25 17:54 UTC #14 lore raw

Quoted 18 lines · Willy Tarreau

On Mon, Nov 24, 2008 at 05:52:23PM -0800, Matt Carlson wrote:
(...)
Quoted 16 lines
Quoted 16 lines
tg3: eth0: transmit timed out, resetting
tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
The ease with which I reproduce it here clearly indicates that this is related to the switch, probably just the fact that it is at 100 Mbps. Unfortunately this evening I must go, but I still have one 100 Mbps switch somewhere at home, I'll reproduce the same test ASAP in order to bisect the issue.

Quoted 8 lines

Quoted 4 lines
Quoted 2 lines
Regards,
Willy
Does turning off flow control help at all?
I have not tested but I will. I hope to be able to trigger the problem on other similar switches, because I'm only once a week connected to the culprit...

I can't say for certain, but I suspect the problem might be more associated with the link speed than the particular switch you are using. Can you try autoneg'ing down to a slower speed and see if that helps make the problem more reproducable?

Willy Tarreau

2008-11-26 21:13 UTC #15 lore raw

Hi Matt,

Quoted 24 lines · Matt Carlson

Quoted 23 lines · Willy Tarreau
On Mon, Nov 24, 2008 at 05:52:23PM -0800, Matt Carlson wrote:
(...)
Quoted 20 lines
Quoted 10 lines
tg3: eth0: transmit timed out, resetting
tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
Quoted 8 lines
The ease with which I reproduce it here clearly indicates that this is related to the switch, probably just the fact that it is at 100 Mbps. Unfortunately this evening I must go, but I still have one 100 Mbps switch somewhere at home, I'll reproduce the same test ASAP in order to bisect the issue.
Regards,
Willy
Does turning off flow control help at all?

Quoted 8 lines

Quoted 3 lines
I have not tested but I will. I hope to be able to trigger the problem on other similar switches, because I'm only once a week connected to the culprit...
I can't say for certain, but I suspect the problem might be more associated with the link speed than the particular switch you are using. Can you try autoneg'ing down to a slower speed and see if that helps make the problem more reproducable?

I've run a new test on a switch I have here at home (another el-cheapo, non-manageable 100 Mbps, netgear this time). Unfortunately I cannot reproduce the problem at all. I have disabled FC on my laptop, it did not have any effect. I have disabled auto-neg and manually forced the speed to 100/Full on my laptop, and could not reproduce the problem either (though the speed was much lower due to the switch obviously negociating 100/Half when not seeing my NWay frames).

I have tried unplugging the cable during transfers and changing negociation during transfers, trying to trigger artifacts, but with no result. So I think that I will really need to debug this on the "faulty" switch on next monday. It does not surprize me much, because we don't see that many reports for a similar problem, eventhough the tg3 is very common in laptops. I just hope it's a recent regression, as I'd prefer avoid having to bisect from a very old kernel.

I'll keep you informed,
Willy

Matt Carlson

2008-11-26 22:54 UTC #16 lore raw

Quoted 36 lines · Willy Tarreau

Hi Matt,
Quoted 24 lines · Matt Carlson
Quoted 23 lines · Willy Tarreau
On Mon, Nov 24, 2008 at 05:52:23PM -0800, Matt Carlson wrote:
(...)
Quoted 20 lines
Quoted 10 lines
tg3: eth0: transmit timed out, resetting
tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000006]
tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000008]
tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4c00 enable_bit=2
tg3: eth0: Link is down.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
Quoted 8 lines
The ease with which I reproduce it here clearly indicates that this is related to the switch, probably just the fact that it is at 100 Mbps. Unfortunately this evening I must go, but I still have one 100 Mbps switch somewhere at home, I'll reproduce the same test ASAP in order to bisect the issue.
Regards,
Willy
Does turning off flow control help at all?
Quoted 8 lines
Quoted 3 lines
I have not tested but I will. I hope to be able to trigger the problem on other similar switches, because I'm only once a week connected to the culprit...
I can't say for certain, but I suspect the problem might be more associated with the link speed than the particular switch you are using. Can you try autoneg'ing down to a slower speed and see if that helps make the problem more reproducable?

Quoted 4 lines

I've run a new test on a switch I have here at home (another el-cheapo, non-manageable 100 Mbps, netgear this time). Unfortunately I cannot reproduce the problem at all. I have disabled FC on my laptop, it did not have any effect.

Disabling FC should have a positive effect, not a negative one. It might be the case that the switch does not advertise nor support FC. If that is true, you might not be able to repro the problem no matter what you did (if your problem is what I think it is). Can you check your link messages and see if it really is negotiated to off? (I see the message above, but I don't think that is with the current switch.)

Quoted 4 lines

I have disabled auto-neg and manually forced the
speed to 100/Full on my laptop, and could not reproduce the problem either (though the speed was much lower due to the switch obviously negociating 100/Half when not seeing my NWay frames).

Yes. If you force the link, both sides must be forced. The switch rightly assumes HD when bringing the link up.

Quoted 10 lines

I have tried unplugging the cable during transfers and changing negociation during transfers, trying to trigger artifacts, but with no result. So I think that I will really need to debug this on the "faulty" switch on next monday. It does not surprize me much, because we don't see that many reports for a similar problem, eventhough the tg3 is very common in laptops. I just hope it's a recent regression, as I'd prefer avoid having to bisect from a very old kernel.
I'll keep you informed,
Willy

O.K.

Willy Tarreau

2008-11-27 05:17 UTC #17 lore raw

On Wed, Nov 26, 2008 at 02:54:21PM -0800, Matt Carlson wrote:
(...)

Quoted 11 lines

Quoted 4 lines
I've run a new test on a switch I have here at home (another el-cheapo, non-manageable 100 Mbps, netgear this time). Unfortunately I cannot reproduce the problem at all. I have disabled FC on my laptop, it did not have any effect.
Disabling FC should have a positive effect, not a negative one. It might be the case that the switch does not advertise nor support FC. If that is true, you might not be able to repro the problem no matter what you did (if your problem is what I think it is). Can you check your link messages and see if it really is negotiated to off? (I see the message above, but I don't think that is with the current switch.)

yes the switch does advertuse FC :

willy@wtap:~$ dmesg|grep eth0
eth0: Tigon3 [partno(BMC5705mA3) rev 3003 PHY(5705)] (PCI:33MHz:32-bit) 10/100/1000Base-T Ethernet 00:0d:9d:91:ef:24
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1]
eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

Quoted 7 lines

Quoted 4 lines
I have disabled auto-neg and manually forced the
speed to 100/Full on my laptop, and could not reproduce the problem either (though the speed was much lower due to the switch obviously negociating 100/Half when not seeing my NWay frames).
Yes. If you force the link, both sides must be forced. The switch rightly assumes HD when bringing the link up.

I know ;-) but not seeing the problem, I started to suspect that the other switch was a little bit ill and tried to reproduce some problems I might incidently have been encountering on it.

Regards,
Willy

Frantisek Hanzlik

2008-11-27 10:06 UTC #18 lore raw

Quoted 29 lines · Willy Tarreau

On Wed, Nov 26, 2008 at 02:54:21PM -0800, Matt Carlson wrote:
(...)
Quoted 10 lines
Quoted 4 lines
I've run a new test on a switch I have here at home (another el-cheapo, non-manageable 100 Mbps, netgear this time). Unfortunately I cannot reproduce the problem at all. I have disabled FC on my laptop, it did not have any effect.
Disabling FC should have a positive effect, not a negative one. It might be the case that the switch does not advertise nor support FC. If that is true, you might not be able to repro the problem no matter what you did (if your problem is what I think it is). Can you check your link messages and see if it really is negotiated to off? (I see the message above, but I don't think that is with the current switch.)
yes the switch does advertuse FC :
willy@wtap:~$ dmesg|grep eth0
eth0: Tigon3 [partno(BMC5705mA3) rev 3003 PHY(5705)] (PCI:33MHz:32-bit) 10/100/1000Base-T Ethernet 00:0d:9d:91:ef:24
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1]
eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
Quoted 6 lines
Quoted 4 lines
I have disabled auto-neg and manually forced the
speed to 100/Full on my laptop, and could not reproduce the problem either (though the speed was much lower due to the switch obviously negociating 100/Half when not seeing my NWay frames).
Yes. If you force the link, both sides must be forced. The switch rightly assumes HD when bringing the link up.

Quoted 7 lines

I know ;-) but not seeing the problem, I started to suspect that the other switch was a little bit ill and tried to reproduce some problems I might incidently have been encountering on it.
Regards,
Willy

I got same issue two days ago on PCI-X Fiber 1000BASE-SX D-Link Adapter DGE-550SX/dl2k driver (I report it yesterday). This card work fine in old DEC Alphaserver 800/Fedora Core 5/kernel 2.6.17. We use it for our internet connections, which is limited to approx. 50 Mb/sec by ISP, then I think there should be no bottleneck in LAN card.
We just tried install new i386 based router (Core2Duo E8500/4GB RAM/ 4x Realtek 8111C and this D-Link from old Alphaserver, on Fedora 10, kernel 2.6.27.5). Router is connected to several gigabit backbones over other ois gigabit cards, then I'm not sure when any crapped internal machines could overload this DGE-550SX. Card in new router stop working after several secs/minutes - packed transmitting freeze (what is interesting, in one case after it sent exactly 8192 packets, and other cases are some multiples of 8 too) and stop transmit packets. Only workaround (but for a while) is rmmod + modprobe its dl2k driver.

/var/log/messages contains following messages:

...

Nov 25 19:04:52 ns kernel: Sundance Technology IPG Triple-Speed Ethernet 0000:09:00.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
Nov 25 19:04:52 ns kernel: 0000:09:00.0: D-Link NIC
...
Nov 25 19:05:12 ns kernel: ------------[ cut here ]------------
Nov 25 19:05:12 ns kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xda/0x12d()
Nov 25 19:05:12 ns kernel: NETDEV WATCHDOG: eth1 (Sundance Technology IPG Triple-Speed Ethernet): transmit timed out
Nov 25 19:05:12 ns kernel: Modules linked in: hwmon_vid hwmon nf_nat_ftp nf_conntrack_ftp xt_comment iptable_nat nf_nat cpufreq_ondemand acpi_cpufreq dm_multipath uinput snd_hda_intel snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device dl2k snd_pcm_oss snd_mixer_oss snd_pcm ipg snd_timer snd_page_alloc snd_hwdep snd i2c_i801 firewire_ohci i2c_core soundcore pcspkr r8169 mii firewire_core crc_itu_t raid456 async_xor async_memcpy async_tx xor raid1 [last unloaded: microcode]
Nov 25 19:05:12 ns kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5-117.fc10.i686 #1
Nov 25 19:05:12 ns kernel: [<c042bb3a>] warn_slowpath+0x4b/0x6c
Nov 25 19:05:12 ns kernel: [<c046fb00>] ? mempool_resize+0x15c/0x183
Nov 25 19:05:12 ns kernel: [<c048ca25>] ? __slab_free+0x63/0x26e
Nov 25 19:05:12 ns kernel: [<c048ce4e>] ? kmem_cache_free+0x71/0xa7
Nov 25 19:05:12 ns kernel: [<c048ca25>] ? __slab_free+0x63/0x26e
Nov 25 19:05:12 ns kernel: [<c050cb98>] ? blk_remove_plug+0x66/0x92
Nov 25 19:05:12 ns kernel: [<c050a19f>] ? elv_queue_empty+0x20/0x22
Nov 25 19:05:12 ns kernel: [<c050d29a>] ? blk_run_queue+0x28/0x2c
Nov 25 19:05:12 ns kernel: [<c059f8c5>] ? scsi_run_queue+0x250/0x27c
Nov 25 19:05:12 ns kernel: [<c0518eb6>] ? kobject_put+0x37/0x3c
Nov 25 19:05:12 ns kernel: [<c051bf6e>] ? strlcpy+0x17/0x49
Nov 25 19:05:12 ns kernel: [<c063e554>] dev_watchdog+0xda/0x12d
Nov 25 19:05:12 ns kernel: [<c05a0cef>] ? scsi_device_unbusy+0x6b/0x70
Nov 25 19:05:12 ns kernel: [<c04341f5>] run_timer_softirq+0x14b/0x1bb
Nov 25 19:05:12 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:05:12 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:05:12 ns kernel: [<c043076f>] __do_softirq+0x84/0x109
Nov 25 19:05:12 ns kernel: [<c04306eb>] ? __do_softirq+0x0/0x109
Nov 25 19:05:12 ns kernel: [<c0405eec>] do_softirq+0x77/0xdb
Nov 25 19:05:12 ns kernel: [<c04640bb>] ? handle_fasteoi_irq+0x0/0xc0
Nov 25 19:05:12 ns kernel: [<c04303d6>] irq_exit+0x44/0x83
Nov 25 19:05:12 ns kernel: [<c0405e5e>] do_IRQ+0xe7/0xfe
Nov 25 19:05:12 ns kernel: [<c0404654>] common_interrupt+0x28/0x30
Nov 25 19:05:12 ns kernel: [<c0564b95>] ? acpi_idle_enter_simple+0x162/0x19d
Nov 25 19:05:12 ns kernel: [<c0613efd>] cpuidle_idle_call+0x67/0x97
Nov 25 19:05:12 ns kernel: [<c0402c4d>] cpu_idle+0x101/0x134
Nov 25 19:05:12 ns kernel: [<c06a32de>] start_secondary+0x197/0x19f
Nov 25 19:05:12 ns kernel: =======================
Nov 25 19:05:12 ns kernel: ---[ end trace fd7fa9607e312047 ]---

=====

Nov 25 19:29:33 ns kernel: ------------[ cut here ]------------
Nov 25 19:29:33 ns kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xda/0x12d()
Nov 25 19:29:33 ns kernel: NETDEV WATCHDOG: eth1 (Sundance Technology IPG Triple-Speed Ethernet): transmit timed out
Nov 25 19:29:33 ns kernel: Modules linked in: hwmon_vid hwmon nf_nat_ftp nf_conntrack_ftp xt_comment iptable_nat nf_nat cpufreq_ondemand acpi_cpufreq dm_multipath uinput dl2k i2c_i801 pcspkr r8169 ipg mii i2c_core firewire_ohci firewire_core crc_itu_t raid456 async_xor async_memcpy async_tx xor raid1 [last unloaded: microcode]
Nov 25 19:29:33 ns kernel: Pid: 9941, comm: mail Not tainted 2.6.27.5-117.fc10.i686 #1
Nov 25 19:29:33 ns kernel: [<c042bb3a>] warn_slowpath+0x4b/0x6c
Nov 25 19:29:33 ns kernel: [<c046fb00>] ? mempool_resize+0x15c/0x183
Nov 25 19:29:33 ns kernel: [<c04b0b3b>] ? bio_free+0x40/0x44
Nov 25 19:29:33 ns kernel: [<c04b0b4d>] ? bio_fs_destructor+0xe/0x11
Nov 25 19:29:33 ns kernel: [<c04af8a5>] ? bio_put+0x26/0x28
Nov 25 19:29:33 ns kernel: [<c048a971>] ? virt_to_head_page+0x22/0x2e
Nov 25 19:29:33 ns kernel: [<c050c0d4>] ? queue_flag_clear+0x18/0x54
Nov 25 19:29:33 ns kernel: [<c050c17b>] ? __freed_request+0x6b/0x72
Nov 25 19:29:33 ns kernel: [<c0420940>] ? __enqueue_entity+0xe3/0xeb
Nov 25 19:29:33 ns kernel: [<c04223aa>] ? enqueue_entity+0x203/0x20b
Nov 25 19:29:33 ns kernel: [<c051bf6e>] ? strlcpy+0x17/0x49
Nov 25 19:29:33 ns kernel: [<c063e554>] dev_watchdog+0xda/0x12d
Nov 25 19:29:33 ns kernel: [<c0405e5e>] ? do_IRQ+0xe7/0xfe
Nov 25 19:29:33 ns kernel: [<c04341f5>] run_timer_softirq+0x14b/0x1bb
Nov 25 19:29:33 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:29:33 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:29:33 ns kernel: [<c043076f>] __do_softirq+0x84/0x109
Nov 25 19:29:33 ns kernel: [<c04306eb>] ? __do_softirq+0x0/0x109
Nov 25 19:29:33 ns kernel: [<c0405eec>] do_softirq+0x77/0xdb
Nov 25 19:29:33 ns kernel: [<c04303d6>] irq_exit+0x44/0x83
Nov 25 19:29:33 ns kernel: [<c0413ee9>] smp_apic_timer_interrupt+0x6e/0x7c
Nov 25 19:29:33 ns kernel: [<c0404759>] apic_timer_interrupt+0x2d/0x34
Nov 25 19:29:33 ns kernel: [<c041007b>] ? speedstep_target+0x23/0x7e
Nov 25 19:29:33 ns kernel: [<c0419e89>] ? native_flush_tlb_single+0x6/0x8
Nov 25 19:29:33 ns kernel: [<c041d37d>] kunmap_atomic+0x67/0xa7
Nov 25 19:29:33 ns kernel: [<c047a368>] follow_page+0x1c5/0x23b
Nov 25 19:29:33 ns kernel: [<c047b98e>] get_user_pages+0x289/0x2fe
Nov 25 19:29:33 ns kernel: [<c0493d9e>] get_arg_page+0x2d/0x80
Nov 25 19:29:33 ns kernel: [<c051dcb9>] ? strnlen_user+0x2f/0x4d
Nov 25 19:29:33 ns kernel: [<c0493eb4>] copy_strings+0xc3/0x160
Nov 25 19:29:33 ns kernel: [<c0495079>] do_execve+0x14e/0x215
Nov 25 19:29:33 ns kernel: [<c04023a3>] sys_execve+0x29/0x50
Nov 25 19:29:33 ns kernel: [<c0403c76>] syscall_call+0x7/0xb
Nov 25 19:29:33 ns kernel: [<c06a007b>] ? init_intel_cacheinfo+0x0/0x421
Nov 25 19:29:33 ns kernel: =======================
Nov 25 19:29:33 ns kernel: ---[ end trace 45c41aa8555c94fa ]---

=====

Nov 25 19:41:00 ns kernel: ------------[ cut here ]------------
Nov 25 19:41:00 ns kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xda/0x12d()
Nov 25 19:41:00 ns kernel: NETDEV WATCHDOG: eth1 (Sundance Technology IPG Triple-Speed Ethernet): transmit timed out
Nov 25 19:41:00 ns kernel: Modules linked in: hwmon_vid hwmon nf_nat_ftp nf_conntrack_ftp xt_comment iptable_nat nf_nat cpufreq_ondemand acpi_cpufreq dm_multipath uinput dl2k ipg firewire_ohci firewire_core crc_itu_t pcspkr r8169 mii i2c_i801 i2c_core raid456 async_xor async_memcpy async_tx xor raid1 [last unloaded: microcode]
Nov 25 19:41:00 ns kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5-117.fc10.i686 #1
Nov 25 19:41:00 ns kernel: [<c042bb3a>] warn_slowpath+0x4b/0x6c
Nov 25 19:41:00 ns kernel: [<c0420900>] ? __enqueue_entity+0xa3/0xeb
Nov 25 19:41:00 ns kernel: [<c04223aa>] ? enqueue_entity+0x203/0x20b
Nov 25 19:41:00 ns kernel: [<c04223ed>] ? enqueue_task_fair+0x3b/0x3f
Nov 25 19:41:00 ns kernel: [<c041d92c>] ? resched_task+0x3a/0x6e
Nov 25 19:41:00 ns kernel: [<c06a76b3>] ? _spin_unlock_irqrestore+0x22/0x38
Nov 25 19:41:00 ns kernel: [<c0426237>] ? try_to_wake_up+0x221/0x22b
Nov 25 19:41:00 ns kernel: [<c06a77cc>] ? _spin_lock_irqsave+0x29/0x30
Nov 25 19:41:00 ns kernel: [<c051bf6e>] ? strlcpy+0x17/0x49
Nov 25 19:41:00 ns kernel: [<c063e554>] dev_watchdog+0xda/0x12d
Nov 25 19:41:00 ns kernel: [<c043a716>] ? __queue_work+0x26/0x2b
Nov 25 19:41:00 ns kernel: [<c04341f5>] run_timer_softirq+0x14b/0x1bb
Nov 25 19:41:00 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:41:00 ns kernel: [<c063e47a>] ? dev_watchdog+0x0/0x12d
Nov 25 19:41:00 ns kernel: [<c043076f>] __do_softirq+0x84/0x109
Nov 25 19:41:00 ns kernel: [<c04306eb>] ? __do_softirq+0x0/0x109
Nov 25 19:41:00 ns kernel: [<c0405eec>] do_softirq+0x77/0xdb
Nov 25 19:41:00 ns kernel: [<c04303d6>] irq_exit+0x44/0x83
Nov 25 19:41:00 ns kernel: [<c0413ee9>] smp_apic_timer_interrupt+0x6e/0x7c
Nov 25 19:41:00 ns kernel: [<c0404759>] apic_timer_interrupt+0x2d/0x34
Nov 25 19:41:00 ns kernel: [<c05649e4>] ? acpi_idle_enter_bm+0x277/0x2c6
Nov 25 19:41:00 ns kernel: [<c0613efd>] cpuidle_idle_call+0x67/0x97
Nov 25 19:41:00 ns kernel: [<c0402c4d>] cpu_idle+0x101/0x134
Nov 25 19:41:00 ns kernel: [<c0697406>] rest_init+0x4e/0x50
Nov 25 19:41:00 ns kernel: =======================
Nov 25 19:41:00 ns kernel: ---[ end trace 9ce93cc9159b5214 ]---

It's there any thing for help with it?

Thx, Franta Hanzlik

Willy Tarreau

2008-11-27 20:33 UTC #19 lore raw

Quoted 38 lines · Frantisek Hanzlik

Quoted 30 lines · Willy Tarreau
On Wed, Nov 26, 2008 at 02:54:21PM -0800, Matt Carlson wrote:
(...)
Quoted 10 lines
Quoted 4 lines
I've run a new test on a switch I have here at home (another el-cheapo, non-manageable 100 Mbps, netgear this time). Unfortunately I cannot reproduce the problem at all. I have disabled FC on my laptop, it did not have any effect.
Disabling FC should have a positive effect, not a negative one. It might be the case that the switch does not advertise nor support FC. If that is true, you might not be able to repro the problem no matter what you did (if your problem is what I think it is). Can you check your link messages and see if it really is negotiated to off? (I see the message above, but I don't think that is with the current switch.)
yes the switch does advertuse FC :
willy@wtap:~$ dmesg|grep eth0
eth0: Tigon3 [partno(BMC5705mA3) rev 3003 PHY(5705)] (PCI:33MHz:32-bit) 
10/100/1000Base-T Ethernet 00:0d:9d:91:ef:24
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1]
eth0: dma_rwctrl[763f0000] dma_mask[64-bit]
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
Quoted 6 lines
Quoted 4 lines
I have disabled auto-neg and manually forced the
speed to 100/Full on my laptop, and could not reproduce the problem either (though the speed was much lower due to the switch obviously negociating 100/Half when not seeing my NWay frames).
Yes. If you force the link, both sides must be forced. The switch rightly assumes HD when bringing the link up.
Quoted 7 lines
I know ;-) but not seeing the problem, I started to suspect that the other switch was a little bit ill and tried to reproduce some problems I might incidently have been encountering on it.
Regards,
Willy

Quoted 14 lines

I got same issue two days ago on PCI-X Fiber 1000BASE-SX D-Link Adapter DGE-550SX/dl2k driver (I report it yesterday). This card work fine in old DEC Alphaserver 800/Fedora Core 5/kernel 2.6.17. We use it for our internet connections, which is limited to approx. 50 Mb/sec by ISP, then I think there should be no bottleneck in LAN card.
We just tried install new i386 based router (Core2Duo E8500/4GB RAM/ 4x Realtek 8111C and this D-Link from old Alphaserver, on Fedora 10, kernel 2.6.27.5). Router is connected to several gigabit backbones over other ois gigabit cards, then I'm not sure when any crapped internal machines could overload this DGE-550SX. Card in new router stop working after several secs/minutes - packed transmitting freeze (what is interesting, in one case after it sent exactly 8192 packets, and other cases are some multiples of 8 too) and stop transmit packets. Only workaround (but for a while) is rmmod + modprobe its dl2k driver.

I had been facing this same issue 5-6 years ago when I tried my first DL2K. It always stopped transmitting after a few hundreds of packets on my machine while it was OK on another one. Mine was dual-athlon with an AMD chipset. A working one was a P3-933 with a ServerWorks chipset. I noticed that it would only lock up when plugged into a 64-bit slot, and it was fine in 32-bit slots (though it did work well on the P3 in a 64-bit slot). It was such a crappy chip that I finally threw it away, so I could not compare anymore. I'd bet that your new machine is too fast for this beast and it triggers some bugs in the chip, probably on the PCI side.

Regards,
Willy

Willy Tarreau

2008-12-02 22:55 UTC #20 lore raw

In reply to Matt Carlson

Hi Matt,

I ran a lot of tests last night. I have a few more information. The issue sometimes takes longer to reproduce so it caused me to identify wrong culprits among the 29 patches affecting tg3 between 2.6.25 and 2.6.27.7. I was finally able to reproduce the issue by running the plain 2.6.25 driver (v3.90) on 2.6.27.7, but not at all when running on 2.6.25, even after ten minutes (in 2.6.27.7, it takes between 5s and 1mn to get a tx timeout).

Later, I noticed that 2.6.27's driver uses libphy, which was never removed between tests. I wonder if it can interfer with my tests. Maybe it initializes the phy differently from plain 2.6.25, causing delayed issues, I don't know. Unfortunately, I cannot run 2.6.27's driver on 2.6.25 because of the libphy dependency (that's how I discovered it).

I'm also now 100% certain that enabling/disabling FC does not change anything with either kernel. So unless the hardware still interpretes pause frames when disabled, it should not come from there.

I suspect that the switch is getting ill : The problem happens more often when it's been transfering at full speed for some time. Since it's a cheap one lying on a desk, it might have burned out capacitors in it causing some randomly corrupt frames to go out from time to time (maybe even pause frames preventing the NIC from sending). That was also a problem for my tests, because after patching/unpatching and compilation phases, it had some time to rest and took longer to reproduce the issue.

I will re-run some tests on 2.6.27 + tg3 v3.90 (from 2.6.25) without ever loading libphy from the power up, in order to clearly identify if the problem is caused by the driver or something else in the kernel. If it's something else, the bisect will take a few weeks since I'm not there long enough to run about 15 full builds and wait long enough for the problem to (not) occur.

But I'm keeping hope, there's no reason not to find it!

Regards,
Willy

Matt Carlson

2008-11-20 03:00 UTC #21 lore raw

In reply to Roger Heflin

Quoted 63 lines · Roger Heflin

Quoted 8 lines · Peter Zijlstra
(netdev CC'ed)
Quoted 3 lines · Roger Heflin
I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.
How do more recent kernels do?
I did not try more recent kernels, more testing seems to indicate that at the very least it the bug depends on a certain version of either tg3 and/or firmware to happen, as my second tg3 port does not
have it happen. More about this below.
Quoted 33 lines
Quoted 33 lines
So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.
Unloading all of the network modules and reloading them did not correct the problem.
Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?
http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238
The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).
Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.
I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.
Quoted 5 lines
Quoted 5 lines
The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.
I am first turning off all offload capabilities on tg3 and going to see if that changes anything.
This made no difference in the error.
Quoted 3 lines
Quoted 3 lines
The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.
Did not try.
Quoted 1 line
Quoted 1 line

Quoted 9 lines

Quoted 2 lines
Quoted 2 lines
I also have a second tg3 port that is slightly different, so I may try that eventually.
I tried this, and with the second port I don't appear to be getting
the error. The first port is a 5789-v3.29a and the second port is a
5788-v3.04, I know the first port is faster (pcie-x1) than the second
port (pci bus-built-in, unknown exact connection). The second port
will sustain about 50MB/second, were as the first port will get
Quoted 1 line
90MB/second.

The 5789 is a PCIe device, the 5788 is a PCI device. There's a lot of room for differences between the two devices.

Quoted 4 lines

It seems to me to likely be the firmware on the tg3, and it would seem unlikely that the driver could do anything more than work around the issue that is in the firmware, and currently my system works on the second port, and the second port is fast enough for my needs.

I don't think the 5789 should have any firmware running at the time of the failure. There's always the possibility that there is a problem with the bootcode, but I'm guessing the problem is elsewhere.

Quoted 5 lines

If someone else runs into this issue, since I have 2 ports I would be able to do some testing on it, right now my first port is locked up, and the machine is running fine on the second port.
lspci -vvv for the first (bad) port:

Ah. There it is.

Quoted 24 lines

02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit Ethernet PCI Express (rev 11)
Subsystem: Foxconn International, Inc. Unknown device 0cc1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K] Expansion ROM at <ignored> [disabled]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D3 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: 0101b8102a0f7b0c Data: f21e
Capabilities: [d0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <4us, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

Quoted 8 lines

Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s <2us, L1 <64us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Virtual Channel

Hmmm. No smoking gun. Perhaps the register dump will help.

Quoted 4 lines

Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out

Usually the tg3_tx_timeout function dumps a few registers before resetting the chip, but I don't see that here. Have you seen any dumps since then?

Quoted 52 lines

Quoted 37 lines
Quoted 37 lines
Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs 
w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 
[uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
Quoted 9 lines
Quoted 8 lines
Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Roger Heflin

author

2008-11-20 10:07 UTC #22 lore raw

Quoted 66 lines · Matt Carlson

Quoted 56 lines · Roger Heflin
Quoted 7 lines · Peter Zijlstra
(netdev CC'ed)
Quoted 3 lines · Roger Heflin
I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.
How do more recent kernels do?
I did not try more recent kernels, more testing seems to indicate that at the very least it the bug depends on a certain version of either tg3 and/or firmware to happen, as my second tg3 port does not
have it happen. More about this below.
Quoted 32 lines
Quoted 32 lines
So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.
Unloading all of the network modules and reloading them did not correct the problem.
Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?
http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238
The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).
Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.
I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.
Quoted 5 lines
Quoted 5 lines
The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.
I am first turning off all offload capabilities on tg3 and going to see if that changes anything.
This made no difference in the error.
Quoted 2 lines
Quoted 2 lines
The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.
Did not try.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
I also have a second tg3 port that is slightly different, so I may try that eventually.
I tried this, and with the second port I don't appear to be getting
the error. The first port is a 5789-v3.29a and the second port is a
5788-v3.04, I know the first port is faster (pcie-x1) than the second
port (pci bus-built-in, unknown exact connection). The second port
will sustain about 50MB/second, were as the first port will get
Quoted 1 line
90MB/second.

Quoted 2 lines

The 5789 is a PCIe device, the 5788 is a PCI device. There's a lot of room for differences between the two devices.

I expected that there were going to be large changes between the two.

Quoted 9 lines

Quoted 5 lines
It seems to me to likely be the firmware on the tg3, and it would seem unlikely that the driver could do anything more than work around the issue that is in the firmware, and currently my system works on the second port, and the second port is fast enough for my needs.
I don't think the 5789 should have any firmware running at the time of the failure. There's always the possibility that there is a problem with the bootcode, but I'm guessing the problem is elsewhere.

Where does the offload code and other similar stuff run?

I know on a tg3 before I had difficulties when a BMC card was in the system as the tg3 had a bug that it grabbed UDP packets that it though were going to the BMC card, in that case it was incorrectly thinking that UDP data was a port
number (this is only the case on the first UDP packet of a set) when it was actually data (on the UDP packets after the first one) in some of the udp packets. Is that done from the firmware, or is that completely done in the hardware and basically not fixable?

Quoted 41 lines

Quoted 6 lines
If someone else runs into this issue, since I have 2 ports I would be able to do some testing on it, right now my first port is locked up, and the machine is running fine on the second port.
lspci -vvv for the first (bad) port:
Ah. There it is.
Quoted 23 lines
02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit Ethernet PCI Express (rev 11)
Subsystem: Foxconn International, Inc. Unknown device 0cc1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K]
Expansion ROM at <ignored> [disabled]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D3 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: 0101b8102a0f7b0c Data: f21e
Capabilities: [d0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <4us, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Quoted 8 lines
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s <2us, L1 <64us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Virtual Channel

Quoted 2 lines

Hmmm. No smoking gun. Perhaps the register dump will help.

driver: tg3
version: 3.94
firmware-version: 5789-v3.29a
bus-info: 0000:02:00.0

tg3.c:v3.94 (August 14, 2008)
tg3 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19 tg3 0000:02:00.0: setting latency timer to 64
tg3 0000:05:01.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.

Right now I brought the interface back up (it is still broken) and setup a network ip on it that other machines can ping.

The registers are included at the end of the email.

Quoted 8 lines

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Usually the tg3_tx_timeout function dumps a few registers before resetting the chip, but I don't see that here. Have you seen any dumps since then?

Is this the dump?

Nov 12 14:58:13 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 14:58:13 computer kernel: tg3: DEBUG: MAC_TX_STATUS[00000008] MAC_RX_STATUS[00000006]
Nov 12 14:58:13 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000010] WDMAC_STATUS[00000000]
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=1400 enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: eth0: Link is down.
Nov 12 14:58:16 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 14:58:16 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 15:20:37 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 15:20:37 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000000]
Nov 12 15:20:37 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000000]
Nov 12 15:20:37 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
Nov 12 15:20:37 computer kernel: tg3: eth0: Link is down.
Nov 12 15:20:40 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 15:20:40 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 15:48:16 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 15:48:16 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000000]
Nov 12 15:48:16 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000000]
Nov 12 15:48:16 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
Nov 12 15:48:16 computer kernel: tg3: eth0: Link is down.
Nov 12 15:48:19 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 15:48:19 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:10:44 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:10:44 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000000]
Nov 12 16:10:44 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000000]
Nov 12 16:10:44 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
Nov 12 16:10:44 computer kernel: tg3: eth0: Link is down.
Nov 12 16:10:47 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:10:47 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:18:43 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:18:43 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000000]
Nov 12 16:18:43 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000000]
Nov 12 16:18:43 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
Nov 12 16:18:43 computer kernel: tg3: eth0: Link is down.
Nov 12 16:18:46 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:18:46 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:27:02 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:27:02 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000000]
Nov 12 16:27:02 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000000]
Nov 12 16:27:02 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
Nov 12 16:27:02 computer kernel: tg3: eth0: Link is down.
Nov 12 16:27:06 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:27:06 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:35:17 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:35:17 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b] MAC_RX_STATUS[00000000]
Nov 12 16:35:17 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000] WDMAC_STATUS[00000000]
Nov 12 16:35:17 computer kernel: tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
Nov 12 16:35:17 computer kernel: tg3: eth0: Link is down.
Nov 12 16:35:20 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:35:20 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.

Quoted 20 lines

Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 [uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e

Quoted 10 lines

Quoted 8 lines

Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5

Offset Value
------ ----------
0x0000 0x169d14e4
0x0004 0x00100006
0x0008 0x02000011
0x000c 0x00000008
0x0010 0xfd8f0004
0x0014 0x00000000
0x0018 0x00000000
0x001c 0x00000000
0x0020 0x00000000
0x0024 0x00000000
0x0028 0x00000007
0x002c 0x0cc1105b
0x0030 0x02aa0000
0x0034 0x00000048
0x0038 0x00000000
0x003c 0x0000010b
0x0040 0x00000000
0x0044 0x00000000
0x0048 0xc0025001
0x004c 0x64002000
0x0050 0x00d05803
0x0054 0x587d4b8a
0x0058 0x0086d005
0x005c 0x2a0f7b0c
0x0060 0x0101b810
0x0064 0x0000f21e
0x0068 0x41010298
0x006c 0x76180000
0x0070 0x000010f2
0x0074 0x000000c0
0x0078 0x00000050
0x007c 0x00000000
0x0080 0x00d05803
0x0084 0x00000000
0x0088 0x04130034
0x008c 0x3c081082
0x0090 0x01000609
0x0094 0x02000000
0x0098 0x00000000
0x009c 0x000000c8
0x00a0 0x00000000
0x00a4 0x00000000
0x00a8 0x00000000
0x00ac 0x00000000
0x00b0 0x00000000
0x00b4 0x00000000
0x00b8 0x00000000
0x00bc 0x00000000
0x00c0 0x00000000
0x00c4 0x00000000
0x00c8 0x00000000
0x00cc 0x00000000
0x00d0 0x00000000
0x00d4 0x00000000
0x00d8 0x00000000
0x00dc 0x00000000
0x00e0 0x00000000
0x00e4 0x00000000
0x00e8 0x00000000
0x00ec 0x00000000
0x00f0 0x00000000
0x00f4 0x00000000
0x00f8 0x00000000
0x00fc 0x00000000
0x0100 0x00000000
0x0104 0x00000000
0x0108 0x00000000
0x010c 0x00000000
0x0110 0x00000000
0x0114 0x00000000
0x0118 0x00000000
0x011c 0x00000000
0x0120 0x00000000
0x0124 0x00000000
0x0128 0x00000000
0x012c 0x00000000
0x0130 0x00000000
0x0134 0x00000000
0x0138 0x00000000
0x013c 0x00000000
0x0140 0x00000000
0x0144 0x00000000
0x0148 0x00000000
0x014c 0x00000000
0x0150 0x00000000
0x0154 0x00000000
0x0158 0x00000000

0x0200 0x00000000
0x0204 0x02000000
0x0208 0x00000000
0x020c 0x02000000
0x0210 0x00000000
0x0214 0x02000000
0x0218 0x00000000
0x021c 0x02000000
0x0220 0x00000000
0x0224 0x02000000
0x0228 0x00000000
0x022c 0x02000000
0x0230 0x00000000
0x0234 0x02000000
0x0238 0x00000000
0x023c 0x02000000
0x0240 0x00000000
0x0244 0x02000000
0x0248 0x00000000
0x024c 0x02000000
0x0250 0x00000000
0x0254 0x02000000
0x0258 0x00000000
0x025c 0x02000000
0x0260 0x00000000
0x0264 0x02000000
0x0268 0x00000000
0x026c 0x000000c8
0x0270 0x00000000
0x0274 0x02000000
0x0278 0x00000000
0x027c 0x02000000
0x0280 0x00000000
0x0284 0x00000000
0x0288 0x00000000
0x028c 0x02000000
0x0290 0x00000000
0x0294 0x02000000
0x0298 0x00000000
0x029c 0x02000000
0x02a0 0x00000000
0x02a4 0x02000000
0x02a8 0x00000000
0x02ac 0x02000000
0x02b0 0x00000000
0x02b4 0x02000000
0x02b8 0x00000000
0x02bc 0x02000000
0x02c0 0x00000000
0x02c4 0x02000000
0x02c8 0x00000000
0x02cc 0x02000000
0x02d0 0x00000000
0x02d4 0x02000000
0x02d8 0x00000000
0x02dc 0x02000000
0x02e0 0x00000000
0x02e4 0x02000000
0x02e8 0x00000000
0x02ec 0x02000000
0x02f0 0x00000000
0x02f4 0x02000000
0x02f8 0x00000000
0x02fc 0x02000000
0x0300 0x00000000
0x0304 0x00000024
0x0308 0x00000000
0x030c 0x02000024
0x0310 0x00000000
0x0314 0x02000024
0x0318 0x00000000
0x031c 0x02000024
0x0320 0x00000000
0x0324 0x02000024
0x0328 0x00000000
0x032c 0x02000024
0x0330 0x00000000
0x0334 0x02000024
0x0338 0x00000000
0x033c 0x02000024
0x0340 0x00000000
0x0344 0x02000024
0x0348 0x00000000
0x034c 0x02000024
0x0350 0x00000000
0x0354 0x02000024
0x0358 0x00000000
0x035c 0x02000024
0x0360 0x00000000
0x0364 0x02000024
0x0368 0x00000000
0x036c 0x02000024
0x0370 0x00000000
0x0374 0x02000024
0x0378 0x00000000
0x037c 0x02000024
0x0380 0x00000000
0x0384 0x00000000
0x0388 0x00000000
0x038c 0x02000000
0x0390 0x00000000
0x0394 0x02000000
0x0398 0x00000000
0x039c 0x02000000
0x03a0 0x00000000
0x03a4 0x02000000
0x03a8 0x00000000
0x03ac 0x02000000
0x03b0 0x00000000
0x03b4 0x02000000
0x03b8 0x00000000
0x03bc 0x02000000
0x03c0 0x00000000
0x03c4 0x02000000
0x03c8 0x00000000
0x03cc 0x02000000
0x03d0 0x00000000
0x03d4 0x02000000
0x03d8 0x00000000
0x03dc 0x02000000
0x03e0 0x00000000
0x03e4 0x02000000
0x03e8 0x00000000
0x03ec 0x02000000
0x03f0 0x00000000
0x03f4 0x02000000
0x03f8 0x00000000
0x03fc 0x02000000

0x0400 0x00e04808
0x0404 0x00400000
0x0408 0x00001000
0x040c 0x00000880
0x0410 0x00000001
0x0414 0x6ce59c51
0x0418 0x00000001
0x041c 0x6ce59c51
0x0420 0x00000001
0x0424 0x6ce59c51
0x0428 0x00000001
0x042c 0x6ce59c51
0x0430 0x00000000
0x0434 0x00000000
0x0438 0x000000da
0x043c 0x000005f2
0x0440 0x00000000
0x0444 0x00000000
0x0448 0x00000000
0x044c 0x0825c5e1
0x0450 0x00000001
0x0454 0x000c0000
0x0458 0x00000000
0x045c 0x00000012
0x0460 0x00000008
0x0464 0x00002620
0x0468 0x00000406
0x046c 0x00000000
0x0470 0x80000000
0x0474 0x00000000
0x0478 0x00000080
0x047c 0x40200000
0x0480 0x42000000
0x0484 0x7fffffff
0x0488 0x06000004
0x048c 0x7fffffff
0x0490 0x00000000
0x0494 0x00000000
0x0498 0x00000000
0x049c 0x00000000
0x04a0 0x00000000
0x04a4 0x00000000
0x04a8 0x00000000
0x04ac 0x00000000
0x04b0 0x00000000
0x04b4 0x00000000
0x04b8 0x00000000
0x04bc 0x00000000
0x04c0 0x00000000
0x04c4 0x00000000
0x04c8 0x00000000
0x04cc 0x00000000
0x04d0 0x00000000
0x04d4 0x00000000
0x04d8 0x00000000
0x04dc 0x00000000
0x04e0 0x00000000
0x04e4 0x00000000
0x04e8 0x00000000
0x04ec 0x00000000
0x04f0 0x00000000
0x04f4 0x00000000
0x04f8 0x00000000
0x04fc 0x00000000
0x0500 0x00000008
0x0504 0x00000002
0x0508 0x00000000
0x050c 0x00000000
0x0510 0x00000000
0x0514 0x00000000
0x0518 0x00000000
0x051c 0x00000000
0x0520 0x00000000
0x0524 0x00000000
0x0528 0x00000000
0x052c 0x00000000
0x0530 0x00000000
0x0534 0x00000000
0x0538 0x00000000
0x053c 0x00000000
0x0540 0x00000000
0x0544 0x00000000
0x0548 0x00000000
0x054c 0x00000000
0x0550 0x00000000
0x0554 0x00000000
0x0558 0x00000000
0x055c 0x00000000
0x0560 0x00000000
0x0564 0x00000000
0x0568 0x00000000
0x056c 0x00000000
0x0570 0x00000000
0x0574 0x00000000
0x0578 0x00000000
0x057c 0x00000000
0x0580 0x00000000
0x0584 0x00000000
0x0588 0x00000000
0x058c 0x00000000
0x0590 0x00e00000
0x0594 0x00000000
0x0598 0x00000000
0x059c 0x00000000
0x05a0 0x00000000
0x05a4 0x00000000
0x05a8 0x00000000
0x05ac 0x00000000
0x05b0 0x00000000
0x05b4 0x00000000
0x05b8 0x00000000
0x05bc 0x00000000
0x05c0 0x00000000
0x05c4 0x00000000
0x05c8 0x00000000
0x05cc 0x00000000
0x05d0 0x00000000
0x05d4 0x00000000
0x05d8 0x00000000
0x05dc 0x00000000
0x05e0 0x00000000
0x05e4 0x00000000
0x05e8 0x00000000
0x05ec 0x00000000
0x05f0 0x00000000
0x05f4 0x00000000
0x05f8 0x00000000
0x05fc 0x00000000
0x0600 0xffffffff
0x0604 0x00f80011
0x0608 0x00000000
0x060c 0x00001f04
0x0610 0xffffffff
0x0614 0x00000000
0x0618 0x07c00004
0x061c 0x30000000
0x0620 0x00000040
0x0624 0x00000000
0x0628 0x00000000
0x062c 0x00000000
0x0630 0x00000000
0x0634 0x00000000
0x0638 0x00000000
0x063c 0x00000000
0x0640 0x00000000
0x0644 0x00000000
0x0648 0x00000000
0x064c 0x00000000
0x0650 0x00000000
0x0654 0x00000000
0x0658 0x00000000
0x065c 0x00000000
0x0660 0x00000000
0x0664 0x00000000
0x0668 0x00000000
0x066c 0x00000000
0x0670 0x00000000
0x0674 0x00000000
0x0678 0x00000000
0x067c 0x00000000
0x0680 0x00000000
0x0684 0x00000000
0x0688 0x00000000
0x068c 0x00000000
0x0690 0x00000000
0x0694 0x00000000
0x0698 0x00000000
0x069c 0x00000000
0x06a0 0x00000000
0x06a4 0x00000000
0x06a8 0x00000000
0x06ac 0x00000000
0x06b0 0x00000000
0x06b4 0x00000000
0x06b8 0x00000000
0x06bc 0x00000000
0x06c0 0x00000000
0x06c4 0x00000000
0x06c8 0x00000000
0x06cc 0x00000000
0x06d0 0x00000000
0x06d4 0x00000000
0x06d8 0x00000000
0x06dc 0x00000000
0x06e0 0x00000000
0x06e4 0x00000000
0x06e8 0x00000000
0x06ec 0x00000000
0x06f0 0x00000000
0x06f4 0x00000000
0x06f8 0x00000000
0x06fc 0x00000000
0x0700 0x00000000
0x0704 0x00000000
0x0708 0x00000000
0x070c 0x00000000
0x0710 0x00000000
0x0714 0x00000000
0x0718 0x00000000
0x071c 0x00000000
0x0720 0x00000000
0x0724 0x00000000
0x0728 0x00000000
0x072c 0x00000000
0x0730 0x00000000
0x0734 0x00000000
0x0738 0x00000000
0x073c 0x00000000
0x0740 0x00000000
0x0744 0x00000000
0x0748 0x00000000
0x074c 0x00000000
0x0750 0x00000000
0x0754 0x00000000
0x0758 0x00000000
0x075c 0x00000000
0x0760 0x00000000
0x0764 0x00000000
0x0768 0x00000000
0x076c 0x00000000
0x0770 0x00000000
0x0774 0x00000000
0x0778 0x00000000
0x077c 0x00000000
0x0780 0x00000000
0x0784 0x00000000
0x0788 0x00000000
0x078c 0x00000000
0x0790 0x00000000
0x0794 0x00000000
0x0798 0x00000000
0x079c 0x00000000
0x07a0 0x00000000
0x07a4 0x00000000
0x07a8 0x00000000
0x07ac 0x00000000
0x07b0 0x00000000
0x07b4 0x00000000
0x07b8 0x00000000
0x07bc 0x00000000
0x07c0 0x00000000
0x07c4 0x00000000
0x07c8 0x00000000
0x07cc 0x00000000
0x07d0 0x00000000
0x07d4 0x00000000
0x07d8 0x00000000
0x07dc 0x00000000
0x07e0 0x00000000
0x07e4 0x00000000
0x07e8 0x00000000
0x07ec 0x00000000
0x07f0 0x00000000
0x07f4 0x00000000
0x07f8 0x00000000
0x07fc 0x00000000
0x0800 0x00000000
0x0804 0xffffffff
0x0808 0x00000000
0x080c 0x00000000
0x0810 0x00000000
0x0814 0xffffffff
0x0818 0x00000000
0x081c 0x00000000
0x0820 0x00000000
0x0824 0x00000000
0x0828 0xffffffff
0x082c 0x00000000
0x0830 0x00000000
0x0834 0xffffffff
0x0838 0xffffffff
0x083c 0xffffffff
0x0840 0xffffffff
0x0844 0xffffffff
0x0848 0xffffffff
0x084c 0xffffffff
0x0850 0xffffffff
0x0854 0xffffffff
0x0858 0xffffffff
0x085c 0xffffffff
0x0860 0xffffffff
0x0864 0xffffffff
0x0868 0xffffffff
0x086c 0x00000000
0x0870 0x00000000
0x0874 0x00000000
0x0878 0x00000000
0x087c 0x00000000
0x0880 0x00000000
0x0884 0x00000c80
0x0888 0x00000000
0x088c 0x00000000
0x0890 0x00000000
0x0894 0x00000000
0x0898 0x00000000
0x089c 0x00000000
0x08a0 0x00000000
0x08a4 0x00000000
0x08a8 0x00000000
0x08ac 0x00000000
0x08b0 0x00000000
0x08b4 0x00000000
0x08b8 0x00000000
0x08bc 0x00000000
0x08c0 0x00000000
0x08c4 0x00000000
0x08c8 0x00000000
0x08cc 0x00000000
0x08d0 0x00000000
0x08d4 0x00000000
0x08d8 0x00000000
0x08dc 0x00000000
0x08e0 0x00000000
0x08e4 0x00000000
0x08e8 0x00000000
0x08ec 0x00000000

0x0c00 0x0000000a
0x0c04 0x00000000
0x0c08 0x00000003
0x0c0c 0x00000001
0x0c10 0x00000000
0x0c14 0x00000000
0x0c18 0x00000000
0x0c1c 0x00000000
0x0c20 0x00000000
0x0c24 0x00000000
0x0c28 0x00000000
0x0c2c 0x00000000
0x0c30 0x00000000
0x0c34 0x00000000
0x0c38 0x00000000
0x0c3c 0x00000000
0x0c40 0x00000000
0x0c44 0x00000000
0x0c48 0x00000000
0x0c4c 0x00000000
0x0c50 0x00000000
0x0c54 0x00000000
0x0c58 0x00000000
0x0c5c 0x00000000
0x0c60 0x00000000
0x0c64 0x00000000
0x0c68 0x00000000
0x0c6c 0x00000000
0x0c70 0x00000000
0x0c74 0x00000000
0x0c78 0x00000000
0x0c7c 0x00000000
0x0c80 0x00000000
0x0c84 0x00000000
0x0c88 0x00000000
0x0c8c 0x00000000
0x0c90 0x00000000
0x0c94 0x00000000
0x0c98 0x00000000
0x0c9c 0x00000000
0x0ca0 0x00000000
0x0ca4 0x00000000
0x0ca8 0x00000000
0x0cac 0x00000000
0x0cb0 0x00000000
0x0cb4 0x00000000
0x0cb8 0x00000000
0x0cbc 0x00000000
0x0cc0 0x00000000
0x0cc4 0x00000000
0x0cc8 0x00000000
0x0ccc 0x00000000
0x0cd0 0x00000000
0x0cd4 0x00000000
0x0cd8 0x00000000
0x0cdc 0x00000000

0x1000 0x00000002

0x1400 0x00000006
0x1404 0x00000000
0x1408 0x00000000
0x140c 0x00000000
0x1410 0x00000000
0x1414 0x00000000
0x1418 0x00000000
0x141c 0x00000000
0x1420 0x00000000
0x1424 0x00000000
0x1428 0x00000000
0x142c 0x00000000
0x1430 0x00000000
0x1434 0x00000000
0x1438 0x00000000
0x143c 0x00000000
0x1440 0x00000000
0x1444 0x00000000
0x1448 0x00000000
0x144c 0x00000000
0x1450 0x00000000
0x1454 0x00000000
0x1458 0x00000000
0x145c 0x00000000
0x1460 0x00000000
0x1464 0x00000000
0x1468 0x00000000
0x146c 0x00000000
0x1470 0x00000000
0x1474 0x00000000
0x1478 0x00000000
0x147c 0x00000000

0x1800 0x00000016
0x1804 0x00000000
0x1808 0x00000001
0x180c 0x00000000
0x1810 0x00000000
0x1814 0x00000000
0x1818 0x00000000
0x181c 0x00000000
0x1820 0x00000000
0x1824 0x00000000
0x1828 0x00000000
0x182c 0x00000000
0x1830 0x00000000
0x1834 0x00000000
0x1838 0x00000000
0x183c 0x00000000
0x1840 0x00000000
0x1844 0x00000000

0x1c00 0x00000002

0x2000 0x00000002
0x2004 0x00000000
0x2008 0x00000000
0x200c 0x00000000
0x2010 0x00000181
0x2014 0x00000001
0x2018 0x00790407
0x201c 0x00000000
0x2020 0x00000000
0x2024 0x00000000
0x2028 0x00000000
0x202c 0x00000000
0x2030 0x00000000
0x2034 0x00000000
0x2038 0x00000000
0x203c 0x00000000
0x2040 0x00000000
0x2044 0x00000000
0x2048 0x00000000
0x204c 0x00000000
0x2050 0x00000000
0x2054 0x00000000
0x2058 0x00000000
0x205c 0x00000000
0x2060 0x00000000
0x2064 0x00000000
0x2068 0x00000000
0x206c 0x00000000
0x2070 0x00000000
0x2074 0x00000000
0x2078 0x00000000
0x207c 0x00000000
0x2080 0x00000000
0x2084 0x00000000
0x2088 0x00000000
0x208c 0x00000000
0x2090 0x00000000
0x2094 0x00000000
0x2098 0x00000000
0x209c 0x00000000
0x20a0 0x00000000
0x20a4 0x00000000
0x20a8 0x00000000
0x20ac 0x00000000
0x20b0 0x00000000
0x20b4 0x00000000
0x20b8 0x00000000
0x20bc 0x00000000
0x20c0 0x00000000
0x20c4 0x00000000
0x20c8 0x00000000
0x20cc 0x00000000
0x20d0 0x00000000
0x20d4 0x00000000
0x20d8 0x00000000
0x20dc 0x00000000
0x20e0 0x00000000
0x20e4 0x00000000
0x20e8 0x00000000
0x20ec 0x00000000
0x20f0 0x00000000
0x20f4 0x00000000
0x20f8 0x00000000
0x20fc 0x00000000
0x2100 0x00086633
0x2104 0x00090683
0x2108 0x0000004d
0x210c 0x00000000
0x2110 0x00086633
0x2114 0x00090683
0x2118 0x0000004d
0x211c 0x00000000
0x2120 0x00086633
0x2124 0x00090683
0x2128 0x0000004d
0x212c 0x00000000
0x2130 0x00086633
0x2134 0x00090683
0x2138 0x0000004d
0x213c 0x00000000
0x2140 0x00086633
0x2144 0x00090683
0x2148 0x0000004d
0x214c 0x00000000
0x2150 0x00086633
0x2154 0x00090683
0x2158 0x0000004d
0x215c 0x00000000
0x2160 0x00086633
0x2164 0x00090683
0x2168 0x0000004d
0x216c 0x00000000
0x2170 0x00086633
0x2174 0x00090683
0x2178 0x0000004d
0x217c 0x00000000
0x2180 0x00086633
0x2184 0x00090683
0x2188 0x0000004d
0x218c 0x00000000
0x2190 0x00086633
0x2194 0x00090683
0x2198 0x0000004d
0x219c 0x00000000
0x21a0 0x00086633
0x21a4 0x00090683
0x21a8 0x0000004d
0x21ac 0x00000000
0x21b0 0x00086633
0x21b4 0x00090683
0x21b8 0x0000004d
0x21bc 0x00000000
0x21c0 0x00086633
0x21c4 0x00090683
0x21c8 0x0000004d
0x21cc 0x00000000
0x21d0 0x00086633
0x21d4 0x00090683
0x21d8 0x0000004d
0x21dc 0x00000000
0x21e0 0x00086633
0x21e4 0x00090683
0x21e8 0x0000004d
0x21ec 0x00000000
0x21f0 0x00086633
0x21f4 0x00090683
0x21f8 0x0000004d
0x21fc 0x00000000
0x2200 0x00000037
0x2204 0x00000000
0x2208 0x00000000
0x220c 0x00000000
0x2210 0x00000000
0x2214 0x00000000
0x2218 0x00000000
0x221c 0x00000000
0x2220 0x00000000
0x2224 0x00000000
0x2228 0x00000000
0x222c 0x00000000
0x2230 0x00000000
0x2234 0x00000000
0x2238 0x00000000
0x223c 0x00000000
0x2240 0x00000000
0x2244 0x00000000
0x2248 0x00000000
0x224c 0x00000000
0x2250 0x00000000
0x2254 0x00000000
0x2258 0x00000000

0x2400 0x00000012
0x2404 0x00000000
0x2408 0x00000000
0x240c 0x00000000
0x2410 0x00000000
0x2414 0x00000000
0x2418 0x00000000
0x241c 0x00000000
0x2420 0x00000000
0x2424 0x00000000
0x2428 0x00000000
0x242c 0x00000000
0x2430 0x00000000
0x2434 0x00000000
0x2438 0x00000000
0x243c 0x00000000
0x2440 0x00000000
0x2444 0x00000000
0x2448 0x00000000
0x244c 0x00000000
0x2450 0x00000000
0x2454 0x030d4000
0x2458 0x02000000
0x245c 0x00006000
0x2460 0x00000000
0x2464 0x00000000
0x2468 0x00000000
0x246c 0x00000000
0x2470 0x00000000
0x2474 0x00000000
0x2478 0x00000000
0x247c 0x00000000
0x2480 0x00000000
0x2484 0x00000000
0x2488 0x00000000
0x248c 0x00000000
0x2490 0x00000000
0x2494 0x00000000
0x2498 0x00000000
0x249c 0x00000000
0x24a0 0x00000000
0x24a4 0x00000000
0x24a8 0x00000000
0x24ac 0x00000000
0x24b0 0x00000000
0x24b4 0x00000000
0x24b8 0x00000000
0x24bc 0x00000000
0x24c0 0x08000010

0x2800 0x00000006

0x2c00 0x00000006
0x2c04 0x00000000
0x2c08 0x00000000
0x2c0c 0x0000007f
0x2c10 0x00000000
0x2c14 0x00000000
0x2c18 0x00000008
0x2c1c 0x00000000

0x3000 0x00000006
0x3004 0x00000000
0x3008 0x00000000
0x300c 0x00000000
0x3010 0x00000000

0x3400 0x00000000
0x3404 0x00000000

0x3800 0x00000000
0x3804 0x00000000

0x3c00 0x00000302
0x3c04 0x00000000
0x3c08 0x00000064
0x3c0c 0x00000048
0x3c10 0x0000000a
0x3c14 0x00000035
0x3c18 0x00000000
0x3c1c 0x00000000
0x3c20 0x0000000a
0x3c24 0x00000005
0x3c28 0x00000000
0x3c2c 0x00000000
0x3c30 0x00000000
0x3c34 0x00000000
0x3c38 0x00000000
0x3c3c 0x03270000
0x3c40 0x00000000
0x3c44 0x00000b00
0x3c48 0x00000000
0x3c4c 0x00000000
0x3c50 0x00000000
0x3c54 0x00000000
0x3c58 0x00000000
0x3c5c 0x00000000
0x3c60 0x00000000
0x3c64 0x00000000
0x3c68 0x00000000
0x3c6c 0x00000000
0x3c70 0x00000000
0x3c74 0x00000000
0x3c78 0x00000000
0x3c7c 0x00000000
0x3c80 0x00000000
0x3c84 0x00000000
0x3c88 0x00000000
0x3c8c 0x00000000
0x3c90 0x00000000
0x3c94 0x00000000
0x3c98 0x00000000
0x3c9c 0x00000000
0x3ca0 0x00000000
0x3ca4 0x00000000
0x3ca8 0x00000000
0x3cac 0x00000000
0x3cb0 0x00000000
0x3cb4 0x00000000
0x3cb8 0x00000000
0x3cbc 0x00000000
0x3cc0 0x00000000
0x3cc4 0x00000000
0x3cc8 0x00000000
0x3ccc 0x00000000
0x3cd0 0x00000000
0x3cd4 0x00000000
0x3cd8 0x00000000
0x3cdc 0x00000000
0x3ce0 0x00000000
0x3ce4 0x00000000
0x3ce8 0x00000000
0x3cec 0x00000000
0x3cf0 0x00000000
0x3cf4 0x00000000
0x3cf8 0x00000000
0x3cfc 0x00000000

0x4000 0x00000002
0x4004 0x00000000
0x4008 0x000b72fd
0x400c 0x001d174b

0x4400 0x00000006
0x4404 0x00000000
0x4408 0x00010f00
0x440c 0x0000e100
0x4410 0x00000000
0x4414 0x00000010
0x4418 0x00000060
0x441c 0x00000000
0x4420 0x0000003d
0x4424 0x00000000
0x4428 0x00000000
0x442c 0x00000000
0x4430 0x00000000
0x4434 0x00000000
0x4438 0x00000000
0x443c 0x00000000
0x4440 0x00000000
0x4444 0x00000000
0x4448 0x00000000
0x444c 0x01207401
0x4450 0x005e0364
0x4454 0x00090067

0x4800 0x080303fe
0x4804 0x00000000

0x4c00 0x000003fe
0x4c04 0x00000000

0x5000 0x00009820
0x5004 0x80008000
0x5008 0x00000000
0x500c 0x00000000
0x5010 0x00000000
0x5014 0x00000000
0x5018 0x00000000
0x501c 0x00010d24
0x5020 0x00000000
0x5024 0x00000000
0x5028 0x00000000
0x502c 0x00000000
0x5030 0x00000000
0x5034 0x0000001d
0x5038 0x00000000
0x503c 0x00000000
0x5040 0x00000000
0x5044 0x00000000
0x5048 0x00000000
0x504c 0x00000000
0x5050 0x00000000
0x5054 0x00000000
0x5058 0x00000000
0x505c 0x00000000
0x5060 0x00000000
0x5064 0x00000000
0x5068 0x00000000
0x506c 0x00000000
0x5070 0x00000000
0x5074 0x00000000
0x5078 0x00000000
0x507c 0x00000000
0x5080 0x00000000
0x5084 0x00000000
0x5088 0x00000000
0x508c 0x00000000
0x5090 0x00000000
0x5094 0x00000000
0x5098 0x00000000
0x509c 0x00000000
0x50a0 0x00000000
0x50a4 0x00000000
0x50a8 0x00000000
0x50ac 0x00000000
0x50b0 0x00000000
0x50b4 0x00000000
0x50b8 0x00000000
0x50bc 0x00000000
0x50c0 0x00000000
0x50c4 0x00000000
0x50c8 0x00000000
0x50cc 0x00000000
0x50d0 0x00000000
0x50d4 0x00000000
0x50d8 0x00000000
0x50dc 0x00000000
0x50e0 0x00000000
0x50e4 0x00000000
0x50e8 0x00000000
0x50ec 0x00000000
0x50f0 0x00000000
0x50f4 0x00000000
0x50f8 0x00000000
0x50fc 0x00000000
0x5100 0x00000000
0x5104 0x00000000
0x5108 0x00000000
0x510c 0x00000000
0x5110 0x00000000
0x5114 0x00000000
0x5118 0x00000000
0x511c 0x00000000
0x5120 0x00000000
0x5124 0x00000000
0x5128 0x00000000
0x512c 0x00000000
0x5130 0x00000000
0x5134 0x00000000
0x5138 0x00000000
0x513c 0x00000000
0x5140 0x00000000
0x5144 0x00000000
0x5148 0x00000000
0x514c 0x00000000
0x5150 0x00000000
0x5154 0x00000000
0x5158 0x00000000
0x515c 0x00000000
0x5160 0x00000000
0x5164 0x00000000
0x5168 0x00000000
0x516c 0x00000000
0x5170 0x00000000
0x5174 0x00000000
0x5178 0x00000000
0x517c 0x00000000
0x5180 0x00000000
0x5184 0x00000000
0x5188 0x00000000
0x518c 0x00000000
0x5190 0x00000000
0x5194 0x00000000
0x5198 0x00000000
0x519c 0x00000000
0x51a0 0x00000000
0x51a4 0x00000000
0x51a8 0x00000000
0x51ac 0x00000000
0x51b0 0x00000000
0x51b4 0x00000000
0x51b8 0x00000000
0x51bc 0x00000000
0x51c0 0x00000000
0x51c4 0x00000000
0x51c8 0x00000000
0x51cc 0x00000000
0x51d0 0x00000000
0x51d4 0x00000000
0x51d8 0x00000000
0x51dc 0x00000000
0x51e0 0x00000000
0x51e4 0x00000000
0x51e8 0x00000000
0x51ec 0x00000000
0x51f0 0x00000000
0x51f4 0x00000000
0x51f8 0x00000000
0x51fc 0x00000000
0x5200 0x00000000
0x5204 0x00000000
0x5208 0x00000000
0x520c 0x00000000
0x5210 0x00000000
0x5214 0x00000000
0x5218 0x00000000
0x521c 0x00000000
0x5220 0x00000000
0x5224 0x00000000
0x5228 0x00000000
0x522c 0x00000000
0x5230 0x00000000
0x5234 0x00000000
0x5238 0x00000000
0x523c 0x00000000
0x5240 0x00000000
0x5244 0x00000000
0x5248 0x00000000
0x524c 0x00000000
0x5250 0x00000000
0x5254 0x00000000
0x5258 0x00000000
0x525c 0x00000000
0x5260 0x00000000
0x5264 0x00000000
0x5268 0x00000000
0x526c 0x00000000
0x5270 0x00000000
0x5274 0x00000000
0x5278 0x00000000
0x527c 0x00000000

0x5400 0x00000000
0x5404 0x00000000
0x5408 0x00000000
0x540c 0x00000000
0x5410 0x00000000
0x5414 0x00000000
0x5418 0x00000000
0x541c 0x00000000
0x5420 0x00000000
0x5424 0x00000000
0x5428 0x00000000
0x542c 0x00000000
0x5430 0x00000000
0x5434 0x00000000
0x5438 0x00000000
0x543c 0x00000000
0x5440 0x00000000
0x5444 0x00000000
0x5448 0x00000000
0x544c 0x00000000
0x5450 0x00000000
0x5454 0x00000000
0x5458 0x00000000
0x545c 0x00000000
0x5460 0x00000000
0x5464 0x00000000
0x5468 0x00000000
0x546c 0x00000000
0x5470 0x00000000
0x5474 0x00000000
0x5478 0x00000000
0x547c 0x00000000
0x5480 0x00000000
0x5484 0x00000000
0x5488 0x00000000
0x548c 0x00000000
0x5490 0x00000000
0x5494 0x00000000
0x5498 0x00000000
0x549c 0x00000000
0x54a0 0x00000000
0x54a4 0x00000000
0x54a8 0x00000000
0x54ac 0x00000000
0x54b0 0x00000000
0x54b4 0x00000000
0x54b8 0x00000000
0x54bc 0x00000000
0x54c0 0x00000000
0x54c4 0x00000000
0x54c8 0x00000000
0x54cc 0x00000000
0x54d0 0x00000000
0x54d4 0x00000000
0x54d8 0x00000000
0x54dc 0x00000000
0x54e0 0x00000000
0x54e4 0x00000000
0x54e8 0x00000000
0x54ec 0x00000000
0x54f0 0x00000000
0x54f4 0x00000000
0x54f8 0x00000000
0x54fc 0x00000000
0x5500 0x00000000
0x5504 0x00000000
0x5508 0x00000000
0x550c 0x00000000
0x5510 0x00000000
0x5514 0x00000000
0x5518 0x00000000
0x551c 0x00000000
0x5520 0x00000000
0x5524 0x00000000
0x5528 0x00000000
0x552c 0x00000000
0x5530 0x00000000
0x5534 0x00000000
0x5538 0x00000000
0x553c 0x00000000
0x5540 0x00000000
0x5544 0x00000000
0x5548 0x00000000
0x554c 0x00000000
0x5550 0x00000000
0x5554 0x00000000
0x5558 0x00000000
0x555c 0x00000000
0x5560 0x00000000
0x5564 0x00000000
0x5568 0x00000000
0x556c 0x00000000
0x5570 0x00000000
0x5574 0x00000000
0x5578 0x00000000
0x557c 0x00000000
0x5580 0x00000000
0x5584 0x00000000
0x5588 0x00000000
0x558c 0x00000000
0x5590 0x00000000
0x5594 0x00000000
0x5598 0x00000000
0x559c 0x00000000
0x55a0 0x00000000
0x55a4 0x00000000
0x55a8 0x00000000
0x55ac 0x00000000
0x55b0 0x00000000
0x55b4 0x00000000
0x55b8 0x00000000
0x55bc 0x00000000
0x55c0 0x00000000
0x55c4 0x00000000
0x55c8 0x00000000
0x55cc 0x00000000
0x55d0 0x00000000
0x55d4 0x00000000
0x55d8 0x00000000
0x55dc 0x00000000
0x55e0 0x00000000
0x55e4 0x00000000
0x55e8 0x00000000
0x55ec 0x00000000
0x55f0 0x00000000
0x55f4 0x00000000
0x55f8 0x00000000
0x55fc 0x00000000
0x5600 0x00000000
0x5604 0x00000000
0x5608 0x00000000
0x560c 0x00000000
0x5610 0x00000000
0x5614 0x00000000
0x5618 0x00000000
0x561c 0x00000000
0x5620 0x00000000
0x5624 0x00000000
0x5628 0x00000000
0x562c 0x00000000
0x5630 0x00000000
0x5634 0x00000000
0x5638 0x00000000
0x563c 0x00000000
0x5640 0x00000000
0x5644 0x00000000
0x5648 0x00000000
0x564c 0x00000000
0x5650 0x00000000
0x5654 0x00000000
0x5658 0x00000000
0x565c 0x00000000
0x5660 0x00000000
0x5664 0x00000000
0x5668 0x00000000
0x566c 0x00000000
0x5670 0x00000000
0x5674 0x00000000
0x5678 0x00000000
0x567c 0x00000000

0x5800 0x00000000
0x5804 0x02000000
0x5808 0x00000000
0x580c 0x02000000
0x5810 0x00000000
0x5814 0x02000000
0x5818 0x00000000
0x581c 0x02000000
0x5820 0x00000000
0x5824 0x02000000
0x5828 0x00000000
0x582c 0x02000000
0x5830 0x00000000
0x5834 0x02000000
0x5838 0x00000000
0x583c 0x02000000
0x5840 0x00000000
0x5844 0x02000000
0x5848 0x00000000
0x584c 0x02000000
0x5850 0x00000000
0x5854 0x02000000
0x5858 0x00000000
0x585c 0x02000000
0x5860 0x00000000
0x5864 0x02000000
0x5868 0x00000000
0x586c 0x000000c8
0x5870 0x00000000
0x5874 0x02000000
0x5878 0x00000000
0x587c 0x02000000
0x5880 0x00000000
0x5884 0x00000000
0x5888 0x00000000
0x588c 0x02000000
0x5890 0x00000000
0x5894 0x02000000
0x5898 0x00000000
0x589c 0x02000000
0x58a0 0x00000000
0x58a4 0x02000000
0x58a8 0x00000000
0x58ac 0x02000000
0x58b0 0x00000000
0x58b4 0x02000000
0x58b8 0x00000000
0x58bc 0x02000000
0x58c0 0x00000000
0x58c4 0x02000000
0x58c8 0x00000000
0x58cc 0x02000000
0x58d0 0x00000000
0x58d4 0x02000000
0x58d8 0x00000000
0x58dc 0x02000000
0x58e0 0x00000000
0x58e4 0x02000000
0x58e8 0x00000000
0x58ec 0x02000000
0x58f0 0x00000000
0x58f4 0x02000000
0x58f8 0x00000000
0x58fc 0x02000000
0x5900 0x00000000
0x5904 0x00000024
0x5908 0x00000000
0x590c 0x02000000
0x5910 0x00000000
0x5914 0x00000000
0x5918 0x00000000
0x591c 0x00000000
0x5920 0x00000000
0x5924 0x00000000
0x5928 0x00000000
0x592c 0x00000000
0x5930 0x00000000
0x5934 0x00000000
0x5938 0x00000000
0x593c 0x00000000
0x5940 0x00000000
0x5944 0x00000000
0x5948 0x00000000
0x594c 0x00000000
0x5950 0x00000000
0x5954 0x00000000
0x5958 0x00000000
0x595c 0x00000000
0x5960 0x00000000
0x5964 0x00000000
0x5968 0x00000000
0x596c 0x00000000
0x5970 0x00000000
0x5974 0x00000000
0x5978 0x00000000
0x597c 0x00000000
0x5980 0x00000000
0x5984 0x00000000
0x5988 0x00000000
0x598c 0x00000000
0x5990 0x00000000
0x5994 0x00000000
0x5998 0x00000000
0x599c 0x00000000
0x59a0 0x00000000
0x59a4 0x00000000
0x59a8 0x00000000
0x59ac 0x00000000
0x59b0 0x00000000
0x59b4 0x00000000
0x59b8 0x00000000
0x59bc 0x00000000
0x59c0 0x00000000
0x59c4 0x00000000
0x59c8 0x00000000
0x59cc 0x00000000
0x59d0 0x00000000
0x59d4 0x00000000
0x59d8 0x00000000
0x59dc 0x00000000
0x59e0 0x00000000
0x59e4 0x00000000
0x59e8 0x00000000
0x59ec 0x00000000
0x59f0 0x00000000
0x59f4 0x00000000
0x59f8 0x00000000
0x59fc 0x00000000
0x5a00 0x00000000
0x5a04 0x00000000
0x5a08 0x00000000
0x5a0c 0x00000000

0x5c00 0x00000000
0x5c04 0x00000000
0x5c08 0x00000000
0x5c0c 0x00000000
0x5c10 0x00000000
0x5c14 0x00000000
0x5c18 0x00000000
0x5c1c 0x00000000
0x5c20 0x00000000
0x5c24 0x00000000
0x5c28 0x00000000
0x5c2c 0x00000000
0x5c30 0x00000000
0x5c34 0x00000000
0x5c38 0x00000000
0x5c3c 0x00000000
0x5c40 0x00000000
0x5c44 0x00000000
0x5c48 0x00000000
0x5c4c 0x00000000
0x5c50 0x00000000
0x5c54 0x00000000
0x5c58 0x00000000
0x5c5c 0x00000000
0x5c60 0x00000000
0x5c64 0x00000000
0x5c68 0x00000000
0x5c6c 0x00000000
0x5c70 0x00000000
0x5c74 0x00000000
0x5c78 0x00000000
0x5c7c 0x00000000
0x5c80 0x00000000
0x5c84 0x00000000
0x5c88 0x00000000
0x5c8c 0x00000000
0x5c90 0x00000000
0x5c94 0x00000000
0x5c98 0x00000000
0x5c9c 0x00000000
0x5ca0 0x00000000
0x5ca4 0x00000000
0x5ca8 0x00000000
0x5cac 0x00000000
0x5cb0 0x00000000
0x5cb4 0x00000000
0x5cb8 0x00000000
0x5cbc 0x00000000
0x5cc0 0x00000000
0x5cc4 0x00000000
0x5cc8 0x00000000
0x5ccc 0x00000000
0x5cd0 0x00000000
0x5cd4 0x00000000
0x5cd8 0x00000000
0x5cdc 0x00000000
0x5ce0 0x00000000
0x5ce4 0x00000000
0x5ce8 0x00000000
0x5cec 0x00000000
0x5cf0 0x00000000
0x5cf4 0x00000000
0x5cf8 0x00000000
0x5cfc 0x00000000
0x5d00 0x00000000
0x5d04 0x00000000
0x5d08 0x00000000
0x5d0c 0x00000000
0x5d10 0x00000000
0x5d14 0x00000000
0x5d18 0x00000000
0x5d1c 0x00000000

0x6000 0x00000000
0x6004 0x00000000
0x6008 0x00000004

0x6800 0x04130034
0x6804 0x3c081082
0x6808 0x01000609
0x680c 0x16695a00
0x6810 0x81020010
0x6814 0xffffffff
0x6818 0x00000000
0x681c 0x00000000
0x6820 0x00000000
0x6824 0x00000000
0x6828 0x00000000
0x682c 0x00000000
0x6830 0xfffc3ccf
0x6834 0xfffc0fff
0x6838 0x00000000
0x683c 0x00000000
0x6840 0x00000004
0x6844 0x00000000

0x7000 0x00000188
0x7004 0x00000000
0x7008 0x00000000
0x700c 0x000000c0
0x7010 0x0a000064
0x7014 0x02008273
0x7018 0x00570081
0x701c 0x68848353
0x7020 0x00000000
0x7024 0x00000000
0x7028 0x00000000
0x702c 0x00000000
0x7030 0x00000000

0x7c00 0x00000000
0x7c04 0x00000000
0x7c08 0x00000000
0x7c0c 0x00000000
0x7c10 0x00000000
0x7c14 0x00000000
0x7c18 0x00000000
0x7c1c 0x00000000
0x7c20 0x00000000
0x7c24 0x00000000
0x7c28 0x00000000
0x7c2c 0x00000000
0x7c30 0x00000000
0x7c34 0x00000000
0x7c38 0x00000000
0x7c3c 0x00000000
0x7c40 0x00000000
0x7c44 0x00000000
0x7c48 0x00000000
0x7c4c 0x00000000
0x7c50 0x00000000
0x7c54 0x00000000
0x7c58 0x00000000
0x7c5c 0x00000000
0x7c60 0x00000000
0x7c64 0x00000000
0x7c68 0x00000000
0x7c6c 0x00000000
0x7c70 0x00000000
0x7c74 0x00000000
0x7c78 0x00000000
0x7c7c 0x00000000
0x7c80 0x00000000
0x7c84 0x00000000
0x7c88 0x00000000
0x7c8c 0x00000000
0x7c90 0x00000000
0x7c94 0x00000000
0x7c98 0x00000000
0x7c9c 0x00000000
0x7ca0 0x00000000
0x7ca4 0x00000000
0x7ca8 0x00000000
0x7cac 0x00000000
0x7cb0 0x00000000
0x7cb4 0x00000000
0x7cb8 0x00000000
0x7cbc 0x00000000
0x7cc0 0x00000000
0x7cc4 0x00000000
0x7cc8 0x00000000
0x7ccc 0x00000000
0x7cd0 0x00000000
0x7cd4 0x00000000
0x7cd8 0x00000000
0x7cdc 0x00000000
0x7ce0 0x00000000
0x7ce4 0x00000000
0x7ce8 0x00000000
0x7cec 0x00000000
0x7cf0 0x00000000
0x7cf4 0x00000000
0x7cf8 0x00000000
0x7cfc 0x00000000
0x7d00 0x00000000
0x7d04 0x00000000
0x7d08 0x00000000
0x7d0c 0x00000000
0x7d10 0x00000000
0x7d14 0x00000000
0x7d18 0x00000000
0x7d1c 0x00000000
0x7d20 0x00000000
0x7d24 0x00000000
0x7d28 0x00000000
0x7d2c 0x00000000
0x7d30 0x00000000
0x7d34 0x00000000
0x7d38 0x00000000
0x7d3c 0x00000000
0x7d40 0x00000000
0x7d44 0x00000000
0x7d48 0x00000000
0x7d4c 0x00000000
0x7d50 0x00000000
0x7d54 0x00000000
0x7d58 0x00000000
0x7d5c 0x00000000
0x7d60 0x00000000
0x7d64 0x00000000
0x7d68 0x00000000
0x7d6c 0x00000000
0x7d70 0x00000000
0x7d74 0x00000000
0x7d78 0x00000000
0x7d7c 0x00000000
0x7d80 0x00000000
0x7d84 0x00000000
0x7d88 0x00000000
0x7d8c 0x00000000
0x7d90 0x00000000
0x7d94 0x00000000
0x7d98 0x00000000
0x7d9c 0x00000000
0x7da0 0x00000000
0x7da4 0x00000000
0x7da8 0x00000000
0x7dac 0x00000000
0x7db0 0x00000000
0x7db4 0x00000000
0x7db8 0x00000000
0x7dbc 0x00000000
0x7dc0 0x00000000
0x7dc4 0x00000000
0x7dc8 0x00000000
0x7dcc 0x00000000
0x7dd0 0x00000000
0x7dd4 0x00000000
0x7dd8 0x00000000
0x7ddc 0x00000000
0x7de0 0x00000000
0x7de4 0x00000000
0x7de8 0x00000000
0x7dec 0x00000000
0x7df0 0x00000000
0x7df4 0x00000000
0x7df8 0x00000000
0x7dfc 0x00000000
0x7e00 0x00000000
0x7e04 0x00000000
0x7e08 0x00000000
0x7e0c 0x00000000
0x7e10 0x00000000
0x7e14 0x00000000
0x7e18 0x00000000
0x7e1c 0x00000000
0x7e20 0x00000000
0x7e24 0x00000000
0x7e28 0x00000000
0x7e2c 0x00000000
0x7e30 0x00000000
0x7e34 0x00000000
0x7e38 0x00000000
0x7e3c 0x00000000

Matt Carlson

2008-11-20 17:11 UTC #23 lore raw

Quoted 77 lines · Roger Heflin

Quoted 66 lines · Matt Carlson
Quoted 56 lines · Roger Heflin
Quoted 7 lines · Peter Zijlstra
(netdev CC'ed)
Quoted 3 lines · Roger Heflin
I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.
How do more recent kernels do?
I did not try more recent kernels, more testing seems to indicate that at the very least it the bug depends on a certain version of either tg3 and/or firmware to happen, as my second tg3 port does not
have it happen. More about this below.
Quoted 32 lines
Quoted 32 lines
So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.
Unloading all of the network modules and reloading them did not correct the problem.
Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?
http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238
The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).
Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.
I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.
Quoted 5 lines
Quoted 5 lines
The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.
I am first turning off all offload capabilities on tg3 and going to see if that changes anything.
This made no difference in the error.
Quoted 2 lines
Quoted 2 lines
The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.
Did not try.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
I also have a second tg3 port that is slightly different, so I may try that eventually.
I tried this, and with the second port I don't appear to be getting
the error. The first port is a 5789-v3.29a and the second port is a
5788-v3.04, I know the first port is faster (pcie-x1) than the second
port (pci bus-built-in, unknown exact connection). The second port
will sustain about 50MB/second, were as the first port will get
Quoted 1 line
90MB/second.
Quoted 2 lines
The 5789 is a PCIe device, the 5788 is a PCI device. There's a lot of room for differences between the two devices.
I expected that there were going to be large changes between the two.
Quoted 6 lines
Quoted 5 lines
It seems to me to likely be the firmware on the tg3, and it would seem unlikely that the driver could do anything more than work around the issue that is in the firmware, and currently my system works on the second port, and the second port is fast enough for my needs.

Quoted 5 lines

Quoted 3 lines
I don't think the 5789 should have any firmware running at the time of the failure. There's always the possibility that there is a problem with the bootcode, but I'm guessing the problem is elsewhere.
Where does the offload code and other similar stuff run?

On the 5789, LSO is implemented directly in the hardware. Checksum offload was always in the hardware.

Quoted 7 lines

I know on a tg3 before I had difficulties when a BMC card was in the system as the tg3 had a bug that it grabbed UDP packets that it though were going to the BMC card, in that case it was incorrectly thinking that UDP data was a port number (this is only the case on the first UDP packet of a set) when it was actually data (on the UDP packets after the first one) in some of the udp
packets. Is that done from the firmware, or is that completely done in the
hardware and basically not fixable?

Yes, I remember hearing something about this problem too. That is a firmware problem though. The 5789 does not have any management firmware, so that shouldn't be the case here.

Quoted 41 lines

Quoted 41 lines
Quoted 5 lines
If someone else runs into this issue, since I have 2 ports I would be able to do some testing on it, right now my first port is locked up, and the machine is running fine on the second port.
lspci -vvv for the first (bad) port:
Ah. There it is.
Quoted 32 lines
02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit Ethernet PCI Express (rev 11)
Subsystem: Foxconn International, Inc. Unknown device 0cc1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K] Expansion ROM at <ignored> [disabled]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D3 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: 0101b8102a0f7b0c Data: f21e
Capabilities: [d0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <4us, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s <2us, L1 <64us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Virtual Channel

Quoted 7 lines

Quoted 2 lines
Hmmm. No smoking gun. Perhaps the register dump will help.
driver: tg3
version: 3.94
firmware-version: 5789-v3.29a
bus-info: 0000:02:00.0

O.K. I'll see if I can find any problems like this in the firmware archives.

Quoted 11 lines

tg3.c:v3.94 (August 14, 2008)
tg3 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19 tg3 0000:02:00.0: setting latency timer to 64
tg3 0000:05:01.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
Right now I brought the interface back up (it is still broken) and setup a network ip on it that other machines can ping.
The registers are included at the end of the email.

O.K. I'll pour over the dump and get back to you.

More below.

Quoted 10 lines

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219
dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Usually the tg3_tx_timeout function dumps a few registers before resetting the chip, but I don't see that here. Have you seen any dumps since then?
Is this the dump?

This would be it. Thanks.

Quoted 5 lines

Nov 12 14:58:13 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 14:58:13 computer kernel: tg3: DEBUG: MAC_TX_STATUS[00000008]
MAC_RX_STATUS[00000006]
Nov 12 14:58:13 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000010]

WDMAC_STATUS[00000000]

Here the Read DMA Status register is reporting a Read DMA PCI Parity Error. I've seen this before...very recently in fact. The problem was that the chipset was not programmed by the BIOS correctly. In that particular case, a BIOS upgrade solved the problem. YMMV.

Quoted 13 lines

Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=2c00
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=1400
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: eth0: Link is down.
Nov 12 14:58:16 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 14:58:16 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 15:20:37 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 15:20:37 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
MAC_RX_STATUS[00000000]
Nov 12 15:20:37 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]

Quoted 1 line

WDMAC_STATUS[00000000]

Here the MAC TX Status register is reporting that the link is up, but the device is sending pause frames and rx is currently rx off'd.

Does the same problem happen if flow control is disabled?

Quoted 1,784 lines

Nov 12 15:20:37 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
enable_bit=2
Nov 12 15:20:37 computer kernel: tg3: eth0: Link is down.
Nov 12 15:20:40 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 15:20:40 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 15:48:16 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 15:48:16 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
MAC_RX_STATUS[00000000]
Nov 12 15:48:16 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
WDMAC_STATUS[00000000]
Nov 12 15:48:16 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
enable_bit=2
Nov 12 15:48:16 computer kernel: tg3: eth0: Link is down.
Nov 12 15:48:19 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 15:48:19 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:10:44 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:10:44 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
MAC_RX_STATUS[00000000]
Nov 12 16:10:44 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
WDMAC_STATUS[00000000]
Nov 12 16:10:44 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
enable_bit=2
Nov 12 16:10:44 computer kernel: tg3: eth0: Link is down.
Nov 12 16:10:47 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:10:47 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:18:43 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:18:43 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
MAC_RX_STATUS[00000000]
Nov 12 16:18:43 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
WDMAC_STATUS[00000000]
Nov 12 16:18:43 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
enable_bit=2
Nov 12 16:18:43 computer kernel: tg3: eth0: Link is down.
Nov 12 16:18:46 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:18:46 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:27:02 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:27:02 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
MAC_RX_STATUS[00000000]
Nov 12 16:27:02 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
WDMAC_STATUS[00000000]
Nov 12 16:27:02 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
enable_bit=2
Nov 12 16:27:02 computer kernel: tg3: eth0: Link is down.
Nov 12 16:27:06 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:27:06 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 16:35:17 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 16:35:17 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
MAC_RX_STATUS[00000000]
Nov 12 16:35:17 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]
WDMAC_STATUS[00000000]
Nov 12 16:35:17 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
enable_bit=2
Nov 12 16:35:17 computer kernel: tg3: eth0: Link is down.
Nov 12 16:35:20 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 16:35:20 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.

Quoted 33 lines

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs
w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0
[uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e

Quoted 10 lines

Quoted 8 lines

Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5

Offset  Value
------  ----------
0x0000  0x169d14e4
0x0004  0x00100006
0x0008  0x02000011
0x000c  0x00000008
0x0010  0xfd8f0004
0x0014  0x00000000
0x0018  0x00000000
0x001c  0x00000000
0x0020  0x00000000
0x0024  0x00000000
0x0028  0x00000007
0x002c  0x0cc1105b
0x0030  0x02aa0000
0x0034  0x00000048
0x0038  0x00000000
0x003c  0x0000010b
0x0040  0x00000000
0x0044  0x00000000
0x0048  0xc0025001
0x004c  0x64002000
0x0050  0x00d05803
0x0054  0x587d4b8a
0x0058  0x0086d005
0x005c  0x2a0f7b0c
0x0060  0x0101b810
0x0064  0x0000f21e
0x0068  0x41010298
0x006c  0x76180000
0x0070  0x000010f2
0x0074  0x000000c0
0x0078  0x00000050
0x007c  0x00000000
0x0080  0x00d05803
0x0084  0x00000000
0x0088  0x04130034
0x008c  0x3c081082
0x0090  0x01000609
0x0094  0x02000000
0x0098  0x00000000
0x009c  0x000000c8
0x00a0  0x00000000
0x00a4  0x00000000
0x00a8  0x00000000
0x00ac  0x00000000
0x00b0  0x00000000
0x00b4  0x00000000
0x00b8  0x00000000
0x00bc  0x00000000
0x00c0  0x00000000
0x00c4  0x00000000
0x00c8  0x00000000
0x00cc  0x00000000
0x00d0  0x00000000
0x00d4  0x00000000
0x00d8  0x00000000
0x00dc  0x00000000
0x00e0  0x00000000
0x00e4  0x00000000
0x00e8  0x00000000
0x00ec  0x00000000
0x00f0  0x00000000
0x00f4  0x00000000
0x00f8  0x00000000
0x00fc  0x00000000
0x0100  0x00000000
0x0104  0x00000000
0x0108  0x00000000
0x010c  0x00000000
0x0110  0x00000000
0x0114  0x00000000
0x0118  0x00000000
0x011c  0x00000000
0x0120  0x00000000
0x0124  0x00000000
0x0128  0x00000000
0x012c  0x00000000
0x0130  0x00000000
0x0134  0x00000000
0x0138  0x00000000
0x013c  0x00000000
0x0140  0x00000000
0x0144  0x00000000
0x0148  0x00000000
0x014c  0x00000000
0x0150  0x00000000
0x0154  0x00000000
0x0158  0x00000000

0x0200  0x00000000
0x0204  0x02000000
0x0208  0x00000000
0x020c  0x02000000
0x0210  0x00000000
0x0214  0x02000000
0x0218  0x00000000
0x021c  0x02000000
0x0220  0x00000000
0x0224  0x02000000
0x0228  0x00000000
0x022c  0x02000000
0x0230  0x00000000
0x0234  0x02000000
0x0238  0x00000000
0x023c  0x02000000
0x0240  0x00000000
0x0244  0x02000000
0x0248  0x00000000
0x024c  0x02000000
0x0250  0x00000000
0x0254  0x02000000
0x0258  0x00000000
0x025c  0x02000000
0x0260  0x00000000
0x0264  0x02000000
0x0268  0x00000000
0x026c  0x000000c8
0x0270  0x00000000
0x0274  0x02000000
0x0278  0x00000000
0x027c  0x02000000
0x0280  0x00000000
0x0284  0x00000000
0x0288  0x00000000
0x028c  0x02000000
0x0290  0x00000000
0x0294  0x02000000
0x0298  0x00000000
0x029c  0x02000000
0x02a0  0x00000000
0x02a4  0x02000000
0x02a8  0x00000000
0x02ac  0x02000000
0x02b0  0x00000000
0x02b4  0x02000000
0x02b8  0x00000000
0x02bc  0x02000000
0x02c0  0x00000000
0x02c4  0x02000000
0x02c8  0x00000000
0x02cc  0x02000000
0x02d0  0x00000000
0x02d4  0x02000000
0x02d8  0x00000000
0x02dc  0x02000000
0x02e0  0x00000000
0x02e4  0x02000000
0x02e8  0x00000000
0x02ec  0x02000000
0x02f0  0x00000000
0x02f4  0x02000000
0x02f8  0x00000000
0x02fc  0x02000000
0x0300  0x00000000
0x0304  0x00000024
0x0308  0x00000000
0x030c  0x02000024
0x0310  0x00000000
0x0314  0x02000024
0x0318  0x00000000
0x031c  0x02000024
0x0320  0x00000000
0x0324  0x02000024
0x0328  0x00000000
0x032c  0x02000024
0x0330  0x00000000
0x0334  0x02000024
0x0338  0x00000000
0x033c  0x02000024
0x0340  0x00000000
0x0344  0x02000024
0x0348  0x00000000
0x034c  0x02000024
0x0350  0x00000000
0x0354  0x02000024
0x0358  0x00000000
0x035c  0x02000024
0x0360  0x00000000
0x0364  0x02000024
0x0368  0x00000000
0x036c  0x02000024
0x0370  0x00000000
0x0374  0x02000024
0x0378  0x00000000
0x037c  0x02000024
0x0380  0x00000000
0x0384  0x00000000
0x0388  0x00000000
0x038c  0x02000000
0x0390  0x00000000
0x0394  0x02000000
0x0398  0x00000000
0x039c  0x02000000
0x03a0  0x00000000
0x03a4  0x02000000
0x03a8  0x00000000
0x03ac  0x02000000
0x03b0  0x00000000
0x03b4  0x02000000
0x03b8  0x00000000
0x03bc  0x02000000
0x03c0  0x00000000
0x03c4  0x02000000
0x03c8  0x00000000
0x03cc  0x02000000
0x03d0  0x00000000
0x03d4  0x02000000
0x03d8  0x00000000
0x03dc  0x02000000
0x03e0  0x00000000
0x03e4  0x02000000
0x03e8  0x00000000
0x03ec  0x02000000
0x03f0  0x00000000
0x03f4  0x02000000
0x03f8  0x00000000
0x03fc  0x02000000

0x0400  0x00e04808
0x0404  0x00400000
0x0408  0x00001000
0x040c  0x00000880
0x0410  0x00000001
0x0414  0x6ce59c51
0x0418  0x00000001
0x041c  0x6ce59c51
0x0420  0x00000001
0x0424  0x6ce59c51
0x0428  0x00000001
0x042c  0x6ce59c51
0x0430  0x00000000
0x0434  0x00000000
0x0438  0x000000da
0x043c  0x000005f2
0x0440  0x00000000
0x0444  0x00000000
0x0448  0x00000000
0x044c  0x0825c5e1
0x0450  0x00000001
0x0454  0x000c0000
0x0458  0x00000000
0x045c  0x00000012
0x0460  0x00000008
0x0464  0x00002620
0x0468  0x00000406
0x046c  0x00000000
0x0470  0x80000000
0x0474  0x00000000
0x0478  0x00000080
0x047c  0x40200000
0x0480  0x42000000
0x0484  0x7fffffff
0x0488  0x06000004
0x048c  0x7fffffff
0x0490  0x00000000
0x0494  0x00000000
0x0498  0x00000000
0x049c  0x00000000
0x04a0  0x00000000
0x04a4  0x00000000
0x04a8  0x00000000
0x04ac  0x00000000
0x04b0  0x00000000
0x04b4  0x00000000
0x04b8  0x00000000
0x04bc  0x00000000
0x04c0  0x00000000
0x04c4  0x00000000
0x04c8  0x00000000
0x04cc  0x00000000
0x04d0  0x00000000
0x04d4  0x00000000
0x04d8  0x00000000
0x04dc  0x00000000
0x04e0  0x00000000
0x04e4  0x00000000
0x04e8  0x00000000
0x04ec  0x00000000
0x04f0  0x00000000
0x04f4  0x00000000
0x04f8  0x00000000
0x04fc  0x00000000
0x0500  0x00000008
0x0504  0x00000002
0x0508  0x00000000
0x050c  0x00000000
0x0510  0x00000000
0x0514  0x00000000
0x0518  0x00000000
0x051c  0x00000000
0x0520  0x00000000
0x0524  0x00000000
0x0528  0x00000000
0x052c  0x00000000
0x0530  0x00000000
0x0534  0x00000000
0x0538  0x00000000
0x053c  0x00000000
0x0540  0x00000000
0x0544  0x00000000
0x0548  0x00000000
0x054c  0x00000000
0x0550  0x00000000
0x0554  0x00000000
0x0558  0x00000000
0x055c  0x00000000
0x0560  0x00000000
0x0564  0x00000000
0x0568  0x00000000
0x056c  0x00000000
0x0570  0x00000000
0x0574  0x00000000
0x0578  0x00000000
0x057c  0x00000000
0x0580  0x00000000
0x0584  0x00000000
0x0588  0x00000000
0x058c  0x00000000
0x0590  0x00e00000
0x0594  0x00000000
0x0598  0x00000000
0x059c  0x00000000
0x05a0  0x00000000
0x05a4  0x00000000
0x05a8  0x00000000
0x05ac  0x00000000
0x05b0  0x00000000
0x05b4  0x00000000
0x05b8  0x00000000
0x05bc  0x00000000
0x05c0  0x00000000
0x05c4  0x00000000
0x05c8  0x00000000
0x05cc  0x00000000
0x05d0  0x00000000
0x05d4  0x00000000
0x05d8  0x00000000
0x05dc  0x00000000
0x05e0  0x00000000
0x05e4  0x00000000
0x05e8  0x00000000
0x05ec  0x00000000
0x05f0  0x00000000
0x05f4  0x00000000
0x05f8  0x00000000
0x05fc  0x00000000
0x0600  0xffffffff
0x0604  0x00f80011
0x0608  0x00000000
0x060c  0x00001f04
0x0610  0xffffffff
0x0614  0x00000000
0x0618  0x07c00004
0x061c  0x30000000
0x0620  0x00000040
0x0624  0x00000000
0x0628  0x00000000
0x062c  0x00000000
0x0630  0x00000000
0x0634  0x00000000
0x0638  0x00000000
0x063c  0x00000000
0x0640  0x00000000
0x0644  0x00000000
0x0648  0x00000000
0x064c  0x00000000
0x0650  0x00000000
0x0654  0x00000000
0x0658  0x00000000
0x065c  0x00000000
0x0660  0x00000000
0x0664  0x00000000
0x0668  0x00000000
0x066c  0x00000000
0x0670  0x00000000
0x0674  0x00000000
0x0678  0x00000000
0x067c  0x00000000
0x0680  0x00000000
0x0684  0x00000000
0x0688  0x00000000
0x068c  0x00000000
0x0690  0x00000000
0x0694  0x00000000
0x0698  0x00000000
0x069c  0x00000000
0x06a0  0x00000000
0x06a4  0x00000000
0x06a8  0x00000000
0x06ac  0x00000000
0x06b0  0x00000000
0x06b4  0x00000000
0x06b8  0x00000000
0x06bc  0x00000000
0x06c0  0x00000000
0x06c4  0x00000000
0x06c8  0x00000000
0x06cc  0x00000000
0x06d0  0x00000000
0x06d4  0x00000000
0x06d8  0x00000000
0x06dc  0x00000000
0x06e0  0x00000000
0x06e4  0x00000000
0x06e8  0x00000000
0x06ec  0x00000000
0x06f0  0x00000000
0x06f4  0x00000000
0x06f8  0x00000000
0x06fc  0x00000000
0x0700  0x00000000
0x0704  0x00000000
0x0708  0x00000000
0x070c  0x00000000
0x0710  0x00000000
0x0714  0x00000000
0x0718  0x00000000
0x071c  0x00000000
0x0720  0x00000000
0x0724  0x00000000
0x0728  0x00000000
0x072c  0x00000000
0x0730  0x00000000
0x0734  0x00000000
0x0738  0x00000000
0x073c  0x00000000
0x0740  0x00000000
0x0744  0x00000000
0x0748  0x00000000
0x074c  0x00000000
0x0750  0x00000000
0x0754  0x00000000
0x0758  0x00000000
0x075c  0x00000000
0x0760  0x00000000
0x0764  0x00000000
0x0768  0x00000000
0x076c  0x00000000
0x0770  0x00000000
0x0774  0x00000000
0x0778  0x00000000
0x077c  0x00000000
0x0780  0x00000000
0x0784  0x00000000
0x0788  0x00000000
0x078c  0x00000000
0x0790  0x00000000
0x0794  0x00000000
0x0798  0x00000000
0x079c  0x00000000
0x07a0  0x00000000
0x07a4  0x00000000
0x07a8  0x00000000
0x07ac  0x00000000
0x07b0  0x00000000
0x07b4  0x00000000
0x07b8  0x00000000
0x07bc  0x00000000
0x07c0  0x00000000
0x07c4  0x00000000
0x07c8  0x00000000
0x07cc  0x00000000
0x07d0  0x00000000
0x07d4  0x00000000
0x07d8  0x00000000
0x07dc  0x00000000
0x07e0  0x00000000
0x07e4  0x00000000
0x07e8  0x00000000
0x07ec  0x00000000
0x07f0  0x00000000
0x07f4  0x00000000
0x07f8  0x00000000
0x07fc  0x00000000
0x0800  0x00000000
0x0804  0xffffffff
0x0808  0x00000000
0x080c  0x00000000
0x0810  0x00000000
0x0814  0xffffffff
0x0818  0x00000000
0x081c  0x00000000
0x0820  0x00000000
0x0824  0x00000000
0x0828  0xffffffff
0x082c  0x00000000
0x0830  0x00000000
0x0834  0xffffffff
0x0838  0xffffffff
0x083c  0xffffffff
0x0840  0xffffffff
0x0844  0xffffffff
0x0848  0xffffffff
0x084c  0xffffffff
0x0850  0xffffffff
0x0854  0xffffffff
0x0858  0xffffffff
0x085c  0xffffffff
0x0860  0xffffffff
0x0864  0xffffffff
0x0868  0xffffffff
0x086c  0x00000000
0x0870  0x00000000
0x0874  0x00000000
0x0878  0x00000000
0x087c  0x00000000
0x0880  0x00000000
0x0884  0x00000c80
0x0888  0x00000000
0x088c  0x00000000
0x0890  0x00000000
0x0894  0x00000000
0x0898  0x00000000
0x089c  0x00000000
0x08a0  0x00000000
0x08a4  0x00000000
0x08a8  0x00000000
0x08ac  0x00000000
0x08b0  0x00000000
0x08b4  0x00000000
0x08b8  0x00000000
0x08bc  0x00000000
0x08c0  0x00000000
0x08c4  0x00000000
0x08c8  0x00000000
0x08cc  0x00000000
0x08d0  0x00000000
0x08d4  0x00000000
0x08d8  0x00000000
0x08dc  0x00000000
0x08e0  0x00000000
0x08e4  0x00000000
0x08e8  0x00000000
0x08ec  0x00000000

0x0c00  0x0000000a
0x0c04  0x00000000
0x0c08  0x00000003
0x0c0c  0x00000001
0x0c10  0x00000000
0x0c14  0x00000000
0x0c18  0x00000000
0x0c1c  0x00000000
0x0c20  0x00000000
0x0c24  0x00000000
0x0c28  0x00000000
0x0c2c  0x00000000
0x0c30  0x00000000
0x0c34  0x00000000
0x0c38  0x00000000
0x0c3c  0x00000000
0x0c40  0x00000000
0x0c44  0x00000000
0x0c48  0x00000000
0x0c4c  0x00000000
0x0c50  0x00000000
0x0c54  0x00000000
0x0c58  0x00000000
0x0c5c  0x00000000
0x0c60  0x00000000
0x0c64  0x00000000
0x0c68  0x00000000
0x0c6c  0x00000000
0x0c70  0x00000000
0x0c74  0x00000000
0x0c78  0x00000000
0x0c7c  0x00000000
0x0c80  0x00000000
0x0c84  0x00000000
0x0c88  0x00000000
0x0c8c  0x00000000
0x0c90  0x00000000
0x0c94  0x00000000
0x0c98  0x00000000
0x0c9c  0x00000000
0x0ca0  0x00000000
0x0ca4  0x00000000
0x0ca8  0x00000000
0x0cac  0x00000000
0x0cb0  0x00000000
0x0cb4  0x00000000
0x0cb8  0x00000000
0x0cbc  0x00000000
0x0cc0  0x00000000
0x0cc4  0x00000000
0x0cc8  0x00000000
0x0ccc  0x00000000
0x0cd0  0x00000000
0x0cd4  0x00000000
0x0cd8  0x00000000
0x0cdc  0x00000000

0x1000 0x00000002

0x1400  0x00000006
0x1404  0x00000000
0x1408  0x00000000
0x140c  0x00000000
0x1410  0x00000000
0x1414  0x00000000
0x1418  0x00000000
0x141c  0x00000000
0x1420  0x00000000
0x1424  0x00000000
0x1428  0x00000000
0x142c  0x00000000
0x1430  0x00000000
0x1434  0x00000000
0x1438  0x00000000
0x143c  0x00000000
0x1440  0x00000000
0x1444  0x00000000
0x1448  0x00000000
0x144c  0x00000000
0x1450  0x00000000
0x1454  0x00000000
0x1458  0x00000000
0x145c  0x00000000
0x1460  0x00000000
0x1464  0x00000000
0x1468  0x00000000
0x146c  0x00000000
0x1470  0x00000000
0x1474  0x00000000
0x1478  0x00000000
0x147c  0x00000000

0x1800  0x00000016
0x1804  0x00000000
0x1808  0x00000001
0x180c  0x00000000
0x1810  0x00000000
0x1814  0x00000000
0x1818  0x00000000
0x181c  0x00000000
0x1820  0x00000000
0x1824  0x00000000
0x1828  0x00000000
0x182c  0x00000000
0x1830  0x00000000
0x1834  0x00000000
0x1838  0x00000000
0x183c  0x00000000
0x1840  0x00000000
0x1844  0x00000000

0x1c00 0x00000002

0x2000  0x00000002
0x2004  0x00000000
0x2008  0x00000000
0x200c  0x00000000
0x2010  0x00000181
0x2014  0x00000001
0x2018  0x00790407
0x201c  0x00000000
0x2020  0x00000000
0x2024  0x00000000
0x2028  0x00000000
0x202c  0x00000000
0x2030  0x00000000
0x2034  0x00000000
0x2038  0x00000000
0x203c  0x00000000
0x2040  0x00000000
0x2044  0x00000000
0x2048  0x00000000
0x204c  0x00000000
0x2050  0x00000000
0x2054  0x00000000
0x2058  0x00000000
0x205c  0x00000000
0x2060  0x00000000
0x2064  0x00000000
0x2068  0x00000000
0x206c  0x00000000
0x2070  0x00000000
0x2074  0x00000000
0x2078  0x00000000
0x207c  0x00000000
0x2080  0x00000000
0x2084  0x00000000
0x2088  0x00000000
0x208c  0x00000000
0x2090  0x00000000
0x2094  0x00000000
0x2098  0x00000000
0x209c  0x00000000
0x20a0  0x00000000
0x20a4  0x00000000
0x20a8  0x00000000
0x20ac  0x00000000
0x20b0  0x00000000
0x20b4  0x00000000
0x20b8  0x00000000
0x20bc  0x00000000
0x20c0  0x00000000
0x20c4  0x00000000
0x20c8  0x00000000
0x20cc  0x00000000
0x20d0  0x00000000
0x20d4  0x00000000
0x20d8  0x00000000
0x20dc  0x00000000
0x20e0  0x00000000
0x20e4  0x00000000
0x20e8  0x00000000
0x20ec  0x00000000
0x20f0  0x00000000
0x20f4  0x00000000
0x20f8  0x00000000
0x20fc  0x00000000
0x2100  0x00086633
0x2104  0x00090683
0x2108  0x0000004d
0x210c  0x00000000
0x2110  0x00086633
0x2114  0x00090683
0x2118  0x0000004d
0x211c  0x00000000
0x2120  0x00086633
0x2124  0x00090683
0x2128  0x0000004d
0x212c  0x00000000
0x2130  0x00086633
0x2134  0x00090683
0x2138  0x0000004d
0x213c  0x00000000
0x2140  0x00086633
0x2144  0x00090683
0x2148  0x0000004d
0x214c  0x00000000
0x2150  0x00086633
0x2154  0x00090683
0x2158  0x0000004d
0x215c  0x00000000
0x2160  0x00086633
0x2164  0x00090683
0x2168  0x0000004d
0x216c  0x00000000
0x2170  0x00086633
0x2174  0x00090683
0x2178  0x0000004d
0x217c  0x00000000
0x2180  0x00086633
0x2184  0x00090683
0x2188  0x0000004d
0x218c  0x00000000
0x2190  0x00086633
0x2194  0x00090683
0x2198  0x0000004d
0x219c  0x00000000
0x21a0  0x00086633
0x21a4  0x00090683
0x21a8  0x0000004d
0x21ac  0x00000000
0x21b0  0x00086633
0x21b4  0x00090683
0x21b8  0x0000004d
0x21bc  0x00000000
0x21c0  0x00086633
0x21c4  0x00090683
0x21c8  0x0000004d
0x21cc  0x00000000
0x21d0  0x00086633
0x21d4  0x00090683
0x21d8  0x0000004d
0x21dc  0x00000000
0x21e0  0x00086633
0x21e4  0x00090683
0x21e8  0x0000004d
0x21ec  0x00000000
0x21f0  0x00086633
0x21f4  0x00090683
0x21f8  0x0000004d
0x21fc  0x00000000
0x2200  0x00000037
0x2204  0x00000000
0x2208  0x00000000
0x220c  0x00000000
0x2210  0x00000000
0x2214  0x00000000
0x2218  0x00000000
0x221c  0x00000000
0x2220  0x00000000
0x2224  0x00000000
0x2228  0x00000000
0x222c  0x00000000
0x2230  0x00000000
0x2234  0x00000000
0x2238  0x00000000
0x223c  0x00000000
0x2240  0x00000000
0x2244  0x00000000
0x2248  0x00000000
0x224c  0x00000000
0x2250  0x00000000
0x2254  0x00000000
0x2258  0x00000000

0x2400  0x00000012
0x2404  0x00000000
0x2408  0x00000000
0x240c  0x00000000
0x2410  0x00000000
0x2414  0x00000000
0x2418  0x00000000
0x241c  0x00000000
0x2420  0x00000000
0x2424  0x00000000
0x2428  0x00000000
0x242c  0x00000000
0x2430  0x00000000
0x2434  0x00000000
0x2438  0x00000000
0x243c  0x00000000
0x2440  0x00000000
0x2444  0x00000000
0x2448  0x00000000
0x244c  0x00000000
0x2450  0x00000000
0x2454  0x030d4000
0x2458  0x02000000
0x245c  0x00006000
0x2460  0x00000000
0x2464  0x00000000
0x2468  0x00000000
0x246c  0x00000000
0x2470  0x00000000
0x2474  0x00000000
0x2478  0x00000000
0x247c  0x00000000
0x2480  0x00000000
0x2484  0x00000000
0x2488  0x00000000
0x248c  0x00000000
0x2490  0x00000000
0x2494  0x00000000
0x2498  0x00000000
0x249c  0x00000000
0x24a0  0x00000000
0x24a4  0x00000000
0x24a8  0x00000000
0x24ac  0x00000000
0x24b0  0x00000000
0x24b4  0x00000000
0x24b8  0x00000000
0x24bc  0x00000000
0x24c0  0x08000010

0x2800 0x00000006

0x2c00  0x00000006
0x2c04  0x00000000
0x2c08  0x00000000
0x2c0c  0x0000007f
0x2c10  0x00000000
0x2c14  0x00000000
0x2c18  0x00000008
0x2c1c  0x00000000

0x3000  0x00000006
0x3004  0x00000000
0x3008  0x00000000
0x300c  0x00000000
0x3010  0x00000000

0x3400 0x00000000
0x3404 0x00000000

0x3800 0x00000000
0x3804 0x00000000

0x3c00  0x00000302
0x3c04  0x00000000
0x3c08  0x00000064
0x3c0c  0x00000048
0x3c10  0x0000000a
0x3c14  0x00000035
0x3c18  0x00000000
0x3c1c  0x00000000
0x3c20  0x0000000a
0x3c24  0x00000005
0x3c28  0x00000000
0x3c2c  0x00000000
0x3c30  0x00000000
0x3c34  0x00000000
0x3c38  0x00000000
0x3c3c  0x03270000
0x3c40  0x00000000
0x3c44  0x00000b00
0x3c48  0x00000000
0x3c4c  0x00000000
0x3c50  0x00000000
0x3c54  0x00000000
0x3c58  0x00000000
0x3c5c  0x00000000
0x3c60  0x00000000
0x3c64  0x00000000
0x3c68  0x00000000
0x3c6c  0x00000000
0x3c70  0x00000000
0x3c74  0x00000000
0x3c78  0x00000000
0x3c7c  0x00000000
0x3c80  0x00000000
0x3c84  0x00000000
0x3c88  0x00000000
0x3c8c  0x00000000
0x3c90  0x00000000
0x3c94  0x00000000
0x3c98  0x00000000
0x3c9c  0x00000000
0x3ca0  0x00000000
0x3ca4  0x00000000
0x3ca8  0x00000000
0x3cac  0x00000000
0x3cb0  0x00000000
0x3cb4  0x00000000
0x3cb8  0x00000000
0x3cbc  0x00000000
0x3cc0  0x00000000
0x3cc4  0x00000000
0x3cc8  0x00000000
0x3ccc  0x00000000
0x3cd0  0x00000000
0x3cd4  0x00000000
0x3cd8  0x00000000
0x3cdc  0x00000000
0x3ce0  0x00000000
0x3ce4  0x00000000
0x3ce8  0x00000000
0x3cec  0x00000000
0x3cf0  0x00000000
0x3cf4  0x00000000
0x3cf8  0x00000000
0x3cfc  0x00000000

0x4000  0x00000002
0x4004  0x00000000
0x4008  0x000b72fd
0x400c  0x001d174b

0x4400  0x00000006
0x4404  0x00000000
0x4408  0x00010f00
0x440c  0x0000e100
0x4410  0x00000000
0x4414  0x00000010
0x4418  0x00000060
0x441c  0x00000000
0x4420  0x0000003d
0x4424  0x00000000
0x4428  0x00000000
0x442c  0x00000000
0x4430  0x00000000
0x4434  0x00000000
0x4438  0x00000000
0x443c  0x00000000
0x4440  0x00000000
0x4444  0x00000000
0x4448  0x00000000
0x444c  0x01207401
0x4450  0x005e0364
0x4454  0x00090067

0x4800 0x080303fe
0x4804 0x00000000

0x4c00 0x000003fe
0x4c04 0x00000000

0x5000  0x00009820
0x5004  0x80008000
0x5008  0x00000000
0x500c  0x00000000
0x5010  0x00000000
0x5014  0x00000000
0x5018  0x00000000
0x501c  0x00010d24
0x5020  0x00000000
0x5024  0x00000000
0x5028  0x00000000
0x502c  0x00000000
0x5030  0x00000000
0x5034  0x0000001d
0x5038  0x00000000
0x503c  0x00000000
0x5040  0x00000000
0x5044  0x00000000
0x5048  0x00000000
0x504c  0x00000000
0x5050  0x00000000
0x5054  0x00000000
0x5058  0x00000000
0x505c  0x00000000
0x5060  0x00000000
0x5064  0x00000000
0x5068  0x00000000
0x506c  0x00000000
0x5070  0x00000000
0x5074  0x00000000
0x5078  0x00000000
0x507c  0x00000000
0x5080  0x00000000
0x5084  0x00000000
0x5088  0x00000000
0x508c  0x00000000
0x5090  0x00000000
0x5094  0x00000000
0x5098  0x00000000
0x509c  0x00000000
0x50a0  0x00000000
0x50a4  0x00000000
0x50a8  0x00000000
0x50ac  0x00000000
0x50b0  0x00000000
0x50b4  0x00000000
0x50b8  0x00000000
0x50bc  0x00000000
0x50c0  0x00000000
0x50c4  0x00000000
0x50c8  0x00000000
0x50cc  0x00000000
0x50d0  0x00000000
0x50d4  0x00000000
0x50d8  0x00000000
0x50dc  0x00000000
0x50e0  0x00000000
0x50e4  0x00000000
0x50e8  0x00000000
0x50ec  0x00000000
0x50f0  0x00000000
0x50f4  0x00000000
0x50f8  0x00000000
0x50fc  0x00000000
0x5100  0x00000000
0x5104  0x00000000
0x5108  0x00000000
0x510c  0x00000000
0x5110  0x00000000
0x5114  0x00000000
0x5118  0x00000000
0x511c  0x00000000
0x5120  0x00000000
0x5124  0x00000000
0x5128  0x00000000
0x512c  0x00000000
0x5130  0x00000000
0x5134  0x00000000
0x5138  0x00000000
0x513c  0x00000000
0x5140  0x00000000
0x5144  0x00000000
0x5148  0x00000000
0x514c  0x00000000
0x5150  0x00000000
0x5154  0x00000000
0x5158  0x00000000
0x515c  0x00000000
0x5160  0x00000000
0x5164  0x00000000
0x5168  0x00000000
0x516c  0x00000000
0x5170  0x00000000
0x5174  0x00000000
0x5178  0x00000000
0x517c  0x00000000
0x5180  0x00000000
0x5184  0x00000000
0x5188  0x00000000
0x518c  0x00000000
0x5190  0x00000000
0x5194  0x00000000
0x5198  0x00000000
0x519c  0x00000000
0x51a0  0x00000000
0x51a4  0x00000000
0x51a8  0x00000000
0x51ac  0x00000000
0x51b0  0x00000000
0x51b4  0x00000000
0x51b8  0x00000000
0x51bc  0x00000000
0x51c0  0x00000000
0x51c4  0x00000000
0x51c8  0x00000000
0x51cc  0x00000000
0x51d0  0x00000000
0x51d4  0x00000000
0x51d8  0x00000000
0x51dc  0x00000000
0x51e0  0x00000000
0x51e4  0x00000000
0x51e8  0x00000000
0x51ec  0x00000000
0x51f0  0x00000000
0x51f4  0x00000000
0x51f8  0x00000000
0x51fc  0x00000000
0x5200  0x00000000
0x5204  0x00000000
0x5208  0x00000000
0x520c  0x00000000
0x5210  0x00000000
0x5214  0x00000000
0x5218  0x00000000
0x521c  0x00000000
0x5220  0x00000000
0x5224  0x00000000
0x5228  0x00000000
0x522c  0x00000000
0x5230  0x00000000
0x5234  0x00000000
0x5238  0x00000000
0x523c  0x00000000
0x5240  0x00000000
0x5244  0x00000000
0x5248  0x00000000
0x524c  0x00000000
0x5250  0x00000000
0x5254  0x00000000
0x5258  0x00000000
0x525c  0x00000000
0x5260  0x00000000
0x5264  0x00000000
0x5268  0x00000000
0x526c  0x00000000
0x5270  0x00000000
0x5274  0x00000000
0x5278  0x00000000
0x527c  0x00000000

0x5400  0x00000000
0x5404  0x00000000
0x5408  0x00000000
0x540c  0x00000000
0x5410  0x00000000
0x5414  0x00000000
0x5418  0x00000000
0x541c  0x00000000
0x5420  0x00000000
0x5424  0x00000000
0x5428  0x00000000
0x542c  0x00000000
0x5430  0x00000000
0x5434  0x00000000
0x5438  0x00000000
0x543c  0x00000000
0x5440  0x00000000
0x5444  0x00000000
0x5448  0x00000000
0x544c  0x00000000
0x5450  0x00000000
0x5454  0x00000000
0x5458  0x00000000
0x545c  0x00000000
0x5460  0x00000000
0x5464  0x00000000
0x5468  0x00000000
0x546c  0x00000000
0x5470  0x00000000
0x5474  0x00000000
0x5478  0x00000000
0x547c  0x00000000
0x5480  0x00000000
0x5484  0x00000000
0x5488  0x00000000
0x548c  0x00000000
0x5490  0x00000000
0x5494  0x00000000
0x5498  0x00000000
0x549c  0x00000000
0x54a0  0x00000000
0x54a4  0x00000000
0x54a8  0x00000000
0x54ac  0x00000000
0x54b0  0x00000000
0x54b4  0x00000000
0x54b8  0x00000000
0x54bc  0x00000000
0x54c0  0x00000000
0x54c4  0x00000000
0x54c8  0x00000000
0x54cc  0x00000000
0x54d0  0x00000000
0x54d4  0x00000000
0x54d8  0x00000000
0x54dc  0x00000000
0x54e0  0x00000000
0x54e4  0x00000000
0x54e8  0x00000000
0x54ec  0x00000000
0x54f0  0x00000000
0x54f4  0x00000000
0x54f8  0x00000000
0x54fc  0x00000000
0x5500  0x00000000
0x5504  0x00000000
0x5508  0x00000000
0x550c  0x00000000
0x5510  0x00000000
0x5514  0x00000000
0x5518  0x00000000
0x551c  0x00000000
0x5520  0x00000000
0x5524  0x00000000
0x5528  0x00000000
0x552c  0x00000000
0x5530  0x00000000
0x5534  0x00000000
0x5538  0x00000000
0x553c  0x00000000
0x5540  0x00000000
0x5544  0x00000000
0x5548  0x00000000
0x554c  0x00000000
0x5550  0x00000000
0x5554  0x00000000
0x5558  0x00000000
0x555c  0x00000000
0x5560  0x00000000
0x5564  0x00000000
0x5568  0x00000000
0x556c  0x00000000
0x5570  0x00000000
0x5574  0x00000000
0x5578  0x00000000
0x557c  0x00000000
0x5580  0x00000000
0x5584  0x00000000
0x5588  0x00000000
0x558c  0x00000000
0x5590  0x00000000
0x5594  0x00000000
0x5598  0x00000000
0x559c  0x00000000
0x55a0  0x00000000
0x55a4  0x00000000
0x55a8  0x00000000
0x55ac  0x00000000
0x55b0  0x00000000
0x55b4  0x00000000
0x55b8  0x00000000
0x55bc  0x00000000
0x55c0  0x00000000
0x55c4  0x00000000
0x55c8  0x00000000
0x55cc  0x00000000
0x55d0  0x00000000
0x55d4  0x00000000
0x55d8  0x00000000
0x55dc  0x00000000
0x55e0  0x00000000
0x55e4  0x00000000
0x55e8  0x00000000
0x55ec  0x00000000
0x55f0  0x00000000
0x55f4  0x00000000
0x55f8  0x00000000
0x55fc  0x00000000
0x5600  0x00000000
0x5604  0x00000000
0x5608  0x00000000
0x560c  0x00000000
0x5610  0x00000000
0x5614  0x00000000
0x5618  0x00000000
0x561c  0x00000000
0x5620  0x00000000
0x5624  0x00000000
0x5628  0x00000000
0x562c  0x00000000
0x5630  0x00000000
0x5634  0x00000000
0x5638  0x00000000
0x563c  0x00000000
0x5640  0x00000000
0x5644  0x00000000
0x5648  0x00000000
0x564c  0x00000000
0x5650  0x00000000
0x5654  0x00000000
0x5658  0x00000000
0x565c  0x00000000
0x5660  0x00000000
0x5664  0x00000000
0x5668  0x00000000
0x566c  0x00000000
0x5670  0x00000000
0x5674  0x00000000
0x5678  0x00000000
0x567c  0x00000000

0x5800  0x00000000
0x5804  0x02000000
0x5808  0x00000000
0x580c  0x02000000
0x5810  0x00000000
0x5814  0x02000000
0x5818  0x00000000
0x581c  0x02000000
0x5820  0x00000000
0x5824  0x02000000
0x5828  0x00000000
0x582c  0x02000000
0x5830  0x00000000
0x5834  0x02000000
0x5838  0x00000000
0x583c  0x02000000
0x5840  0x00000000
0x5844  0x02000000
0x5848  0x00000000
0x584c  0x02000000
0x5850  0x00000000
0x5854  0x02000000
0x5858  0x00000000
0x585c  0x02000000
0x5860  0x00000000
0x5864  0x02000000
0x5868  0x00000000
0x586c  0x000000c8
0x5870  0x00000000
0x5874  0x02000000
0x5878  0x00000000
0x587c  0x02000000
0x5880  0x00000000
0x5884  0x00000000
0x5888  0x00000000
0x588c  0x02000000
0x5890  0x00000000
0x5894  0x02000000
0x5898  0x00000000
0x589c  0x02000000
0x58a0  0x00000000
0x58a4  0x02000000
0x58a8  0x00000000
0x58ac  0x02000000
0x58b0  0x00000000
0x58b4  0x02000000
0x58b8  0x00000000
0x58bc  0x02000000
0x58c0  0x00000000
0x58c4  0x02000000
0x58c8  0x00000000
0x58cc  0x02000000
0x58d0  0x00000000
0x58d4  0x02000000
0x58d8  0x00000000
0x58dc  0x02000000
0x58e0  0x00000000
0x58e4  0x02000000
0x58e8  0x00000000
0x58ec  0x02000000
0x58f0  0x00000000
0x58f4  0x02000000
0x58f8  0x00000000
0x58fc  0x02000000
0x5900  0x00000000
0x5904  0x00000024
0x5908  0x00000000
0x590c  0x02000000
0x5910  0x00000000
0x5914  0x00000000
0x5918  0x00000000
0x591c  0x00000000
0x5920  0x00000000
0x5924  0x00000000
0x5928  0x00000000
0x592c  0x00000000
0x5930  0x00000000
0x5934  0x00000000
0x5938  0x00000000
0x593c  0x00000000
0x5940  0x00000000
0x5944  0x00000000
0x5948  0x00000000
0x594c  0x00000000
0x5950  0x00000000
0x5954  0x00000000
0x5958  0x00000000
0x595c  0x00000000
0x5960  0x00000000
0x5964  0x00000000
0x5968  0x00000000
0x596c  0x00000000
0x5970  0x00000000
0x5974  0x00000000
0x5978  0x00000000
0x597c  0x00000000
0x5980  0x00000000
0x5984  0x00000000
0x5988  0x00000000
0x598c  0x00000000
0x5990  0x00000000
0x5994  0x00000000
0x5998  0x00000000
0x599c  0x00000000
0x59a0  0x00000000
0x59a4  0x00000000
0x59a8  0x00000000
0x59ac  0x00000000
0x59b0  0x00000000
0x59b4  0x00000000
0x59b8  0x00000000
0x59bc  0x00000000
0x59c0  0x00000000
0x59c4  0x00000000
0x59c8  0x00000000
0x59cc  0x00000000
0x59d0  0x00000000
0x59d4  0x00000000
0x59d8  0x00000000
0x59dc  0x00000000
0x59e0  0x00000000
0x59e4  0x00000000
0x59e8  0x00000000
0x59ec  0x00000000
0x59f0  0x00000000
0x59f4  0x00000000
0x59f8  0x00000000
0x59fc  0x00000000
0x5a00  0x00000000
0x5a04  0x00000000
0x5a08  0x00000000
0x5a0c  0x00000000

0x5c00  0x00000000
0x5c04  0x00000000
0x5c08  0x00000000
0x5c0c  0x00000000
0x5c10  0x00000000
0x5c14  0x00000000
0x5c18  0x00000000
0x5c1c  0x00000000
0x5c20  0x00000000
0x5c24  0x00000000
0x5c28  0x00000000
0x5c2c  0x00000000
0x5c30  0x00000000
0x5c34  0x00000000
0x5c38  0x00000000
0x5c3c  0x00000000
0x5c40  0x00000000
0x5c44  0x00000000
0x5c48  0x00000000
0x5c4c  0x00000000
0x5c50  0x00000000
0x5c54  0x00000000
0x5c58  0x00000000
0x5c5c  0x00000000
0x5c60  0x00000000
0x5c64  0x00000000
0x5c68  0x00000000
0x5c6c  0x00000000
0x5c70  0x00000000
0x5c74  0x00000000
0x5c78  0x00000000
0x5c7c  0x00000000
0x5c80  0x00000000
0x5c84  0x00000000
0x5c88  0x00000000
0x5c8c  0x00000000
0x5c90  0x00000000
0x5c94  0x00000000
0x5c98  0x00000000
0x5c9c  0x00000000
0x5ca0  0x00000000
0x5ca4  0x00000000
0x5ca8  0x00000000
0x5cac  0x00000000
0x5cb0  0x00000000
0x5cb4  0x00000000
0x5cb8  0x00000000
0x5cbc  0x00000000
0x5cc0  0x00000000
0x5cc4  0x00000000
0x5cc8  0x00000000
0x5ccc  0x00000000
0x5cd0  0x00000000
0x5cd4  0x00000000
0x5cd8  0x00000000
0x5cdc  0x00000000
0x5ce0  0x00000000
0x5ce4  0x00000000
0x5ce8  0x00000000
0x5cec  0x00000000
0x5cf0  0x00000000
0x5cf4  0x00000000
0x5cf8  0x00000000
0x5cfc  0x00000000
0x5d00  0x00000000
0x5d04  0x00000000
0x5d08  0x00000000
0x5d0c  0x00000000
0x5d10  0x00000000
0x5d14  0x00000000
0x5d18  0x00000000
0x5d1c  0x00000000

0x6000  0x00000000
0x6004  0x00000000
0x6008  0x00000004

0x6800  0x04130034
0x6804  0x3c081082
0x6808  0x01000609
0x680c  0x16695a00
0x6810  0x81020010
0x6814  0xffffffff
0x6818  0x00000000
0x681c  0x00000000
0x6820  0x00000000
0x6824  0x00000000
0x6828  0x00000000
0x682c  0x00000000
0x6830  0xfffc3ccf
0x6834  0xfffc0fff
0x6838  0x00000000
0x683c  0x00000000
0x6840  0x00000004
0x6844  0x00000000

0x7000  0x00000188
0x7004  0x00000000
0x7008  0x00000000
0x700c  0x000000c0
0x7010  0x0a000064
0x7014  0x02008273
0x7018  0x00570081
0x701c  0x68848353
0x7020  0x00000000
0x7024  0x00000000
0x7028  0x00000000
0x702c  0x00000000
0x7030  0x00000000

0x7c00  0x00000000
0x7c04  0x00000000
0x7c08  0x00000000
0x7c0c  0x00000000
0x7c10  0x00000000
0x7c14  0x00000000
0x7c18  0x00000000
0x7c1c  0x00000000
0x7c20  0x00000000
0x7c24  0x00000000
0x7c28  0x00000000
0x7c2c  0x00000000
0x7c30  0x00000000
0x7c34  0x00000000
0x7c38  0x00000000
0x7c3c  0x00000000
0x7c40  0x00000000
0x7c44  0x00000000
0x7c48  0x00000000
0x7c4c  0x00000000
0x7c50  0x00000000
0x7c54  0x00000000
0x7c58  0x00000000
0x7c5c  0x00000000
0x7c60  0x00000000
0x7c64  0x00000000
0x7c68  0x00000000
0x7c6c  0x00000000
0x7c70  0x00000000
0x7c74  0x00000000
0x7c78  0x00000000
0x7c7c  0x00000000
0x7c80  0x00000000
0x7c84  0x00000000
0x7c88  0x00000000
0x7c8c  0x00000000
0x7c90  0x00000000
0x7c94  0x00000000
0x7c98  0x00000000
0x7c9c  0x00000000
0x7ca0  0x00000000
0x7ca4  0x00000000
0x7ca8  0x00000000
0x7cac  0x00000000
0x7cb0  0x00000000
0x7cb4  0x00000000
0x7cb8  0x00000000
0x7cbc  0x00000000
0x7cc0  0x00000000
0x7cc4  0x00000000
0x7cc8  0x00000000
0x7ccc  0x00000000
0x7cd0  0x00000000
0x7cd4  0x00000000
0x7cd8  0x00000000
0x7cdc  0x00000000
0x7ce0  0x00000000
0x7ce4  0x00000000
0x7ce8  0x00000000
0x7cec  0x00000000
0x7cf0  0x00000000
0x7cf4  0x00000000
0x7cf8  0x00000000
0x7cfc  0x00000000
0x7d00  0x00000000
0x7d04  0x00000000
0x7d08  0x00000000
0x7d0c  0x00000000
0x7d10  0x00000000
0x7d14  0x00000000
0x7d18  0x00000000
0x7d1c  0x00000000
0x7d20  0x00000000
0x7d24  0x00000000
0x7d28  0x00000000
0x7d2c  0x00000000
0x7d30  0x00000000
0x7d34  0x00000000
0x7d38  0x00000000
0x7d3c  0x00000000
0x7d40  0x00000000
0x7d44  0x00000000
0x7d48  0x00000000
0x7d4c  0x00000000
0x7d50  0x00000000
0x7d54  0x00000000
0x7d58  0x00000000
0x7d5c  0x00000000
0x7d60  0x00000000
0x7d64  0x00000000
0x7d68  0x00000000
0x7d6c  0x00000000
0x7d70  0x00000000
0x7d74  0x00000000
0x7d78  0x00000000
0x7d7c  0x00000000
0x7d80  0x00000000
0x7d84  0x00000000
0x7d88  0x00000000
0x7d8c  0x00000000
0x7d90  0x00000000
0x7d94  0x00000000
0x7d98  0x00000000
0x7d9c  0x00000000
0x7da0  0x00000000
0x7da4  0x00000000
0x7da8  0x00000000
0x7dac  0x00000000
0x7db0  0x00000000
0x7db4  0x00000000
0x7db8  0x00000000
0x7dbc  0x00000000
0x7dc0  0x00000000
0x7dc4  0x00000000
0x7dc8  0x00000000
0x7dcc  0x00000000
0x7dd0  0x00000000
0x7dd4  0x00000000
0x7dd8  0x00000000
0x7ddc  0x00000000
0x7de0  0x00000000
0x7de4  0x00000000
0x7de8  0x00000000
0x7dec  0x00000000
0x7df0  0x00000000
0x7df4  0x00000000
0x7df8  0x00000000
0x7dfc  0x00000000
0x7e00  0x00000000
0x7e04  0x00000000
0x7e08  0x00000000
0x7e0c  0x00000000
0x7e10  0x00000000
0x7e14  0x00000000
0x7e18  0x00000000
0x7e1c  0x00000000
0x7e20  0x00000000
0x7e24  0x00000000
0x7e28  0x00000000
0x7e2c  0x00000000
0x7e30  0x00000000
0x7e34  0x00000000
0x7e38  0x00000000
0x7e3c  0x00000000

Roger Heflin

author

2008-11-21 09:34 UTC #24 lore raw

Quoted 8 lines · Matt Carlson

Quoted 1 line · Roger Heflin
Matt Carlson wrote:
Yes, I remember hearing something about this problem too. That is a firmware problem though. The 5789 does not have any management firmware, so that shouldn't be the case here.

Gotcha.

Quoted 82 lines

Quoted 39 lines
Quoted 39 lines
Quoted 5 lines
If someone else runs into this issue, since I have 2 ports I would be able to do some testing on it, right now my first port is locked up, and the machine is running fine on the second port.
lspci -vvv for the first (bad) port:
Ah. There it is.
Quoted 32 lines
02:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5789 Gigabit Ethernet PCI Express (rev 11)
Subsystem: Foxconn International, Inc. Unknown device 0cc1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at fd8f0000 (64-bit, non-prefetchable) [size=64K] Expansion ROM at <ignored> [disabled]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D3 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: 0101b8102a0f7b0c Data: f21e
Capabilities: [d0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <4us, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s <2us, L1 <64us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Virtual Channel
Quoted 6 lines
Quoted 2 lines
Hmmm. No smoking gun. Perhaps the register dump will help.
driver: tg3
version: 3.94
firmware-version: 5789-v3.29a
bus-info: 0000:02:00.0
O.K. I'll see if I can find any problems like this in the firmware archives.
Quoted 11 lines
tg3.c:v3.94 (August 14, 2008)
tg3 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19 tg3 0000:02:00.0: setting latency timer to 64
tg3 0000:05:01.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
tg3: eth0: Link is up at 1000 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
Right now I brought the interface back up (it is still broken) and setup a network ip on it that other machines can ping.
The registers are included at the end of the email.
O.K. I'll pour over the dump and get back to you.
More below.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219
dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Usually the tg3_tx_timeout function dumps a few registers before resetting the chip, but I don't see that here. Have you seen any dumps since then?
Is this the dump?
This would be it. Thanks.
Quoted 5 lines
Nov 12 14:58:13 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 14:58:13 computer kernel: tg3: DEBUG: MAC_TX_STATUS[00000008]
MAC_RX_STATUS[00000006]
Nov 12 14:58:13 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000010]
WDMAC_STATUS[00000000]

Quoted 4 lines

Here the Read DMA Status register is reporting a Read DMA PCI Parity Error. I've seen this before...very recently in fact. The problem was that the chipset was not programmed by the BIOS correctly. In that particular case, a BIOS upgrade solved the problem. YMMV.

The board I have is a OLD board (but new to me) and I have what appears to be the last bios that was officially released for it, and cannot find any newer updates that what I have.

Quoted 16 lines

Quoted 14 lines

Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=2c00
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=1400
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: tg3_stop_block timed out, ofs=4800
enable_bit=2
Nov 12 14:58:13 computer kernel: tg3: eth0: Link is down.
Nov 12 14:58:16 computer kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
Nov 12 14:58:16 computer kernel: tg3: eth0: Flow control is on for TX and on for RX.
Nov 12 15:20:37 computer kernel: tg3: eth0: transmit timed out, resetting
Nov 12 15:20:37 computer kernel: tg3: DEBUG: MAC_TX_STATUS[0000000b]
MAC_RX_STATUS[00000000]
Nov 12 15:20:37 computer kernel: tg3: DEBUG: RDMAC_STATUS[00000000]

Quoted 1 line

WDMAC_STATUS[00000000]

Quoted 5 lines

Here the MAC TX Status register is reporting that the link is up, but the device is sending pause frames and rx is currently rx off'd.
Does the same problem happen if flow control is disabled?

I have disabled flow control (live) but not rebooted yet I won't have time to reboot and test until sometime tomorrow.

Matt Carlson

2008-11-20 02:52 UTC #25 lore raw

In reply to Roger Heflin

Quoted 42 lines · Roger Heflin

I have duplicate this with kernel 2.6.27.2 and 2.6.27.5, no
extra modules, tg3 Gbit networking. I have not yet tested
earlier kernels to see if this has been around for a while.
So far I have had this error happen 5 times (MTBF is maybe 12 hours), 4 of the 5 times resulted in the networking being broken, one time things came back by itself without a reboot, I believe in this case the hang was traffic coming into the machine vs the other times going out of the machine.
Unloading all of the network modules and reloading them did not correct the problem.
Searching google finds a couple of other people getting the same error but they have a different network chipset (e1000 and a rt811C chipset), which makes me thing that there is
something interacting bad with the network. Or does this
error truly mean that the network chipset for some unknown reason locked itself up?
http://www.google.com/url?sa=U&start=4&q=http://kerneltrap.org/mailarchive/linux-netdev/2008/8/6/2838184&ei=rU8ZScysAon8edz5xKgO&sig2=Wxp7IkUtdgORGZiflxvppg&usg=AFQjCNHzPwsCOmLGKmtX4q_FEpk6oubxxg
http://article.gmane.org/gmane.linux.network/110238
The changes I made recently were to upgrade my MB (old was E100 on a 100Mbit network,new is tg3 on a Gbit network, cpu and memory are the same, MB chipset is a intel 955 chipset vs the old being a intel 915 chipset).
Autoneg is turned on all around, the GBit switch is a
8-port Dlink switch. The network seems to otherwise be working
correctly.
I did test the network under decent load and the error did not appear to be any more likely under load, and typically the network is under very light load 2-3MB/second.
The machine originally had 2 HT CPU's showing up, I turned off HT so that only one cpu was showing, but this did not change the error.
I am first turning off all offload capabilities on tg3 and going to see if that changes anything.

Quoted 7 lines

The next thing I am going to be doing is to turn of GB capability on the networking and see if that does anything.
I also have a second tg3 port that is slightly different, so I may try that eventually.
What else can I look at?

Can you cut-and-paste the driver sign-on message and the output of lspci -vvv -xxx, and 'ethtool -i eth0'? Can you also obtain a register dump via ethtool when the problem happens?

Quoted 50 lines

Nov 11 00:44:39 computer kernel: ------------[ cut here ]------------
Nov 11 00:44:39 computer kernel: WARNING: at net/sched/sch_generic.c:219 
dev_watchdog+0xfe/0x17e()
Nov 11 00:44:39 computer kernel: NETDEV WATCHDOG: eth0 (tg3): transmit timed out
Nov 11 00:44:39 computer kernel: Modules linked in: nfsd auth_rpcgss exportfs 
w83627ehf hwmon_vid hwmon nfs lockd nfs_acl sunrpc ipv6 xfs raid456 async_xor async_memcpy async_tx xor video output sbs sbshc battery ac lgdt330x cx88_dvb wm8775 cx88_vp3054_i2c cx25840 tuner_simple tuner_types tda9887 tda8290 tuner mt2131 s5h1409 snd_hda_intel snd_seq_dummy ivtv cx8800 snd_seq_oss cx88_alsa cx8802 cx88xx cx23885 snd_seq_midi_event snd_seq ir_common videodev v4l1_compat i2c_algo_bit cx2341x firewire_ohci iTCO_wdt snd_seq_device compat_ioctl32 videobuf_dvb i2c_i801 firewire_core tveeprom floppy iTCO_vendor_support v4l2_common snd_pcm_oss dvb_core pcspkr tg3 sata_sil i2c_core btcx_risc videobuf_dma_sg crc_itu_t snd_mixer_oss libphy videobuf_core snd_pcm parport_pc parport snd_timer snd soundcore button snd_page_alloc sg dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd [last unloaded: eeprom]
Nov 11 00:44:39 computer kernel: Pid: 0, comm: swapper Not tainted 2.6.27.5 #2
Nov 11 00:44:39 computer kernel:  [<c042524f>] warn_slowpath+0x61/0x83
Nov 11 00:44:39 computer kernel:  [<c05663a4>] usb_hcd_submit_urb+0x75c/0x811
Nov 11 00:44:39 computer kernel:  [<c0594972>] hiddev_hid_event+0x0/0x64
Nov 11 00:44:39 computer kernel:  [<c058ce80>] hid_process_event+0x58/0x5f
Nov 11 00:44:39 computer kernel:  [<c04e13d6>] __next_cpu+0x12/0x21
Nov 11 00:44:39 computer kernel:  [<c041cbe3>] find_busiest_group+0x23e/0x672
Nov 11 00:44:39 computer kernel:  [<c0439d1e>] clocksource_get_next+0x39/0x3f
Nov 11 00:44:39 computer kernel:  [<c0438e51>] update_wall_time+0x567/0x70c
Nov 11 00:44:39 computer kernel:  [<c040783e>] read_tsc+0x6/0x22
Nov 11 00:44:39 computer kernel:  [<c04387e8>] getnstimeofday+0x37/0xc1
Nov 11 00:44:39 computer kernel:  [<f8829a83>] uhci_scan_schedule+0x11b/0x6b0 
[uhci_hcd]
Nov 11 00:44:39 computer kernel:  [<c05b16ba>] dev_watchdog+0xfe/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c66f>] __mod_timer+0x99/0xa3
Nov 11 00:44:39 computer kernel:  [<c05654b6>] rh_timer_func+0x0/0x5
Nov 11 00:44:39 computer kernel:  [<c05654ae>] usb_hcd_poll_rh_status+0x12b/0x133
Nov 11 00:44:39 computer kernel:  [<c043bca8>] tick_dev_program_event+0x1e/0x81
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c042c2b4>] run_timer_softirq+0x10e/0x167
Nov 11 00:44:39 computer kernel:  [<c05b15bc>] dev_watchdog+0x0/0x17e
Nov 11 00:44:39 computer kernel:  [<c0428d3e>] __do_softirq+0x5d/0xc1
Nov 11 00:44:39 computer kernel:  [<c0428dd4>] do_softirq+0x32/0x36
Nov 11 00:44:39 computer kernel:  [<c0412939>] smp_apic_timer_interrupt+0x6e/0x79
Nov 11 00:44:39 computer kernel:  [<c040431c>] apic_timer_interrupt+0x28/0x30
Nov 11 00:44:39 computer kernel:  [<c0408582>] mwait_idle+0x32/0x38
Nov 11 00:44:39 computer kernel:  [<c040255d>] cpu_idle+0xbd/0xd5
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/