Skip to contentSkip to conversation

New: tg3: eth0: The system may be re-ordering memory-mapped I/O cycles

Bugme-new Andrew Morton 3 participants 7 messages Feb 14, 2008 – Feb 15, 2008 1 message not in the archive

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author
#1 lore raw
Quoted 101 lines · On Thu, 14 Feb 2008 01:59:12 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=9990

           Summary: tg3: eth0: The system may be re-ordering memory-mapped
                    I/O cycles
           Product: Drivers
           Version: 2.5
     KernelVersion: 2.6.24-git18
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Latest working kernel version: 2.6.24
Earliest failing kernel version: 2.6.24-git18
Distribution: Debian/testing
Hardware Environment:
Software Environment:
Problem Description:

Feb 11 13:11:52 www kernel: [   12.015569] tg3: eth0: Link is up at 100 Mbps,
full duplex.
Feb 11 13:11:52 www kernel: [   12.015633] tg3: eth0: Flow control is on for TX
and on for RX.
Feb 11 13:33:44 www kernel: [ 1328.538204] tg3: eth0: The system may be
re-ordering memory-mapped I/O cycles to the network
device, attempting to recover. Please report the problem to the driver
maintainer and include system chipset information.
Feb 11 13:33:44 www kernel: [ 1328.667255] tg3: eth0: Link is down.
Feb 11 13:33:46 www kernel: [ 1330.560734] tg3: eth0: Link is up at 100 Mbps,
full duplex.
Feb 11 13:33:46 www kernel: [ 1330.560734] tg3: eth0: Flow control is on for TX

and on for RX.

After that, the machine rebooted (panic?)

Feb 11 13:35:14 www kernel: klogd 1.5.0#1.1, log source = /proc/kmsg started.

lspci -vvv info:
02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
Subsystem: Compaq Computer Corporation NC7782 Gigabit Server Adapter
(PCI-X, 10,100,1000-T)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 64 (16000ns min), Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 19
        Region 0: Memory at fdf70000 (64-bit, non-prefetchable) [size=64K]
        [virtual] Expansion ROM at 88140000 [disabled] [size=64K]
        Capabilities: [40] PCI-X non-bridge device
                Command: DPERE- ERO- RBC=2048 OST=1
                Status: Dev=02:02.0 64bit+ 133MHz+ SCD- USC- DC=simple

DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data <?>
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: fd7ffd6fdf7deeb8 Data: bdfd
Kernel driver in use: tg3
Kernel modules: tg3

02:02.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
Subsystem: Compaq Computer Corporation NC7782 Gigabit Server Adapter
(PCI-X, 10,100,1000-T)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 64 (16000ns min), Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 20
        Region 0: Memory at fdf60000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] PCI-X non-bridge device
                Command: DPERE- ERO+ RBC=512 OST=1
                Status: Dev=02:02.1 64bit+ 133MHz+ SCD- USC- DC=simple

DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data <?>
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: f73feeefffffe7f8 Data: 9bcd
Kernel driver in use: tg3
Kernel modules: tg3

Steps to reproduce:

#2 lore raw
Quoted 100 lines · Andrew Morton
Quoted 98 lines · On Thu, 14 Feb 2008 01:59:12 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=9990

           Summary: tg3: eth0: The system may be re-ordering memory-mapped
                    I/O cycles
           Product: Drivers
           Version: 2.5
     KernelVersion: 2.6.24-git18
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Latest working kernel version: 2.6.24
Earliest failing kernel version: 2.6.24-git18
Distribution: Debian/testing
Hardware Environment:
Software Environment:
Problem Description:

Feb 11 13:11:52 www kernel: [   12.015569] tg3: eth0: Link is up at 100 Mbps,
full duplex.
Feb 11 13:11:52 www kernel: [   12.015633] tg3: eth0: Flow control is on for TX
and on for RX.
Feb 11 13:33:44 www kernel: [ 1328.538204] tg3: eth0: The system may be
re-ordering memory-mapped I/O cycles to the network
device, attempting to recover. Please report the problem to the driver
maintainer and include system chipset information.
Feb 11 13:33:44 www kernel: [ 1328.667255] tg3: eth0: Link is down.
Feb 11 13:33:46 www kernel: [ 1330.560734] tg3: eth0: Link is up at 100 Mbps,
full duplex.
Feb 11 13:33:46 www kernel: [ 1330.560734] tg3: eth0: Flow control is on for TX

and on for RX.

After that, the machine rebooted (panic?)

Feb 11 13:35:14 www kernel: klogd 1.5.0#1.1, log source = /proc/kmsg started.

lspci -vvv info:
02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
Subsystem: Compaq Computer Corporation NC7782 Gigabit Server Adapter
(PCI-X, 10,100,1000-T)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 64 (16000ns min), Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 19
        Region 0: Memory at fdf70000 (64-bit, non-prefetchable) [size=64K]
        [virtual] Expansion ROM at 88140000 [disabled] [size=64K]
        Capabilities: [40] PCI-X non-bridge device
                Command: DPERE- ERO- RBC=2048 OST=1
                Status: Dev=02:02.0 64bit+ 133MHz+ SCD- USC- DC=simple

DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable+ DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data <?>
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: fd7ffd6fdf7deeb8 Data: bdfd
Kernel driver in use: tg3
Kernel modules: tg3

02:02.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
Subsystem: Compaq Computer Corporation NC7782 Gigabit Server Adapter
(PCI-X, 10,100,1000-T)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 64 (16000ns min), Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 20
        Region 0: Memory at fdf60000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] PCI-X non-bridge device
                Command: DPERE- ERO+ RBC=512 OST=1
                Status: Dev=02:02.1 64bit+ 133MHz+ SCD- USC- DC=simple

DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data <?>
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Address: f73feeefffffe7f8 Data: 9bcd
Kernel driver in use: tg3
Kernel modules: tg3

Quoted 3 lines
Quoted 3 lines

Steps to reproduce:

That should be a simple matter of adding the right pci-ids to tg3_get_invariants -- hopefully Ralf will respond and we can get that knocked out quickly.

#3 lore raw
Quoted 22 lines · Andy Gospodarek
Quoted 4 lines · Andrew Morton
Quoted 2 lines · On Thu, 14 Feb 2008 01:59:12 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=9990

Quoted 16 lines
Quoted 6 lines
           Summary: tg3: eth0: The system may be re-ordering memory-mapped
                    I/O cycles
           Product: Drivers
           Version: 2.5
     KernelVersion: 2.6.24-git18
          Platform: All
Quoted 10 lines
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: [email protected]
        ReportedBy: [email protected]
Quoted 5 lines

That should be a simple matter of adding the right pci-ids to tg3_get_invariants -- hopefully Ralf will respond and we can get that knocked out quickly.

It doesn't look like it was re-ordered IO. If it was, it should have self-recovered without hitting the BUG().

One possibility is that the nr_frags in the SKB got corrupted before the TX SKB was freed. The driver relies on the nr_frags in the SKB to find the packet boundaries in the TX ring. If it cannot find the packet boundaries, it will exhibit the same symptom as re-ordered IO, only that it cannot be self-recovered.

Ralf, please try this debug patch with the same traffic condition you ran before. This patch stores the nr_frags when transmitting an SKB. During tx completion, it will compare the stored nr_frags with the one in the SKB and will print out something in dmesg if they don't match.

drivers/net/tg3.c +12 -1
index db606b6..73f1ddd 100644
@@ -3324,12 +3324,20 @@ static void tg3_tx(struct tg3 *tp)
 		struct tx_ring_info *ri = &tp->tx_buffers[sw_idx];
 		struct sk_buff *skb = ri->skb;
 		int i, tx_bug = 0;
+		unsigned short nr_frags = ri->nr_frags;
 
 		if (unlikely(skb == NULL)) {
 			tg3_tx_recover(tp);
 			return;
 		}
 
+		if (nr_frags != skb_shinfo(skb)->nr_frags) {
+			printk(KERN_ALERT "tg3: %s: Tx skb->nr_frags corrupted "
+				"before skb is freed. Expected nr_frags %d, "
+				"corrupted nr_frags %d\n", tp->dev->name,
+				nr_frags, skb_shinfo(skb)->nr_frags);
+		}
+
 		pci_unmap_single(tp->pdev,
 				 pci_unmap_addr(ri, mapping),
 				 skb_headlen(skb),

@@ -3339,7 +3347,7 @@ static void tg3_tx(struct tg3 *tp)
 
 		sw_idx = NEXT_TX(sw_idx);
 
-		for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+		for (i = 0; i < nr_frags; i++) {
 			ri = &tp->tx_buffers[sw_idx];
 			if (unlikely(ri->skb != NULL || sw_idx == hw_idx))
 				tx_bug = 1;

@@ -4105,6 +4113,7 @@ static int tigon3_dma_hwbug_workaround(struct tg3 *tp, struct sk_buff *skb,
 				 len, PCI_DMA_TODEVICE);
 		if (i == 0) {
 			tp->tx_buffers[entry].skb = new_skb;
+			tp->tx_buffers[entry].nr_frags = 0;
 			pci_unmap_addr_set(&tp->tx_buffers[entry], mapping, new_addr);
 		} else {
 			tp->tx_buffers[entry].skb = NULL;

@@ -4211,6 +4220,7 @@ static int tg3_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	mapping = pci_map_single(tp->pdev, skb->data, len, PCI_DMA_TODEVICE);
 
 	tp->tx_buffers[entry].skb = skb;
+	tp->tx_buffers[entry].nr_frags = skb_shinfo(skb)->nr_frags;
 	pci_unmap_addr_set(&tp->tx_buffers[entry], mapping, mapping);
 
 	tg3_set_txd(tp, entry, mapping, len, base_flags,

@@ -4388,6 +4398,7 @@ static int tg3_start_xmit_dma_bug(struct sk_buff *skb, struct net_device *dev)
 	mapping = pci_map_single(tp->pdev, skb->data, len, PCI_DMA_TODEVICE);
 
 	tp->tx_buffers[entry].skb = skb;
+	tp->tx_buffers[entry].nr_frags = skb_shinfo(skb)->nr_frags;
 	pci_unmap_addr_set(&tp->tx_buffers[entry], mapping, mapping);
 
 	would_hit_hwbug = 0;
drivers/net/tg3.h +1 -0
index 3938eb3..d4a3aca 100644
@@ -2098,6 +2098,7 @@ struct tx_ring_info {
 	struct sk_buff			*skb;
 	DECLARE_PCI_UNMAP_ADDR(mapping)
 	u32				prev_vlan_tag;
+	unsigned short			nr_frags;
 };
 
 struct tg3_config_info {
#4 lore raw
Quoted 23 lines · Michael Chan
Quoted 22 lines · Andy Gospodarek
Quoted 20 lines · Andrew Morton
Quoted 18 lines · On Thu, 14 Feb 2008 01:59:12 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=9990

           Summary: tg3: eth0: The system may be re-ordering memory-mapped
                    I/O cycles
           Product: Drivers
           Version: 2.5
     KernelVersion: 2.6.24-git18
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Network
        AssignedTo: [email protected]
        ReportedBy: [email protected]
Quoted 9 lines
Quoted 5 lines

That should be a simple matter of adding the right pci-ids to tg3_get_invariants -- hopefully Ralf will respond and we can get that knocked out quickly.

It doesn't look like it was re-ordered IO. If it was, it should have self-recovered without hitting the BUG().

Good catch, Michael! I missed that it paniced since I expect to see some sort of backtrace when that happens. We should try and get that bridge added to the list though, to avoid repeated complaints that there is a tg3 bug.

#5 lore raw
Quoted 8 lines · Andy Gospodarek
Quoted 7 lines · Michael Chan
Quoted 5 lines · Andy Gospodarek

That should be a simple matter of adding the right pci-ids to tg3_get_invariants -- hopefully Ralf will respond and we can get that knocked out quickly.

Quoted 10 lines
Quoted 3 lines

It doesn't look like it was re-ordered IO. If it was, it should have self-recovered without hitting the BUG().

Good catch, Michael! I missed that it paniced since I expect to see some sort of backtrace when that happens. We should try and get that bridge added to the list though, to avoid repeated complaints that there is a tg3 bug.

Andy, I think you still missed my point. I don't believe this problem was caused by the bridge or the chipset at all. Some corruption caused us to not find the SKB in the TX ring where it was expected. So the driver assumed it was the bridge re-ordering I/O and printed that warning message and took recovery action. The recovery action had no effect in this case since apparently it was caused by something else and the corruption happened again later. This 2nd time, we hit the BUG_ON() seeing that the recovery action did not work.

#6 lore raw
Quoted 20 lines · Michael Chan
Quoted 8 lines · Andy Gospodarek
Quoted 7 lines · Michael Chan
Quoted 5 lines · Andy Gospodarek

That should be a simple matter of adding the right pci-ids to tg3_get_invariants -- hopefully Ralf will respond and we can get that knocked out quickly.

Quoted 10 lines
Quoted 3 lines

It doesn't look like it was re-ordered IO. If it was, it should have self-recovered without hitting the BUG().

Good catch, Michael! I missed that it paniced since I expect to see some sort of backtrace when that happens. We should try and get that bridge added to the list though, to avoid repeated complaints that there is a tg3 bug.

Quoted 10 lines

Andy, I think you still missed my point. I don't believe this problem was caused by the bridge or the chipset at all. Some corruption caused us to not find the SKB in the TX ring where it was expected. So the driver assumed it was the bridge re-ordering I/O and printed that warning message and took recovery action. The recovery action had no effect in this case since apparently it was caused by something else and the corruption happened again later. This 2nd time, we hit the BUG_ON() seeing that the recovery action did not work.

Ah, I see. Due to at leat a 2 second delay between the message and the panic, I figured it would be good data to gather....

#7 lore raw
Quoted 12 lines · Andy Gospodarek
Quoted 10 lines · Michael Chan

Andy, I think you still missed my point. I don't believe this problem was caused by the bridge or the chipset at all. Some corruption caused us to not find the SKB in the TX ring where it was expected. So the driver assumed it was the bridge re-ordering I/O and printed that warning message and took recovery action. The recovery action had no effect in this case since apparently it was caused by something else and the corruption happened again later. This 2nd time, we hit the BUG_ON() seeing that the recovery action did not work.

Quoted 5 lines

Ah, I see. Due to at leat a 2 second delay between the message and the panic, I figured it would be good data to gather....

Yeah, 2 seconds for the link to come up after chip reset to recover. It then panicked sometime later and rebooted about 90 seconds after the initial warning message.

It was also running at the slower 100Mbps link speed. Tx packets stay longer in the TX ring at this slower speed, increasing the window of time that the nr_frags in the SKB can be corrupted.

Ralf, please try the debug patch that I sent out earlier. Thanks.