Skip to contentSkip to conversation

BCM5704 performance questions.

DISCUSSION Ben Greear 5 participants 17 messages Jun 9, 2005 – Jun 10, 2005

Subsystems (2)
lore
Share & tools
lore

Ben Greear

author
#0 lore raw

Hello!

I have a 4-port NIC by silicom-usa.com that uses the BCM5704 (rev 10) chipset.

It's running in a PCI-X bus (100 or 133Mhz). CPUs are dual xeon 2.8Ghz, 1MB cache, 1GB RAM, etc). Kernel is 2.6.11 + my hacks (no hacks to tg3, minor hacks to e1000 and other parts of the networking stacks).

I am trying to bridge as much traffic as possible across two interfaces, using a proprietary kernel module.
The network traffic is 1514 byte packets, generated by a modified version of pktgen running on another machine with similar hardware (Intel NICs).

With the BCM NIC I can get about 600Mbps in one direction and about 800Mbps in the other..with a great deal of dropped packets. With the Intel 4-port NIC (same machine, different PCI slot, and also from Silicom-usa.com) I can get 900+Mbps in both directions with virtually no drops.

So:

* Is the BCM5704 chipset/driver really that much slower?

* Is there some information on tuning the tg3 somewhere?
(I didn't see a Documentation/networking/tg3.txt file, for instance)

* Is there a way to verify the bus speed that the NIC is running at? (ethtool -d ethX gives lots of meaningless (to me) hex)

Please let me know if more information would be useful.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#1 lore raw
Quoted 3 lines · Ben Greear

* Is the BCM5704 chipset/driver really that much slower?

Unfortunately, the 5704 requires the "ONE_DMA" workaround which will limit throughput in a PCIX 100/133 bus. If you comment out the line that sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved performance. However, you may run into some DMA issues on certain systems.

Quoted 6 lines

* Is there some information on tuning the tg3 somewhere?
(I didn't see a Documentation/networking/tg3.txt file, for instance)

* Is there a way to verify the bus speed that the NIC is running at? (ethtool -d ethX gives lots of meaningless (to me) hex)

tg3 probing string for each device will tell you the bus type, width, and speed.

author
#2 lore raw
Quoted 9 lines · Michael Chan
Quoted 2 lines · Ben Greear

* Is the BCM5704 chipset/driver really that much slower?

Unfortunately, the 5704 requires the "ONE_DMA" workaround which will limit throughput in a PCIX 100/133 bus. If you comment out the line that sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved performance. However, you may run into some DMA issues on certain systems.

Is there any way I can tell which systems are affected? It won't be an option for me to purposefully ship possibly busted drivers/hardware, but if I can be certain that my systems are immune, I will try this modification.

Quoted 10 lines
Quoted 6 lines

* Is there some information on tuning the tg3 somewhere?
(I didn't see a Documentation/networking/tg3.txt file, for instance)

* Is there a way to verify the bus speed that the NIC is running at? (ethtool -d ethX gives lots of meaningless (to me) hex)

tg3 probing string for each device will tell you the bus type, width, and speed.

Thanks, I found it.

Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#3 lore raw
Quoted 8 lines · Ben Greear
Quoted 6 lines · Michael Chan

Unfortunately, the 5704 requires the "ONE_DMA" workaround which will limit throughput in a PCIX 100/133 bus. If you comment out the line that sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved performance. However, you may run into some DMA issues on certain systems.

Quoted 5 lines

Is there any way I can tell which systems are affected? It won't be an option for me to purposefully ship possibly busted drivers/hardware, but if I can be certain that my systems are immune, I will try this modification.

I mentioned this so that you could verify that the slow performance was indeed caused by ONE_DMA. Even if your system is affected, it's a very subtle problem that won't show up right away and should allow you to get some performance numbers.

Unfortunately, if indeed it is ONE_DMA, there is no easy way for us to tell which system is affected. And the recommendation is to turn it on for all 5704 in PCIX 100/133.

author
#4 lore raw
Quoted 16 lines · Michael Chan
Quoted 8 lines · Ben Greear
Quoted 5 lines · Michael Chan

Unfortunately, the 5704 requires the "ONE_DMA" workaround which will limit throughput in a PCIX 100/133 bus. If you comment out the line that sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved performance. However, you may run into some DMA issues on certain systems.

Quoted 5 lines

Is there any way I can tell which systems are affected? It won't be an option for me to purposefully ship possibly busted drivers/hardware, but if I can be certain that my systems are immune, I will try this modification.

Quoted 4 lines

I mentioned this so that you could verify that the slow performance was indeed caused by ONE_DMA. Even if your system is affected, it's a very subtle problem that won't show up right away and should allow you to get some performance numbers.

I commented out the code and ran the pktgen test again. It may be a small bit better, but not much: 770Mbps in one direction, 750Mbps in the other.

Have you done any tests with 2 tg3 NICs in a single machine to see if they can run at or near line speed (full duplex)?

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#5 lore raw
Quoted 3 lines · Ben Greear

I commented out the code and ran the pktgen test again. It may be a small bit better, but not much: 770Mbps in one direction, 750Mbps in the other.

The latest tg3 driver will print out the dma_rwctrl at probe time. Can you check the value to make sure ONE_DMA is disabled? Bit 14 sets ONE_DMA.

Quoted 3 lines

Have you done any tests with 2 tg3 NICs in a single machine to see if they can run at or near line speed (full duplex)?

Yes, we have years ago but not with the tg3 driver. We set up the 5704 to bridge (or route, I don't remember) one port to the other. It cannot bridge at line-rate with the ONE_DMA workaround enabled.

author
#6 lore raw
Quoted 9 lines · Michael Chan
Quoted 3 lines · Ben Greear

I commented out the code and ran the pktgen test again. It may be a small bit better, but not much: 770Mbps in one direction, 750Mbps in the other.

The latest tg3 driver will print out the dma_rwctrl at probe time. Can you check the value to make sure ONE_DMA is disabled? Bit 14 sets ONE_DMA.

I don't see any printout in /var/log/messages that looks like it relates to the dma_rwctrl. I'm using the driver in 2.6.11..maybe it is not recent enough to print this information out?

Quoted 7 lines
Quoted 3 lines

Have you done any tests with 2 tg3 NICs in a single machine to see if they can run at or near line speed (full duplex)?

Yes, we have years ago but not with the tg3 driver. We set up the 5704 to bridge (or route, I don't remember) one port to the other. It cannot bridge at line-rate with the ONE_DMA workaround enabled.

Ok. Do you have other chipsets/NICs that can handle faster GigE speeds?

If you'd like to send me some multi-port NICs to play with I'll be happy to report my findings :)

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
maintainer
#7 lore raw

From: Ben Greear <[email protected]>
Date: Fri, 10 Jun 2005 15:35:35 -0700

Quoted 7 lines
Quoted 5 lines · Michael Chan
Quoted 3 lines · Ben Greear

I commented out the code and ran the pktgen test again. It may be a small bit better, but not much: 770Mbps in one direction, 750Mbps in the other.

Quoted 7 lines
Quoted 3 lines

The latest tg3 driver will print out the dma_rwctrl at probe time. Can you check the value to make sure ONE_DMA is disabled? Bit 14 sets ONE_DMA.

I don't see any printout in /var/log/messages that looks like it relates to the dma_rwctrl. I'm using the driver in 2.6.11..maybe it is not recent enough to print this information out?

Yes, your driver is too old. The one in Linus's current GIT tree has this, along with many other enhancements.

#8 lore raw
In reply to Ben Greear
Quoted 3 lines

Have you done any tests with 2 tg3 NICs in a single machine to see if they can run at or near line speed (full duplex)?

It isn't just a question of two tg3 NICs in the same box is it? You are running two NICs on the same bus right? And unless my dimm memory is mistaken, four ports on a card with 5704s means two 5704's a bridge chip right? So, it would be two tg3 NICs going through the same bridge chip, not just the same bus or same system. I'd be worrying about DMA latencies on the system and the bridge chip, and perhaps the efficiency of the PCI-X bus usage (not sure - is there anything in your system's chipset to extract that sort of information?)

What happens when you turn pktgen around/insideout and source packets from the bridging system to each of the (two other?) systems?

Since you are bridging, does having CKO enabled really matter? Mightn't that allow the firmware on the 5704(s) to run a triffle faster? Or does bridging already not request CKO (I suppose it might).

Are your interface interrupts distributed across the CPUs?

rick jones

author
#9 lore raw
Quoted 6 lines · Rick Jones
Quoted 3 lines

Have you done any tests with 2 tg3 NICs in a single machine to see if they can run at or near line speed (full duplex)?

It isn't just a question of two tg3 NICs in the same box is it? You are running two NICs on the same bus right? And unless my dimm memory is mistaken, four ports on a card with 5704s means two 5704's a bridge chip right? So, it would be two tg3 NICs going through the same bridge chip, not just the same bus or same system. I'd be worrying about DMA latencies on the system and the bridge chip, and perhaps the efficiency of the PCI-X bus usage (not sure - is there anything in your system's chipset to extract that sort of information?)

There will be a bridge chip, and indeed I see better performance when I just use a 2-port Intel NIC as opposed to a 4 port, even if I am only actively using 2 of the 4 ports on the 4-port NIC. For the tg3 hardware I only have a 4-port NIC. I do assume that a 2-port tg3 NIC w/out a bridge chip would be faster..but probably not too much.

Quoted 1 line

What happens when you turn pktgen around/insideout and source packets from the bridging system to each of the (two other?) systems?

I looped two ports on the same NIC together for the pktgen tests, so there is only a single machine in question. With Intel I can source/sink about 960Mbps on two ports simultaneously in this configuration. With the tg3 NIC I can only do about 750Mbps.

And, the tg3 is in the faster PCI-X slot (133Mhz v/s 100Mhz). So, to me it appears that the tg3 hardware and/or driver can only handle about 80% of the performance that the intel e1000 can produce. It's possible I have a particularly sub-optimal configuration for tg3, or maybe a poorly designed NIC, which is why I'd like to know what others see...

Quoted 1 line

Since you are bridging, does having CKO enabled really matter? Mightn't that allow the firmware on the 5704(s) to run a triffle faster? Or does bridging already not request CKO (I suppose it might).

CKO == IP checksum offload?

Since Dave doesn't want to debug my bridge setup (and I don't blame him), I am going to try to focus my testing/debug reports on the pktgen tests. If/when pktgen shows better performance with tg3, I can verify that I see the same speedups with my proprietary bridging module. I've no idea if CKO would help or hinder pktgen, nor have I tried to enable or disable it.

Quoted 1 line

Are your interface interrupts distributed across the CPUs?

I'm using FC2, basically a default install. It does seem to have an irq balance daemon running. But, I'm not specifically binding IRQs or anything like that. pktgen tx is running as a single thread, so the rx code could run mostly on the other CPU if locking allows...

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#10 lore raw
Quoted 4 lines

There will be a bridge chip, and indeed I see better performance when I just use a 2-port Intel NIC as opposed to a 4 port, even if I am only actively using 2 of the 4 ports on the 4-port NIC. For the tg3 hardware I only have a 4-port NIC. I do assume that a 2-port tg3 NIC w/out a bridge chip would be
faster..but probably not too much.

I have been taught by several wise old engineers that the proper spelling of assume is ass-u-me :)

Bridge chips can in theory do all sorts of nasty things to performance.

Quoted 1 line

CKO == IP checksum offload?

Yes.

Quoted 5 lines

Since Dave doesn't want to debug my bridge setup (and I don't blame him), I
am going to try to focus my testing/debug reports on the pktgen tests. If/when pktgen shows better performance with tg3, I can verify that I see the same speedups with my proprietary bridging module. I've no idea if CKO would
help or hinder pktgen, nor have I tried to enable or disable it.

Quoted 7 lines
Quoted 1 line

Are your interface interrupts distributed across the CPUs?

I'm using FC2, basically a default install. It does seem to have an irq balance daemon running. But, I'm not specifically binding IRQs or anything like that. pktgen tx is running as a single thread, so the rx code could run mostly on the other CPU if locking allows...

again, never ass-u-me.

rick

author
#11 lore raw
Quoted 9 lines · Rick Jones
Quoted 4 lines

There will be a bridge chip, and indeed I see better performance when I just use a 2-port Intel NIC as opposed to a 4 port, even if I am only actively using 2 of the 4 ports on the 4-port NIC. For the tg3 hardware I only have a 4-port NIC. I do assume that a 2-port tg3 NIC w/out a bridge chip would be
faster..but probably not too much.

I have been taught by several wise old engineers that the proper spelling of assume is ass-u-me :)

Bridge chips can in theory do all sorts of nasty things to performance.

Sure...but the end result is that I need 2 port NICs and I need 4 port NICs. The 4-port ones need a bridge, so I'm stuck with a bridge. The Intel 4-port with a bridge works OK, the tg3 not so good. If someone else has a 2-port BCM NIC that handles full line speed, then that would be a good data point, but I'm unlikely to purchase one just to satisfy my curiousity. If someone wants to send me one, I'll happily stick it in my system and report the results.

Quoted 6 lines
Quoted 4 lines

I'm using FC2, basically a default install. It does seem to have an irq balance daemon running. But, I'm not specifically binding IRQs or anything like that. pktgen tx is running as a single thread, so the rx code could run mostly on the other CPU if locking allows...

again, never ass-u-me.

I'm not assuming anything here...just reporting the setup.

Truth is, the e1000 works really well for my application in most configurations. It was interesting for me to learn that some folks are getting very good tg3 performance for TCP transfers when the e1000 was dropping frames (see thread from the last week or so). So, I was a little supprised that I did not see such good tg3 numbers.

If the answer is that the tg3 just can't do it, no shame there...but if my testing can help the tg3 driver improve, I will try to do my part.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#12 lore raw
In reply to Michael Chan
Quoted 15 lines · [email protected] said:
Quoted 3 lines · Ben Greear

* Is the BCM5704 chipset/driver really that much slower?

Unfortunately, the 5704 requires the "ONE_DMA" workaround which will limit throughput in a PCIX 100/133 bus. If you comment out the line that sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved performance. However, you may run into some DMA issues on certain systems.

Quoted 3 lines

* Is there some information on tuning the tg3 somewhere?
(I didn't see a Documentation/networking/tg3.txt file, for instance)

Quoted 6 lines
Quoted 3 lines

* Is there a way to verify the bus speed that the NIC is running at? (ethtool -d ethX gives lots of meaningless (to me) hex)

tg3 probing string for each device will tell you the bus type, width, and speed.

The patch below from http://article.gmane.org/gmane.linux.network/18734
does this too for e1000. Lennert Buytenhek posted it a while back.

Jason

drivers/net/e1000/e1000_main.c +15 -0
diff -urpN linux-pf/drivers/net/e1000/e1000_main.c linux-bs/drivers/net/e1000/e1000_main.c
@@ -617,6 +617,21 @@ e1000_probe(struct pci_dev *pdev,
 	if(eeprom_data & eeprom_apme_mask)
 		adapter->wol |= E1000_WUFC_MAG;
 
+	/* print bus type/speed/width info */
+	printk(KERN_INFO "%s: e1000 (PCI%s:%s:%s) ", netdev->name,
+		((adapter->hw.bus_type == e1000_bus_type_pcix) ? "X" : ""),
+		((adapter->hw.bus_speed == e1000_bus_speed_133) ? "133MHz" :
+		 (adapter->hw.bus_speed == e1000_bus_speed_120) ? "120MHz" :
+		 (adapter->hw.bus_speed == e1000_bus_speed_100) ? "100MHz" :
+		 (adapter->hw.bus_speed == e1000_bus_speed_66) ? "66MHz" :
+		  "33MHz"),
+		((adapter->hw.bus_width == e1000_bus_width_64) ? "64-bit" :
+		  "32-bit"));
+
+	for (i = 0; i < 6; i++)
+		printk("%2.2x%c", netdev->dev_addr[i],
+			i == 5 ? '\n' : ':');
+
 	/* reset the hardware with the new settings */
 	e1000_reset(adapter);
 
maintainer
#13 lore raw
In reply to Ben Greear

From: Ben Greear <[email protected]>
Date: Thu, 09 Jun 2005 17:38:22 -0700

Quoted 2 lines

I am trying to bridge as much traffic as possible across two interfaces, using a proprietary kernel module.

Ben, I'm going to just mention that I'm not going to look into your bug report. You consistently come here without a test case or setup that other developers can use to reproduce or investigate your problem. You always report things with your proprietary setup of this or that.

You have our code, we don't have your's.

author
#14 lore raw
Quoted 8 lines · David S. Miller

From: Ben Greear <[email protected]>
Date: Thu, 09 Jun 2005 17:38:22 -0700

Quoted 2 lines

I am trying to bridge as much traffic as possible across two interfaces, using a proprietary kernel module.

Quoted 7 lines

Ben, I'm going to just mention that I'm not going to look into your bug report. You consistently come here without a test case or setup that other developers can use to reproduce or investigate your problem. You always report things with your proprietary setup of this or that.

You have our code, we don't have your's.

Fair enough.

I ran a test using pktgen to (try to) send 82kpps, 1514 byte packets between two ports on the tg3 NIC. It can do about 780Mbps in one direction, and 880Mbps in the other direction. Lots of harmless hard-start xmit errors reported (tg3 may not stop it's tx queue correctly, or maybe pktgen is screwed up since e1000 reports similar errors).
Intel e1000 can do about 960Mbps in both directions.

My pktgen is modified. You can find my full patch against 2.6.11 here if you so wish:

http://www.candelatech.com/oss/candela_2.6.11.patch

If you need the exact arguments I used to configure pktgen, I can get those for you as well.

I found the Mhz printout (thanks Michael!)

The tg3 NIC is in the 133Mhz slot. That probably means the intel NIC is only running at 100Mhz.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
maintainer
#15 lore raw

From: Ben Greear <[email protected]>
Date: Thu, 09 Jun 2005 18:20:33 -0700

Quoted 5 lines

I ran a test using pktgen to (try to) send 82kpps, 1514 byte packets between two ports on the tg3 NIC. It can do about 780Mbps in one direction, and 880Mbps in the other direction. Lots of harmless hard-start xmit errors reported (tg3 may not stop it's tx queue correctly, or maybe pktgen is screwed up since e1000 reports similar errors).

There is a known race on SMP with drivers using NETIF_F_LLTX that is still not fixed. It will cause the error message to be reported from the driver's ->hard_start_xmit() routine when you hit this race.

The e1000 driver hits the same race, it just doesn't print any message.

You're definitely on an SMP system if you are triggering that message.

author
#16 lore raw
Quoted 11 lines · David S. Miller

From: Ben Greear <[email protected]>
Date: Thu, 09 Jun 2005 18:20:33 -0700

Quoted 5 lines

I ran a test using pktgen to (try to) send 82kpps, 1514 byte packets between two ports on the tg3 NIC. It can do about 780Mbps in one direction, and 880Mbps in the other direction. Lots of harmless hard-start xmit errors reported (tg3 may not stop it's tx queue correctly, or maybe pktgen is screwed up since e1000 reports similar errors).

Quoted 3 lines

There is a known race on SMP with drivers using NETIF_F_LLTX that is still not fixed. It will cause the error message to be reported from the driver's ->hard_start_xmit() routine when you hit this race.

Ok, it's not a big deal, since I can just retry the packet when the tx queue wakes up again. The message was coming from pktgen, btw.

Quoted 1 line

You're definitely on an SMP system if you are triggering that message.

Yes, dual xeon with HT as well.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com