Skip to contentSkip to conversation

First test release of Tigon3 driver

BETA David S. Miller 10 participants 29 messages Feb 26, 2002 – Feb 27, 2002 Networking

lore
Share & tools
lore
#0 lore raw

This is to announce the first test release of a new Broadcom Tigon3 driver written by Jeff and myself. Get it at:

ftp://ftp.kernel.org/pub/linux/kernel/people/davem/TIGON3/tg3.patch.gz

The patch is against 2.4.18 but there is no reason it should be difficult to apply this patch to any other tree.

It is meant to replace Broadcom's driver because frankly their driver is junk and would never be accepted into the tree. For an example of why their driver is junk, note that the resulting object file from our driver is less than half the size of Broadcom's. That kind of bloat is simply unacceptable. Next, Broadcom's driver is still way non-portable, ioremap() pointers are still dereferenced directly among other things. Finally, their driver is just plain buggy, they have code which tries to use page_address() on pages which are potentially in highmem and that is guarenteed to oops.

There are other problems with their driver too, just ask as I'm in a good mood to grind my axe about this stuff :-)

We would also like to give Broadcom a big "no thanks" because their lawyers refused to give Jeff the documentation for the Tigon3 chipset using an NDA that would allow him to write a GPL'd driver based upon said documentation. This means that all that we know about the hardware has been reverse engineered from Broadcom's GPL'd driver plus some experimenting. It is why this driver has taken so long to finish, because it is hard to find incentive for this kind of brack breaking work when the vendor is totally uncooperative.

I personally think Broadcom has some hidden agenda that requires that their driver be "the one". So instead of just complaining about such stupid policy, we've done something about it by doing our own driver with all the problems fixed.

We'd really appreciate if people would test this thing out as hard as they can. I can tell you that I've personally only been able to test out the following scenerios so far:

1) Both big-endian and little-endian platforms.
2) 64-bit on big-endian side, 32-bit on little-endian side.
3) IOMMU based PCI dma platform on 64-bit/big-endian side, non-IOMMU based PCI dma platform on 32-bit little-endian side.
4) 10baseT half duplex, 100baseT full duplex
5) Copper based connectors, as that is all that Jeff and I have.

Gigabit and fibre are the big areas where our testing has been lacking. We have also not done any tuning of the driver at all, although we do consider the driver feature complete.

#1 lore raw

Can you list what cards/systems use this driver? I at least am totally unsure.
Thanks
Nick

Quoted 56 lines · David S. Miller

This is to announce the first test release of a new Broadcom Tigon3 driver written by Jeff and myself. Get it at:

ftp://ftp.kernel.org/pub/linux/kernel/people/davem/TIGON3/tg3.patch.gz

The patch is against 2.4.18 but there is no reason it should be difficult to apply this patch to any other tree.

It is meant to replace Broadcom's driver because frankly their driver is junk and would never be accepted into the tree. For an example of why their driver is junk, note that the resulting object file from our driver is less than half the size of Broadcom's. That kind of bloat is simply unacceptable. Next, Broadcom's driver is still way non-portable, ioremap() pointers are still dereferenced directly among other things. Finally, their driver is just plain buggy, they have code which tries to use page_address() on pages which are potentially in highmem and that is guarenteed to oops.

There are other problems with their driver too, just ask as I'm in a good mood to grind my axe about this stuff :-)

We would also like to give Broadcom a big "no thanks" because their lawyers refused to give Jeff the documentation for the Tigon3 chipset using an NDA that would allow him to write a GPL'd driver based upon said documentation. This means that all that we know about the hardware has been reverse engineered from Broadcom's GPL'd driver plus some experimenting. It is why this driver has taken so long to finish, because it is hard to find incentive for this kind of brack breaking work when the vendor is totally uncooperative.

I personally think Broadcom has some hidden agenda that requires that their driver be "the one". So instead of just complaining about such stupid policy, we've done something about it by doing our own driver with all the problems fixed.

We'd really appreciate if people would test this thing out as hard as they can. I can tell you that I've personally only been able to test out the following scenerios so far:

1) Both big-endian and little-endian platforms.
2) 64-bit on big-endian side, 32-bit on little-endian side.
3) IOMMU based PCI dma platform on 64-bit/big-endian side, non-IOMMU based PCI dma platform on 32-bit little-endian side.
4) 10baseT half duplex, 100baseT full duplex
5) Copper based connectors, as that is all that Jeff and I have.

Gigabit and fibre are the big areas where our testing has been lacking. We have also not done any tuning of the driver at all, although we do consider the driver feature complete.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw

From: [email protected]
Date: Mon, 25 Feb 2002 22:43:53 -0500 (EST)

Can you list what cards/systems use this driver? I at least am totally unsure.

I can't tell you much more than the list of PCI device IDs and those are contained in the driver sources.

The first two cards I have for testing are 3COM copper gigabit cards but I have no idea what the official model numbers are for these as they came with no packaging.

#3 lore raw
In reply to [email protected]
Quoted 9 lines · David S. Miller

From: Sebastian Heidl <[email protected]>
Date: Tue, 26 Feb 2002 12:22:05 +0100

I think these are 3C996-T. Is there any information about the compatibility of TG3 to TG2 ?

Totally different chip, but there are subtle similarities all over the place. Ie. if you knew how to program the tg2, or even were just familiar with the acenic driver, the tg3 stuff looks familiar.

That's what I noticed looking at tg3.c ;-)
I guess nobody was crazy enough yet to try to load a tg2-firmware on a tg3 ? Only as there are some utils to build a customized firmware for the tg2.

just guessing wildly ;-)
_sh_

#4 lore raw

From: Sebastian Heidl <[email protected]>
Date: Tue, 26 Feb 2002 12:39:49 +0100

I guess nobody was crazy enough yet to try to load a tg2-firmware on a tg3 ? Only as there are some utils to build a customized firmware for the tg2.

Different cpus, tg3 uses StrongARM and tg2 uses a lobotomized MIPS.

#5 lore raw
In reply to David S. Miller
Quoted 6 lines · David S. Miller

From: [email protected]
Date: Mon, 25 Feb 2002 22:43:53 -0500 (EST)

Can you list what cards/systems use this driver? I at least am totally unsure.

Quoted 6 lines

I can't tell you much more than the list of PCI device IDs and those are contained in the driver sources.

The first two cards I have for testing are 3COM copper gigabit cards but I have no idea what the official model numbers are for these as they came with no packaging.

I think these are 3C996-T. Is there any information about the compatibility of TG3 to TG2 ?

_sh_

#6 lore raw

From: Sebastian Heidl <[email protected]>
Date: Tue, 26 Feb 2002 12:22:05 +0100

I think these are 3C996-T. Is there any information about the compatibility of TG3 to TG2 ?

Totally different chip, but there are subtle similarities all over the place. Ie. if you knew how to program the tg2, or even were just familiar with the acenic driver, the tg3 stuff looks familiar.

#7 lore raw
In reply to David S. Miller

David S. Miller:

Quoted 3 lines

Gigabit and fibre are the big areas where our testing has been lacking. We have also not done any tuning of the driver at all, although we do consider the driver feature complete.

I've tested the driver from broadcom (yeah, yeah, the sucky driver :) and now I've tested yours, and this is the result (from dmesg):

tg3.c:v0.90 (Feb 25, 2002)
tg3: Problem fetching invariants of chip, aborting.

This is lspci -vvvxx:
01:08.0 Ethernet controller: BROADCOM Corporation BCM5700 1000BaseTX (rev 12)
Subsystem: Dell Computer Corporation: Unknown device 00d1
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin A routed to IRQ 17
Region 0: Memory at feb00000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
Address: 9b8dc9bb84b8f3a8 Data: bed9
00: e4 14 44 16 02 01 b0 02 12 00 00 02 08 20 00 00
10: 04 00 b0 fe 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 d1 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 40 00

If you need any more info, I'll be happy to provide.

Signature · 1 line
Thomas
#8 lore raw

From: Thomas Langås <[email protected]>
Date: Tue, 26 Feb 2002 14:57:30 +0100

tg3.c:v0.90 (Feb 25, 2002)
tg3: Problem fetching invariants of chip, aborting.

Please apply this patch and tell me what it spits
out, thanks.

net/tg3.c renamed from net/tg3.c.~1~ +10 -2
@@ -4424,12 +4424,20 @@
 	if (grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5700 &&
 	    grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5701 &&
 	    grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5703 &&
-	    grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5703S)
+	    grc_misc_cfg != GRC_MISC_CFG_BOARD_ID_5703S) {
+		printk("DEBUG: Yikes grc_misc_cfg is %08x\n",
+		       grc_misc_cfg);
 		return -ENODEV;
+	}
 
 	err = tg3_phy_probe(tp);
-	if (!err)
+	if (err)
+		printk("DEBUG: phy_probe returns with %d\n", err);
+	if (!err) {
 		err = tg3_read_partno(tp);
+		if (err)
+			printk("DEBUG: read_partno returns with %d\n", err);
+	}
 
 	if (tp->phy_id == PHY_ID_SERDES) {
 		tp->tg3_flags &= ~TG3_FLAG_USE_PHY_IRQ;
#9 lore raw
In reply to Thomas Langås

David S. Miller:

Quoted 2 lines

Great, now add the following patch below and let me know what it does and prints out now.

tg3.c:v0.90 (Feb 25, 2002)
DEBUG: smallest_limit is 9897
DEBUG: read_partno returned -19
tg3: Problem fetching invariants of chip, aborting.

Quoted 8 lines

If the module still fails to load because of the -EBUSY error (ie. the "read_partno returns -19" thing happens again), bring drivers/net/tg3.c into an editor and go to around line 4185 and change the line that reads:
[snipp]
And see if it works then. PLEASE type sync a few times before trying to load the module in this case as it could very well hang your machine.

Didn't work, didn't hang the box either :)

Any more ideas? :)

Signature · 1 line
Thomas
#10 lore raw

From: Thomas Langås <[email protected]>
Date: Wed, 27 Feb 2002 12:05:49 +0100

Quoted 8 lines

If the module still fails to load because of the -EBUSY error (ie. the "read_partno returns -19" thing happens again), bring drivers/net/tg3.c into an editor and go to around line 4185 and change the line that reads:
[snipp]
And see if it works then. PLEASE type sync a few times before trying to load the module in this case as it could very well hang your machine.

Didn't work, didn't hang the box either :)

What did it print out when you changed the code to be "while (1)"? It must print something different.

#11 lore raw
In reply to Thomas Langås

David S. Miller:

Quoted 4 lines

Please, simply make tg3_read_partno always return 0 (or, alternatively, make tg3_get_invariants() ignore tg3_read_partno's return value). It should just work, this string is not critical to the driver's operation.

Do you want a dump of the vpd_data it receives? Here's the dmesg line after the driver is loaded:

tg3.c:v0.90 (Feb 25, 2002)
eth1: Tigon3 [partno() rev 7102 PHY(5401)] (PCI:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:06:5b:05:f9:8e

The machine is a Dell PowerEdge 2550 machine, running RedHat Linux 7.2 (with 2.4.18 kernel). Let me know if you want me to run any specific tests, it's connected to a Cisco switch with gigabit-ports (so I can actually test things like jumboframes if needed).

Signature · 1 line
Thomas
#12 lore raw

From: Thomas Langås <[email protected]>
Date: Wed, 27 Feb 2002 13:24:54 +0100

Do you want a dump of the vpd_data it receives?

It isn't interesting, so no.

Here's the dmesg line after
the driver is loaded:

Thank you.

The machine is a Dell PowerEdge 2550 machine, running RedHat Linux 7.2 (with 2.4.18 kernel). Let me know if you want me to run any specific tests, it's connected to a Cisco switch with gigabit-ports (so I can actually test things like jumboframes if needed).

At this point I'm mostly interested in if it works at all :-)

If the answer is yes, tell me that and then you can feel free to experiment with jumbo frames et al. to discover other bugs in the driver :-)

#13 lore raw
In reply to David S. Miller

David S. Miller:

Quoted 2 lines

Please apply this patch and tell me what it spits
out, thanks.

tg3.c:v0.90 (Feb 25, 2002)
DEBUG: read_partno returns -19
tg3: Problem fetching invariants of chip, aborting.

Signature · 1 line
Thomas
#14 lore raw

__skb_dequeue irq race ?

I'm trying to debug what looks like a race-condition I reach with heavy interrupt traffic, including heavy ethernet traffic.

The result of the race (if it is one) is that the pointers in skb_head_pool in the network code get corrupted, like the list heads ->next pointer points to the zero-page and other nasty things.

I've gotten an oops from different codepaths but one caught my eye especially in that there seems to be a codepath with IRQ's enabled which calls __skb_dequeue, namely do_softirq calls net_tx_action which uses qdisc_restart which calls q->dequeue, which contains pfifo_fast_dequeue which calls __skb_dequeue directly.

Trace; c009a508 <pfifo_fast_dequeue+22/5a>
Trace; c009a16e <qdisc_restart+16/b6>
Trace; c0094b44 <net_tx_action+7c/8c>
Trace; c0008a8c <do_softirq+4c/94>

Before diving deeper into the fray, is this normal ? If __skb_dequeue is called with irq's enabled, couldn't another network interrupt cause a race, corrupting the skb_head_pool linked structure ?

/BW

#15 lore raw
In reply to Thomas Langås

From: Thomas Langås <[email protected]>
Date: Tue, 26 Feb 2002 16:40:44 +0100

tg3.c:v0.90 (Feb 25, 2002)
DEBUG: read_partno returns -19
tg3: Problem fetching invariants of chip, aborting.

Great, now add the following patch below and let me know what it does and prints out now.

If the module still fails to load because of the -EBUSY error (ie. the "read_partno returns -19" thing happens again), bring drivers/net/tg3.c into an editor and go to around line 4185 and change the line that reads:

	while (--limit) {

into:

	while (1) {

And see if it works then. PLEASE type sync a few times before trying to load the module in this case as it could very well hang your machine.

Thanks again for all the testing so far:

net/tg3.c renamed from net/tg3.c.~2~ +8 -2
@@ -4168,16 +4168,18 @@
 static int __devinit tg3_read_partno(struct tg3 *tp)
 {
 	unsigned char vpd_data[256];
+	unsigned int smallest_limit = ~0U;
 	int i;
 
 	/* Enable seeprom accesses. */
-	tw32(GRC_LOCAL_CTRL, GRC_LCLCTRL_AUTO_SEEPROM);
+	tw32(GRC_LOCAL_CTRL,
+	     tr32(GRC_LOCAL_CTRL) | GRC_LCLCTRL_AUTO_SEEPROM);
 	udelay(100);
 
 	for (i = 0; i < 256; i += 4) {
 		u32 tmp;
 		u16 stat;
-		int limit = 5000;
+		unsigned int limit = 100000;
 
 		pci_write_config_word(tp->pdev, TG3PCI_VPD_ADDR_FLAG, i);
 		while (--limit) {
@@ -4188,6 +4190,8 @@
 		}
 		if (!limit)
 			return -EBUSY;
+		if (limit < smallest_limit)
+			smallest_limit = limit;
 
 		pci_read_config_dword(tp->pdev, TG3PCI_VPD_DATA, &tmp);
 
@@ -4196,6 +4200,8 @@
 		vpd_data[i + 2] = ((tmp >> 16) & 0xff);
 		vpd_data[i + 3] = ((tmp >> 24) & 0xff);
 	}
+
+	printk("DEBUG: smallest_limit is %u\n", smallest_limit);
 
 	/* Now parse and find the part number. */
 	for (i = 0; i < 256; ) {
#16 lore raw
In reply to Thomas Langås

David S. Miller:

Quoted 2 lines

What did it print out when you changed the code to be "while (1)"? It must print something different.

Here's the output (counter is something I added inside the while (1) {}-loop, on top in that loop, it's unsigned int, set to 0 to start with):

tg3.c:v0.90 (Feb 25, 2002)
DEBUG: counter 6592
DEBUG: smallest_limit is 10000
DEBUG: read_partno returned -19
tg3: Problem fetching invariants of chip, aborting.

smallest_limit is naturally 10k since limit isn't decremented...

Signature · 1 line
Thomas
#17 lore raw

From: Thomas Langås <[email protected]>
Date: Wed, 27 Feb 2002 12:56:11 +0100

tg3.c:v0.90 (Feb 25, 2002)
DEBUG: counter 6592
DEBUG: smallest_limit is 10000
DEBUG: read_partno returned -19
tg3: Problem fetching invariants of chip, aborting.

smallest_limit is naturally 10k since limit isn't decremented...

Oh I see... ok so it gets the vital product data, but it cannot find the part number string.

Please, simply make tg3_read_partno always return 0 (or, alternatively, make tg3_get_invariants() ignore tg3_read_partno's return value). It should just work, this string is not critical to the driver's operation.

#18 lore raw
In reply to Thomas Langås

David S. Miller:

Quoted 4 lines

At this point I'm mostly interested in if it works at all :-) If the answer is yes, tell me that and then you can feel free to experiment with jumbo frames et al. to discover other bugs in the driver :-)

Just tested with MTU set at 1500 for now, but it seems to work fine, did a netcat between two boxes on the same switch and got around 80MB/sec.

Any programs or anything that could do a serious stresstest? (Both hosts are Dell PowerEdge 2550, RedHat Linux 7.2).

Signature · 1 line
Thomas
#19 lore raw
In reply to Thomas Langås
Quoted 7 lines · Thomas Langås

David S. Miller:

Quoted 4 lines

At this point I'm mostly interested in if it works at all :-) If the answer is yes, tell me that and then you can feel free to experiment with jumbo frames et al. to discover other bugs in the driver :-)

Quoted 5 lines

Just tested with MTU set at 1500 for now, but it seems to work fine, did a netcat between two boxes on the same switch and got around 80MB/sec.

Any programs or anything that could do a serious stresstest? (Both hosts are Dell PowerEdge 2550, RedHat Linux 7.2).

google for nttcp
runs a nice tcp performance test...

#20 lore raw
In reply to Thomas Langås
Quoted 7 lines · Thomas Langås

David S. Miller:

Quoted 4 lines

At this point I'm mostly interested in if it works at all :-) If the answer is yes, tell me that and then you can feel free to experiment with jumbo frames et al. to discover other bugs in the driver :-)

Quoted 5 lines

Just tested with MTU set at 1500 for now, but it seems to work fine, did a netcat between two boxes on the same switch and got around 80MB/sec.

Any programs or anything that could do a serious stresstest? (Both hosts are Dell PowerEdge 2550, RedHat Linux 7.2).

I've had good luck with an infinite loop on sendto(). Can easily saturate a link, and depending on the receiving host, can essentially DOS the machine.

As an example, a Dell OptiPlex GX1 running 2.4.17 becomes completely unusable when receiving 80000 packets/sec.

Chris

Signature · 4 lines
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]
#21 lore raw
In reply to Thomas Langås

Arjan van de Ven:

Quoted 2 lines

google for nttcp
runs a nice tcp performance test...

Ok, here's some numbers:

UDP:

kiwi:~/nttcp-1.47# ./nttcp -T -v -r -u 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=3702
nttcp-l: send window size = 65535
nttcp-l: receive window size = 65535
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=1820, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-u@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: send window size = 65535
nttcp-1: receive window size = 65535
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/udp

nttcp-l: got EOF
nttcp-l: received 7725056 bytes

     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  7725056    0.07    0.07    878.2714    882.8635    1887  26816.93   26957.1

nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
nttcp-l: try to get outstanding messages from 1 remote clients
1 8388608 0.07 0.05 966.3321 1342.1773 2051 29533.31 41020.0
nttcp-1: exiting

TCP:

kiwi:~/nttcp-1.47# ./nttcp -T -v -r 129.241.57.161   
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=3705
nttcp-l: accept from 129.241.57.161
nttcp-l: send window size = 16384
nttcp-l: receive window size = 87380
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=1821, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-1: send window size = 16384
nttcp-1: receive window size = 87380
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/tcp

nttcp-l: received 8388608 bytes

     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.08    0.05    865.7309   1342.1773    2177  28084.16   43540.0

nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
1 8388608 0.08 0.04 864.4930 1677.7216 2048 26382.23 51200.0
nttcp-1: exiting

nttcp on the receiving host is running throuhg xinetd, I don't think that impacts the performance, tho.

Signature · 1 line
Thomas
#22 lore raw
In reply to Thomas Langås
Quoted 5 lines
Quoted 2 lines

Any programs or anything that could do a serious stresstest? (Both hosts are Dell PowerEdge 2550, RedHat Linux 7.2).

google for nttcp
runs a nice tcp performance test...

I dig on gensink:

http://jes.home.cern.ch/jes/gensink/

the attached hack gives it the ability to tx with sendfile() from a ftruncate()ed tmpfile based on the presence of a 5th argument, which can be fun with scatter/csum capable cards. I've put up some rpms for the lazy:

http://www.zabbo.net/rpms/gensink/

5aa66dfd7749c77cb58de22fad96de gensink-4.1-1sendfile.i386.rpm 04912a63bda95a79b6b835f340c924c2 gensink-4.1-1sendfile.src.rpm

enjoy.

- z

gen4.c +97 -28
diff -urN gensink-4.1.orig/gen4.c gensink-4.1/gen4.c
@@ -48,20 +48,101 @@
 
 #include <sys/ioctl.h>
 
+#include <sys/sendfile.h>
+
+void sendfile_loop(int s, int rec_len)
+{
+	char *tmpdir_default = "/tmp";
+	char template[PATH_MAX];
+	char *tmpdir;
+	int fd;
+	int i;
+	ssize_t bytes_written;
+	off_t offset;
+
+	tmpdir = getenv("TMPDIR");
+	if ( tmpdir == NULL )
+		tmpdir = tmpdir_default;
+
+	snprintf(template, PATH_MAX, "%s/XXXXXX", tmpdir);
+
+	fd = mkstemp(template);
+	if ( fd < 0 ) {
+		perror("couldn't mkstemp() sendfile() backing file");
+		exit(-1);
+	}
+
+	unlink(template);
+
+	if ( ftruncate(fd, RECORD_BUFFER_SIZE + rec_len) < 0 ) {
+		perror("frunctate() backing file");
+		exit(-1);
+	}
+
+	offset = 0;
+
+	for (i = 1; i < 4000000; i++)
+	{
+		/* loop writing blocks */
+		if ((bytes_written = sendfile(s,fd,&offset,rec_len)) < 0) 
+			oops("write");
+		if ((bytes = bytes + bytes_written) >= REPORT_FREQ) 
+		{
+			report(1, "source", bytes);
+			bytes = 0;
+		}
+
+		if ( offset >= RECORD_BUFFER_SIZE )
+			offset = 0;
+	}
+
+	close(fd);
+}
+
+void write_loop(int s, int rec_len)
+{
+	char *record_buffer, *orig_record_ptr;
+	int i, bytes_written;
+
+	record_buffer = (char *)malloc(RECORD_BUFFER_SIZE + rec_len);
+	if (!record_buffer) {
+		fprintf(stderr, "Unable to allocate memory for transmit buffer\n");
+		exit(-1);
+	}
+
+	orig_record_ptr = record_buffer;
+	memset(record_buffer, 0, RECORD_BUFFER_SIZE);
+
+	for (i = 1; i < 4000000; i++)
+	{
+		/* loop writing blocks */
+		if ((bytes_written = write(s,record_buffer,rec_len)) < 0) 
+			oops("write");
+		if ((bytes = bytes + bytes_written) >= REPORT_FREQ) 
+		{
+			report(1, "source", bytes);
+			bytes = 0;
+		}
+		record_buffer += rec_len;
+		if ((unsigned long)record_buffer > (unsigned long)(orig_record_ptr + RECORD_BUFFER_SIZE))
+			record_buffer = orig_record_ptr;
+	}
+	free(orig_record_ptr);
+}
+
 int main(argc, argv)
 int argc; 
 char *argv[];
 {
 	struct sockaddr_in saddr;
-	char *record_buffer, *orig_record_ptr;
 	char reptext[256], *hostname;
-	int portnum, rec_len, bytes_written;
-	int i;
+	int portnum, rec_len;
+	int use_sendfile = 0;
 
 	/* check argument count */
-	if(argc != 5) 
+	if(argc < 5 || argc > 6)
 	{
-		fprintf(stderr,"Usage: %s server_host server_port record_length setsockopt\n", argv[0]);
+		fprintf(stderr, "Usage: %s server_host server_port record_length setsockopt [use_sendfile_boolean]\n", argv[0]);
 		exit(-1);
 	}
 
@@ -73,16 +154,11 @@
 		exit(-1);
 	}
 
-	record_buffer = (char *)malloc(RECORD_BUFFER_SIZE + rec_len);
-	if (!record_buffer) {
-		fprintf(stderr, "Unable to allocate memory for transmit buffer\n");
-		exit(-1);
-	}
-	orig_record_ptr = record_buffer;
-	memset(record_buffer, 0, RECORD_BUFFER_SIZE);
-
 	max = atoi(argv[4]);
 
+	if ( argc == 6 ) 
+		use_sendfile = atoi(argv[5]);
+
 	strcpy(reptext,"");
 	/* get the record length */
 	strncat(reptext, " reclen=", 255 - strlen(reptext));
@@ -122,24 +198,17 @@
   
 	printf("Max seg TCP : %d\n",maxseg);
 
+	printf("sending with %s\n", 
+		use_sendfile ? "sendfile(2)" : "write(2)");
+
 	/* initialise reporting */
 	report_start(reptext,1,0,1,argv);
 	report(1, "source", 0); 
 
-	for (i = 1; i < 4000000; i++)
-	{
-		/* loop writing blocks */
-		if ((bytes_written = write(s,record_buffer,rec_len)) < 0) 
-			oops("write");
-		if ((bytes = bytes + bytes_written) >= REPORT_FREQ) 
-		{
-			report(1, "source", bytes);
-			bytes = 0;
-		}
-		record_buffer += rec_len;
-		if ((unsigned long)record_buffer > (unsigned long)(orig_record_ptr + RECORD_BUFFER_SIZE))
-			record_buffer = orig_record_ptr;
-	}
-	free(orig_record_ptr);
+	if ( use_sendfile )
+		sendfile_loop(s, rec_len);
+	else
+		write_loop(s, rec_len);
+
 	return 0;
 }
#23 lore raw
In reply to Thomas Langås
Quoted 2 lines

5aa66dfd7749c77cb58de22fad96de gensink-4.1-1sendfile.i386.rpm 04912a63bda95a79b6b835f340c924c2 gensink-4.1-1sendfile.src.rpm

that should be..

3a5aa66dfd7749c77cb58de22fad96de gensink-4.1-1sendfile.i386.rpm 04912a63bda95a79b6b835f340c924c2 gensink-4.1-1sendfile.src.rpm

*sigh*

- z
[ whats the point of md5sums in an unsigned mail anyway? Who knows. ]

#24 lore raw
In reply to Thomas Langås

Arjan van de Ven:

Quoted 2 lines

google for nttcp
runs a nice tcp performance test...

I did some more benchmarking as Jeff Garzik requested:
[S: sending host, R: receiving host]

1. S: Broadcom's GPLed driver
R: Miller/Garzik's GPLed driver
2. S: Broadcom's GPLed driver
R: Broadcom's GPLed driver
3. S: Miller/Garzik's GPLed driver
R: Broadcom's GPLed driver

(scenario 4, Miller/Garzik on both ends has already been tested, see another email in this thread).

__Scenario 1__
UDP:

kiwi:~/nttcp-1.47# ./nttcp -T -v -r -u 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4354
nttcp-l: send window size = 65535
nttcp-l: receive window size = 65535
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=1930, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-u@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: send window size = 65535
nttcp-1: receive window size = 65535
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/udp

nttcp-l: got EOF
nttcp-l: received 5910528 bytes

     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  5910528    0.07    0.08    656.3334    591.0528    1444  20043.59   18050.0

nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
nttcp-l: try to get outstanding messages from 1 remote clients
1 8388608 0.07 0.07 962.3826 958.6981 2051 29412.61 29300.0
nttcp-1: exiting

TCP:

kiwi:~/nttcp-1.47# ./nttcp -T -v -r 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4355
nttcp-l: accept from 129.241.57.161
nttcp-l: send window size = 16384
nttcp-l: receive window size = 87380
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=1931, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-1: send window size = 16384
nttcp-1: receive window size = 87380
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/tcp

nttcp-l: received 8388608 bytes

     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.09    0.09    754.8719    745.6540    2073  23318.07   23033.3

nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
1 8388608 0.09 0.07 759.8205 958.6981 2048 23187.88 29257.1
nttcp-1: exiting

__Scenario 2__
UDP:

kiwi:~/nttcp-1.47# ./nttcp -T -v -r -u 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4409
nttcp-l: send window size = 65535
nttcp-l: receive window size = 65535
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=2066, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-u@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: send window size = 65535
nttcp-1: receive window size = 65535
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/udp

nttcp-l: got EOF
nttcp-l: received 6094848 bytes

     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  6094848    0.07    0.07    678.4114    696.5541    1489  20717.39   21271.4

nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
nttcp-l: try to get outstanding messages from 1 remote clients
1 8388608 0.07 0.07 936.5683 958.6981 2051 28623.66 29300.0
nttcp-1: exiting

TCP:

kiwi:~/nttcp-1.47# ./nttcp -T -v -r 129.241.57.161   
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4410
nttcp-l: accept from 129.241.57.161
nttcp-l: send window size = 16384
nttcp-l: receive window size = 87380
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=2067, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-1: send window size = 16384
nttcp-1: receive window size = 87380
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/tcp

nttcp-l: received 8388608 bytes

     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.09    0.08    758.2323    838.8608    2061  23286.29   25762.5

nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
1 8388608 0.09 0.04 763.3294 1677.7216 2048 23294.96 51200.0
nttcp-1: exiting

__Scenario 3__
UDP:

kiwi:~/nttcp-1.47# ./nttcp -T -v -r -u 129.241.57.161   
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4509
nttcp-l: send window size = 65535
nttcp-l: receive window size = 65535
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/udp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=2106, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-u@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: send window size = 65535
nttcp-1: receive window size = 65535
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/udp

nttcp-l: got EOF
nttcp-l: received 8192000 bytes

     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8192000    0.07    0.07    914.3367    936.2286    2001  27917.29   28585.7

nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
nttcp-l: try to get outstanding messages from 1 remote clients
1 8388608 0.07 0.06 939.5448 1118.4811 2051 28714.63 34183.3
nttcp-1: exiting

TCP:

kiwi:~/nttcp-1.47# ./nttcp -T -v -r 129.241.57.161
nttcp-l: nttcp, version 1.47
nttcp-l: Pid=4512
nttcp-l: accept from 129.241.57.161
nttcp-l: send window size = 16384
nttcp-l: receive window size = 87380
nttcp-l: buflen=4096, bufcnt=2048, dataport=5038/tcp
nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: Pid=2107, InetPeer= 129.241.57.160
nttcp-1: Optionline="nttcp@-t@-l4096@-n2048@-v@-f@%9b%8.2rt%8.2ct%12.4rbr%12.4cbr%8c%10.2rcr%10.1ccr@-N1"
nttcp-1: from ?: "129.241.57.160" (=dataport: 5038)
nttcp-1: send window size = 16384
nttcp-1: receive window size = 87380
nttcp-1: buflen=4096, bufcnt=2048, dataport=5038/tcp

nttcp-l: received 8388608 bytes

     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.08    0.06    877.2286   1118.4811    2179  28483.29   36316.7

nttcp-l: try to get outstanding messages from 1 remote clients
nttcp-1: transmitted 8388608 bytes
1 8388608 0.08 0.04 872.2687 1677.7216 2048 26619.53 51200.0
nttcp-1: exiting

Hope this helps :)

Signature · 1 line
Thomas
#25 lore raw
In reply to David S. Miller

Just wanted to say thanks for doing this work, the driver works great for me for this device:

01:04.0 Ethernet controller: BROADCOM Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 12)
Subsystem: BROADCOM Corporation NetXtreme BCM5700 1000BaseTX
Flags: bus master, 66Mhz, medium devsel, latency 240, IRQ 42
Memory at f0e00000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] PCI-X non-bridge device.
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-

But I'm only able to run it at 10Mbit :)

thanks,

greg k-h

#26 lore raw

From: Greg KH <[email protected]>
Date: Tue, 26 Feb 2002 09:22:51 -0800

Just wanted to say thanks for doing this work, the driver works great for me for this device:

01:04.0 Ethernet controller: BROADCOM Corporation NetXtreme BCM5700 Gigabit Ethernet (rev 12)
Subsystem: BROADCOM Corporation NetXtreme BCM5700 1000BaseTX
...
But I'm only able to run it at 10Mbit :)

Thanks for the report, but can you also show us the "dmesg" lines that the driver printed out when the module was loaded? That provides more detailed probing information for us.

#27 lore raw
In reply to David S. Miller
Quoted 4 lines · David S. Miller

This is to announce the first test release of a new Broadcom Tigon3 driver written by Jeff and myself. Get it at:

Does/Will this driver support NICE-extensions as does the Broadcom's driver? I've been successfully using broadcom's driver with the nice-patched vlan-code for a couple of months now..

I think Ben Greear is about to submit NICE patch for VLAN to be included in the kernel soon..

(NICE extensions enable hardware vlan-tagging etc for linux VLAN code).

btw. Does any other driver support NICE-extensions?

- Pasi Kärkkäinen

                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.
#28 lore raw

From: Pasi Kärkkäinen <[email protected]>
Date: Tue, 26 Feb 2002 20:08:51 +0200 (EET)

Does/Will this driver support NICE-extensions as does the Broadcom's driver? I've been successfully using broadcom's driver with the nice-patched vlan-code for a couple of months now..

NICE is a gross hack and I would hope that Ben Greear would have better taste in the interface he chooses for hw VLAN acceleration in the 802.1q layer :-)

NICE was explicitly left out and this was done on purpose.