how to tune a pair of e1000 cards on intel e7501-based system?
Subsystems (2)
hi all
i'm trying to understand how to tune a pair of e1000 cards in a server box. the box is a dual xeon 3.06 with hyperthreading, using the intel e7501 chipset, with both cards on hub interface D. the application involves small UDP packets, generally under 300 bytes. i need to maximize the number of packets per second transferred between the two cards.
at the moment, i'm looking at the PCI bus in this box to see what might be tweakable. lspci output for the relevant parts is attached below.
could anyone give me an idea:
- what kind of packets per second i could expect to achieve from this particular system (for small packets)
- what parameters i can tweak at the PCI level (or any other level, for that matter...) to achieve that level of performance
for example, how could i get my e1000 cards to say '64bit+ 133MHz+' to match the secondary side of the 82870P2 bridge?
thank you
-------------------------------------------------------------------------
lspci -t
-------------------------------------------------------------------------
-[00]-+-00.0
+-00.1
+-04.0-[01-03]--+-1c.0
| +-1d.0-[02]--+-01.0
| | \-02.0
| +-1e.0
| \-1f.0-[03]--
+-1d.0
+-1d.1
+-1d.2
+-1e.0-[04]--+-03.0
| \-06.0
+-1f.0
+-1f.1
\-1f.3-------------------------------------------------------------------------
lspci -vv (selected items)
-------------------------------------------------------------------------
0000:00:00.0 Host bridge: Intel Corp. E7501 Memory Controller Hub (rev 01)
Subsystem: Intel Corp. E7501 Memory Controller Hub
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Capabilities: [40] #09 [1105]
0000:00:04.0 PCI bridge: Intel Corp. E7000 Series Hub Interface D PCI-to-PCI Bridge (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32
Bus: primary=00, secondary=01, subordinate=03, sec-latency=0
I/O behind bridge: 0000b000-0000bfff
Memory behind bridge: f9000000-fbffffff
BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
0000:01:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04) (prog-if 20 [IO(X)-APIC])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Region 0: Memory at fb000000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [50] PCI-X non-bridge device.
Command: DPERE- ERO- RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-
0000:01:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, cache line size 10
Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
I/O behind bridge: 0000b000-0000bfff
Memory behind bridge: f9000000-faffffff
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort+ >Reset- FastB2B-
Capabilities: [50] PCI-X bridge device.
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
: Upstream: Capacity=0, Commitment Limit=0
: Downstream: Capacity=0, Commitment Limit=0
0000:02:01.0 Ethernet controller: Intel Corp. 82544EI Gigabit Ethernet Controller (Copper) (rev 02)
Subsystem: Intel Corp. PRO/1000 XT Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (63750ns min), cache line size 08
Interrupt: pin A routed to IRQ 10
Region 0: Memory at fa020000 (64-bit, non-prefetchable) [size=128K]
Region 2: Memory at fa000000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at b000 [size=32]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
0000:02:02.0 Ethernet controller: Intel Corp. 82544EI Gigabit Ethernet Controller (Copper) (rev 02)
Subsystem: Intel Corp. PRO/1000 XT Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (63750ns min), cache line size 08
Interrupt: pin A routed to IRQ 10
Region 0: Memory at fa040000 (64-bit, non-prefetchable) [size=128K]
Region 2: Memory at fa060000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at b400 [size=32]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
Signature · 2 lines
----------------------------------------------------------------------
Ray L <[email protected]>Quoted 4 lines · Ray Lehtiniemi
i'm trying to understand how to tune a pair of e1000 cards in a server box. the box is a dual xeon 3.06 with hyperthreading, using the intel e7501 chipset, with both cards on hub interface D. the application involves small UDP packets, generally under
These are 82544 cards hanging of P64H2, so they're running at PCI-X, but at what speed? Run ethtool -d eth<x> | grep Bus. The cards support 133Mhz, but having them adjacent on the same P64H2 probably bumps them down to 100Mhz. Can you put one on D and the other on another bus?
Quoted 2 lines
300 bytes. i need to maximize the number of packets per second transferred between the two cards.
There is a lot of current traffic on netdev about this topic. netdev is the official e1000 mailing this weekend. :-)
Quoted 8 lines
at the moment, i'm looking at the PCI bus in this box to see what might be tweakable. lspci output for the relevant parts is attached below.
could anyone give me an idea:
- what kind of packets per second i could expect to achieve from this particular system (for small packets)
What kind of numbers are you getting?
What kernel are you using?
What driver tweaks have you made, if any?
-scott
Quoted 8 lines · Scott Feldman
Quoted 4 lines · Ray Lehtiniemi
i'm trying to understand how to tune a pair of e1000 cards in a server box. the box is a dual xeon 3.06 with hyperthreading, using the intel e7501 chipset, with both cards on hub interface D. the application involves small UDP packets, generally under
These are 82544 cards hanging of P64H2, so they're running at PCI-X, but at what speed? Run ethtool -d eth<x> | grep Bus.
:-) just found ethtool and compiled it before i read this email. any other useful tools i should get?
Quoted 3 lines
The cards support
133Mhz, but having them adjacent on the same P64H2 probably bumps them down to 100Mhz.
# ethtool -d eth0 | grep Bus
Bus type: PCI-X
Bus speed: 133MHz
Bus width: 64-bit
# ethtool -d eth1 | grep Bus
Bus type: PCI-X
Bus speed: 133MHz
Bus width: 64-bitso that looks okay...
any idea why lspci -vv shows non-64bit, non-133 MHz? (i am assuming that is what the minus sign means)
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-
Quoted 1 line
Can you put one on D and the other on another bus?
not sure... have to look at the chassis tomorrow morning. a co-worker actually built the box, i've not seen it in person yet.
Quoted 2 lines
There is a lot of current traffic on netdev about this topic. netdev is the official e1000 mailing this weekend. :-)
i noticed that almost half of the messages in the e1000-devel archives since oct 2002 were sent in the last 10 days :-)
Quoted 1 line
What kind of numbers are you getting?
will start testing tomorrow, just starting my background research tonight
Quoted 1 line
What kernel are you using?
2.4.20 with driver 4.4.12-k1, and 2.6.bkcurr with driver 5.bkcurr
Quoted 1 line
What driver tweaks have you made, if any?
none yet. based on mailing list searches, i plan to:
- InterruptThrottleRate 15000
- TxIntDelay 0
i also plan to set rp_filter to zero for all interfaces.
thanks
Signature · 2 lines
----------------------------------------------------------------------
Ray L <[email protected]>Quoted 7 lines · Ray Lehtiniemi
any idea why lspci -vv shows non-64bit, non-133 MHz? (i am assuming that is what the minus sign means)
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-
turns out this is a bug in pci-utils-2.1.11. it was not correctly fetching the config information from the card, and was displaying zeroed data instead. i've included a patch at the end of this email and have also forwarded it to martin mares.
Quoted 4 lines
Quoted 1 line
Can you put one on D and the other on another bus?
not sure... have to look at the chassis tomorrow morning. a co-worker actually built the box, i've not seen it in person yet.
nope, can't move things around. this is a NexGate NSA 2040G, and everything is built into the motherboard.
Quoted 1 line
Quoted 1 line
What kind of numbers are you getting?
i'm seeing about 100Kpps, with all settings at their defaults on the 2.4.20 kernel.
basically, i have a couple of desktop PCs generating 480 streams of UDP data at 50 packets per second. Packet size on the wire, including 96 bits of IFG, is 128 bytes. these packets are forwarded through a user process on the NexGen box to an echoer process which is also running on the traffic generator boxes. the echoer sends them back to the NexGen user process, which forwards them back to the generator process. timestamps are logged for each packet at send, loop and recv points.
anything over 480 streams, and i start to get large latencies and packet drops, as measured by the timestamps in the sender and echoer process.
does 100Kpps sound reasonable for an untweaked 2.4.20 kernel?
thanks
lspci.c +6 -2
@@ -476,16 +476,19 @@static void show_pcix_nobridge (struct device *d,int where) { -u16 command =get_conf_word (d, where +PCI_PCIX_COMMAND ); -u32 status =get_conf_long (d, where +PCI_PCIX_STATUS ); +u16 command; +u32 status;printf ("PCI-X non-bridge device.\n" );if (verbose <2 )return ; +config_fetch (d, where,8 ); + command =get_conf_word (d, where +PCI_PCIX_COMMAND );printf ("\t\tCommand: DPERE%c ERO%c RBC=%d OST=%d\n" ,FLAG (command,PCI_PCIX_COMMAND_DPERE ),FLAG (command,PCI_PCIX_COMMAND_ERO ), ((command &PCI_PCIX_COMMAND_MAX_MEM_READ_BYTE_COUNT ) >>2U ), ((command &PCI_PCIX_COMMAND_MAX_OUTSTANDING_SPLIT_TRANS ) >>4U )); + status =get_conf_long (d, where +PCI_PCIX_STATUS ); printf("\t\tStatus: Bus=%u Dev=%u Func=%u 64bit%c 133MHz%c SCD%c USC%c, DC=%s, DMMRBC=%u, DMOST=%u, DMCRS=%u, RSCEM%c", ((status >> 8) & 0xffU), // bus ((status >> 3) & 0x1fU), // dev
@@ -509,6 +512,7 @@printf ("PCI-X bridge device.\n" );if (verbose <2 )return ; +config_fetch (d, where,8 ); secstatus =get_conf_word (d, where +PCI_PCIX_BRIDGE_SEC_STATUS ); printf("\t\tSecondary Status: 64bit%c, 133MHz%c, SCD%c, USC%c, SCO%c, SRD%c Freq=%d\n", FLAG(secstatus, PCI_PCIX_BRIDGE_SEC_STATUS_64BIT),
Signature · 2 lines
----------------------------------------------------------------------
Ray L <[email protected]>