Getting physical packet counts with LRO enabled with ixgbe?
Subsystems (2)
I notice that with LRO enabled, the interface stats count the LRO'd pkts, not the physical ones on the wire.
I also tried using ethtool -S, but it seems those counters are the same.
Is there any way to get the actual rx/tx packet count on the wire?
Also, for the rx/tx bytes, I assume that isn't counting the protocol headers for the physical pkts that have been merged into a single LRO packet. Is there any way to get the wire stats for bytes as well?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 7 lines · Ben Greear
I notice that with LRO enabled, the interface stats count the LRO'd pkts, not the physical ones on the wire.
I also tried using ethtool -S, but it seems those counters are the same.
Is there any way to get the actual rx/tx packet count on the wire?
Depending on which device you're using ixgbe with, there are slightly different registers to get what you want.
The only suggestion I have for you though is to refer to the datasheets for each device on our SourceForge site (e1000.sf.net). Some of the relevant counters to look at are PRC64, PRC127, etc, and GPRC/GPTC. For the per-queue stuff, you'll need to look at the TQSMR and RQSMR mapping registers. Let me know if you need assistance in using these registers.
Quoted 3 lines
Also, for the rx/tx bytes, I assume that isn't counting the protocol headers for the physical pkts that have been merged into a single LRO packet. Is there any way to get the wire stats for bytes as well?
The counters for per-byte are purely software-based, so if the packet is LRO'd, you can probably do some somewhat trivial math with the MTU to find the actual wire stats. But we only compute what we pass to the stack, so it'd be the LRO'd packet.
Cheers,
-PJ Waskiewicz
Quoted 9 lines · Peter P Waskiewicz Jr
Quoted 7 lines · Ben Greear
I notice that with LRO enabled, the interface stats count the LRO'd pkts, not the physical ones on the wire.
I also tried using ethtool -S, but it seems those counters are the same.
Is there any way to get the actual rx/tx packet count on the wire?
Quoted 8 lines
Depending on which device you're using ixgbe with, there are slightly different registers to get what you want.
The only suggestion I have for you though is to refer to the datasheets for each device on our SourceForge site (e1000.sf.net). Some of the relevant counters to look at are PRC64, PRC127, etc, and GPRC/GPTC. For the per-queue stuff, you'll need to look at the TQSMR and RQSMR mapping registers. Let me know if you need assistance in using these registers.
Thanks, I'll look at the data-sheet.
I don't care about per-queue stats at this time, just over-all NIC stats.
Quoted 8 lines
Quoted 3 lines
Also, for the rx/tx bytes, I assume that isn't counting the protocol headers for the physical pkts that have been merged into a single LRO packet. Is there any way to get the wire stats for bytes as well?
The counters for per-byte are purely software-based, so if the packet is LRO'd, you can probably do some somewhat trivial math with the MTU to find the actual wire stats. But we only compute what we pass to the stack, so it'd be the LRO'd packet.
That's a bummer. I'm guessing you might get close to right on average with some trivial math, but if someone is sending you pkts with size of 1000 and your MTU is 1500, would there be any way to tell that the pkts were originally 1000 bytes instead of 1500?
Next time you guys re-compile your hardware, please consider adding byte counters :)
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 33 lines · Ben Greear
Quoted 9 lines · Peter P Waskiewicz Jr
Quoted 7 lines · Ben Greear
I notice that with LRO enabled, the interface stats count the LRO'd pkts, not the physical ones on the wire.
I also tried using ethtool -S, but it seems those counters are the same.
Is there any way to get the actual rx/tx packet count on the wire?
Quoted 8 lines
Depending on which device you're using ixgbe with, there are slightly different registers to get what you want.
The only suggestion I have for you though is to refer to the datasheets for each device on our SourceForge site (e1000.sf.net). Some of the relevant counters to look at are PRC64, PRC127, etc, and GPRC/GPTC. For the per-queue stuff, you'll need to look at the TQSMR and RQSMR mapping registers. Let me know if you need assistance in using these registers.
Thanks, I'll look at the data-sheet.
I don't care about per-queue stats at this time, just over-all NIC stats.
Quoted 8 lines
Quoted 3 lines
Also, for the rx/tx bytes, I assume that isn't counting the protocol headers for the physical pkts that have been merged into a single LRO packet. Is there any way to get the wire stats for bytes as well?
The counters for per-byte are purely software-based, so if the packet is LRO'd, you can probably do some somewhat trivial math with the MTU to find the actual wire stats. But we only compute what we pass to the stack, so it'd be the LRO'd packet.
Quoted 4 lines
That's a bummer. I'm guessing you might get close to right on average with some trivial math, but if someone is sending you pkts with size of 1000 and your MTU is 1500, would there be any way to tell that the pkts were originally 1000 bytes instead of 1500?
Good point.
Quoted 1 line
Next time you guys re-compile your hardware, please consider adding byte counters :)
On 10G adapters, byte counters can skyrocket quickly, so we'd need to read them often to avoid them wrapping. But I will forward your request to our HW design folks and see if they have other ideas to implement these counters and make them efficient.
Cheers,
-PJ
Quoted 5 lines · Peter P Waskiewicz Jr
Quoted 4 lines
That's a bummer. I'm guessing you might get close to right on average with some trivial math, but if someone is sending you pkts with size of 1000 and your MTU is 1500, would there be any way to tell that the pkts were originally 1000 bytes instead of 1500?
Quoted 8 lines
Good point.
Quoted 1 line
Next time you guys re-compile your hardware, please consider adding byte counters :)
On 10G adapters, byte counters can skyrocket quickly, so we'd need to read them often to avoid them wrapping. But I will forward your request to our HW design folks and see if they have other ideas to implement these counters and make them efficient.
It still takes a while to wrap 64-bit counters :)
But, you do have to read every 3 secs or so if you're using 32-bit counters.
This can be dealt with in user-space easily enough as long as polling the NIC for counters is efficient.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 5 lines
Quoted 1 line
Next time you guys re-compile your hardware, please consider adding byte counters :)
On 10G adapters, byte counters can skyrocket quickly, so we'd need to read them often to avoid them wrapping.
10G Ethernet is ~1.16 GB/s (GiB/s for purists I guess) for simplicity, call that 2GB/s or 2^31 bytes per second. If the counter is 64 bits, that would suggest wrap in 2^64/2^31 or 2^33 seconds right? Or have I made some nasty math error?
I'm having quit a difficult time imagining that someone would have 32 bit
counters in a 10G NIC.
rick jones
Quoted 7 lines · Peter P Waskiewicz Jr
Quoted 5 lines
That's a bummer. I'm guessing you might get close to right on average with some trivial math, but if someone is sending you pkts with size of 1000 and your MTU is 1500, would there be any way to tell that the pkts were originally 1000 bytes instead of 1500?
Good point.
Well if you did GRO instead of LRO then none of this would matter :)
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 10 lines · Herbert Xu
Quoted 7 lines · Peter P Waskiewicz Jr
Quoted 5 lines
That's a bummer. I'm guessing you might get close to right on average with some trivial math, but if someone is sending you pkts with size of 1000 and your MTU is 1500, would there be any way to tell that the pkts were originally 1000 bytes instead of 1500?
Good point.
Well if you did GRO instead of LRO then none of this would matter :)
I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very much a useful feature for me.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
David Miller
maintainerFrom: Ben Greear <[email protected]>
Date: Fri, 25 Sep 2009 14:50:44 -0700
Quoted 2 lines
I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very much a useful feature for me.
And it'll stay like that until you work with Herbert to fix that performance difference.
Quoted 8 lines · David Miller
From: Ben Greear <[email protected]>
Date: Fri, 25 Sep 2009 14:50:44 -0700Quoted 2 lines
I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very much a useful feature for me.
And it'll stay like that until you work with Herbert to fix that performance difference.
Actually I believe he's talking about hardware LRO which is a different beast.
AFAICS the ixgbe driver doesn't use software LRO anymore.
So if it is hardware LRO then what we need is for the hardware to switch over :)
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 10 lines · Herbert Xu
Quoted 8 lines · David Miller
From: Ben Greear<[email protected]>
Date: Fri, 25 Sep 2009 14:50:44 -0700Quoted 2 lines
I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very much a useful feature for me.
And it'll stay like that until you work with Herbert to fix that performance difference.
Quoted 7 lines
Actually I believe he's talking about hardware LRO which is a different beast.
AFAICS the ixgbe driver doesn't use software LRO anymore.
So if it is hardware LRO then what we need is for the hardware to switch over :)
Either way, I will be happy to test & benchmark patches of this nature if someone wants to post them.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 14 lines · Ben Greear
Quoted 13 lines · Herbert Xu
Quoted 8 lines · David Miller
From: Ben Greear<[email protected]>
Date: Fri, 25 Sep 2009 14:50:44 -0700Quoted 2 lines
I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very much a useful feature for me.
And it'll stay like that until you work with Herbert to fix that performance difference.
Actually I believe he's talking about hardware LRO which is a different beast.
Quoted 8 lines
Quoted 4 lines
AFAICS the ixgbe driver doesn't use software LRO anymore.
So if it is hardware LRO then what we need is for the hardware to switch over :)
Either way, I will be happy to test & benchmark patches of this nature if someone wants to post them.
I can't post a patch that changes my hardware behavior. :-)
So to be clear, ixgbe uses GRO, and does not use software LRO. In 82599 adapters, we have a HW LRO engine. So we run HW LRO and GRO together. What Herbert is referring to is that we change our HW design of the HW LRO engine to do more of what GRO is doing today. That would require a silicon spin, so that wouldn't happen soon unfortunately.
-PJ
Quoted 5 lines · Ben Greear
Quoted 2 lines
Well if you did GRO instead of LRO then none of this would matter :)
I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very much a useful feature for me.
Sorry we are each talking about different things. The performance difference you observed is software GRO vs. hardware LRO. What I was talking about above is switching the Intel hardware to do GRO instead of LRO.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 10 lines · Herbert Xu
Quoted 7 lines · Peter P Waskiewicz Jr
Quoted 5 lines
That's a bummer. I'm guessing you might get close to right on average with some trivial math, but if someone is sending you pkts with size of 1000 and your MTU is 1500, would there be any way to tell that the pkts were originally 1000 bytes instead of 1500?
Good point.
Well if you did GRO instead of LRO then none of this would matter :)
ixgbe uses GRO, but we have HW RSC/LRO running on 82599 adapters underneath GRO.
-PJ
Quoted 5 lines · Peter P Waskiewicz Jr
Quoted 2 lines
Well if you did GRO instead of LRO then none of this would matter :)
ixgbe uses GRO, but we have HW RSC/LRO running on 82599 adapters underneath GRO.
I know. What I was suggesting is that the hardware should also switch away from LRO to GRO, i.e., restrict what you merge so that resegmentation can be performed.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtI noticed that 'ethtool -d' has some counters that may be right, including bytes.
(I'm looking at the 82599 chipset currently)
0x04074: gprc (Good Packets Received Count) 0xF611E2C2 0x04078: bprc (Broadcast Packets Rx Count) 0x00000000 0x0407C: mprc (Multicast Packets Rx Count) 0x0000000C 0x04080: gptc (Good Packets Transmitted Count) 0xC9962C10 0x04088: gorcl (Good Octets Rx Count Low) 0x9A23F0B0 0x0408C: gorch (Good Octets Rx Count High) 0x00000000 0x04090: gotcl (Good Octets Tx Count Low) 0x5DFAACD4 0x04094: gotch (Good Octets Tx Count High) 0x00000000
Any suggestions on a more efficient way to get these than dumping 'ethtool -d' and searching all that data? Maybe put it in 'ethtool -S' as well ?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
I'm poking at the ixgbe_main code in 2.6.31.
It seems from the spec sheet that the 82599 supports the GORCH.
/* 82598 hardware only has a 32 bit counter in the high register */
if (hw->mac.type == ixgbe_mac_82599EB) {
adapter->stats.gorc += IXGBE_READ_REG(hw, IXGBE_GORCL);
IXGBE_READ_REG(hw, IXGBE_GORCH); /* to clear */stats.gorc is 64-bit, so any reason not to grab the 4 high-bits out of GORCL and add them to stats.gorc instead of just clearing them as this code seems to do?
That gives us some precious extra seconds to read counters before they wrap :)
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com