How to count tx and rx bytes?

Is there an agreed upon standard for exactly what ethernet drivers should be counting for rx-bytes and tx-bytes? For example, should the counters include the 4-byte FCS? Should they include the ethernet header?

From what I can tell, various drivers count the rx and tx bytes differently, which cannot be correct....

Thanks,
Ben

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Good question.

It should be that all drivers use what skb->len ends up with at rx/tx time.

However, it is often faster to just let the hardware keep track of these statistics (tg3 is one example of a chip that can do this). And sometimes these mechanisms take the FCS or whatever into account and this as you note makes the numbers different.

On Mon, 15 Dec 2003 14:17:29 -0800 "David S. Miller" <[email protected]> wrote:

| On Mon, 15 Dec 2003 12:03:58 -0800
| Ben Greear <[email protected]> wrote:
| 
| > Is there an agreed upon standard for exactly what ethernet drivers
| > should be counting for rx-bytes and tx-bytes?  For example, should the
| > counters include the 4-byte FCS?  Should they include the ethernet header?
| 
| Good question.
| 
| It should be that all drivers use what skb->len ends up with at
| rx/tx time.
| 
| However, it is often faster to just let the hardware keep track
| of these statistics (tg3 is one example of a chip that can do
| this).  And sometimes these mechanisms take the FCS or whatever
| into account and this as you note makes the numbers different.

RFC 1573:
RFC 2233:
RFC 1213:
all agree that on an "interface" the number of octets received (InOctets) is:
The total number of octets received on the interface, including framing characters.

so I read that as including MAC headers and FCS. Without reading these, I would have said include MAC headers but not FCS.

Similar for OutOctets.

--
~Randy
MOTD: Always include version info.

Yes.

Yes, but not the preamble.
[[ Ethernet packets have an 8 byte preamble before the station addresses. It doesn't make sense to count this, as
all modern NICs strip it off, and
a few leading bits may be dropped at each hop
]]

Not quite. The skb->len value omits the 4 byte FCS/CRC.

The driver should correct, although this is non-critical. Using hardware counters where available used to make sense, but today it's better to have the software driver maintain the statistics. The only thing the hardware counters are needed for is otherwise-unobservable errors, such as missed packets and CRC errors.

This is imprecise. "Framing" might be misread to include the preamble, which should be omitted from the count.

The original Linux errors counters were influenced by what the dp8390 chip reported. Later, the definitions in appendix B of the dc21040 manual served as a more general guideline (although the software counters were never a one-for-one copy of any specific hardware semantics).

Donald Becker				[email protected]
Scyld Computing Corporation		http://www.scyld.com
914 Bay Ridge Road, Suite 220		Scyld Beowulf cluster systems
Annapolis MD 21403			410-990-9993

On Mon, 15 Dec 2003 18:19:27 -0500 (EST) Donald Becker <[email protected]> wrote:

| On Mon, 15 Dec 2003, Randy.Dunlap wrote:
| 
| > On Mon, 15 Dec 2003 14:17:29 -0800 "David S. Miller" <[email protected]> wrote:
| > 
| > | On Mon, 15 Dec 2003 12:03:58 -0800
| > | Ben Greear <[email protected]> wrote:
| > | 
| > | > Is there an agreed upon standard for exactly what ethernet drivers
| > | > should be counting for rx-bytes and tx-bytes?  For example, should the
| > | > counters include the 4-byte FCS?
| 
| Yes.
| 
| > | >  Should they include the ethernet header?
| 
| Yes, but not the preamble.
|   [[ Ethernet packets have an 8 byte preamble before the station
|   addresses.  It doesn't make sense to count this, as
|     all modern NICs strip it off, and
|     a few leading bits may be dropped at each hop
|   ]]
| 
| > | It should be that all drivers use what skb->len ends up with at
| > | rx/tx time.
| 
| Not quite.  The skb->len value omits the 4 byte FCS/CRC.
| 
| > | However, it is often faster to just let the hardware keep track
| > | of these statistics (tg3 is one example of a chip that can do
| > | this).  And sometimes these mechanisms take the FCS or whatever
| > | into account and this as you note makes the numbers different.
| 
| The driver should correct, although this is non-critical.
| Using hardware counters where available used to make sense, but today
| it's better to have the software driver maintain the statistics.  The
| only thing the hardware counters are needed for is
| otherwise-unobservable errors, such as missed packets and CRC errors.
| 
| > RFC 1573:
| > RFC 2233:
| > RFC 1213:
| > all agree that on an "interface" the number of octets received (InOctets)
| > is:
| >  The total number of octets received on the interface, including
| >  framing characters.
| 
| This is imprecise.  "Framing" might be misread to include the preamble,
| which should be omitted from the count.

Yes, sorry.

| The original Linux errors counters were influenced by what the dp8390
| chip reported.  Later, the definitions in appendix B of the dc21040
| manual served as a more general guideline (although the software
| counters were never a one-for-one copy of any specific hardware
| semantics).

I agree with Don's comments, based on my experiences in former lives. I don't know about the DC21040, but I worked at/on both National and Intel LAN drivers.

--
~Randy
MOTD: Always include version info.

A more noticeable problem is what I found in e100: It was adding skb->len after doing eth_type_trans, which yanks off the ethernet header... I moved it before the eth_type_trans and I am getting much better numbers. It does appear that the e1000 is counting the FCS in transmitted bytes, however, and unless we tell the e100 to receive the FCS, it will not count those on receive.

Maybe the stats-polling code in the driver could do the necessary subtraction to remove the FCS from the results (ie, subtract (packets-since-last-sample * 4) bytes.

Thanks,
Ben

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

The whole reason to use the chip internally computed stats is to avoid having to do "stats->foo++" at all, your suggestion basically eliminates this purpose.

Well, I was assuming that the stats polling is a fixed cost O(1), where-as the per-packet calculation is O(n). I am quite sure this assumption is true for e1000, but I have not looked at tg3. So, for e1000, the cost of subtracting out the FCS would be basically free.

All that said, from Randy's email, it appears we should be including the FCS anyway...

Ben

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

I see, I misunderstood your idea. That would work, and yes all of the effort would be expended at netdev->get_stats() time.

Yes, indeed. So the drivers that do not have hardware statistics doing this, and are using skb->len, need to add on the FCS length (which is 4 bytes right?) when accumulating stats->* values.

Sounds like a nice janitor job :)

Should they include the frame padding in tx-bytes? Some do, some don't. IMHO they should, because you do actually transmit those bytes on the wire.

Regards,
Rask Ingemann Lambertsen

On Tue, 16 Dec 2003 14:38:16 +0100 Rask Ingemann Lambertsen <[email protected]> wrote:

| On Mon, Dec 15, 2003 at 12:03:58PM -0800, Ben Greear wrote:
| > 
| > Is there an agreed upon standard for exactly what ethernet drivers
| > should be counting for rx-bytes and tx-bytes?  For example, should the
| > counters include the 4-byte FCS?  Should they include the ethernet header?
| 
| Should they include the frame padding in tx-bytes? Some do, some don't.
| IMHO they should, because you do actually transmit those bytes on the wire.

If you are referring to padding that is put there by software, then Yes.

--
~Randy

The padding is often done in the NIC itself, and since the bytes are put on the wire, they should be counted...

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com