Skip to contentSkip to conversation

ixgbe: ierrors counter spuriously increasing in DPDK 2.1

PATCH Superseded Martin Weiser 4 participants 12 messages Oct 21, 2015 – Nov 4, 2015 Network drivers

lore
Share & tools
lore
#0 lore raw

Hi,

with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device:

./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
...
testpmd> show port stats all

######################## NIC statistics for port 0 ########################

  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1

RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################

######################## NIC statistics for port 1 ########################

  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1

RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################

When packet forwarding is started the ports perform normally and properly forward all packets but a huge number of ierrors is counted:

testpmd> start
...
testpmd> show port stats all

######################## NIC statistics for port 0 ########################

  RX-packets: 9011857    RX-missed: 0          RX-bytes:  5020932992
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9011753

RX-nombuf: 0
TX-packets: 9026250 TX-errors: 0 TX-bytes: 2922375542
############################################################################

######################## NIC statistics for port 1 ########################

  RX-packets: 9026250    RX-missed: 0          RX-bytes:  2922375542
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9026138

RX-nombuf: 0
TX-packets: 9011857 TX-errors: 0 TX-bytes: 5020932992
############################################################################

When running the exact same test with DPDK version 2.0 no ierrors are reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic cards with DPDK 2.1?

Best regards,
Martin

#1 lore raw

Yes Marcin,
The issue was discussed here:
http://dpdk.org/ml/archives/dev/2015-September/023229.html

You can either fix the ierrors in ixgbe_dev_stats_get() or implement a workaround in your app getting the extended statistics and counting out some of extended counters from the ierrors.

Here is an example:
https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306

Regards,
Andriy

Quoted 56 lines · Martin Weiser

Hi,

with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device:

./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
...
testpmd> show port stats all

######################## NIC statistics for port 0 ########################

  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1

RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################

######################## NIC statistics for port 1 ########################

  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1

RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################

When packet forwarding is started the ports perform normally and properly forward all packets but a huge number of ierrors is counted:

testpmd> start
...
testpmd> show port stats all

######################## NIC statistics for port 0 ########################

  RX-packets: 9011857    RX-missed: 0          RX-bytes:  5020932992
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9011753

RX-nombuf: 0
TX-packets: 9026250 TX-errors: 0 TX-bytes: 2922375542
############################################################################

######################## NIC statistics for port 1 ########################

  RX-packets: 9026250    RX-missed: 0          RX-bytes:  2922375542
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9026138

RX-nombuf: 0
TX-packets: 9011857 TX-errors: 0 TX-bytes: 5020932992
############################################################################

When running the exact same test with DPDK version 2.0 no ierrors are reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic cards with DPDK 2.1?

Best regards,
Martin

Signature · 1 line
Andriy Berestovskyy
#2 lore raw

Hi Andriy,

thank you for pointing this discussion out to me. I somehow missed it. Unfortunately it looks like the discussion stopped after Maryam made a good proposal so I will vote in on that and hopefully get things started again.

Best regards,
Martin

Quoted 74 lines · Andriy Berestovskyy

Yes Marcin,
The issue was discussed here:
http://dpdk.org/ml/archives/dev/2015-September/023229.html

You can either fix the ierrors in ixgbe_dev_stats_get() or implement a workaround in your app getting the extended statistics and counting out some of extended counters from the ierrors.

Here is an example:
https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306

Regards,
Andriy

Quoted 56 lines · Martin Weiser

Hi,

with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device:

./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
...
testpmd> show port stats all

######################## NIC statistics for port 0 ########################

  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1

RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################

######################## NIC statistics for port 1 ########################

  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1

RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################

When packet forwarding is started the ports perform normally and properly forward all packets but a huge number of ierrors is counted:

testpmd> start
...
testpmd> show port stats all

######################## NIC statistics for port 0 ########################

  RX-packets: 9011857    RX-missed: 0          RX-bytes:  5020932992
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9011753

RX-nombuf: 0
TX-packets: 9026250 TX-errors: 0 TX-bytes: 2922375542
############################################################################

######################## NIC statistics for port 1 ########################

  RX-packets: 9026250    RX-missed: 0          RX-bytes:  2922375542
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9026138

RX-nombuf: 0
TX-packets: 9011857 TX-errors: 0 TX-bytes: 5020932992
############################################################################

When running the exact same test with DPDK version 2.0 no ierrors are reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic cards with DPDK 2.1?

Best regards,
Martin

#3 lore raw

Hi Martin,
We agreed on the main point: it's an issue. IMO the implementation details are up to Maryam.

There have been few patches, so I guess it will be fixed in 2.2.

Andriy

Quoted 89 lines · Martin Weiser

Hi Andriy,

thank you for pointing this discussion out to me. I somehow missed it. Unfortunately it looks like the discussion stopped after Maryam made a good proposal so I will vote in on that and hopefully get things started again.

Best regards,
Martin

Quoted 74 lines · Andriy Berestovskyy

Yes Marcin,
The issue was discussed here:
http://dpdk.org/ml/archives/dev/2015-September/023229.html

You can either fix the ierrors in ixgbe_dev_stats_get() or implement a workaround in your app getting the extended statistics and counting out some of extended counters from the ierrors.

Here is an example:
https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306

Regards,
Andriy

Quoted 56 lines · Martin Weiser

Hi,

with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device:

./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
...
testpmd> show port stats all

######################## NIC statistics for port 0 ########################

  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1

RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################

######################## NIC statistics for port 1 ########################

  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1

RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################

When packet forwarding is started the ports perform normally and properly forward all packets but a huge number of ierrors is counted:

testpmd> start
...
testpmd> show port stats all

######################## NIC statistics for port 0 ########################

  RX-packets: 9011857    RX-missed: 0          RX-bytes:  5020932992
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9011753

RX-nombuf: 0
TX-packets: 9026250 TX-errors: 0 TX-bytes: 2922375542
############################################################################

######################## NIC statistics for port 1 ########################

  RX-packets: 9026250    RX-missed: 0          RX-bytes:  2922375542
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9026138

RX-nombuf: 0
TX-packets: 9011857 TX-errors: 0 TX-bytes: 5020932992
############################################################################

When running the exact same test with DPDK version 2.0 no ierrors are reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic cards with DPDK 2.1?

Best regards,
Martin

Signature · 1 line
Andriy Berestovskyy
#4 lore raw
In reply to Martin Weiser
Quoted 9 lines

From: dev [mailto:[email protected]] On Behalf Of Martin Weiser
Sent: Wednesday, October 21, 2015 9:38 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device

Hi Martin,

To dig up an old thread - I've recently experienced similar issues that the rx error count increases on 82599, so I dug into the sources and realized that the MAC fault counters were being added to the rx error count.

The following patch removes the MAC local and remote fault counts from rx errors, and this has fixed the issue here:
http://dpdk.org/dev/patchwork/patch/8525/

If this issue is still open perhaps try the patch and see if it improves the situation.

On a related note, I'm currently implementing extended statistics for ixgbe, which showed me that the MAC faults were the root cause of the rx increase in my case.

-Harry

#5 lore raw

Hi Harry,

first of all thank you for your effort.
I have done some test with the current master (which includes you patch). It improves the situation but does not fix it entirely. The rx-error which showed up immediately after starting the interface is gone since this was probably caused by mac_remote_errors. But we still see a huge number of rx-errors although all packets are received properly and when looking at the extended stats those come from the rx_l3_l4_xsum_error counter.
In our setup we are dealing with lots of UDP traffic which does have the UDP checksum set to 0 (which to my knowledge is allowed for UDP). This traffic seems to cause those rx_l3_l4_xsum_errors. When doing the same test with other NICs (e.g. XL710) no rx-errors are accounted. For the generic stats interface I would prefer only packets that could not be received to be accounted in the rx-error counter regardless of the actual NIC. What do you think?

Regards,
Martin

Quoted 20 lines · Harry
Quoted 8 lines

From: dev [mailto:[email protected]] On Behalf Of Martin Weiser
Sent: Wednesday, October 21, 2015 9:38 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1 with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device

Hi Martin,

To dig up an old thread - I've recently experienced similar issues that the rx error count increases on 82599, so I dug into the sources and realized that the MAC fault counters were being added to the rx error count.

The following patch removes the MAC local and remote fault counts from rx errors, and this has fixed the issue here:
http://dpdk.org/dev/patchwork/patch/8525/

If this issue is still open perhaps try the patch and see if it improves the situation.

On a related note, I'm currently implementing extended statistics for ixgbe, which showed me that the MAC faults were the root cause of the rx increase in my case.

-Harry

#6 lore raw

PATCH 1/1 ixgbe: remove l3_l4_xsum_errors from rx errors

This patch removes l3_l4_xsum_errors from rx errors.

The reason to remove it is that UDP packets have an optional checksum, and when not calculated the checksum field should be set to zero. When the checksum is not calculated (zero-ed out), the hardware still counts a valid UDP packet as an l3_l4_xsum_error.

This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"

The solution is to remove l3_l4_xsum_errors from rx_errors, as discussed on
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/

---
 drivers/net/ixgbe/ixgbe_ethdev.c | 1 -
 1 file changed, 1 deletion(-)
drivers/net/ixgbe/ixgbe_ethdev.c +0 -1
index 25966ef..9ed5b49 100644
@@ -2480,7 +2480,6 @@ ixgbe_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 	                  total_missed_rx +
 	                  hw_stats->illerrc +
 	                  hw_stats->errbc +
-	                  hw_stats->xec +
 	                  hw_stats->rfc +
 	                  hw_stats->fccrc +
 	                  hw_stats->fclast;
Signature · 1 line
1.9.1
#8 lore raw
In reply to Martin Weiser
Quoted 6 lines

From: Martin Weiser [mailto:[email protected]]
Subject: Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

The
rx-error which showed up immediately after starting the interface is gone since this was probably caused by mac_remote_errors.

Improvement - that's good.

Quoted 4 lines

But we still
see a huge number of rx-errors although all packets are received properly and when looking at the extended stats those come from the rx_l3_l4_xsum_error counter.

That is useful information, good to know that statistic is the root cause.

Quoted 2 lines

In our setup we are dealing with lots of UDP traffic which does have the UDP checksum set to 0 (which to my knowledge is allowed for UDP).

Yes indeed checksum calculation for UDP is optional, and should be set to zero when not performed.

Quoted 4 lines

This
traffic seems to cause those rx_l3_l4_xsum_errors. When doing the same
test with other NICs (e.g. XL710) no rx-errors are accounted.

So this is a ixgbe bug, and listed in the errata, item 43 "Integrity Error Reported for IPv4/UDP Packets with Zero Checksum" in http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/82599-10-gbe-controller-spec-update.pdf

Quoted 3 lines

For the generic stats interface I would prefer only packets that could not be received to be accounted in the rx-error counter regardless of the actual NIC. What do you think?

Agreed. I've sent a patch that removes "xec", the register name for l3_l4_xsum_errors:
http://dpdk.org/dev/patchwork/patch/8678/

Would you test it please? -Harry

#7 lore raw

Re: [PATCH] ixgbe: remove l3_l4_xsum_errors from rx errors

In reply to Harry van Haaren

2015-11-04 15:50, Harry van Haaren:

Quoted 9 lines

This patch removes l3_l4_xsum_errors from rx errors.

The reason to remove it is that UDP packets have an optional checksum, and when not calculated the checksum field should be set to zero. When the checksum is not calculated (zero-ed out), the hardware still counts a valid UDP packet as an l3_l4_xsum_error.

This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"

Please Harry, could you add a Fixes tag to track the origin of the bug?

#10 lore raw

PATCH v2 1/1 ixgbe: remove l3_l4_xsum_errors from rx errors

In reply to Martin Weiser

This patch removes l3_l4_xsum_errors from rx errors.

The reason to remove it is that UDP packets have an optional checksum, and when not calculated the checksum field should be set to zero. When the checksum is not calculated (zero-ed out), the hardware still counts a valid UDP packet as an l3_l4_xsum_error.

This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"

The solution is to remove l3_l4_xsum_errors from rx_errors, as discussed on
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/

Fixes f6bf669b9900 ("ixgbe: account more Rx errors")

---

v2: Add fixes line

 drivers/net/ixgbe/ixgbe_ethdev.c | 1 -
 1 file changed, 1 deletion(-)
drivers/net/ixgbe/ixgbe_ethdev.c +0 -1
index 25966ef..9ed5b49 100644
@@ -2480,7 +2480,6 @@ ixgbe_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 	                  total_missed_rx +
 	                  hw_stats->illerrc +
 	                  hw_stats->errbc +
-	                  hw_stats->xec +
 	                  hw_stats->rfc +
 	                  hw_stats->fccrc +
 	                  hw_stats->fclast;
Signature · 1 line
1.9.1
#9 lore raw
In reply to Van Haaren, Harry
Quoted 23 lines · Harry
Quoted 5 lines

From: Martin Weiser [mailto:[email protected]]
Subject: Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1 The
rx-error which showed up immediately after starting the interface is gone since this was probably caused by mac_remote_errors.

Improvement - that's good.

Quoted 4 lines

But we still
see a huge number of rx-errors although all packets are received properly and when looking at the extended stats those come from the rx_l3_l4_xsum_error counter.

That is useful information, good to know that statistic is the root cause.

Quoted 2 lines

In our setup we are dealing with lots of UDP traffic which does have the UDP checksum set to 0 (which to my knowledge is allowed for UDP).

Yes indeed checksum calculation for UDP is optional, and should be set to zero when not performed.

Quoted 4 lines

This
traffic seems to cause those rx_l3_l4_xsum_errors. When doing the same
test with other NICs (e.g. XL710) no rx-errors are accounted.

So this is a ixgbe bug, and listed in the errata, item 43 "Integrity Error Reported for IPv4/UDP Packets with Zero Checksum" in http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/82599-10-gbe-controller-spec-update.pdf

Quoted 7 lines
Quoted 3 lines

For the generic stats interface I would prefer only packets that could not be received to be accounted in the rx-error counter regardless of the actual NIC. What do you think?

Agreed. I've sent a patch that removes "xec", the register name for l3_l4_xsum_errors:
http://dpdk.org/dev/patchwork/patch/8678/

Would you test it please? -Harry

I have tested the patch and now everything looks good.

Best regards, Martin

#11 lore raw

Re: [PATCH v2] ixgbe: remove l3_l4_xsum_errors from rx errors

In reply to Harry van Haaren

2015-11-04 16:14, Harry van Haaren:

Quoted 10 lines

This patch removes l3_l4_xsum_errors from rx errors.

The reason to remove it is that UDP packets have an optional checksum, and when not calculated the checksum field should be set to zero. When the checksum is not calculated (zero-ed out), the hardware still counts a valid UDP packet as an l3_l4_xsum_error.

This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"

Quoted 7 lines

The solution is to remove l3_l4_xsum_errors from rx_errors, as discussed on
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/

Fixes f6bf669b9900 ("ixgbe: account more Rx errors")

Applied, thanks