ixgbe: ierrors counter spuriously increasing in DPDK 2.1
Hi,
with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device:
./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
...
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-badcrc: 0 RX-badlen: 0 RX-errors: 1
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################
######################## NIC statistics for port 1 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-badcrc: 0 RX-badlen: 0 RX-errors: 1
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################
When packet forwarding is started the ports perform normally and properly forward all packets but a huge number of ierrors is counted:
testpmd> start
...
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 9011857 RX-missed: 0 RX-bytes: 5020932992 RX-badcrc: 0 RX-badlen: 0 RX-errors: 9011753
RX-nombuf: 0
TX-packets: 9026250 TX-errors: 0 TX-bytes: 2922375542
############################################################################
######################## NIC statistics for port 1 ########################
RX-packets: 9026250 RX-missed: 0 RX-bytes: 2922375542 RX-badcrc: 0 RX-badlen: 0 RX-errors: 9026138
RX-nombuf: 0
TX-packets: 9011857 TX-errors: 0 TX-bytes: 5020932992
############################################################################
When running the exact same test with DPDK version 2.0 no ierrors are reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic cards with DPDK 2.1?
Best regards,
Martin
Yes Marcin,
The issue was discussed here:
http://dpdk.org/ml/archives/dev/2015-September/023229.html
You can either fix the ierrors in ixgbe_dev_stats_get() or implement a workaround in your app getting the extended statistics and counting out some of extended counters from the ierrors.
Here is an example:
https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306
Regards,
Andriy
Quoted 56 lines · Martin Weiser
Hi,
with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device:
./testpmd -c
0xfe-n 4 -- --portmask0x3--interactive
...
testpmd> show port stats all######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-badcrc: 0 RX-badlen: 0 RX-errors: 1RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
#################################################################################################### NIC statistics for port 1 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-badcrc: 0 RX-badlen: 0 RX-errors: 1RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################When packet forwarding is started the ports perform normally and properly forward all packets but a huge number of ierrors is counted:
testpmd> start
...
testpmd> show port stats all######################## NIC statistics for port 0 ########################
RX-packets: 9011857 RX-missed: 0 RX-bytes: 5020932992 RX-badcrc: 0 RX-badlen: 0 RX-errors: 9011753RX-nombuf: 0
TX-packets: 9026250 TX-errors: 0 TX-bytes: 2922375542
#################################################################################################### NIC statistics for port 1 ########################
RX-packets: 9026250 RX-missed: 0 RX-bytes: 2922375542 RX-badcrc: 0 RX-badlen: 0 RX-errors: 9026138RX-nombuf: 0
TX-packets: 9011857 TX-errors: 0 TX-bytes: 5020932992
############################################################################When running the exact same test with DPDK version 2.0 no ierrors are reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic cards with DPDK 2.1?Best regards,
Martin
Signature · 1 line
Andriy Berestovskyy
Hi Andriy,
thank you for pointing this discussion out to me. I somehow missed it. Unfortunately it looks like the discussion stopped after Maryam made a good proposal so I will vote in on that and hopefully get things started again.
Best regards,
Martin
Quoted 74 lines · Andriy Berestovskyy
Yes Marcin,
The issue was discussed here:
http://dpdk.org/ml/archives/dev/2015-September/023229.htmlYou can either fix the ierrors in
ixgbe_dev_stats_get()or implement a workaround in your app getting the extended statistics and counting out some of extended counters from the ierrors.Here is an example:
https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306Regards,
AndriyQuoted 56 lines · Martin Weiser
Hi,
with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device:
./testpmd -c
0xfe-n 4 -- --portmask0x3--interactive
...
testpmd> show port stats all######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-badcrc: 0 RX-badlen: 0 RX-errors: 1RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
#################################################################################################### NIC statistics for port 1 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-badcrc: 0 RX-badlen: 0 RX-errors: 1RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################When packet forwarding is started the ports perform normally and properly forward all packets but a huge number of ierrors is counted:
testpmd> start
...
testpmd> show port stats all######################## NIC statistics for port 0 ########################
RX-packets: 9011857 RX-missed: 0 RX-bytes: 5020932992 RX-badcrc: 0 RX-badlen: 0 RX-errors: 9011753RX-nombuf: 0
TX-packets: 9026250 TX-errors: 0 TX-bytes: 2922375542
#################################################################################################### NIC statistics for port 1 ########################
RX-packets: 9026250 RX-missed: 0 RX-bytes: 2922375542 RX-badcrc: 0 RX-badlen: 0 RX-errors: 9026138RX-nombuf: 0
TX-packets: 9011857 TX-errors: 0 TX-bytes: 5020932992
############################################################################When running the exact same test with DPDK version 2.0 no ierrors are reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic cards with DPDK 2.1?Best regards,
Martin
Hi Martin,
We agreed on the main point: it's an issue. IMO the implementation details are up to Maryam.
There have been few patches, so I guess it will be fixed in 2.2.
Andriy
Quoted 89 lines · Martin Weiser
Hi Andriy,
thank you for pointing this discussion out to me. I somehow missed it. Unfortunately it looks like the discussion stopped after Maryam made a good proposal so I will vote in on that and hopefully get things started again.
Best regards,
MartinQuoted 74 lines · Andriy Berestovskyy
Yes Marcin,
The issue was discussed here:
http://dpdk.org/ml/archives/dev/2015-September/023229.htmlYou can either fix the ierrors in
ixgbe_dev_stats_get()or implement a workaround in your app getting the extended statistics and counting out some of extended counters from the ierrors.Here is an example:
https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306Regards,
AndriyQuoted 56 lines · Martin Weiser
Hi,
with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device:
./testpmd -c
0xfe-n 4 -- --portmask0x3--interactive
...
testpmd> show port stats all######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-badcrc: 0 RX-badlen: 0 RX-errors: 1RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
#################################################################################################### NIC statistics for port 1 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-badcrc: 0 RX-badlen: 0 RX-errors: 1RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
############################################################################When packet forwarding is started the ports perform normally and properly forward all packets but a huge number of ierrors is counted:
testpmd> start
...
testpmd> show port stats all######################## NIC statistics for port 0 ########################
RX-packets: 9011857 RX-missed: 0 RX-bytes: 5020932992 RX-badcrc: 0 RX-badlen: 0 RX-errors: 9011753RX-nombuf: 0
TX-packets: 9026250 TX-errors: 0 TX-bytes: 2922375542
#################################################################################################### NIC statistics for port 1 ########################
RX-packets: 9026250 RX-missed: 0 RX-bytes: 2922375542 RX-badcrc: 0 RX-badlen: 0 RX-errors: 9026138RX-nombuf: 0
TX-packets: 9011857 TX-errors: 0 TX-bytes: 5020932992
############################################################################When running the exact same test with DPDK version 2.0 no ierrors are reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic cards with DPDK 2.1?Best regards,
Martin
Signature · 1 line
Andriy Berestovskyy
Quoted 9 lines
From: dev [mailto:[email protected]] On Behalf Of Martin Weiser
Sent: Wednesday, October 21, 2015 9:38 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the device
Hi Martin,
To dig up an old thread - I've recently experienced similar issues that the rx error count increases on 82599, so I dug into the sources and realized that the MAC fault counters were being added to the rx error count.
The following patch removes the MAC local and remote fault counts from rx errors, and this has fixed the issue here:
http://dpdk.org/dev/patchwork/patch/8525/
If this issue is still open perhaps try the patch and see if it improves the situation.
On a related note, I'm currently implementing extended statistics for ixgbe, which showed me that the MAC faults were the root cause of the rx increase in my case.
-Harry
Hi Harry,
first of all thank you for your effort.
I have done some test with the current master (which includes you patch). It improves the situation but does not fix it entirely. The rx-error which showed up immediately after starting the interface is gone since this was probably caused by mac_remote_errors. But we still see a huge number of rx-errors although all packets are received properly and when looking at the extended stats those come from the rx_l3_l4_xsum_error counter.
In our setup we are dealing with lots of UDP traffic which does have the UDP checksum set to 0 (which to my knowledge is allowed for UDP). This traffic seems to cause those rx_l3_l4_xsum_errors. When doing the same test with other NICs (e.g. XL710) no rx-errors are accounted. For the generic stats interface I would prefer only packets that could not be received to be accounted in the rx-error counter regardless of the actual NIC. What do you think?
Regards,
Martin
Quoted 20 lines · Harry
Quoted 8 lines
From: dev [mailto:[email protected]] On Behalf Of Martin Weiser
Sent: Wednesday, October 21, 2015 9:38 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1 with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES ports without reason. Even directly after starting test-pmd the error counter immediately is 1 without even a single packet being sent to the deviceHi Martin,
To dig up an old thread - I've recently experienced similar issues that the rx error count increases on 82599, so I dug into the sources and realized that the MAC fault counters were being added to the rx error count.
The following patch removes the MAC local and remote fault counts from rx errors, and this has fixed the issue here:
http://dpdk.org/dev/patchwork/patch/8525/If this issue is still open perhaps try the patch and see if it improves the situation.
On a related note, I'm currently implementing extended statistics for ixgbe, which showed me that the MAC faults were the root cause of the rx increase in my case.
-Harry
PATCH 1/1 ixgbe: remove l3_l4_xsum_errors from rx errors
This patch removes l3_l4_xsum_errors from rx errors.
The reason to remove it is that UDP packets have an optional checksum, and when not calculated the checksum field should be set to zero. When the checksum is not calculated (zero-ed out), the hardware still counts a valid UDP packet as an l3_l4_xsum_error.
This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"
The solution is to remove l3_l4_xsum_errors from rx_errors, as discussed on
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/
--- drivers/net/ixgbe/ixgbe_ethdev.c | 1 - 1 file changed, 1 deletion(-)
drivers/net/ixgbe/ixgbe_ethdev.c +0 -1
@@ -2480,7 +2480,6 @@ ixgbe_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) total_missed_rx + hw_stats->illerrc + hw_stats->errbc + - hw_stats->xec + hw_stats->rfc + hw_stats->fccrc + hw_stats->fclast;
Signature · 1 line
1.9.1
Quoted 6 lines
From: Martin Weiser [mailto:[email protected]]
Subject: Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1The
rx-error which showed up immediately after starting the interface is gone since this was probably caused bymac_remote_errors.
Improvement - that's good.
Quoted 4 lines
But we still
see a huge number of rx-errors although all packets are received properly and when looking at the extended stats those come from therx_l3_l4_xsum_errorcounter.
That is useful information, good to know that statistic is the root cause.
Quoted 2 lines
In our setup we are dealing with lots of UDP traffic which does have the UDP checksum set to 0 (which to my knowledge is allowed for UDP).
Yes indeed checksum calculation for UDP is optional, and should be set to zero when not performed.
Quoted 4 lines
This
traffic seems to cause thoserx_l3_l4_xsum_errors. When doing the same
test with other NICs (e.g. XL710) no rx-errors are accounted.
So this is a ixgbe bug, and listed in the errata, item 43 "Integrity Error Reported for IPv4/UDP Packets with Zero Checksum" in http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/82599-10-gbe-controller-spec-update.pdf
Quoted 3 lines
For the generic stats interface I would prefer only packets that could not be received to be accounted in the rx-error counter regardless of the actual NIC. What do you think?
Agreed. I've sent a patch that removes "xec", the register name for l3_l4_xsum_errors:
http://dpdk.org/dev/patchwork/patch/8678/
Would you test it please? -Harry
Re: [PATCH] ixgbe: remove l3_l4_xsum_errors from rx errors
In reply to Harry van Haaren2015-11-04 15:50, Harry van Haaren:
Quoted 9 lines
This patch removes
l3_l4_xsum_errorsfrom rx errors.The reason to remove it is that UDP packets have an optional checksum, and when not calculated the checksum field should be set to zero. When the checksum is not calculated (zero-ed out), the hardware still counts a valid UDP packet as an
l3_l4_xsum_error.This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"
Please Harry, could you add a Fixes tag to track the origin of the bug?
PATCH v2 1/1 ixgbe: remove l3_l4_xsum_errors from rx errors
In reply to Martin WeiserThis patch removes l3_l4_xsum_errors from rx errors.
The reason to remove it is that UDP packets have an optional checksum, and when not calculated the checksum field should be set to zero. When the checksum is not calculated (zero-ed out), the hardware still counts a valid UDP packet as an l3_l4_xsum_error.
This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"
The solution is to remove l3_l4_xsum_errors from rx_errors, as discussed on
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/
---
v2: Add fixes line
drivers/net/ixgbe/ixgbe_ethdev.c | 1 - 1 file changed, 1 deletion(-)
drivers/net/ixgbe/ixgbe_ethdev.c +0 -1
@@ -2480,7 +2480,6 @@ ixgbe_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) total_missed_rx + hw_stats->illerrc + hw_stats->errbc + - hw_stats->xec + hw_stats->rfc + hw_stats->fccrc + hw_stats->fclast;
Signature · 1 line
1.9.1
Quoted 23 lines · Harry
Quoted 5 lines
From: Martin Weiser [mailto:[email protected]]
Subject: Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1 The
rx-error which showed up immediately after starting the interface is gone since this was probably caused bymac_remote_errors.Improvement - that's good.
Quoted 4 lines
But we still
see a huge number of rx-errors although all packets are received properly and when looking at the extended stats those come from therx_l3_l4_xsum_errorcounter.That is useful information, good to know that statistic is the root cause.
Quoted 2 lines
In our setup we are dealing with lots of UDP traffic which does have the UDP checksum set to 0 (which to my knowledge is allowed for UDP).
Yes indeed checksum calculation for UDP is optional, and should be set to zero when not performed.
Quoted 4 lines
This
traffic seems to cause thoserx_l3_l4_xsum_errors. When doing the same
test with other NICs (e.g. XL710) no rx-errors are accounted.So this is a ixgbe bug, and listed in the errata, item 43 "Integrity Error Reported for IPv4/UDP Packets with Zero Checksum" in http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/82599-10-gbe-controller-spec-update.pdf
Quoted 7 lines
Quoted 3 lines
For the generic stats interface I would prefer only packets that could not be received to be accounted in the rx-error counter regardless of the actual NIC. What do you think?
Agreed. I've sent a patch that removes "xec", the register name for
l3_l4_xsum_errors:
http://dpdk.org/dev/patchwork/patch/8678/Would you test it please? -Harry
I have tested the patch and now everything looks good.
Best regards, Martin
Thomas Monjalon
AppliedRe: [PATCH v2] ixgbe: remove l3_l4_xsum_errors from rx errors
In reply to Harry van Haaren2015-11-04 16:14, Harry van Haaren:
Quoted 10 lines
This patch removes
l3_l4_xsum_errorsfrom rx errors.The reason to remove it is that UDP packets have an optional checksum, and when not calculated the checksum field should be set to zero. When the checksum is not calculated (zero-ed out), the hardware still counts a valid UDP packet as an
l3_l4_xsum_error.This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"
Quoted 7 lines
The solution is to remove
l3_l4_xsum_errorsfromrx_errors, as discussed on
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/Fixes f6bf669b9900 ("ixgbe: account more Rx errors")Suggested-by Martin Weiser <[email protected]>Signed-off-by Harry van Haaren <[email protected]>
Applied, thanks