Skip to contentSkip to conversation

i40e card Tx resets

Intel-wired-lan Sowmini Varadhan 3 participants 14 messages Mar 14, 2016 – Mar 18, 2016

lore
Share & tools
lore
#0 lore raw

Hi,

I am trying out some DB stress tests on both i40e and ixgbe. The stress test that I use is rds-stress (http://linux.die.net/man/1/rds-stress),
and I can list out the entire set of steps and parameters that I am using to run this if that info is interesting.

My test bed is a pair of X5-2 (Haswell) servers, each with a Niantic (X540-AT2) card and a Fortville card. The Niantic/fortville cards are connected back-to-back, so I essentially have a 10G connection and a 40G connection.

Everything else (kernel, RDS modules, stress test and parameters) remaining the same, I get the expected throughput on the 10G connection, but the i40e connection goes through a lot of TX errors that result in console messages like this:

i40e 0000:81:00.0: TX driver issue detected, PF reset issued
i40e 0000:81:00.0 eth2: adding 68:05:ca:30:db:30 vid=0
i40e 0000:81:00.0: TX driver issue detected, PF reset issued
i40e 0000:81:00.0 eth2: VSI_seid 390, Hung TX queue 32, tx_pending: 82, NTC:0xeb, HWB: 0xeb, NTU: 0x13d, TAIL: 0x13d i40e 0000:81:00.0 eth2: VSI_seid 390, Issuing force_wb for TX queue 32, Interrupt Reg

I understand these are "mdd errors", but how can I find out what triggered these errors, any hints?

The other data-point here is that if I disable tso, and fall back to gso, there are no tx errors, and throughput matches the 10G connection (for the same set of test parameters).

Please let me know if there is any other info that would help. The kernel is a 4.5.0-rc2 kernel. Info for the i40e card is

    # ethtool -i eth3
    driver: i40e
    version: 1.4.8-k
    firmware-version: 5.02 0x80002285 0.0.0
    bus-info: 0000:81:00.1 
       :

Thanks in advance,
--Sowmini

#1 lore raw

Hi,

I have the similar problem. Would you like to make tests with pktgen tools? Maybe the test result is not related with tso.

Zhu Yanjun

Quoted 54 lines · Sowmini Varadhan

Hi,

I am trying out some DB stress tests on both i40e and ixgbe. The stress test that I use is rds-stress (http://linux.die.net/man/1/rds-stress),
and I can list out the entire set of steps and parameters that I am using to run this if that info is interesting.

My test bed is a pair of X5-2 (Haswell) servers, each with a Niantic (X540-AT2) card and a Fortville card. The Niantic/fortville cards are connected back-to-back, so I essentially have a 10G connection and a 40G connection.

Everything else (kernel, RDS modules, stress test and parameters) remaining the same, I get the expected throughput on the 10G connection, but the i40e connection goes through a lot of TX errors that result in console messages like this:

   i40e 0000:81:00.0: TX driver issue detected, PF reset issued
   i40e 0000:81:00.0 eth2: adding 68:05:ca:30:db:30 vid=0
   i40e 0000:81:00.0: TX driver issue detected, PF reset issued
   i40e 0000:81:00.0 eth2: VSI_seid 390, Hung TX queue 32, tx_pending: 82, NTC:0xeb, HWB: 0xeb, NTU: 0x13d, TAIL: 0x13d
   i40e 0000:81:00.0 eth2: VSI_seid 390, Issuing force_wb for TX queue 32, Interrupt Reg

I understand these are "mdd errors", but how can I find out what triggered these errors, any hints?

The other data-point here is that if I disable tso, and fall back to gso, there are no tx errors, and throughput matches the 10G connection (for the same set of test parameters).

Please let me know if there is any other info that would help. The kernel is a 4.5.0-rc2 kernel. Info for the i40e card is

     # ethtool -i eth3
     driver: i40e
     version: 1.4.8-k
     firmware-version: 5.02 0x80002285 0.0.0
     bus-info: 0000:81:00.1
        :

Thanks in advance,
--Sowmini

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
_______________________________________________
E1000-devel mailing list
E1000-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired

#2 lore raw
Quoted 5 lines · zhuyj

Hi,

I have the similar problem. Would you like to make tests with pktgen tools?
Maybe the test result is not related with tso.

Sorry. I explain this in details.
I have an similar problem. At first, I think it is related with tso. Then I made tests with pktgen tools and found that this similar problem still occurred whether
tso is enabled or not.

So I suggest to make tests with pktgen tools to exclude tso.

Best Regards!
Zhu Yanjun

Quoted 64 lines

Zhu Yanjun

Quoted 59 lines · Sowmini Varadhan

Hi,

I am trying out some DB stress tests on both i40e and ixgbe. The stress test that I use is rds-stress
(http://linux.die.net/man/1/rds-stress),
and I can list out the entire set of steps and parameters that I am using to run this if that info is interesting.

My test bed is a pair of X5-2 (Haswell) servers, each with a Niantic (X540-AT2) card and a Fortville card. The Niantic/fortville cards are connected back-to-back, so I essentially have a 10G connection and a 40G connection.

Everything else (kernel, RDS modules, stress test and parameters) remaining the same, I get the expected throughput on the 10G connection, but the i40e connection goes through a lot of TX errors that result in console messages like this:

i40e 0000:81:00.0: TX driver issue detected, PF reset issued i40e 0000:81:00.0 eth2: adding 68:05:ca:30:db:30 vid=0 i40e 0000:81:00.0: TX driver issue detected, PF reset issued i40e 0000:81:00.0 eth2: VSI_seid 390, Hung TX queue 32,
tx_pending: 82, NTC:0xeb, HWB: 0xeb, NTU: 0x13d, TAIL: 0x13d
i40e 0000:81:00.0 eth2: VSI_seid 390, Issuing force_wb for TX
queue 32, Interrupt Reg

I understand these are "mdd errors", but how can I find out what triggered these errors, any hints?

The other data-point here is that if I disable tso, and fall back to gso, there are no tx errors, and throughput matches the 10G connection (for the same set of test parameters).

Please let me know if there is any other info that would help. The kernel is a 4.5.0-rc2 kernel. Info for the i40e card is

     # ethtool -i eth3
     driver: i40e
     version: 1.4.8-k
     firmware-version: 5.02 0x80002285 0.0.0
     bus-info: 0000:81:00.1
        :

Thanks in advance,
--Sowmini

------------------------------------------------------------------------------

Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
_______________________________________________
E1000-devel mailing list
E1000-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired

#3 lore raw
Quoted 8 lines · zhuyj

Sorry. I explain this in details.
I have an similar problem. At first, I think it is related with tso. Then I made tests with pktgen tools and found that this similar problem still occurred whether
tso is enabled or not.

So I suggest to make tests with pktgen tools to exclude tso.

I realize that TSO might not be the root cause (Tushar also pointed that out) but might just be triggering the issue...

I dont think we need pktgen at this point- it's quite easy to reproduce this on commodity Haswell servers, and by installing the rds-stress from the rpm below:

http://public-yum.oracle.com/repo/OracleLinux/OL6/ofed_UEK/x86_64//getPackageSource/rds-tools-2.0.7-1.12.el6.src.rpm

To run it, set up 2 nodes connected on i40e. I shall call them "client" and "server" though both will send traffic in the test

Start the listener:
server# modprobe rds-tcp
server# rds-stress -r <server addr>

Start the test:

client# modprobe rds-tcp
client# rds-stress -r <client addr> -s <server-addr> -q 256 -a 8192 -d16 -t16 -T30

(all params are explained in the rds-stress man page)

If you do this on ixgbe, you will see that the column for "tx+rx K/s" shows a steady throughput, whereas i40e numbers are bursty and low.

Also, for i40e, you will see messages about TX hang on on the console.

I think that, to find the root-cause, we need to see what is triggering the mdd error.

Would be good if someone from Intel could provide some hints on how to do that (or try the above tests!)

--Sowmini

#4 lore raw
Quoted 32 lines · Sowmini Varadhan
Quoted 8 lines · zhuyj

Sorry. I explain this in details.
I have an similar problem. At first, I think it is related with tso. Then I made tests with pktgen tools and found that this similar problem still occurred whether
tso is enabled or not.

So I suggest to make tests with pktgen tools to exclude tso.

I realize that TSO might not be the root cause (Tushar also pointed that out) but might just be triggering the issue...

I dont think we need pktgen at this point- it's quite easy to reproduce this on commodity Haswell servers, and by installing the rds-stress from the rpm below:

http://public-yum.oracle.com/repo/OracleLinux/OL6/ofed_UEK/x86_64//getPackageSource/rds-tools-2.0.7-1.12.el6.src.rpm

To run it, set up 2 nodes connected on i40e. I shall call them "client" and "server" though both will send traffic in the test

Start the listener:
server# modprobe rds-tcp
server# rds-stress -r <server addr>

Start the test:

client# modprobe rds-tcp
client# rds-stress -r <client addr> -s <server-addr> -q 256 -a 8192 -d16 -t16 -T30

(all params are explained in the rds-stress man page)

Quoted 7 lines

If you do this on ixgbe, you will see that the column for "tx+rx K/s" shows a steady throughput, whereas i40e numbers are bursty and low.

Also, for i40e, you will see messages about TX hang on on the console.

I think that, to find the root-cause, we need to see what is triggering the mdd error.

Hi,

Thanks for your reply.
Yesterday I made tests with rds-tools. It is very pity that I can not reproduce my problem with rds-tools.

But with pktgen tools, I can reproduce this problem easily. And I found that if I set the packet size to 17792 or less than this size, this problem would not occur. But if I set the packet size > 17792, for example 17793, my problem would occur.

As such, maybe the packet size triggers my problem. I am not sure that the packet size will trigger your size.

Best Regards!
Zhu Yanjun

Quoted 5 lines

Would be good if someone from Intel could provide some hints on how to do that (or try the above tests!)

--Sowmini

#5 lore raw
Quoted 9 lines · zhuyj

Thanks for your reply.
Yesterday I made tests with rds-tools. It is very pity that I can not reproduce my problem with rds-tools.

But with pktgen tools, I can reproduce this problem easily. And I found that if I set the packet size to 17792 or less than this size, this problem would not occur. But if I set the packet size > 17792, for example 17793, my problem would occur.

I think it might have to do with number of sockets/cpu/cores and the irq balancing issues.

Quoted 2 lines

As such, maybe the packet size triggers my problem. I am not sure that the packet size will trigger your size.

In my case I did try against netperf request-response (but that is single threaded) and iperf (but that is unidirectional, i.e., it is not a bidirectional request-response test)

perhaps if you share the pktgen config (did you change the code itself)? some of the i40e experts at intel (who are also on the to/cc lists of this mail) can try it out themselves?

I still think that the fundamental problem can be identified by looking for what's causing the mdd event.

It must be some type of bug, because ixgbe works fine for me, and this seems like a regression on that performance.

--Sowmini

#6 lore raw
Quoted 22 lines · Sowmini Varadhan
Quoted 8 lines · zhuyj

Thanks for your reply.
Yesterday I made tests with rds-tools. It is very pity that I can not reproduce my problem with rds-tools.

But with pktgen tools, I can reproduce this problem easily. And I found that if I set the packet size to 17792 or less than this size, this problem would not occur. But if I set the packet size > 17792, for example 17793, my problem would occur.

I think it might have to do with number of sockets/cpu/cores and the irq balancing issues.

Quoted 2 lines

As such, maybe the packet size triggers my problem. I am not sure that the packet size will trigger your size.

In my case I did try against netperf request-response (but that is single threaded) and iperf (but that is unidirectional, i.e., it is not a bidirectional request-response test)

perhaps if you share the pktgen config (did you change the code itself)? some of the i40e experts at intel (who are also on the to/cc lists of this mail) can try it out themselves?

Quoted 8 lines

I still think that the fundamental problem can be identified by looking for what's causing the mdd event.

It must be some type of bug, because ixgbe works fine for me, and this seems like a regression on that performance.

--Sowmini

It is busy today. Tomorrow I will share the steps about pktgen tools.

Best Regards!
Zhu Yanjun

#7 lore raw
Quoted 1 line · zhuyj

It is busy today. Tomorrow I will share the steps about pktgen tools.

Ok. I can give that a try on my machine. It would be good to have some way to reproduce this as simply as possible, maybe we can take the discussion to netdev to see if others there have experienced this.

--Sowmini

#8 lore raw
Quoted 10 lines · Sowmini Varadhan
Quoted 1 line · zhuyj

It is busy today. Tomorrow I will share the steps about pktgen tools.

Ok. I can give that a try on my machine. It would be good to have some way to reproduce this as simply as possible, maybe we can take the discussion to netdev to see if others there have experienced this.

--Sowmini

1. modprobe NET_PKTGEN

2. download the tar file and uncompress to any directory. This tar file is from kernel. It is in samples/pktgen/

3. cd pktgen

4. pktgen_sample02_multiqueue.sh -i ethx -s size -t cpu_number

If size is set to a big number, the similar defect will occur. Adjust this size to a appropriate number, my defect will not occur.

In the test, I found some types igb nic, such as i210, will work well no matter the size is a big number.
some nic, such as 82580, it will not work well if the size is too big.

As such, I think my problem results from the hardware and the big size triggers this problem.

I hope this can help us all.

Zhu Yanjun

#9 lore raw
Quoted 9 lines · zhuyj
Quoted 8 lines · Sowmini Varadhan
Quoted 1 line · zhuyj

It is busy today. Tomorrow I will share the steps about pktgen tools.

Ok. I can give that a try on my machine. It would be good to have some way to reproduce this as simply as possible, maybe we can take the discussion to netdev to see if others there have experienced this.

Quoted 6 lines
Quoted 2 lines

--Sowmini

1. modprobe NET_PKTGEN

2. download the tar file and uncompress to any directory. This tar file is from kernel. It is in samples/pktgen/

Sorry. The tar file is in the attachment. Please check it.

Zhu Yanjun

Quoted 12 lines

3. cd pktgen

4. pktgen_sample02_multiqueue.sh -i ethx -s size -t cpu_number

If size is set to a big number, the similar defect will occur. Adjust this size to a appropriate number, my defect will not occur.

In the test, I found some types igb nic, such as i210, will work well no matter the size is a big number.
some nic, such as 82580, it will not work well if the size is too big.

Quoted 7 lines

As such, I think my problem results from the hardware and the big size triggers this problem.

I hope this can help us all.

Zhu Yanjun

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pktgen.tgz
Type: application/x-compressed-tar
Size: 6257 bytes
Desc: not available
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160317/607a4184/attachment-0001.bin>

#10 lore raw
In reply to zhuyj
Quoted 8 lines · zhuyj

1. modprobe NET_PKTGEN

2. download the tar file and uncompress to any directory. This tar file is from kernel. It is in samples/pktgen/

3. cd pktgen

4. pktgen_sample02_multiqueue.sh -i ethx -s size -t cpu_number

Indeed, I see the same thing as you, and it was very easy to reproduce. It was very interesting that the problem can happen with as few as 3 threads, at which point I see the TX hang at exactly -s 12305

I see:
i40e 0000:82:00.0: TX driver issue detected, PF reset issued
i40e 0000:82:00.0 eth2: VSI_seid 390, Hung TX queue 0, tx_pending: 492, NTC:0x140, HWB: 0x140, NTU: 0x12c, TAIL: 0x12c

I think the common factor in both our test cases is that we have some kernel thread that can efficiently send packets without any context switches.

Has anyone here seen this before? I'll see if I can find some cycles to figure this out, if not, maybe its worth bringing up on netdev, to see if others have seen this, and to draw some patterns.

Quoted 14 lines

If size is set to a big number, the similar defect will occur. Adjust this size to a appropriate number, my defect will not occur.

In the test, I found some types igb nic, such as i210, will work well no matter the size is a big number.
some nic, such as 82580, it will not work well if the size is too big.

As such, I think my problem results from the hardware and the big size triggers this problem.

I hope this can help us all.

Zhu Yanjun

#11 lore raw
Quoted 6 lines · On Thu, 17 Mar 2016 14:56:14 -0400 Sowmini Varadhan…
Quoted 5 lines · zhuyj

1. modprobe NET_PKTGEN

2. download the tar file and uncompress to any directory. This tar file is from kernel. It is in samples/pktgen/

Quoted 8 lines
Quoted 3 lines

3. cd pktgen

4. pktgen_sample02_multiqueue.sh -i ethx -s size -t cpu_number

Indeed, I see the same thing as you, and it was very easy to reproduce. It was very interesting that the problem can happen with as few as 3 threads, at which point I see the TX hang at exactly -s 12305

Okay, sorry I hadn't jumped into this thread yet.

I can uniquivically tell you that what Sowmini saw with the MDD with stack based RDS-STRESS testing is *NOT* the same as what you're seeing while using pktgen with invalid huge skb->data buffers.

We can ask on netdev if the driver should defend against this kind of input to hard_start_xmit (transmit routine), but the driver doesn't check the maximum length of the skb to see if it is invalid, because the stack can never build (only pktgen can) these invalid SKBs.

The issue is that pktgen builds skb->data with a contiguous buffer of whatever size transmit requested, (regardless of MTU) and then sends it straight to the transmit routine, no segmentation flags, no MSS set.

This causes the driver to build a transmit descriptor with an invalid length, which the hardware then "ASSERTS" on by issuing an MDD interrupt and freezing the bad acting queue.

Quoted 7 lines

I see:
i40e 0000:82:00.0: TX driver issue detected, PF reset issued
i40e 0000:82:00.0 eth2: VSI_seid 390, Hung TX queue 0, tx_pending: 492, NTC:0x140, HWB: 0x140, NTU: 0x12c, TAIL: 0x12c

I think the common factor in both our test cases is that we have some kernel thread that can efficiently send packets without any context switches.

You've found a red herring (mistakenly connected two separate events) so I think you can stop going down this path (pktgen).

Quoted 3 lines

Has anyone here seen this before? I'll see if I can find some cycles to figure this out, if not, maybe its worth bringing up on netdev, to see if others have seen this, and to draw some patterns.

we don't need to bring it up on netdev. We have a way to troubleshoot MDDs that I can send to you, if you want to do the work. Otherwise we need to have some time to reproduce here.

Quoted 6 lines
Quoted 6 lines

If size is set to a big number, the similar defect will occur. Adjust this size to a appropriate number, my defect will not occur.

In the test, I found some types igb nic, such as i210, will work well no matter the size is a big number.
some nic, such as 82580, it will not work well if the size is too big.

This is mostly a combination of driver implementation and how the hardware handles a descriptor that is too large. The driver *could* check to make sure the skb->data is never too large, but in that same vein, we *could* fix pktgen to never send a frame greater than MTU down to the driver.

Quoted 5 lines
Quoted 5 lines

As such, I think my problem results from the hardware and the big size triggers this problem.

I hope this can help us all.

Unfortunately Zhu's problem with pktgen is not a reproducer of Sowmini's problem.

In the case of pktgen, it is a "don't do that, because it hurts" kind of bug. In the case of rds-stress, we need to reproduce it here and figure out what hardware constraint the driver is violating during set up of the transmit.

#12 lore raw
Quoted 8 lines · Jesse Brandeburg

We can ask on netdev if the driver should defend against this kind of input to hard_start_xmit (transmit routine), but the driver doesn't check the maximum length of the skb to see if it is invalid, because the stack can never build (only pktgen can) these invalid SKBs.

The issue is that pktgen builds skb->data with a contiguous buffer of whatever size transmit requested, (regardless of MTU) and then sends it straight to the transmit routine, no segmentation flags, no MSS set.

I see. And after you mentioned it, I checked with ixgbe, sure enough, that also results in a tx-hang for the pktgen test case (whereas there were no issues with the (rds-stress , ixgbe) test.

I would surmise that pktgen is a bit of an outlier, more interesting to focus on those cases that use the regular stack.

I dont know if dpdk can create the same issues as pktgen?

Quoted 3 lines

we don't need to bring it up on netdev. We have a way to troubleshoot MDDs that I can send to you, if you want to do the work. Otherwise we need to have some time to reproduce here.

yes, I can do the work, since I already have this nicely set up. Just need some hings on how to trouble-shoot the mdd.

--Sowmini

#13 lore raw
In reply to Jesse Brandeburg
Quoted 55 lines · Jesse Brandeburg
Quoted 6 lines · On Thu, 17 Mar 2016 14:56:14 -0400 Sowmini Varadhan…
Quoted 5 lines · zhuyj

1. modprobe NET_PKTGEN

2. download the tar file and uncompress to any directory. This tar file is from kernel. It is in samples/pktgen/

Quoted 7 lines
Quoted 3 lines

3. cd pktgen

4. pktgen_sample02_multiqueue.sh -i ethx -s size -t cpu_number

Indeed, I see the same thing as you, and it was very easy to reproduce. It was very interesting that the problem can happen with as few as 3 threads, at which point I see the TX hang at exactly -s 12305

Okay, sorry I hadn't jumped into this thread yet.

I can uniquivically tell you that what Sowmini saw with the MDD with stack based RDS-STRESS testing is *NOT* the same as what you're seeing while using pktgen with invalid huge skb->data buffers.

We can ask on netdev if the driver should defend against this kind of input to hard_start_xmit (transmit routine), but the driver doesn't check the maximum length of the skb to see if it is invalid, because the stack can never build (only pktgen can) these invalid SKBs.

The issue is that pktgen builds skb->data with a contiguous buffer of whatever size transmit requested, (regardless of MTU) and then sends it straight to the transmit routine, no segmentation flags, no MSS set.

This causes the driver to build a transmit descriptor with an invalid length, which the hardware then "ASSERTS" on by issuing an MDD interrupt and freezing the bad acting queue.

Quoted 7 lines

I see:
i40e 0000:82:00.0: TX driver issue detected, PF reset issued
i40e 0000:82:00.0 eth2: VSI_seid 390, Hung TX queue 0, tx_pending: 492, NTC:0x140, HWB: 0x140, NTU: 0x12c, TAIL: 0x12c

I think the common factor in both our test cases is that we have some kernel thread that can efficiently send packets without any context switches.

You've found a red herring (mistakenly connected two separate events) so I think you can stop going down this path (pktgen).

Quoted 3 lines

Has anyone here seen this before? I'll see if I can find some cycles to figure this out, if not, maybe its worth bringing up on netdev, to see if others have seen this, and to draw some patterns.

we don't need to bring it up on netdev. We have a way to troubleshoot MDDs that I can send to you, if you want to do the work. Otherwise we need to have some time to reproduce here.

Quoted 3 lines
Quoted 3 lines

If size is set to a big number, the similar defect will occur. Adjust this size to a appropriate number, my defect will not occur.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

In the test, I found some types igb nic, such as i210, will work well no matter the size is a big number.
some nic, such as 82580, it will not work well if the size is too big.

This is mostly a combination of driver implementation and how the hardware handles a descriptor that is too large. The driver *could* check to make sure the skb->data is never too large, but in that same vein, we *could* fix pktgen to never send a frame greater than MTU down to the driver.

Do you mean this is not a bug in nic?
And it is unnecessary to fix it?

But if a test tool makes tests like pktgen, how to handle it?

We just suggests not to make such tests?

Best Regards!
Zhu Yanjun

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines

As such, I think my problem results from the hardware and the big size triggers this problem.

I hope this can help us all.

Unfortunately Zhu's problem with pktgen is not a reproducer of Sowmini's problem.

In the case of pktgen, it is a "don't do that, because it hurts" kind of bug. In the case of rds-stress, we need to reproduce it here and figure out what hardware constraint the driver is violating during set up of the transmit.