RFC: i40e xmit path HW limitation

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 7:58 AM
To: [email protected]; Ananyev, Konstantin; Zhang, Helin
Subject: RFC: i40e xmit path HW limitation

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload buffers)."

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.

In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse() corresponding.
In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device driver since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.

Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some error code for a case when one of the clusters it's given has to be linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original cluster to that new cluster (using the private data) and release it when the send is done.

The first is a change in the API and would require from the application some additional handling (linearization). The second would require some additional memory but would keep all dirty details inside the driver and would leave the rest of the code intact.

Pls., comment.

thanks,
vlad

Quoted 12 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 7:58 AM
To: [email protected]; Ananyev, Konstantin; Zhang, Helin
Subject: RFC: i40e xmit path HW limitation

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

Quoted 6 lines

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload buffers)."

Yes, I remember the RX side just supports 5 segments per packet receiving. But what's the possible issue you thought about?

Quoted 3 lines

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.

What type of size of the small fragments? Basically 2KB is the default size of mbuf of most example applications. 2KB x 8 is bigger than 1.5KB. So it is enough for the maximum packet size we supported.
If 1KB mbuf is used, don't expect it can transmit more than 8KB size of packet.

Quoted 38 lines

In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse() corresponding.
In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device driver since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.
Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some error code for a case when one of the clusters it's given has to be linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original cluster to that new cluster (using the private data) and release it when the send is done.

The first is a change in the API and would require from the application some additional handling (linearization). The second would require some additional memory but would keep all dirty details inside the driver and would leave the rest of the code intact.

Pls., comment.

thanks,
vlad

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 9:44 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: [email protected]
Subject: Re: i40e xmit path HW limitation

Quoted 18 lines · Helin

Quoted 18 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 7:58 AM
To: [email protected]; Ananyev, Konstantin; Zhang, Helin
Subject: RFC: i40e xmit path HW limitation

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload buffers)."

Quoted 2 lines

Yes, I remember the RX side just supports 5 segments per packet receiving. But what's the possible issue you thought about?

Note that it's a Tx size we are talking about.

See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next repo. If such a cluster arrives and you post it on the HW ring - HW will shut this HW ring down permanently. The application will see that it's ring is stuck.

Quoted 5 lines

Quoted 4 lines

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.

What type of size of the small fragments? Basically 2KB is the default size of

mbuf of most

Quoted 1 line

example applications. 2KB x 8 is bigger than 1.5KB. So it is enough for the

maximum

Quoted 2 lines

packet size we supported.
If 1KB mbuf is used, don't expect it can transmit more than 8KB size of packet.

I kinda lost u here. Again, we talk about the Tx side here and buffers are not obligatory completely filled. Namely there may be a cluster with 15 fragments 100 bytes each.

Quoted 6 lines

Quoted 6 lines

In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse() corresponding. In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device

driver

Quoted 33 lines

Quoted 33 lines

since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.
Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some error code for a case when one of the clusters it's given has to be linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original cluster to that new cluster (using the private data) and release it when the send is done.

The first is a change in the API and would require from the application some additional handling (linearization). The second would require some additional memory but would keep all dirty details inside the driver and would leave the rest of the code intact.

Pls., comment.

thanks,
vlad

Quoted 28 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 9:44 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: [email protected]
Subject: Re: i40e xmit path HW limitation

Quoted 17 lines · Helin

Quoted 17 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 7:58 AM
To: [email protected]; Ananyev, Konstantin; Zhang, Helin
Subject: RFC: i40e xmit path HW limitation

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload buffers)."

Quoted 7 lines

Quoted 2 lines

Yes, I remember the RX side just supports 5 segments per packet receiving. But what's the possible issue you thought about?

Note that it's a Tx size we are talking about.

See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next repo. If such a cluster arrives and you post it on the HW ring - HW will shut this HW ring down permanently. The application will see that it's ring is stuck.

That issue was because of using more than 8 descriptors for a packet for TSO.

Quoted 5 lines

Quoted 4 lines

Quoted 3 lines

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.

What type of size of the small fragments? Basically 2KB is the default size of

mbuf of most

Quoted 7 lines

Quoted 1 line

example applications. 2KB x 8 is bigger than 1.5KB. So it is enough for the

maximum

Quoted 2 lines

packet size we supported.
If 1KB mbuf is used, don't expect it can transmit more than 8KB size of packet.

I kinda lost u here. Again, we talk about the Tx side here and buffers are not obligatory completely filled. Namely there may be a cluster with 15 fragments 100 bytes each.

The root cause is using more than 8 descriptors for a packet.

Linux driver can help
on reducing number of descriptors to be used by merging small size of payload together, right?
It is not for TSO, it is just for packet transmitting. 2 options in my mind:
1. Use should ensure it will not use more than 8 descriptors per packet for transmitting.

2. DPDK driver should try to merge small packet together for such case, like Linux kernel driver. I prefer to use option 1, users should ensure that in the application or up layer software, and keep the PMD driver as simple as possible.

But I have a thought that the maximum number of RX/TX descriptor should be able to be queried somewhere.

Regards,
Helin

Quoted 38 lines

Quoted 4 lines

Quoted 4 lines

In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse() corresponding. In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device

driver

Quoted 33 lines

Quoted 33 lines

since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.
Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some error code for a case when one of the clusters it's given has to be linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original cluster to that new cluster (using the private data) and release it when the send is done.

The first is a change in the API and would require from the application some additional handling (linearization). The second would require some additional memory but would keep all dirty details inside the driver and would leave the rest of the code intact.

Pls., comment.

thanks,
vlad

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 10:56 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: [email protected]
Subject: Re: i40e xmit path HW limitation

Quoted 28 lines · Helin

Quoted 28 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 9:44 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: [email protected]
Subject: Re: i40e xmit path HW limitation

Quoted 17 lines · Helin

Quoted 17 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 7:58 AM
To: [email protected]; Ananyev, Konstantin; Zhang, Helin
Subject: RFC: i40e xmit path HW limitation

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload buffers)."

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

Yes, I remember the RX side just supports 5 segments per packet receiving. But what's the possible issue you thought about?

Note that it's a Tx size we are talking about.

See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next

repo.

Quoted 2 lines

Quoted 2 lines

If such a cluster arrives and you post it on the HW ring - HW will shut this HW ring down permanently. The application will see that it's ring is

stuck.

Quoted 1 line

That issue was because of using more than 8 descriptors for a packet for TSO.

There is no problem in transmitting the TSO packet with more than 8 fragments. On the opposite - one can't transmit a non-TSO packet with more than 8 fragments.
One also can't transmit the TSO packet that would contain more than 8 fragments in a single TSO segment including the TSO headers.

Pls., read the HW spec as I quoted above for more details.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.

Quoted 8 lines

Quoted 8 lines

Quoted 2 lines

What type of size of the small fragments? Basically 2KB is the default size of

mbuf of most

Quoted 2 lines

example applications. 2KB x 8 is bigger than 1.5KB. So it is enough for the

maximum

Quoted 2 lines

packet size we supported.
If 1KB mbuf is used, don't expect it can transmit more than 8KB size of

packet.

Quoted 5 lines

Quoted 4 lines

I kinda lost u here. Again, we talk about the Tx side here and buffers are not obligatory completely filled. Namely there may be a cluster with
15 fragments 100 bytes each.

The root cause is using more than 8 descriptors for a packet.

That would be if u would like to SUPER simplify the HW limitation above. In that case u would significantly limit the different packets that may be sent without the linearization.

Quoted 5 lines

Linux driver can help
on reducing number of descriptors to be used by merging small size of payload together, right?
It is not for TSO, it is just for packet transmitting. 2 options in my mind:
1. Use should ensure it will not use more than 8 descriptors per packet for

transmitting.

This requirement is too restricting. Pls., see above.

Quoted 1 line

2. DPDK driver should try to merge small packet together for such case, like

Linux kernel driver.

Quoted 2 lines

I prefer to use option 1, users should ensure that in the application or up layer software, and keep the PMD driver as simple as possible.

The above statement is super confusing: on the one hand u suggest the DPDK driver to merge the small packet (fragments?) together (how?) and then u immediately propose the user application to do that. Could u, pls., clarify what exactly u suggest here?
If that's to leave it to the application - note that it would demand patching all existing DPDK applications that send TCP packets.

Quoted 4 lines

But I have a thought that the maximum number of RX/TX descriptor should be able to be queried somewhere.

There is no such thing as maximum number of Tx fragments in a TSO case. It's only limited by the Tx ring size.

Quoted 6 lines

Regards,
Helin

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse()

corresponding.

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

Quoted 3 lines

In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device

driver

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.
Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return

some

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

error code for a case when one of the clusters it's given has to be linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers

would

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not

taking

Quoted 18 lines

Quoted 18 lines

Quoted 18 lines

Quoted 18 lines

into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original cluster to that new cluster (using the private data) and release it when the send is done.

The first is a change in the API and would require from the application some additional handling (linearization). The second would require some additional memory but would keep all dirty details inside the driver and would leave the rest of the code intact.

Pls., comment.

thanks,
vlad

Quoted 30 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 10:56 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: [email protected]
Subject: Re: i40e xmit path HW limitation

Quoted 17 lines · Helin

Quoted 17 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 9:44 AM
To: Zhang, Helin; Ananyev, Konstantin
Cc: [email protected]
Subject: Re: i40e xmit path HW limitation

Quoted 6 lines · Helin

Quoted 6 lines

-----Original Message-----
From: Vlad Zolotarov [mailto:[email protected]]
Sent: Thursday, July 30, 2015 7:58 AM
To: [email protected]; Ananyev, Konstantin; Zhang, Helin
Subject: RFC: i40e xmit path HW limitation

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yes, I remember the RX side just supports 5 segments per packet

Quoted 7 lines

Quoted 4 lines

Quoted 4 lines

Quoted 1 line

But what's the possible issue you thought about?

Note that it's a Tx size we are talking about.

See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next

repo.

Quoted 2 lines

Quoted 2 lines

If such a cluster arrives and you post it on the HW ring - HW will shut this HW ring down permanently. The application will see that

Quoted 2 lines

stuck.

Quoted 1 line

That issue was because of using more than 8 descriptors for a packet

Quoted 2 lines

There is no problem in transmitting the TSO packet with more than 8

Quoted 3 lines

On the opposite - one can't transmit a non-TSO packet with more than 8 fragments.
One also can't transmit the TSO packet that would contain more than 8

Quoted 3 lines

in a single TSO segment including the TSO headers.

Pls., read the HW spec as I quoted above for more details.

I meant a packet to be transmitted by the hardware, but not the TSO

It could be a segment in TSO packet in memory.
The linearize check in kernel driver is not for TSO only, it is for both

NON-TSO cases.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 2 lines

What type of size of the small fragments? Basically 2KB is the default size of

mbuf of most

Quoted 2 lines

example applications. 2KB x 8 is bigger than 1.5KB. So it is enough for the

maximum

Quoted 2 lines

packet size we supported.
If 1KB mbuf is used, don't expect it can transmit more than 8KB

Quoted 9 lines

packet.

Quoted 5 lines

Quoted 4 lines

I kinda lost u here. Again, we talk about the Tx side here and buffers are not obligatory completely filled. Namely there may be a cluster with
15 fragments 100 bytes each.

The root cause is using more than 8 descriptors for a packet.

That would be if u would like to SUPER simplify the HW limitation above. In that case u would significantly limit the different packets that may

Quoted 6 lines

without the linearization.

Quoted 4 lines

Linux driver can help
on reducing number of descriptors to be used by merging small size of payload together, right?
It is not for TSO, it is just for packet transmitting. 2 options in

Quoted 1 line

Quoted 1 line

1. Use should ensure it will not use more than 8 descriptors per

Quoted 5 lines

transmitting.

This requirement is too restricting. Pls., see above.

Quoted 1 line

2. DPDK driver should try to merge small packet together for such

Quoted 5 lines

Linux kernel driver.

Quoted 2 lines

I prefer to use option 1, users should ensure that in the application or up layer software, and keep the PMD driver as simple as possible.

The above statement is super confusing: on the one hand u suggest the

Quoted 2 lines

driver to merge the small packet (fragments?) together (how?) and then u immediately propose the user application to do that. Could u, pls.,

Quoted 2 lines

exactly u suggest here?
If that's to leave it to the application - note that it would demand

Quoted 1 line

existing DPDK applications that send TCP packets.

Those are two of obvious options. One is to do that in PMD, the other one

that in up layer. I did not mean it needs to do both!

Quoted 9 lines

Quoted 6 lines

But I have a thought that the maximum number of RX/TX descriptor should be able to be queried somewhere.

There is no such thing as maximum number of Tx fragments in a TSO case. It's only limited by the Tx ring size.

Again, it is not for TSO case only. You are talking about how to

Anything missed can be added, as long as it is reasonable.

Regards,
Helin

Quoted 16 lines

Quoted 6 lines

Regards,
Helin

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse()

corresponding.

Quoted 9 lines

Quoted 9 lines

Quoted 3 lines

Quoted 3 lines

In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device

driver

Quoted 5 lines

Quoted 5 lines

since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.
Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

cluster has to be linearized and it should always be called

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

rte_eth_tx_burst(). Alternatively it may be called from

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

rte_eth_tx_burst() and rte_eth_tx_burst() is changed to

Quoted 2 lines

some

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

error code for a case when one of the clusters it's given

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

linearized.
2. Another option is to allocate a mempool in the driver with

Quoted 3 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

elements consuming a single page each (standard 2KB buffers

would

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

do). Number of elements in the pool should be as Tx ring

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

multiplied by "64KB/(linear data length of the buffer in the

Quoted 3 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

above)". Here I use 64KB as a maximum packet length and not

taking

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

into an account esoteric things like "Giant" TSO mentioned

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

spec above. Then we may actually go and linearize the

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

needed on top of the buffers from the pool above, post the

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

from the mempool above on the HW ring, link the original

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

that new cluster (using the private data) and release it

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

send is done.

The first is a change in the API and would require from the application some additional handling (linearization). The second would require some additional memory but would keep all dirty details inside the driver and would leave the rest of the code

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Pls., comment.

thanks,
vlad

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload buffers)."

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.
In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse() corresponding. In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device driver
since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.

Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some error code for a case when one of the clusters it's given has to be linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length

multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original cluster to that new cluster (using the private data) and release it when the send is done.

Quoted 32 lines · On Thu, 30 Jul 2015 17:57:33 +0300 Vlad Zolotarov <[email protected]>…

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload buffers)."

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.
In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse() corresponding. In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device driver
since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.

Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some error code for a case when one of the clusters it's given has to be linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length

Quoted 8 lines

multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original cluster to that new cluster (using the private data) and release it when the send is done.

Or just silently drop heavily scattered packets (and increment oerrors) with a PMD_TX_LOG debug message.

I think a DPDK driver doesn't have to accept all possible mbufs and do extra work. It seems reasonable to expect caller to be well behaved in this restricted ecosystem.

Quoted 49 lines · Stephen Hemminger

Quoted 35 lines · On Thu, 30 Jul 2015 17:57:33 +0300 Vlad Zolotarov <[email protected]>…

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors per segment for both the TSO header and the segment payload buffers)."

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.
In more standard environments like Linux or FreeBSD drivers the solution is straight forward - call skb_linearize()/m_collapse() corresponding. In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device driver
since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.

Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return
some
error code for a case when one of the clusters it's given has
to be
linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool

Quoted 8 lines

above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original
cluster to
that new cluster (using the private data) and release it when the send is done.

Or just silently drop heavily scattered packets (and increment oerrors) with a PMD_TX_LOG debug message.

Quoted 4 lines

I think a DPDK driver doesn't have to accept all possible mbufs and do extra work. It seems reasonable to expect caller to be well behaved in this restricted ecosystem.

How can the caller know what's well behaved? It's device dependent.

Quoted 56 lines · Avi Kivity

Quoted 53 lines · Stephen Hemminger

Quoted 39 lines · On Thu, 30 Jul 2015 17:57:33 +0300 Vlad Zolotarov <[email protected]>…

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including
both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is
unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors
per segment for both the TSO header and the segment payload buffers)."

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.
In more standard environments like Linux or FreeBSD drivers the solution
is straight forward - call skb_linearize()/m_collapse() corresponding. In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device driver
since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.

Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return
some
error code for a case when one of the clusters it's given has
to be
linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool

Quoted 8 lines

above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original
cluster to
that new cluster (using the private data) and release it when the send is done.

Or just silently drop heavily scattered packets (and increment oerrors) with a PMD_TX_LOG debug message.

Quoted 6 lines

Quoted 4 lines

I think a DPDK driver doesn't have to accept all possible mbufs and do extra work. It seems reasonable to expect caller to be well behaved in this restricted ecosystem.

How can the caller know what's well behaved? It's device dependent.

+1

Stephen, how do you imagine this well-behaved application? Having switch case by an underlying device type and then "well-behaving" correspondingly? Not to mention that to "well-behave" the application writer has to read HW specs and understand them, which would limit the amount of DPDK developers to a very small amount of people... ;) Not to mention that the mentioned above switch-case would be a super ugly thing to be found in an application that would raise a big question about the justification of a DPDK existence as as SDK providing device drivers interface. ;)

Quoted 64 lines · On Thu, 30 Jul 2015 19:50:27 +0300 Vlad Zolotarov <[email protected]>…

Quoted 52 lines · Avi Kivity

Quoted 50 lines · Stephen Hemminger

Quoted 47 lines · On Thu, 30 Jul 2015 17:57:33 +0300 Vlad Zolotarov <[email protected]>…

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including
both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is
unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors
per segment for both the TSO header and the segment payload buffers)."

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.
In more standard environments like Linux or FreeBSD drivers the solution
is straight forward - call skb_linearize()/m_collapse() corresponding. In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device driver
since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.

Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return
some
error code for a case when one of the clusters it's given has
to be
linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original
cluster to
that new cluster (using the private data) and release it when the send is done.

Quoted 8 lines

Quoted 7 lines

Or just silently drop heavily scattered packets (and increment oerrors) with a PMD_TX_LOG debug message.

I think a DPDK driver doesn't have to accept all possible mbufs and do extra work. It seems reasonable to expect caller to be well behaved in this restricted ecosystem.

How can the caller know what's well behaved? It's device dependent.

+1

Quoted 9 lines

Stephen, how do you imagine this well-behaved application? Having switch case by an underlying device type and then "well-behaving" correspondingly? Not to mention that to "well-behave" the application writer has to read HW specs and understand them, which would limit the amount of DPDK developers to a very small amount of people... ;) Not to mention that the mentioned above switch-case would be a super ugly thing to be found in an application that would raise a big question about the justification of a DPDK existence as as SDK providing device drivers interface. ;)

Either have a RTE_MAX_MBUF_SEGMENTS

that is global or
a mbuf_linearize function? Driver already can stash the mbuf pool used for Rx and reuse it for the transient Tx buffers.

Quoted 64 lines · On Thu, 30 Jul 2015 19:50:27 +0300 Vlad Zolotarov <[email protected]>…

Quoted 52 lines · Avi Kivity

Quoted 50 lines · Stephen Hemminger

Quoted 47 lines · On Thu, 30 Jul 2015 17:57:33 +0300 Vlad Zolotarov <[email protected]>…

Hi, Konstantin, Helin,
there is a documented limitation of xl710 controllers (i40e driver) which is not handled in any way by a DPDK driver.
From the datasheet chapter 8.4.1:

"• A single transmit packet may span up to 8 buffers (up to 8 data descriptors per packet including
both the header and payload buffers).
• The total number of data descriptors for the whole TSO (explained later on in this chapter) is
unlimited as long as each segment within the TSO obeys the previous rule (up to 8 data descriptors
per segment for both the TSO header and the segment payload buffers)."

This means that, for instance, long cluster with small fragments has to be linearized before it may be placed on the HW ring.
In more standard environments like Linux or FreeBSD drivers the solution
is straight forward - call skb_linearize()/m_collapse() corresponding. In the non-conformist environment like DPDK life is not that easy - there is no easy way to collapse the cluster into a linear buffer from inside the device driver
since device driver doesn't allocate memory in a fast path and utilizes the user allocated pools only.

Here are two proposals for a solution:

1. We may provide a callback that would return a user TRUE if a give cluster has to be linearized and it should always be called before rte_eth_tx_burst(). Alternatively it may be called from inside the rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return
some
error code for a case when one of the clusters it's given has
to be
linearized.
2. Another option is to allocate a mempool in the driver with the elements consuming a single page each (standard 2KB buffers would do). Number of elements in the pool should be as Tx ring length multiplied by "64KB/(linear data length of the buffer in the pool above)". Here I use 64KB as a maximum packet length and not taking into an account esoteric things like "Giant" TSO mentioned in the spec above. Then we may actually go and linearize the cluster if needed on top of the buffers from the pool above, post the buffer from the mempool above on the HW ring, link the original
cluster to
that new cluster (using the private data) and release it when the send is done.

Quoted 8 lines

Quoted 7 lines

Or just silently drop heavily scattered packets (and increment oerrors) with a PMD_TX_LOG debug message.

I think a DPDK driver doesn't have to accept all possible mbufs and do extra work. It seems reasonable to expect caller to be well behaved in this restricted ecosystem.

How can the caller know what's well behaved? It's device dependent.

+1

Quoted 9 lines

Stephen, how do you imagine this well-behaved application? Having switch case by an underlying device type and then "well-behaving" correspondingly? Not to mention that to "well-behave" the application writer has to read HW specs and understand them, which would limit the amount of DPDK developers to a very small amount of people... ;) Not to mention that the mentioned above switch-case would be a super ugly thing to be found in an application that would raise a big question about the justification of a DPDK existence as as SDK providing device drivers interface. ;)

Either have a RTE_MAX_MBUF_SEGMENTS that is global or a mbuf_linearize function? Driver already can stash the mbuf pool used for Rx and reuse it for the transient Tx buffers.