ixgbe vPMD RX functions and buffer number minimum requirement

DISCUSSION Zoltan Kiss 3 participants 7 messages Jul 24, 2015 – Jul 29, 2015

Share & tools

Zoltan Kiss

author

2015-07-24 14:59 UTC #0 lore raw

Hi,

I was thinking how to handle the situation when you call rte_eth_rx_burst with less than RTE_IXGBE_VPMD_RX_BURST buffers. In ODP-DPDK unfortunately we can't force this requirement onto the calling application.
One idea I had to check in ixgbe_recv_pkts_vec() if nb_pkts < RTE_IXGBE_VPMD_RX_BURST, and call ixgbe_recv_pkts_bulk_alloc in that case. Accordingly, in ixgbe_recv_scattered_pkts_vec() we could call ixgbe_recv_scattered_pkts() in this case. A branch predictor can easily eliminate the performance penalty of this, and applications can use whatever burst size feasible for them. The obvious problem could be whether you can mix the receive functions this way. I have a feeling it wouldn't fly, but I wanted to ask first before spending time investigate this option further.

Regards,

Zoltan

Ananyev, Konstantin

2015-07-24 16:43 UTC #1 lore raw

Hi Zoltan,

Quoted 8 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, July 24, 2015 4:00 PM
To: Richardson, Bruce; [email protected]
Subject: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Hi,

Quoted 13 lines

I was thinking how to handle the situation when you call rte_eth_rx_burst with less than RTE_IXGBE_VPMD_RX_BURST buffers. In ODP-DPDK unfortunately we can't force this requirement onto the calling application.
One idea I had to check in ixgbe_recv_pkts_vec() if nb_pkts < RTE_IXGBE_VPMD_RX_BURST, and call ixgbe_recv_pkts_bulk_alloc in that case. Accordingly, in ixgbe_recv_scattered_pkts_vec() we could call ixgbe_recv_scattered_pkts() in this case. A branch predictor can easily eliminate the performance penalty of this, and applications can use whatever burst size feasible for them.
The obvious problem could be whether you can mix the receive functions this way. I have a feeling it wouldn't fly, but I wanted to ask first before spending time investigate this option further.

No, it is not possible to mix different RX functions, they setup/use ixgbe_rx_queue fields in a different manner.
Konstantin

Quoted 4 lines

Regards,
Zoltan

Zoltan Kiss

author

2015-07-27 11:38 UTC #2 lore raw

Hi Konstantin,

Thanks! Another question I would have: why does _recv_raw_pkts_vec() insist on (nb_pkts > RTE_IXGBE_VPMD_RX_BURST)? Looking at the code it should be able to return packets when nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP. The split_flags check in ixgbe_recv_scattered_pkts_vec() would be a bit more complicated, and therefore maybe would have a tiny performance overhead as well, but I don't it would be anything serious.

Regards,

Zoltan

Quoted 32 lines · Konstantin

Hi Zoltan,
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, July 24, 2015 4:00 PM
To: Richardson, Bruce; [email protected]
Subject: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Hi,
Quoted 13 lines
I was thinking how to handle the situation when you call rte_eth_rx_burst with less than RTE_IXGBE_VPMD_RX_BURST buffers. In ODP-DPDK unfortunately we can't force this requirement onto the calling application.
One idea I had to check in ixgbe_recv_pkts_vec() if nb_pkts < RTE_IXGBE_VPMD_RX_BURST, and call ixgbe_recv_pkts_bulk_alloc in that case. Accordingly, in ixgbe_recv_scattered_pkts_vec() we could call ixgbe_recv_scattered_pkts() in this case. A branch predictor can easily eliminate the performance penalty of this, and applications can use whatever burst size feasible for them.
The obvious problem could be whether you can mix the receive functions this way. I have a feeling it wouldn't fly, but I wanted to ask first before spending time investigate this option further.
No, it is not possible to mix different RX functions, they setup/use ixgbe_rx_queue fields in a different manner.
Konstantin
Quoted 4 lines
Regards,
Zoltan

Ananyev, Konstantin

2015-07-28 00:10 UTC #3 lore raw

Hi Zoltan,

Quoted 6 lines

-----Original Message-----
From: Zoltan Kiss [mailto:[email protected]]
Sent: Monday, July 27, 2015 12:38 PM
To: Ananyev, Konstantin; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement

Quoted 6 lines

Hi Konstantin,
Thanks! Another question I would have: why does _recv_raw_pkts_vec() insist on (nb_pkts > RTE_IXGBE_VPMD_RX_BURST)? Looking at the code it should be able to return packets when nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP.

Yes, that seems pretty trivial modification. Don't know any good reason why it wasn't done that way.

Quoted 4 lines

The split_flags check in
ixgbe_recv_scattered_pkts_vec() would be a bit more complicated, and therefore maybe would have a tiny performance overhead as well, but I don't it would be anything serious.

I think, if the performance wouldn't be affected, that will be really useful change. So it is definitely worth to try.
Probably even _recv_raw_pkts_vec() for first nb_pkts & ~(RTE_IXGBE_VPMD_RX_BURST - 1), and then sort of scalar analog for the remainder.
Konstantin

Quoted 39 lines

Regards,
Zoltan
Quoted 32 lines · Konstantin
Hi Zoltan,
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, July 24, 2015 4:00 PM
To: Richardson, Bruce; [email protected]
Subject: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Hi,
Quoted 13 lines
I was thinking how to handle the situation when you call rte_eth_rx_burst with less than RTE_IXGBE_VPMD_RX_BURST buffers. In ODP-DPDK unfortunately we can't force this requirement onto the calling application.
One idea I had to check in ixgbe_recv_pkts_vec() if nb_pkts < RTE_IXGBE_VPMD_RX_BURST, and call ixgbe_recv_pkts_bulk_alloc in that case. Accordingly, in ixgbe_recv_scattered_pkts_vec() we could call ixgbe_recv_scattered_pkts() in this case. A branch predictor can easily eliminate the performance penalty of this, and applications can use whatever burst size feasible for them.
The obvious problem could be whether you can mix the receive functions this way. I have a feeling it wouldn't fly, but I wanted to ask first before spending time investigate this option further.
No, it is not possible to mix different RX functions, they setup/use ixgbe_rx_queue fields in a different manner.
Konstantin
Quoted 4 lines
Regards,
Zoltan

Liang, Cunming

2015-07-28 08:49 UTC #4 lore raw

Hi,

Quoted 8 lines · Konstantin

Hi Zoltan,
Quoted 6 lines
-----Original Message-----
From: Zoltan Kiss [mailto:[email protected]]
Sent: Monday, July 27, 2015 12:38 PM
To: Ananyev, Konstantin; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement

Quoted 8 lines

Quoted 6 lines
Hi Konstantin,
Thanks! Another question I would have: why does _recv_raw_pkts_vec() insist on (nb_pkts > RTE_IXGBE_VPMD_RX_BURST)? Looking at the code it should be able to return packets when nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP.
Yes, that seems pretty trivial modification. Don't know any good reason why it wasn't done that way.

It's true, looks too strict condition check. I tend to set "nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_IXGBE_DESCS_PER_LOOP)" instead. And do loop in "pos < nb_pkts" manner.

Quoted 47 lines

Quoted 5 lines
The split_flags check in
ixgbe_recv_scattered_pkts_vec() would be a bit more complicated, and therefore maybe would have a tiny performance overhead as well, but I don't it would be anything serious.
I think, if the performance wouldn't be affected, that will be really useful change. So it is definitely worth to try.
Probably even _recv_raw_pkts_vec() for first nb_pkts & ~(RTE_IXGBE_VPMD_RX_BURST - 1), and then sort of scalar analog for the remainder.
Konstantin
Quoted 36 lines
Regards,
Zoltan
Quoted 30 lines · Konstantin
Hi Zoltan,
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, July 24, 2015 4:00 PM
To: Richardson, Bruce; [email protected]
Subject: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Hi,
Quoted 13 lines
I was thinking how to handle the situation when you call rte_eth_rx_burst with less than RTE_IXGBE_VPMD_RX_BURST buffers. In ODP-DPDK unfortunately we can't force this requirement onto the calling application.
One idea I had to check in ixgbe_recv_pkts_vec() if nb_pkts < RTE_IXGBE_VPMD_RX_BURST, and call ixgbe_recv_pkts_bulk_alloc in that case. Accordingly, in ixgbe_recv_scattered_pkts_vec() we could call ixgbe_recv_scattered_pkts() in this case. A branch predictor can easily eliminate the performance penalty of this, and applications can use whatever burst size feasible for them.
The obvious problem could be whether you can mix the receive functions this way. I have a feeling it wouldn't fly, but I wanted to ask first before spending time investigate this option further.
No, it is not possible to mix different RX functions, they setup/use ixgbe_rx_queue fields in a different manner.
Konstantin
Quoted 3 lines
Regards,
Zoltan

Zoltan Kiss

author

2015-07-29 09:40 UTC #5 lore raw

In reply to Ananyev, Konstantin

Hi,

Quoted 23 lines · Konstantin

Hi Zoltan,
Quoted 6 lines
-----Original Message-----
From: Zoltan Kiss [mailto:[email protected]]
Sent: Monday, July 27, 2015 12:38 PM
To: Ananyev, Konstantin; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Quoted 6 lines
Hi Konstantin,
Thanks! Another question I would have: why does _recv_raw_pkts_vec() insist on (nb_pkts > RTE_IXGBE_VPMD_RX_BURST)? Looking at the code it should be able to return packets when nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP.
Yes, that seems pretty trivial modification. Don't know any good reason why it wasn't done that way.
Quoted 4 lines
The split_flags check in
ixgbe_recv_scattered_pkts_vec() would be a bit more complicated, and therefore maybe would have a tiny performance overhead as well, but I don't it would be anything serious.

Quoted 4 lines

I think, if the performance wouldn't be affected, that will be really useful change. So it is definitely worth to try.
Probably even _recv_raw_pkts_vec() for first nb_pkts & ~(RTE_IXGBE_VPMD_RX_BURST - 1), and then sort of scalar analog for the remainder.

Ok, I'll give it a go.
Another question, regarding performance: what setup you used to show a performance difference? I've tried to compare the vector function with the normal bulk alloc with receiving a 10 Gbps stream of 64 bytes UDP packets (and forward them out on the other port), but both yielded ~14 Mpps. I have a i5-4570 CPU @ 3.20GHz, maybe I should limit the clock speed?

Regards,

Zoltan

Quoted 41 lines

Konstantin
Quoted 39 lines
Regards,
Zoltan
Quoted 32 lines · Konstantin
Hi Zoltan,
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, July 24, 2015 4:00 PM
To: Richardson, Bruce; [email protected]
Subject: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Hi,
Quoted 13 lines
I was thinking how to handle the situation when you call rte_eth_rx_burst with less than RTE_IXGBE_VPMD_RX_BURST buffers. In ODP-DPDK unfortunately we can't force this requirement onto the calling application.
One idea I had to check in ixgbe_recv_pkts_vec() if nb_pkts < RTE_IXGBE_VPMD_RX_BURST, and call ixgbe_recv_pkts_bulk_alloc in that case. Accordingly, in ixgbe_recv_scattered_pkts_vec() we could call ixgbe_recv_scattered_pkts() in this case. A branch predictor can easily eliminate the performance penalty of this, and applications can use whatever burst size feasible for them.
The obvious problem could be whether you can mix the receive functions this way. I have a feeling it wouldn't fly, but I wanted to ask first before spending time investigate this option further.
No, it is not possible to mix different RX functions, they setup/use ixgbe_rx_queue fields in a different manner.
Konstantin
Quoted 4 lines
Regards,
Zoltan

Ananyev, Konstantin

2015-07-29 10:03 UTC #6 lore raw

Hi Zoltan,

Quoted 37 lines

-----Original Message-----
From: Zoltan Kiss [mailto:[email protected]]
Sent: Wednesday, July 29, 2015 10:40 AM
To: Ananyev, Konstantin; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Hi,
Quoted 23 lines · Konstantin
Hi Zoltan,
Quoted 6 lines
-----Original Message-----
From: Zoltan Kiss [mailto:[email protected]]
Sent: Monday, July 27, 2015 12:38 PM
To: Ananyev, Konstantin; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Quoted 6 lines
Hi Konstantin,
Thanks! Another question I would have: why does _recv_raw_pkts_vec() insist on (nb_pkts > RTE_IXGBE_VPMD_RX_BURST)? Looking at the code it should be able to return packets when nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP.
Yes, that seems pretty trivial modification. Don't know any good reason why it wasn't done that way.
Quoted 4 lines
The split_flags check in
ixgbe_recv_scattered_pkts_vec() would be a bit more complicated, and therefore maybe would have a tiny performance overhead as well, but I don't it would be anything serious.
Quoted 4 lines
I think, if the performance wouldn't be affected, that will be really useful change. So it is definitely worth to try.
Probably even _recv_raw_pkts_vec() for first nb_pkts & ~(RTE_IXGBE_VPMD_RX_BURST - 1), and then sort of scalar analog for the remainder.

Quoted 6 lines

Ok, I'll give it a go.
Another question, regarding performance: what setup you used to show a performance difference? I've tried to compare the vector function with the normal bulk alloc with receiving a 10 Gbps stream of 64 bytes UDP packets (and forward them out on the other port), but both yielded ~14 Mpps. I have a i5-4570 CPU @ 3.20GHz, maybe I should limit the clock speed?

One port is not enough here if you are running @ 3.2 GHz.
For PMD - I personally use a box with 4x10G ports, one port per pci line, run testpmd io fwd mode, all 4 ports over 1 core @ 2.8 GHz.
If you don't have similar box - let us know when you done with your changes, we can give it a try.
Yes, reducing clock speed to minimal is another good option. Though, I think just 1x10G port wouldn't be enough even for 1.2 GHz. Probably 2x10G would do.
Konstantin

Quoted 46 lines

Regards,
Zoltan
Quoted 41 lines
Konstantin
Quoted 39 lines
Regards,
Zoltan
Quoted 32 lines · Konstantin
Hi Zoltan,
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, July 24, 2015 4:00 PM
To: Richardson, Bruce; [email protected]
Subject: [dpdk-dev] ixgbe vPMD RX functions and buffer number minimum requirement
Hi,
Quoted 13 lines
I was thinking how to handle the situation when you call rte_eth_rx_burst with less than RTE_IXGBE_VPMD_RX_BURST buffers. In ODP-DPDK unfortunately we can't force this requirement onto the calling application.
One idea I had to check in ixgbe_recv_pkts_vec() if nb_pkts < RTE_IXGBE_VPMD_RX_BURST, and call ixgbe_recv_pkts_bulk_alloc in that case. Accordingly, in ixgbe_recv_scattered_pkts_vec() we could call ixgbe_recv_scattered_pkts() in this case. A branch predictor can easily eliminate the performance penalty of this, and applications can use whatever burst size feasible for them.
The obvious problem could be whether you can mix the receive functions this way. I have a feeling it wouldn't fly, but I wanted to ask first before spending time investigate this option further.
No, it is not possible to mix different RX functions, they setup/use ixgbe_rx_queue fields in a different manner.
Konstantin
Quoted 4 lines
Regards,
Zoltan