ixgbe TX function selection
Hi,
I've noticed that ixgbe_set_tx_function() selects the non-SG function even if (dev->data->scattered_rx == 1). That seems a bit dangerous, as you can turn that on inadvertently when you don't set max_rx_pkt_len and buffer size in certain ways. I've learnt it in the hard way, as my segmented packets were leaking memory on the TX path, which doesn't cries if you send out segmented packets. How should this case be treated? Assert on the non-SG TX side for the 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other interfaces still can have SG turned on.
Regards,
Zoltan
Hi Zoltan,
Quoted 9 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe TX function selectionHi,
I've noticed that
ixgbe_set_tx_function()selects the non-SG function even if
Thanks for let us know the problem. But I don't catch your point. Do you really mean TX here? After a quick look at the code, I don’t find the SG/non-SG functions for TX. Do I miss something?
Quoted 5 lines
(
dev->data->scattered_rx== 1). That seems a bit dangerous, as you can turn that on inadvertently when you don't setmax_rx_pkt_lenand buffer size in certain ways. I've learnt it in the hard way, as my segmented packets were leaking memory on the TX path, which doesn't cries if you send out segmented packets.
Which one will cause problem? SG or non-SG packets? And where does the memory leak happen?
Quoted 7 lines
How should this case be treated? Assert on the non-SG TX side for the 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other interfaces still can have SG turned on.
Regards,
Zoltan
Quoted 8 lines · Wenzhuo
Hi Zoltan,
Quoted 6 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe TX function selection
Quoted 4 lines
Quoted 3 lines
Hi,
I've noticed that
ixgbe_set_tx_function()selects the non-SG function even ifThanks for let us know the problem. But I don't catch your point. Do you really mean TX here? After a quick look at the code, I don’t find the SG/non-SG functions for TX. Do I miss something?
The simple code path doesn't handle multisegmented packets. ixgbe_txq_vec_setup() and ixgbe_xmit_pkts_simple() doesn't even check the next pointer of the mbuf, just put the first one on the descriptor ring, and when TX completion happens, the memory is leaked because it just sets ->next to NULL, and calls rte_mempool_put[_bulk] ixgbe_xmit_pkts() puts all the segments on the descriptor ring, therefore when the descriptors are released they are released as well.
This is what these functions supposed to do, but my point is it's very easy to send a multisegmented packet to the simple code path.
Quoted 15 lines
Quoted 6 lines
(
dev->data->scattered_rx== 1). That seems a bit dangerous, as you can turn that on inadvertently when you don't setmax_rx_pkt_lenand buffer size in certain ways. I've learnt it in the hard way, as my segmented packets were leaking memory on the TX path, which doesn't cries if you send out segmented packets.Which one will cause problem? SG or non-SG packets? And where does the memory leak happen?
Quoted 7 lines
How should this case be treated? Assert on the non-SG TX side for the 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other interfaces still can have SG turned on.
Regards,
Zoltan
Hi, Zoltan
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe TX function selectionHi,
Quoted 10 lines
I've noticed that
ixgbe_set_tx_function()selects the non-SG function even if (dev->data->scattered_rx== 1). That seems a bit dangerous, as you can turn that on inadvertently when you don't setmax_rx_pkt_lenand buffer size in certain ways. I've learnt it in the hard way, as my segmented packets were leaking memory on the TX path, which doesn't cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other interfaces still can have SG turned on.
If you look into the ixgbe_set_tx_function, you will find tx function selection is decided by the tx_flags on queue configure, which is passed by rte_eth_txconf. So even you set dev->data->scattered_rx to 1, if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple is still selected as tx function. So, you'd better to set tx_flags=0, and have a try.
Quoted 3 lines
Regards,
Zoltan
Quoted 21 lines · Jingjing
Hi, Zoltan
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe TX function selectionHi,
Quoted 10 lines
I've noticed that
ixgbe_set_tx_function()selects the non-SG function even if (dev->data->scattered_rx== 1). That seems a bit dangerous, as you can turn that on inadvertently when you don't setmax_rx_pkt_lenand buffer size in certain ways. I've learnt it in the hard way, as my segmented packets were leaking memory on the TX path, which doesn't cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other interfaces still can have SG turned on.
Quoted 5 lines
If you look into the
ixgbe_set_tx_function, you will find tx function selection is decided by thetx_flagson queue configure, which is passed byrte_eth_txconf. So even you setdev->data->scattered_rxto 1, if thetx_flagsisETH_TXQ_FLAGS_NOMULTSEGS,ixgbe_xmit_pkts_simpleis still selected as tx function. So, you'd better to set tx_flags=0, and have a try.
You mean getting default_txconf from rte_eth_dev_info_get() and explicitly turn ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with zeros doesn't work very well) That's a way to solve it for me, but I'm rather talking about using defaults which doesn't cause memory leak quite easily.
Quoted 4 lines
Quoted 4 lines
Regards,
Zoltan
Hi Zoltan,
Quoted 36 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, March 18, 2016 1:11 AM
To: Wu, Jingjing; [email protected]
Subject: Re: [dpdk-dev] ixgbe TX function selectionQuoted 21 lines · Jingjing
Hi, Zoltan
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe TX function selectionHi,
Quoted 10 lines
I've noticed that
ixgbe_set_tx_function()selects the non-SG function even if (dev->data->scattered_rx== 1). That seems a bit dangerous, as you can turn that on inadvertently when you don't setmax_rx_pkt_lenand buffer size in certain ways. I've learnt it in the hard way, as my segmented packets were leaking memory on the TX path, which doesn't cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other interfaces still can have SG turned on.Quoted 5 lines
If you look into the
ixgbe_set_tx_function, you will find tx function selection is decided by thetx_flagson queue configure, which is passed byrte_eth_txconf. So even you setdev->data->scattered_rxto
1, if thetx_flagsisETH_TXQ_FLAGS_NOMULTSEGS,ixgbe_xmit_pkts_simpleis still selected as tx function. So, you'd better to set tx_flags=0, and have a try.
Quoted 4 lines
You mean getting
default_txconffromrte_eth_dev_info_get()and explicitly turnETH_TXQ_FLAGS_NOMULTSEGSto 0? (fillingtx_flagswith zeros doesn't work very well) That's a way to solve it for me, but I'm rather talking about using defaults which doesn't cause memory leak quite easily.
Yes, ETH_TXQ_FLAGS_NOMULTSEGS only can be set to 1 when you know all your packets will not be segmented.
I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will be segmented or not. You don't need to set tx_flags to 0, only the ETH_TXQ_FLAGS_NOMULTSEGS bit should be 0, the other bits can be 1 if needed.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Regards,
Zoltan
Quoted 39 lines · Wenzhuo
Hi Zoltan,
Quoted 36 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, March 18, 2016 1:11 AM
To: Wu, Jingjing; [email protected]
Subject: Re: [dpdk-dev] ixgbe TX function selectionQuoted 26 lines · Jingjing
Hi, Zoltan
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe TX function selectionHi,
Quoted 10 lines
I've noticed that
ixgbe_set_tx_function()selects the non-SG function even if (dev->data->scattered_rx== 1). That seems a bit dangerous, as you can turn that on inadvertently when you don't setmax_rx_pkt_lenand buffer size in certain ways. I've learnt it in the hard way, as my segmented packets were leaking memory on the TX path, which doesn't cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other interfaces still can have SG turned on.If you look into the
ixgbe_set_tx_function, you will find tx function selection is decided by thetx_flagson queue configure, which is passed byrte_eth_txconf. So even you setdev->data->scattered_rxto
1, if thetx_flagsisETH_TXQ_FLAGS_NOMULTSEGS,ixgbe_xmit_pkts_simpleis still selected as tx function. So, you'd better to set tx_flags=0, and have a try.
Quoted 7 lines
Quoted 4 lines
You mean getting
default_txconffromrte_eth_dev_info_get()and explicitly turnETH_TXQ_FLAGS_NOMULTSEGSto 0? (fillingtx_flagswith zeros doesn't work very well) That's a way to solve it for me, but I'm rather talking about using defaults which doesn't cause memory leak quite easily.Yes,
ETH_TXQ_FLAGS_NOMULTSEGSonly can be set to 1 when you know all your packets will not be segmented.
I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will be segmented or not. You don't need to settx_flagsto 0, only theETH_TXQ_FLAGS_NOMULTSEGSbit should be 0, the other bits can be 1 if needed.
So can we agree that the default settings should set ETH_TXQ_FLAGS_NOMULTSEGS to 0?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Regards,
Zoltan
Hi Zoltan,
Quoted 52 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, March 18, 2016 1:33 PM
To: Lu, Wenzhuo; Wu, Jingjing; [email protected]
Subject: Re: [dpdk-dev] ixgbe TX function selectionQuoted 43 lines · Wenzhuo
Hi Zoltan,
Quoted 40 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Friday, March 18, 2016 1:11 AM
To: Wu, Jingjing; [email protected]
Subject: Re: [dpdk-dev] ixgbe TX function selectionQuoted 21 lines · Jingjing
Hi, Zoltan
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: [email protected]
Subject: [dpdk-dev] ixgbe TX function selectionHi,
Quoted 10 lines
I've noticed that
ixgbe_set_tx_function()selects the non-SG function even if (dev->data->scattered_rx== 1). That seems a bit dangerous, as you can turn that on inadvertently when you don't setmax_rx_pkt_lenand buffer size in certain ways. I've learnt it in the hard way, as my segmented packets were leaking memory on the TX path, which doesn't cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the 'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other interfaces still can have SG turned on.Quoted 5 lines
If you look into the
ixgbe_set_tx_function, you will find tx function selection is decided by thetx_flagson queue configure, which is passed byrte_eth_txconf. So even you setdev->data->scattered_rxto
1, if thetx_flagsisETH_TXQ_FLAGS_NOMULTSEGS,ixgbe_xmit_pkts_simpleis still selected as tx function. So, you'd better to set tx_flags=0, and have a try.You mean getting
default_txconffromrte_eth_dev_info_get()and explicitly turnETH_TXQ_FLAGS_NOMULTSEGSto 0? (fillingtx_flagswith zeros doesn't work very well) That's a way to solve it for me, but I'm rather talking about using defaults which doesn't cause memory leak quite easily.
Quoted 7 lines
Quoted 2 lines
Yes,
ETH_TXQ_FLAGS_NOMULTSEGSonly can be set to 1 when you know all your packets will not be segmented. I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will besegmented or not.
Quoted 1 line
You don't need to set
tx_flagsto 0, only theETH_TXQ_FLAGS_NOMULTSEGSbit should be 0, the other bits can be 1 if needed.So can we agree that the default settings should set
ETH_TXQ_FLAGS_NOMULTSEGSto 0?
I'd prefer to keep things as they are right now.
There always will be arguments and supporters for both alternatives:
Should the fastest or the most comprehensive path be the default one. Again default txq_flags can vary from one PMD to another.
So, I think the right behaviour for the app would be not to rely on default value but set it up manually to the desired value.
Konstantin
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Regards,
Zoltan