Miscellaneous fixes
Subsystems (2)
cover Miscellaneous fixes
This series of patches includes fixes for issues seen with 6WIND fast path, built on DPDK.
The patches are in 6WIND version of DPDK, and should have been upstreamed a long time ago.
Guo Fengtian (1):
net/ixgbevf: fix stats update after a PF reset
Laurent Hardy (1):
net/i40e: set speed to undefined for default case in link update
Olivier Matz (5):
ethdev: fix description of tx descriptor status
net/e1000: fix Tx descriptor status api (igb)
net/e1000: fix Tx descriptor status api (em)
net/ixgbe: fix Tx descriptor status api
net/i40e: fix Tx descriptor status api
Thibaut Collet (2):
virtio: fix rx stats with vectorized functions
virtio: get all pending rx packets with vectorized functions
Thierry Herbelot (3):
drivers/crypto/openssl: use a local copy for the session contexts drivers/crypto/dpaa_sec: update DPAA iova table in dpaa_mem_vtop drivers/crypto/octeontx: enable unbinding for the OcteonTx crypto
engines
drivers/crypto/dpaa_sec/dpaa_sec.c | 5 +++- drivers/crypto/octeontx/otx_cryptodev.c | 1 + drivers/crypto/openssl/rte_openssl_pmd.c | 34 +++++++++++++++------ drivers/net/e1000/em_rxtx.c | 33 +++++++++++++------- drivers/net/e1000/igb_rxtx.c | 9 +++--- drivers/net/i40e/i40e_ethdev.c | 4 +-- drivers/net/i40e/i40e_ethdev_vf.c | 8 +++-- drivers/net/i40e/i40e_rxtx.c | 37 ++++++++++++++++------- drivers/net/ixgbe/ixgbe_ethdev.c | 6 ++-- drivers/net/ixgbe/ixgbe_rxtx.c | 45 ++++++++++++++++++++-------- drivers/net/ixgbe/ixgbe_rxtx.h | 1 + drivers/net/virtio/virtio_rxtx.c | 2 +- drivers/net/virtio/virtio_rxtx.h | 2 ++ drivers/net/virtio/virtio_rxtx_simple_neon.c | 11 +++++-- drivers/net/virtio/virtio_rxtx_simple_sse.c | 11 +++++-- lib/librte_ethdev/rte_ethdev.h | 4 +-- 16 files changed, 153 insertions(+), 60 deletions(-)
Signature · 1 line
2.11.0
PATCH 1/12 net/ixgbevf: fix stats update after a PF reset
From: Guo Fengtian <[email protected]>
When PF is set down, in VF, the value of stats register is zero. So only increase stats when it's non zero.
--- drivers/net/ixgbe/ixgbe_ethdev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/net/ixgbe/ixgbe_ethdev.c +4 -2
@@ -385,7 +385,8 @@ static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);#define UPDATE_VF_STAT (reg, last, cur) \ { \ uint32_t latest = IXGBE_READ_REG(hw, reg); \ - cur += (latest - last) & UINT_MAX; \ + if (latest) \ + cur += (latest - last) & UINT_MAX; \ last = latest; \ }
@@ -394,7 +395,8 @@ static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);u64 new_lsb =IXGBE_READ_REG (hw, lsb); \u64 new_msb =IXGBE_READ_REG (hw, msb); \u64 latest = ((new_msb <<32 ) | new_lsb); \ - cur += (0x1000000000LL + latest - last) &0xFFFFFFFFFLL ; \ +if (latest) \ + cur += (0x1000000000LL + latest - last) &0xFFFFFFFFFLL ;\ last = latest; \ }
Signature · 1 line
2.11.0
PATCH 2/12 ethdev: fix description of tx descriptor status
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The API comment of rte_eth_tx_descriptor_status() was incorrect. The reference descriptor (when offset = 0) is not where the next packet will be sent, but where the latest packet has been enqueued.
--- lib/librte_ethdev/rte_ethdev.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/librte_ethdev/rte_ethdev.h +2 -2
@@ -4245,8 +4245,8 @@ rte_eth_rx_descriptor_status(uint16_t port_id, uint16_t queue_id,* @param queue_id * A valid Tx queue identifier on this port. * @param offset -* The offset of the descriptor starting from tail (0 is the place where -* the next packet will be send). +* The offset of the descriptor starting from tail (0 is the last written +* descriptor). * * @return * - (RTE_ETH_TX_DESC_FULL) Descriptor is being processed by the hw, i.e.
Signature · 1 line
2.11.0
PATCH 3/12 net/e1000: fix Tx descriptor status api (igb)
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
--- drivers/net/e1000/igb_rxtx.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/net/e1000/igb_rxtx.c +5 -4
@@ -1835,14 +1835,15 @@ eth_igb_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct igb_tx_queue *txq = tx_queue;volatile uint32_t *status; -uint32_t desc; +int32_t desc;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; - desc = txq->tx_tail + offset; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; + desc = txq->sw_ring[desc].last_id; status = &txq->tx_ring[desc].wb.status;if (*status &rte_cpu_to_le_32 (E1000_TXD_STAT_DD ))
Signature · 1 line
2.11.0
PATCH 4/12 net/e1000: fix Tx descriptor status api (em)
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/e1000/em_rxtx.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-)
drivers/net/e1000/em_rxtx.c +23 -10
@@ -152,6 +152,7 @@ struct em_tx_queue {uint64_t tx_ring_phys_addr;/**< TX ring DMA address. */ struct em_tx_entry *sw_ring;/**< virtual address of SW ring. */ volatile uint32_t *tdt_reg_addr;/**< Address of TDT register. */ +volatile uint32_t *tdh_reg_addr;/**< Address of TDH register. */ uint16_t nb_tx_desc;/**< number of TX descriptors. */ uint16_t tx_tail;/**< Current value of TDT register. */ /**< Start freeing TX buffers if there are less free descriptors than
@@ -1304,6 +1305,7 @@ eth_em_tx_queue_setup(struct rte_eth_dev *dev, txq->port_id = dev->data->port_id; txq->tdt_reg_addr =E1000_PCI_REG_ADDR (hw,E1000_TDT (queue_idx)); + txq->tdh_reg_addr =E1000_PCI_REG_ADDR (hw,E1000_TDH (queue_idx)); txq->tx_ring_phys_addr = tz->iova; txq->tx_ring = (struct e1000_data_desc *) tz->addr;
@@ -1557,22 +1559,33 @@ eth_em_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct em_tx_queue *txq = tx_queue;volatile uint8_t *status; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; + +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =e1000_read_addr (txq->tdh_reg_addr); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].upper.fields.status; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].upper.fields.status;if (*status &E1000_TXD_STAT_DD )return RTE_ETH_TX_DESC_DONE ;
Signature · 1 line
2.11.0
PATCH 5/12 net/ixgbe: fix Tx descriptor status api
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/ixgbe/ixgbe_rxtx.c | 45 +++++++++++++++++++++++++++++++----------- drivers/net/ixgbe/ixgbe_rxtx.h | 1 + 2 files changed, 34 insertions(+), 12 deletions(-)
drivers/net/ixgbe/ixgbe_rxtx.c +33 -12
@@ -2627,10 +2627,15 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev *dev, hw->mac.type == ixgbe_mac_X540_vf || hw->mac.type == ixgbe_mac_X550_vf || hw->mac.type == ixgbe_mac_X550EM_x_vf || - hw->mac.type == ixgbe_mac_X550EM_a_vf) + hw->mac.type == ixgbe_mac_X550EM_a_vf) { txq->tdt_reg_addr = IXGBE_PCI_REG_ADDR(hw, IXGBE_VFTDT(queue_idx)); - else + txq->tdh_reg_addr =IXGBE_PCI_REG_ADDR (hw, +IXGBE_VFTDH (queue_idx)); + } else { txq->tdt_reg_addr =IXGBE_PCI_REG_ADDR (hw,IXGBE_TDT (txq->reg_idx)); + txq->tdh_reg_addr =IXGBE_PCI_REG_ADDR (hw, +IXGBE_TDH (txq->reg_idx)); + } txq->tx_ring_phys_addr = tz->iova; txq->tx_ring = (union ixgbe_adv_tx_desc *) tz->addr;
@@ -3163,22 +3168,38 @@ ixgbe_dev_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct ixgbe_tx_queue *txq = tx_queue;volatile uint32_t *status; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; + + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =ixgbe_read_addr (txq->tdh_reg_addr); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].wb.status; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + +/* In full featured mode, RS bit is only set in the last descriptor */ +/* of a multisegments packet */ +if (!((txq->offloads ==0 ) && + (txq->tx_rs_thresh >=RTE_PMD_IXGBE_TX_MAX_BURST ))) + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].wb.status;if (*status &rte_cpu_to_le_32 (IXGBE_ADVTXD_STAT_DD ))return RTE_ETH_TX_DESC_DONE ;
drivers/net/ixgbe/ixgbe_rxtx.h +1 -0
@@ -201,6 +201,7 @@ struct ixgbe_tx_queue {struct ixgbe_tx_entry_v *sw_ring_v;/**< address of SW ring for vector PMD */ };volatile uint32_t *tdt_reg_addr;/**< Address of TDT register. */ +volatile uint32_t *tdh_reg_addr;/**< Address of TDH register. */ uint16_t nb_tx_desc;/**< number of TX descriptors. */ uint16_t tx_tail;/**< current value of TDT reg. */ /**< Start freeing TX buffers if there are less free descriptors than
Signature · 1 line
2.11.0
PATCH 6/12 net/i40e: fix Tx descriptor status api
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/i40e/i40e_rxtx.c | 37 +++++++++++++++++++++++++++---------- 1 file changed, 27 insertions(+), 10 deletions(-)
drivers/net/i40e/i40e_rxtx.c +27 -10
@@ -2031,22 +2031,39 @@ i40e_dev_tx_descriptor_status(void *tx_queue, uint16_t offset)struct i40e_tx_queue *txq = tx_queue;volatile uint64_t *status;uint64_t mask, expect; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; + + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =I40E_READ_REG (I40E_VSI_TO_HW (txq->vsi), +I40E_QTX_HEAD (txq->reg_idx)); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].cmd_type_offset_bsz; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + +/* In full featured mode, RS bit is only set in the last descriptor */ +/* of a multisegments packet */ +if (!((txq->offloads ==0 ) && + (txq->tx_rs_thresh >=RTE_PMD_I40E_TX_MAX_BURST ))) + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].cmd_type_offset_bsz; mask =rte_le_to_cpu_64 (I40E_TXD_QW1_DTYPE_MASK ); expect =rte_cpu_to_le_64 (I40E_TX_DESC_DTYPE_DESC_DONE <<I40E_TXD_QW1_DTYPE_SHIFT );
Signature · 1 line
2.11.0
PATCH 7/12 net/i40e: set speed to undefined for default case in link update
In reply to Thierry HerbelotFrom: Laurent Hardy <[email protected]>
During PF/VF link update, a default speed value of 100M will be set if get_link_info has failed or speed is unknown.
Consequently if PF is put in no-carrier state, VFs will switch to "in carrier" state due to a link up + a link speed set to 100M (default value if no speed detected).
To be consistent with linux drivers on which PF and VFs are in same carrier state, sets default speed to undefined (instead of 100M) and updates a link status of VF only if link is up and speed is different from undefined.
--- drivers/net/i40e/i40e_ethdev.c | 4 ++-- drivers/net/i40e/i40e_ethdev_vf.c | 8 +++++--- 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/net/i40e/i40e_ethdev.c +2 -2
@@ -2743,7 +2743,7 @@ update_link_aq(struct i40e_hw *hw, struct rte_eth_link *link, status =i40e_aq_get_link_info (hw, enable_lse, &link_status,NULL );if (unlikely (status !=I40E_SUCCESS )) { - link->link_speed =ETH_SPEED_NUM_100M ; + link->link_speed =ETH_SPEED_NUM_NONE ; link->link_duplex =ETH_LINK_FULL_DUPLEX ;PMD_DRV_LOG (ERR ,"Failed to get link info" );return ;
@@ -2777,7 +2777,7 @@ update_link_aq(struct i40e_hw *hw, struct rte_eth_link *link, link->link_speed =ETH_SPEED_NUM_40G ;break ;default : - link->link_speed =ETH_SPEED_NUM_100M ; + link->link_speed =ETH_SPEED_NUM_NONE ;break ; } }
drivers/net/i40e/i40e_ethdev_vf.c +5 -3
@@ -2143,13 +2143,15 @@ i40evf_dev_link_update(struct rte_eth_dev *dev, new_link.link_speed =ETH_SPEED_NUM_40G ;break ;default : - new_link.link_speed =ETH_SPEED_NUM_100M ; + new_link.link_speed =ETH_SPEED_NUM_NONE ;break ; }/* full duplex only */ new_link.link_duplex =ETH_LINK_FULL_DUPLEX ; - new_link.link_status = vf->link_up ?ETH_LINK_UP : -ETH_LINK_DOWN ; + new_link.link_status = vf->link_up + && new_link.link_speed !=ETH_SPEED_NUM_NONE + ?ETH_LINK_UP + :ETH_LINK_DOWN ; new_link.link_autoneg = !(dev->data->dev_conf.link_speeds Ð_LINK_SPEED_FIXED );
Signature · 1 line
2.11.0
PATCH 8/12 virtio: fix rx stats with vectorized functions
In reply to Thierry HerbelotFrom: Thibaut Collet <[email protected]>
With vectorized functions, only the rx stats for number of packets is incremented.
Update also the other statistics.
Performance impact is about 2%
--- drivers/net/virtio/virtio_rxtx.c | 2 +- drivers/net/virtio/virtio_rxtx.h | 2 ++ drivers/net/virtio/virtio_rxtx_simple_neon.c | 6 ++++++ drivers/net/virtio/virtio_rxtx_simple_sse.c | 6 ++++++ 4 files changed, 15 insertions(+), 1 deletion(-)
drivers/net/virtio/virtio_rxtx.c +1 -1
@@ -1083,7 +1083,7 @@ virtio_discard_rxbuf_inorder(struct virtqueue *vq, struct rte_mbuf *m) } } -static inline void +void virtio_update_packet_stats (struct virtnet_stats *stats,struct rte_mbuf *mbuf) {uint32_t s = mbuf->pkt_len;
drivers/net/virtio/virtio_rxtx.h +2 -0
@@ -59,5 +59,7 @@ struct virtnet_ctl { };int virtio_rxq_vec_setup (struct virtnet_rx *rxvq); +void virtio_update_packet_stats (struct virtnet_stats *stats, +struct rte_mbuf *mbuf);#endif /* _VIRTIO_RXTX_H_ */
drivers/net/virtio/virtio_rxtx_simple_neon.c +6 -0
@@ -47,6 +47,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct vring_used_elem *rused;struct rte_mbuf **sw_ring;struct rte_mbuf **sw_ring_end; +struct rte_mbuf **ref_rx_pkts;uint16_t nb_pkts_received =0 ;uint8x16_t shuf_msk1 = {
@@ -105,6 +106,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; nb_pkts_received < nb_used;) {uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];
@@ -204,5 +206,9 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, vq->vq_used_cons_idx += nb_pkts_received; vq->vq_free_cnt += nb_pkts_received; rxvq->stats.packets += nb_pkts_received; +for (nb_used =0 ; nb_used < nb_pkts_received; nb_used++) { + rxvq->stats.bytes += ref_rx_pkts[nb_used]->pkt_len; +virtio_update_packet_stats (&rxvq->stats, ref_rx_pkts[nb_used]); + }return nb_pkts_received; }
drivers/net/virtio/virtio_rxtx_simple_sse.c +6 -0
@@ -48,6 +48,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct vring_used_elem *rused;struct rte_mbuf **sw_ring;struct rte_mbuf **sw_ring_end; +struct rte_mbuf **ref_rx_pkts;uint16_t nb_pkts_received =0 ;__m128i shuf_msk1, shuf_msk2, len_adjust;
@@ -107,6 +108,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; nb_pkts_received < nb_used;) {__m128i desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];
@@ -190,5 +192,9 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, vq->vq_used_cons_idx += nb_pkts_received; vq->vq_free_cnt += nb_pkts_received; rxvq->stats.packets += nb_pkts_received; +for (nb_used =0 ; nb_used < nb_pkts_received; nb_used++) { + rxvq->stats.bytes += ref_rx_pkts[nb_used]->pkt_len; +virtio_update_packet_stats (&rxvq->stats, ref_rx_pkts[nb_used]); + }return nb_pkts_received; }
Signature · 1 line
2.11.0
PATCH 9/12 virtio: get all pending rx packets with vectorized functions
In reply to Thierry HerbelotFrom: Thibaut Collet <[email protected]>
The loop to read packets does not take all packets as the number of available packets (nb_used) is decremented in the loop. Take all available packets provides a performance improvement of 3%.
--- drivers/net/virtio/virtio_rxtx_simple_neon.c | 5 +++-- drivers/net/virtio/virtio_rxtx_simple_sse.c | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-)
drivers/net/virtio/virtio_rxtx_simple_neon.c +3 -2
@@ -42,7 +42,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct virtnet_rx *rxvq = rx_queue;struct virtqueue *vq = rxvq->vq;struct virtio_hw *hw = vq->hw; -uint16_t nb_used; +uint16_t nb_used, nb_total;uint16_t desc_idx;struct vring_used_elem *rused;struct rte_mbuf **sw_ring;
@@ -106,9 +106,10 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + nb_total = nb_used; ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; - nb_pkts_received < nb_used;) { + nb_pkts_received < nb_total;) {uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];uint64x2_t mbp[RTE_VIRTIO_DESC_PER_LOOP /2 ];uint64x2_t pkt_mb[RTE_VIRTIO_DESC_PER_LOOP ];
drivers/net/virtio/virtio_rxtx_simple_sse.c +3 -2
@@ -43,7 +43,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct virtnet_rx *rxvq = rx_queue;struct virtqueue *vq = rxvq->vq;struct virtio_hw *hw = vq->hw; -uint16_t nb_used; +uint16_t nb_used, nb_total;uint16_t desc_idx;struct vring_used_elem *rused;struct rte_mbuf **sw_ring;
@@ -108,9 +108,10 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + nb_total = nb_used; ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; - nb_pkts_received < nb_used;) { + nb_pkts_received < nb_total;) {__m128i desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];__m128i mbp[RTE_VIRTIO_DESC_PER_LOOP /2 ];__m128i pkt_mb[RTE_VIRTIO_DESC_PER_LOOP ];
Signature · 1 line
2.11.0
PATCH 10/12 drivers/crypto/openssl: use a local copy for the session contexts
In reply to Thierry HerbelotSession contexts are used for temporary storage when processing a packet.
If packets for the same session are to be processed simultaneously on multiple cores, separate contexts must be used.
Note: with openssl 1.1.1 EVP_CIPHER_CTX can no longer be defined as a variable on the stack: it must be allocated. This in turn reduces the performance.
--- drivers/crypto/openssl/rte_openssl_pmd.c | 34 +++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
drivers/crypto/openssl/rte_openssl_pmd.c +25 -9
@@ -1290,6 +1290,7 @@ process_openssl_combined_opint srclen, aadlen, status =-1 ;uint32_t offset;uint8_t taglen; +EVP_CIPHER_CTX *ctx_copy;/* * Segmented destination buffer is not supported for
@@ -1326,6 +1327,8 @@ process_openssl_combined_op } taglen = sess->auth.digest_length; + ctx_copy =EVP_CIPHER_CTX_new (); +EVP_CIPHER_CTX_copy (ctx_copy, sess->cipher.ctx);if (sess->cipher.direction ==RTE_CRYPTO_CIPHER_OP_ENCRYPT ) { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1333,12 +1336,12 @@ process_openssl_combined_op status =process_openssl_auth_encryption_gcm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx_copy); else status =process_openssl_auth_encryption_ccm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx_copy); }else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1346,14 +1349,15 @@ process_openssl_combined_op status =process_openssl_auth_decryption_gcm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx_copy); else status =process_openssl_auth_decryption_ccm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx_copy); } +EVP_CIPHER_CTX_free (ctx_copy); if (status != 0) { if (status == (-EFAULT) && sess->auth.operation ==
@@ -1372,6 +1376,7 @@ process_openssl_cipher_op {uint8_t *dst, *iv;int srclen, status; +EVP_CIPHER_CTX *ctx_copy;/* * Segmented destination buffer is not supported for
@@ -1388,22 +1393,25 @@ process_openssl_cipher_op iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); + ctx_copy =EVP_CIPHER_CTX_new (); +EVP_CIPHER_CTX_copy (ctx_copy, sess->cipher.ctx);if (sess->cipher.mode ==OPENSSL_CIPHER_LIB )if (sess->cipher.direction ==RTE_CRYPTO_CIPHER_OP_ENCRYPT ) status =process_openssl_cipher_encrypt (mbuf_src, dst, op->sym->cipher.data.offset, iv, - srclen, sess->cipher.ctx); + srclen, ctx_copy);else status =process_openssl_cipher_decrypt (mbuf_src, dst, op->sym->cipher.data.offset, iv, - srclen, sess->cipher.ctx); + srclen, ctx_copy);else status =process_openssl_cipher_des3ctr (mbuf_src, dst, op->sym->cipher.data.offset, iv, sess->cipher.key.data, srclen, - sess->cipher.ctx); + ctx_copy); +EVP_CIPHER_CTX_free (ctx_copy);if (status !=0 ) op->status =RTE_CRYPTO_OP_STATUS_ERROR ; }
@@ -1507,6 +1515,8 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op, {uint8_t *dst;int srclen, status; +EVP_MD_CTX *ctx_a; +HMAC_CTX *ctx_h; srclen = op->sym->auth.data.length;
@@ -1514,14 +1524,20 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,switch (sess->auth.mode) {case OPENSSL_AUTH_AS_AUTH : + ctx_a =EVP_MD_CTX_create (); +EVP_MD_CTX_copy_ex (ctx_a, sess->auth.auth.ctx); status =process_openssl_auth (mbuf_src, dst, op->sym->auth.data.offset,NULL ,NULL , srclen, - sess->auth.auth.ctx, sess->auth.auth.evp_algo); + ctx_a, sess->auth.auth.evp_algo); +EVP_MD_CTX_destroy (ctx_a);break ;case OPENSSL_AUTH_AS_HMAC : + ctx_h =HMAC_CTX_new (); +HMAC_CTX_copy (ctx_h, sess->auth.hmac.ctx); status =process_openssl_auth_hmac (mbuf_src, dst, op->sym->auth.data.offset, srclen, - sess->auth.hmac.ctx); + ctx_h); +HMAC_CTX_free (ctx_h);break ;default : status =-1 ;
Signature · 1 line
2.11.0
PATCH 11/12 drivers/crypto/dpaa_sec: update DPAA iova table in dpaa_mem_vtop
In reply to Thierry Herbelotdpaa_sec needs translations between physical and virtual addresses. V to P translation is relatively fast, as memory is managed in contiguous segments.
The result of each V to P translation is used to update the DPAA iova table, which should be updated by a Mem event callback, but is not. Then the DPAA iova table has entries for all needed memory ranges.
With this patch, dpaa_mem_ptov will always use dpaax_iova_table_get_va, which ensures optimal performance.
--- drivers/crypto/dpaa_sec/dpaa_sec.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/crypto/dpaa_sec/dpaa_sec.c +4 -1
@@ -38,6 +38,7 @@#include <rte_dpaa_bus.h> #include <dpaa_sec.h> #include <dpaa_sec_log.h> +#include <dpaax_iova_table.h> enum rta_sec_era rta_sec_era;
@@ -100,8 +101,10 @@ dpaa_mem_vtop(void *vaddr)const struct rte_memseg *ms; ms =rte_mem_virt2memseg (vaddr,NULL ); -if (ms) +if (ms) { +dpaax_iova_table_update (ms->iova, (void *)ms->addr_64, ms->len);return ms->iova +RTE_PTR_DIFF (vaddr, ms->addr); + }return (size_t )NULL ; }
Signature · 1 line
2.11.0
PATCH 12/12 drivers/crypto/octeontx: enable unbinding for the OcteonTx crypto engines
In reply to Thierry HerbelotLike for Ethernet ports, the OcteonTx crypto engines must first be unbound from their kernel module, then rebound to vfio-pci, before being usable in DPDK.
As this capability is detected at runtime by dpdk-pmdinfo, add the info in the PMD registering directives.
Then an external script can be used for bind and unbind.
--- drivers/crypto/octeontx/otx_cryptodev.c | 1 + 1 file changed, 1 insertion(+)
drivers/crypto/octeontx/otx_cryptodev.c +1 -0
@@ -118,6 +118,7 @@ static struct cryptodev_driver otx_cryptodev_drv;RTE_PMD_REGISTER_PCI (CRYPTODEV_NAME_OCTEONTX_PMD , otx_cryptodev_pmd);RTE_PMD_REGISTER_PCI_TABLE (CRYPTODEV_NAME_OCTEONTX_PMD , pci_id_cpt_table); +RTE_PMD_REGISTER_KMOD_DEP (CRYPTODEV_NAME_OCTEONTX_PMD ,"* igb_uio | uio_pci_generic | vfio-pci" );RTE_PMD_REGISTER_CRYPTO_DRIVER (otx_cryptodev_drv, otx_cryptodev_pmd.driver, otx_cryptodev_driver_id);
Signature · 1 line
2.11.0
v2 cover Miscellaneous fixes
In reply to Thierry HerbelotThis series of patches includes fixes for issues seen with 6WIND fast path, built on DPDK.
The patches are in 6WIND version of DPDK, and should have been upstreamed a long time ago.
V2 changes:
fix checkpatch issues
Guo Fengtian (1):
net/ixgbevf: fix stats update after a PF reset
Laurent Hardy (1):
net/i40e: set speed to undefined for default case in link update
Olivier Matz (5):
ethdev: fix description of tx descriptor status
net/e1000: fix Tx descriptor status api (igb)
net/e1000: fix Tx descriptor status api (em)
net/ixgbe: fix Tx descriptor status api
net/i40e: fix Tx descriptor status api
Thibaut Collet (2):
virtio: fix rx stats with vectorized functions
virtio: get all pending rx packets with vectorized functions
Thierry Herbelot (3):
drivers/crypto/openssl: use a local copy for the session contexts drivers/crypto/dpaa_sec: update DPAA iova table in dpaa_mem_vtop drivers/crypto/octeontx: enable unbinding for the OcteonTx crypto
engines
drivers/crypto/dpaa_sec/dpaa_sec.c | 5 +++- drivers/crypto/octeontx/otx_cryptodev.c | 1 + drivers/crypto/openssl/rte_openssl_pmd.c | 34 +++++++++++++++------ drivers/net/e1000/em_rxtx.c | 33 +++++++++++++------- drivers/net/e1000/igb_rxtx.c | 9 +++--- drivers/net/i40e/i40e_ethdev.c | 4 +-- drivers/net/i40e/i40e_ethdev_vf.c | 8 +++-- drivers/net/i40e/i40e_rxtx.c | 37 ++++++++++++++++------- drivers/net/ixgbe/ixgbe_ethdev.c | 6 ++-- drivers/net/ixgbe/ixgbe_rxtx.c | 45 ++++++++++++++++++++-------- drivers/net/ixgbe/ixgbe_rxtx.h | 1 + drivers/net/virtio/virtio_rxtx.c | 2 +- drivers/net/virtio/virtio_rxtx.h | 2 ++ drivers/net/virtio/virtio_rxtx_simple_neon.c | 11 +++++-- drivers/net/virtio/virtio_rxtx_simple_sse.c | 11 +++++-- lib/librte_ethdev/rte_ethdev.h | 4 +-- 16 files changed, 153 insertions(+), 60 deletions(-)
Signature · 1 line
2.11.0
PATCH v2 1/12 net/ixgbevf: fix stats update after a PF reset
From: Guo Fengtian <[email protected]>
When PF is set down, in VF, the value of stats register is zero. So only increase stats when it's non zero.
--- drivers/net/ixgbe/ixgbe_ethdev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/net/ixgbe/ixgbe_ethdev.c +4 -2
@@ -385,7 +385,8 @@ static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);#define UPDATE_VF_STAT (reg, last, cur) \ { \ uint32_t latest = IXGBE_READ_REG(hw, reg); \ - cur += (latest - last) & UINT_MAX; \ + if (latest) \ + cur += (latest - last) & UINT_MAX; \ last = latest; \ }
@@ -394,7 +395,8 @@ static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);u64 new_lsb =IXGBE_READ_REG (hw, lsb); \u64 new_msb =IXGBE_READ_REG (hw, msb); \u64 latest = ((new_msb <<32 ) | new_lsb); \ - cur += (0x1000000000LL + latest - last) &0xFFFFFFFFFLL ; \ +if (latest) \ + cur += (0x1000000000LL + latest - last) &0xFFFFFFFFFLL ;\ last = latest; \ }
Signature · 1 line
2.11.0
PATCH v2 2/12 ethdev: fix description of tx descriptor status
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The API comment of rte_eth_tx_descriptor_status() was incorrect. The reference descriptor (when offset = 0) is not where the next packet will be sent, but where the latest packet has been enqueued.
--- lib/librte_ethdev/rte_ethdev.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/librte_ethdev/rte_ethdev.h +2 -2
@@ -4245,8 +4245,8 @@ rte_eth_rx_descriptor_status(uint16_t port_id, uint16_t queue_id,* @param queue_id * A valid Tx queue identifier on this port. * @param offset -* The offset of the descriptor starting from tail (0 is the place where -* the next packet will be send). +* The offset of the descriptor starting from tail (0 is the last written +* descriptor). * * @return * - (RTE_ETH_TX_DESC_FULL) Descriptor is being processed by the hw, i.e.
Signature · 1 line
2.11.0
PATCH v2 3/12 net/e1000: fix Tx descriptor status api (igb)
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
--- drivers/net/e1000/igb_rxtx.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/net/e1000/igb_rxtx.c +5 -4
@@ -1835,14 +1835,15 @@ eth_igb_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct igb_tx_queue *txq = tx_queue;volatile uint32_t *status; -uint32_t desc; +int32_t desc;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; - desc = txq->tx_tail + offset; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; + desc = txq->sw_ring[desc].last_id; status = &txq->tx_ring[desc].wb.status;if (*status &rte_cpu_to_le_32 (E1000_TXD_STAT_DD ))
Signature · 1 line
2.11.0
PATCH v2 4/12 net/e1000: fix Tx descriptor status api (em)
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/e1000/em_rxtx.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-)
drivers/net/e1000/em_rxtx.c +23 -10
@@ -152,6 +152,7 @@ struct em_tx_queue {uint64_t tx_ring_phys_addr;/**< TX ring DMA address. */ struct em_tx_entry *sw_ring;/**< virtual address of SW ring. */ volatile uint32_t *tdt_reg_addr;/**< Address of TDT register. */ +volatile uint32_t *tdh_reg_addr;/**< Address of TDH register. */ uint16_t nb_tx_desc;/**< number of TX descriptors. */ uint16_t tx_tail;/**< Current value of TDT register. */ /**< Start freeing TX buffers if there are less free descriptors than
@@ -1304,6 +1305,7 @@ eth_em_tx_queue_setup(struct rte_eth_dev *dev, txq->port_id = dev->data->port_id; txq->tdt_reg_addr =E1000_PCI_REG_ADDR (hw,E1000_TDT (queue_idx)); + txq->tdh_reg_addr =E1000_PCI_REG_ADDR (hw,E1000_TDH (queue_idx)); txq->tx_ring_phys_addr = tz->iova; txq->tx_ring = (struct e1000_data_desc *) tz->addr;
@@ -1557,22 +1559,33 @@ eth_em_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct em_tx_queue *txq = tx_queue;volatile uint8_t *status; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; + +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =e1000_read_addr (txq->tdh_reg_addr); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].upper.fields.status; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].upper.fields.status;if (*status &E1000_TXD_STAT_DD )return RTE_ETH_TX_DESC_DONE ;
Signature · 1 line
2.11.0
PATCH v2 5/12 net/ixgbe: fix Tx descriptor status api
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/ixgbe/ixgbe_rxtx.c | 45 +++++++++++++++++++++++++++++++----------- drivers/net/ixgbe/ixgbe_rxtx.h | 1 + 2 files changed, 34 insertions(+), 12 deletions(-)
drivers/net/ixgbe/ixgbe_rxtx.c +33 -12
@@ -2627,10 +2627,15 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev *dev, hw->mac.type == ixgbe_mac_X540_vf || hw->mac.type == ixgbe_mac_X550_vf || hw->mac.type == ixgbe_mac_X550EM_x_vf || - hw->mac.type == ixgbe_mac_X550EM_a_vf) + hw->mac.type == ixgbe_mac_X550EM_a_vf) { txq->tdt_reg_addr = IXGBE_PCI_REG_ADDR(hw, IXGBE_VFTDT(queue_idx)); - else + txq->tdh_reg_addr =IXGBE_PCI_REG_ADDR (hw, +IXGBE_VFTDH (queue_idx)); + } else { txq->tdt_reg_addr =IXGBE_PCI_REG_ADDR (hw,IXGBE_TDT (txq->reg_idx)); + txq->tdh_reg_addr =IXGBE_PCI_REG_ADDR (hw, +IXGBE_TDH (txq->reg_idx)); + } txq->tx_ring_phys_addr = tz->iova; txq->tx_ring = (union ixgbe_adv_tx_desc *) tz->addr;
@@ -3163,22 +3168,38 @@ ixgbe_dev_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct ixgbe_tx_queue *txq = tx_queue;volatile uint32_t *status; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; + + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =ixgbe_read_addr (txq->tdh_reg_addr); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].wb.status; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + +/* In full featured mode, RS bit is only set in the last descriptor */ +/* of a multisegments packet */ +if (!(txq->offloads ==0 && + txq->tx_rs_thresh >=RTE_PMD_IXGBE_TX_MAX_BURST )) + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].wb.status;if (*status &rte_cpu_to_le_32 (IXGBE_ADVTXD_STAT_DD ))return RTE_ETH_TX_DESC_DONE ;
drivers/net/ixgbe/ixgbe_rxtx.h +1 -0
@@ -201,6 +201,7 @@ struct ixgbe_tx_queue {struct ixgbe_tx_entry_v *sw_ring_v;/**< address of SW ring for vector PMD */ };volatile uint32_t *tdt_reg_addr;/**< Address of TDT register. */ +volatile uint32_t *tdh_reg_addr;/**< Address of TDH register. */ uint16_t nb_tx_desc;/**< number of TX descriptors. */ uint16_t tx_tail;/**< current value of TDT reg. */ /**< Start freeing TX buffers if there are less free descriptors than
Signature · 1 line
2.11.0
PATCH v2 6/12 net/i40e: fix Tx descriptor status api
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/i40e/i40e_rxtx.c | 37 +++++++++++++++++++++++++++---------- 1 file changed, 27 insertions(+), 10 deletions(-)
drivers/net/i40e/i40e_rxtx.c +27 -10
@@ -2031,22 +2031,39 @@ i40e_dev_tx_descriptor_status(void *tx_queue, uint16_t offset)struct i40e_tx_queue *txq = tx_queue;volatile uint64_t *status;uint64_t mask, expect; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; + + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =I40E_READ_REG (I40E_VSI_TO_HW (txq->vsi), +I40E_QTX_HEAD (txq->reg_idx)); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].cmd_type_offset_bsz; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + +/* In full featured mode, RS bit is only set in the last descriptor */ +/* of a multisegments packet */ +if (!(txq->offloads ==0 && + txq->tx_rs_thresh >=RTE_PMD_I40E_TX_MAX_BURST )) + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].cmd_type_offset_bsz; mask =rte_le_to_cpu_64 (I40E_TXD_QW1_DTYPE_MASK ); expect =rte_cpu_to_le_64 (I40E_TX_DESC_DTYPE_DESC_DONE <<I40E_TXD_QW1_DTYPE_SHIFT );
Signature · 1 line
2.11.0
PATCH v2 7/12 net/i40e: set speed to undefined for default case in link update
In reply to Thierry HerbelotFrom: Laurent Hardy <[email protected]>
During PF/VF link update, a default speed value of 100M will be set if get_link_info has failed or speed is unknown.
Consequently if PF is put in no-carrier state, VFs will switch to "in carrier" state due to a link up + a link speed set to 100M (default value if no speed detected).
To be consistent with linux drivers on which PF and VFs are in same carrier state, sets default speed to undefined (instead of 100M) and updates a link status of VF only if link is up and speed is different from undefined.
--- drivers/net/i40e/i40e_ethdev.c | 4 ++-- drivers/net/i40e/i40e_ethdev_vf.c | 8 +++++--- 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/net/i40e/i40e_ethdev.c +2 -2
@@ -2743,7 +2743,7 @@ update_link_aq(struct i40e_hw *hw, struct rte_eth_link *link, status =i40e_aq_get_link_info (hw, enable_lse, &link_status,NULL );if (unlikely (status !=I40E_SUCCESS )) { - link->link_speed =ETH_SPEED_NUM_100M ; + link->link_speed =ETH_SPEED_NUM_NONE ; link->link_duplex =ETH_LINK_FULL_DUPLEX ;PMD_DRV_LOG (ERR ,"Failed to get link info" );return ;
@@ -2777,7 +2777,7 @@ update_link_aq(struct i40e_hw *hw, struct rte_eth_link *link, link->link_speed =ETH_SPEED_NUM_40G ;break ;default : - link->link_speed =ETH_SPEED_NUM_100M ; + link->link_speed =ETH_SPEED_NUM_NONE ;break ; } }
drivers/net/i40e/i40e_ethdev_vf.c +5 -3
@@ -2143,13 +2143,15 @@ i40evf_dev_link_update(struct rte_eth_dev *dev, new_link.link_speed =ETH_SPEED_NUM_40G ;break ;default : - new_link.link_speed =ETH_SPEED_NUM_100M ; + new_link.link_speed =ETH_SPEED_NUM_NONE ;break ; }/* full duplex only */ new_link.link_duplex =ETH_LINK_FULL_DUPLEX ; - new_link.link_status = vf->link_up ?ETH_LINK_UP : -ETH_LINK_DOWN ; + new_link.link_status = vf->link_up && + new_link.link_speed !=ETH_SPEED_NUM_NONE + ?ETH_LINK_UP + :ETH_LINK_DOWN ; new_link.link_autoneg = !(dev->data->dev_conf.link_speeds Ð_LINK_SPEED_FIXED );
Signature · 1 line
2.11.0
PATCH v2 8/12 virtio: fix rx stats with vectorized functions
In reply to Thierry HerbelotFrom: Thibaut Collet <[email protected]>
With vectorized functions, only the rx stats for number of packets is incremented.
Update also the other statistics.
Performance impact is about 2%
--- drivers/net/virtio/virtio_rxtx.c | 2 +- drivers/net/virtio/virtio_rxtx.h | 2 ++ drivers/net/virtio/virtio_rxtx_simple_neon.c | 6 ++++++ drivers/net/virtio/virtio_rxtx_simple_sse.c | 6 ++++++ 4 files changed, 15 insertions(+), 1 deletion(-)
drivers/net/virtio/virtio_rxtx.c +1 -1
@@ -1083,7 +1083,7 @@ virtio_discard_rxbuf_inorder(struct virtqueue *vq, struct rte_mbuf *m) } } -static inline void +void virtio_update_packet_stats (struct virtnet_stats *stats,struct rte_mbuf *mbuf) {uint32_t s = mbuf->pkt_len;
drivers/net/virtio/virtio_rxtx.h +2 -0
@@ -59,5 +59,7 @@ struct virtnet_ctl { };int virtio_rxq_vec_setup (struct virtnet_rx *rxvq); +void virtio_update_packet_stats (struct virtnet_stats *stats, +struct rte_mbuf *mbuf);#endif /* _VIRTIO_RXTX_H_ */
drivers/net/virtio/virtio_rxtx_simple_neon.c +6 -0
@@ -47,6 +47,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct vring_used_elem *rused;struct rte_mbuf **sw_ring;struct rte_mbuf **sw_ring_end; +struct rte_mbuf **ref_rx_pkts;uint16_t nb_pkts_received =0 ;uint8x16_t shuf_msk1 = {
@@ -105,6 +106,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; nb_pkts_received < nb_used;) {uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];
@@ -204,5 +206,9 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, vq->vq_used_cons_idx += nb_pkts_received; vq->vq_free_cnt += nb_pkts_received; rxvq->stats.packets += nb_pkts_received; +for (nb_used =0 ; nb_used < nb_pkts_received; nb_used++) { + rxvq->stats.bytes += ref_rx_pkts[nb_used]->pkt_len; +virtio_update_packet_stats (&rxvq->stats, ref_rx_pkts[nb_used]); + }return nb_pkts_received; }
drivers/net/virtio/virtio_rxtx_simple_sse.c +6 -0
@@ -48,6 +48,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct vring_used_elem *rused;struct rte_mbuf **sw_ring;struct rte_mbuf **sw_ring_end; +struct rte_mbuf **ref_rx_pkts;uint16_t nb_pkts_received =0 ;__m128i shuf_msk1, shuf_msk2, len_adjust;
@@ -107,6 +108,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; nb_pkts_received < nb_used;) {__m128i desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];
@@ -190,5 +192,9 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, vq->vq_used_cons_idx += nb_pkts_received; vq->vq_free_cnt += nb_pkts_received; rxvq->stats.packets += nb_pkts_received; +for (nb_used =0 ; nb_used < nb_pkts_received; nb_used++) { + rxvq->stats.bytes += ref_rx_pkts[nb_used]->pkt_len; +virtio_update_packet_stats (&rxvq->stats, ref_rx_pkts[nb_used]); + }return nb_pkts_received; }
Signature · 1 line
2.11.0
PATCH v2 9/12 virtio: get all pending rx packets with vectorized functions
In reply to Thierry HerbelotFrom: Thibaut Collet <[email protected]>
The loop to read packets does not take all packets as the number of available packets (nb_used) is decremented in the loop. Take all available packets provides a performance improvement of 3%.
--- drivers/net/virtio/virtio_rxtx_simple_neon.c | 5 +++-- drivers/net/virtio/virtio_rxtx_simple_sse.c | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-)
drivers/net/virtio/virtio_rxtx_simple_neon.c +3 -2
@@ -42,7 +42,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct virtnet_rx *rxvq = rx_queue;struct virtqueue *vq = rxvq->vq;struct virtio_hw *hw = vq->hw; -uint16_t nb_used; +uint16_t nb_used, nb_total;uint16_t desc_idx;struct vring_used_elem *rused;struct rte_mbuf **sw_ring;
@@ -106,9 +106,10 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + nb_total = nb_used; ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; - nb_pkts_received < nb_used;) { + nb_pkts_received < nb_total;) {uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];uint64x2_t mbp[RTE_VIRTIO_DESC_PER_LOOP /2 ];uint64x2_t pkt_mb[RTE_VIRTIO_DESC_PER_LOOP ];
drivers/net/virtio/virtio_rxtx_simple_sse.c +3 -2
@@ -43,7 +43,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct virtnet_rx *rxvq = rx_queue;struct virtqueue *vq = rxvq->vq;struct virtio_hw *hw = vq->hw; -uint16_t nb_used; +uint16_t nb_used, nb_total;uint16_t desc_idx;struct vring_used_elem *rused;struct rte_mbuf **sw_ring;
@@ -108,9 +108,10 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + nb_total = nb_used; ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; - nb_pkts_received < nb_used;) { + nb_pkts_received < nb_total;) {__m128i desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];__m128i mbp[RTE_VIRTIO_DESC_PER_LOOP /2 ];__m128i pkt_mb[RTE_VIRTIO_DESC_PER_LOOP ];
Signature · 1 line
2.11.0
PATCH v2 10/12 drivers/crypto/openssl: use a local copy for the session contexts
In reply to Thierry HerbelotSession contexts are used for temporary storage when processing a packet.
If packets for the same session are to be processed simultaneously on multiple cores, separate contexts must be used.
Note: with openssl 1.1.1 EVP_CIPHER_CTX can no longer be defined as a variable on the stack: it must be allocated. This in turn reduces the performance.
--- drivers/crypto/openssl/rte_openssl_pmd.c | 34 +++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
drivers/crypto/openssl/rte_openssl_pmd.c +25 -9
@@ -1290,6 +1290,7 @@ process_openssl_combined_opint srclen, aadlen, status =-1 ;uint32_t offset;uint8_t taglen; +EVP_CIPHER_CTX *ctx_copy;/* * Segmented destination buffer is not supported for
@@ -1326,6 +1327,8 @@ process_openssl_combined_op } taglen = sess->auth.digest_length; + ctx_copy =EVP_CIPHER_CTX_new (); +EVP_CIPHER_CTX_copy (ctx_copy, sess->cipher.ctx);if (sess->cipher.direction ==RTE_CRYPTO_CIPHER_OP_ENCRYPT ) { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1333,12 +1336,12 @@ process_openssl_combined_op status =process_openssl_auth_encryption_gcm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx_copy); else status =process_openssl_auth_encryption_ccm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx_copy); }else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1346,14 +1349,15 @@ process_openssl_combined_op status =process_openssl_auth_decryption_gcm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx_copy); else status =process_openssl_auth_decryption_ccm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx_copy); } +EVP_CIPHER_CTX_free (ctx_copy); if (status != 0) { if (status == (-EFAULT) && sess->auth.operation ==
@@ -1372,6 +1376,7 @@ process_openssl_cipher_op {uint8_t *dst, *iv;int srclen, status; +EVP_CIPHER_CTX *ctx_copy;/* * Segmented destination buffer is not supported for
@@ -1388,22 +1393,25 @@ process_openssl_cipher_op iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); + ctx_copy =EVP_CIPHER_CTX_new (); +EVP_CIPHER_CTX_copy (ctx_copy, sess->cipher.ctx);if (sess->cipher.mode ==OPENSSL_CIPHER_LIB )if (sess->cipher.direction ==RTE_CRYPTO_CIPHER_OP_ENCRYPT ) status =process_openssl_cipher_encrypt (mbuf_src, dst, op->sym->cipher.data.offset, iv, - srclen, sess->cipher.ctx); + srclen, ctx_copy);else status =process_openssl_cipher_decrypt (mbuf_src, dst, op->sym->cipher.data.offset, iv, - srclen, sess->cipher.ctx); + srclen, ctx_copy);else status =process_openssl_cipher_des3ctr (mbuf_src, dst, op->sym->cipher.data.offset, iv, sess->cipher.key.data, srclen, - sess->cipher.ctx); + ctx_copy); +EVP_CIPHER_CTX_free (ctx_copy);if (status !=0 ) op->status =RTE_CRYPTO_OP_STATUS_ERROR ; }
@@ -1507,6 +1515,8 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op, {uint8_t *dst;int srclen, status; +EVP_MD_CTX *ctx_a; +HMAC_CTX *ctx_h; srclen = op->sym->auth.data.length;
@@ -1514,14 +1524,20 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,switch (sess->auth.mode) {case OPENSSL_AUTH_AS_AUTH : + ctx_a =EVP_MD_CTX_create (); +EVP_MD_CTX_copy_ex (ctx_a, sess->auth.auth.ctx); status =process_openssl_auth (mbuf_src, dst, op->sym->auth.data.offset,NULL ,NULL , srclen, - sess->auth.auth.ctx, sess->auth.auth.evp_algo); + ctx_a, sess->auth.auth.evp_algo); +EVP_MD_CTX_destroy (ctx_a);break ;case OPENSSL_AUTH_AS_HMAC : + ctx_h =HMAC_CTX_new (); +HMAC_CTX_copy (ctx_h, sess->auth.hmac.ctx); status =process_openssl_auth_hmac (mbuf_src, dst, op->sym->auth.data.offset, srclen, - sess->auth.hmac.ctx); + ctx_h); +HMAC_CTX_free (ctx_h);break ;default : status =-1 ;
Signature · 1 line
2.11.0
PATCH v2 11/12 drivers/crypto/dpaa_sec: update DPAA iova table in dpaa_mem_vtop
In reply to Thierry Herbelotdpaa_sec needs translations between physical and virtual addresses. V to P translation is relatively fast, as memory is managed in contiguous segments.
The result of each V to P translation is used to update the DPAA iova table, which should be updated by a Mem event callback, but is not. Then the DPAA iova table has entries for all needed memory ranges.
With this patch, dpaa_mem_ptov will always use dpaax_iova_table_get_va, which ensures optimal performance.
--- drivers/crypto/dpaa_sec/dpaa_sec.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/crypto/dpaa_sec/dpaa_sec.c +4 -1
@@ -38,6 +38,7 @@#include <rte_dpaa_bus.h> #include <dpaa_sec.h> #include <dpaa_sec_log.h> +#include <dpaax_iova_table.h> enum rta_sec_era rta_sec_era;
@@ -100,8 +101,10 @@ dpaa_mem_vtop(void *vaddr)const struct rte_memseg *ms; ms =rte_mem_virt2memseg (vaddr,NULL ); -if (ms) +if (ms) { +dpaax_iova_table_update (ms->iova, (void *)ms->addr_64, ms->len);return ms->iova +RTE_PTR_DIFF (vaddr, ms->addr); + }return (size_t )NULL ; }
Signature · 1 line
2.11.0
PATCH v2 12/12 drivers/crypto/octeontx: enable unbinding for the OcteonTx crypto engines
In reply to Thierry HerbelotLike for Ethernet ports, the OcteonTx crypto engines must first be unbound from their kernel module, then rebound to vfio-pci, before being usable in DPDK.
As this capability is detected at runtime by dpdk-pmdinfo, add the info in the PMD registering directives.
Then an external script can be used for bind and unbind.
--- drivers/crypto/octeontx/otx_cryptodev.c | 1 + 1 file changed, 1 insertion(+)
drivers/crypto/octeontx/otx_cryptodev.c +1 -0
@@ -118,6 +118,7 @@ static struct cryptodev_driver otx_cryptodev_drv;RTE_PMD_REGISTER_PCI (CRYPTODEV_NAME_OCTEONTX_PMD , otx_cryptodev_pmd);RTE_PMD_REGISTER_PCI_TABLE (CRYPTODEV_NAME_OCTEONTX_PMD , pci_id_cpt_table); +RTE_PMD_REGISTER_KMOD_DEP (CRYPTODEV_NAME_OCTEONTX_PMD ,"* igb_uio | uio_pci_generic | vfio-pci" );RTE_PMD_REGISTER_CRYPTO_DRIVER (otx_cryptodev_drv, otx_cryptodev_pmd.driver, otx_cryptodev_driver_id);
Signature · 1 line
2.11.0
Re: [dpdk-dev] [dpdk-stable] [PATCH 19.11 V2 08/12] virtio: fix rx stats with vectorized functions
In reply to Thierry HerbelotQuoted 6 lines · Thierry Herbelot
From: Thibaut Collet <[email protected]>
With vectorized functions, only the rx stats for number of packets is incremented.
Update also the other statistics.
Performance impact is about 2%
Could you share some details about your performance test?
Quoted 21 lines
drivers/net/virtio/virtio_rxtx_simple_sse.c +2
-0@@ -48,6 +48,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct vring_used_elem *rused;struct rte_mbuf **sw_ring;struct rte_mbuf **sw_ring_end; +struct rte_mbuf **ref_rx_pkts;uint16_t nb_pkts_received =0 ;__m128i shuf_msk1, shuf_msk2, len_adjust;@@ -107,6 +108,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; nb_pkts_received < nb_used;) {__m128i desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];
drivers/net/virtio/virtio_rxtx_simple_sse.c virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, · line 197
Quoted 6 lines
drivers/net/virtio/virtio_rxtx_simple_sse.c +3
-0@@ -190,5 +192,9 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, vq->vq_used_cons_idx += nb_pkts_received; vq->vq_free_cnt += nb_pkts_received; rxvq->stats.packets += nb_pkts_received; +for (nb_used =0 ; nb_used < nb_pkts_received; nb_used++) { + rxvq->stats.bytes += ref_rx_pkts[nb_used]->pkt_len; +virtio_update_packet_stats (&rxvq->stats, ref_rx_pkts[nb_used]);
The stats.bytes was updated twice by above code.
Quoted 6 lines
drivers/net/virtio/virtio_rxtx_simple_sse.c +1
-0+ }return nb_pkts_received; }Signature · 1 line
2.11.0
Re: [dpdk-dev] [dpdk-stable] [PATCH 19.11 V2 08/12] virtio: fix rx stats with vectorized functions
Quoted 10 lines · Tiwei Bie
Quoted 6 lines · Thierry Herbelot
From: Thibaut Collet <[email protected]>
With vectorized functions, only the rx stats for number of packets is incremented.
Update also the other statistics.
Performance impact is about 2%Could you share some details about your performance test?
The test has been done with a 6wind application based on dpdk and I have not keep details. With test pmd impact is maybe a little bit higher.
Quoted 18 lines
Quoted 18 lines
drivers/net/virtio/virtio_rxtx_simple_sse.c +1
-0@@ -48,6 +48,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct vring_used_elem *rused;struct rte_mbuf **sw_ring;struct rte_mbuf **sw_ring_end; +struct rte_mbuf **ref_rx_pkts;uint16_t nb_pkts_received =0 ;__m128i shuf_msk1, shuf_msk2, len_adjust;@@ -107,6 +108,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); }
Quoted 13 lines
drivers/net/virtio/virtio_rxtx_simple_sse.c virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, · line 197
Quoted 11 lines
drivers/net/virtio/virtio_rxtx_simple_sse.c +4
-0+ ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; nb_pkts_received < nb_used;) {__m128i desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];@@ -190,5 +192,9 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, vq->vq_used_cons_idx += nb_pkts_received; vq->vq_free_cnt += nb_pkts_received; rxvq->stats.packets += nb_pkts_received; +for (nb_used =0 ; nb_used < nb_pkts_received; nb_used++) { + rxvq->stats.bytes += ref_rx_pkts[nb_used]->pkt_len; +virtio_update_packet_stats (&rxvq->stats, ref_rx_pkts[nb_used]);The stats.bytes was updated twice by above code.
Agree. My correction is an old one, done on the old 18.11 dpdk, and so done before patch
https://git.dpdk.org/dpdk/commit/drivers/net/virtio/virtio_rxtx.c?id=81e5cdf19e583c742040d3be83b8cc79b451e243.
So the line
+ rxvq->stats.bytes += ref_rx_pkts[nb_used]->pkt_len;
must be removed to be compliant with Ilya Maximets patch.
Quoted 7 lines
Quoted 7 lines
drivers/net/virtio/virtio_rxtx_simple_sse.c +1
-1+ }return nb_pkts_received; } --2.11.0
v3 cover Miscellaneous fixes
In reply to Thierry HerbelotThis series of patches includes fixes for issues seen with 6WIND fast path, built on DPDK.
The patches are in 6WIND version of DPDK, and should have been upstreamed a long time ago.
V3 changes:
fix double increment of stats in virtio vectorized functions
V2 changes:
fix checkpatch issues
Guo Fengtian (1):
net/ixgbevf: fix stats update after a PF reset
Laurent Hardy (1):
net/i40e: set speed to undefined for default case in link update
Olivier Matz (5):
ethdev: fix description of tx descriptor status
net/e1000: fix Tx descriptor status api (igb)
net/e1000: fix Tx descriptor status api (em)
net/ixgbe: fix Tx descriptor status api
net/i40e: fix Tx descriptor status api
Thibaut Collet (2):
virtio: fix rx stats with vectorized functions
virtio: get all pending rx packets with vectorized functions
Thierry Herbelot (3):
drivers/crypto/openssl: use a local copy for the session contexts drivers/crypto/dpaa_sec: update DPAA iova table in dpaa_mem_vtop drivers/crypto/octeontx: enable unbinding for the OcteonTx crypto
engines
drivers/crypto/dpaa_sec/dpaa_sec.c | 5 +++- drivers/crypto/octeontx/otx_cryptodev.c | 1 + drivers/crypto/openssl/rte_openssl_pmd.c | 34 +++++++++++++++------ drivers/net/e1000/em_rxtx.c | 33 +++++++++++++------- drivers/net/e1000/igb_rxtx.c | 9 +++--- drivers/net/i40e/i40e_ethdev.c | 4 +-- drivers/net/i40e/i40e_ethdev_vf.c | 8 +++-- drivers/net/i40e/i40e_rxtx.c | 37 ++++++++++++++++------- drivers/net/ixgbe/ixgbe_ethdev.c | 6 ++-- drivers/net/ixgbe/ixgbe_rxtx.c | 45 ++++++++++++++++++++-------- drivers/net/ixgbe/ixgbe_rxtx.h | 1 + drivers/net/virtio/virtio_rxtx.c | 2 +- drivers/net/virtio/virtio_rxtx.h | 2 ++ drivers/net/virtio/virtio_rxtx_simple_neon.c | 10 +++++-- drivers/net/virtio/virtio_rxtx_simple_sse.c | 10 +++++-- lib/librte_ethdev/rte_ethdev.h | 4 +-- 16 files changed, 151 insertions(+), 60 deletions(-)
Signature · 1 line
2.11.0
PATCH v3 1/12 net/ixgbevf: fix stats update after a PF reset
From: Guo Fengtian <[email protected]>
When PF is set down, in VF, the value of stats register is zero. So only increase stats when it's non zero.
--- drivers/net/ixgbe/ixgbe_ethdev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/net/ixgbe/ixgbe_ethdev.c +4 -2
@@ -385,7 +385,8 @@ static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);#define UPDATE_VF_STAT (reg, last, cur) \ { \ uint32_t latest = IXGBE_READ_REG(hw, reg); \ - cur += (latest - last) & UINT_MAX; \ + if (latest) \ + cur += (latest - last) & UINT_MAX; \ last = latest; \ }
@@ -394,7 +395,8 @@ static void ixgbe_l2_tunnel_conf(struct rte_eth_dev *dev);u64 new_lsb =IXGBE_READ_REG (hw, lsb); \u64 new_msb =IXGBE_READ_REG (hw, msb); \u64 latest = ((new_msb <<32 ) | new_lsb); \ - cur += (0x1000000000LL + latest - last) &0xFFFFFFFFFLL ; \ +if (latest) \ + cur += (0x1000000000LL + latest - last) &0xFFFFFFFFFLL ;\ last = latest; \ }
Signature · 1 line
2.11.0
PATCH v3 2/12 ethdev: fix description of tx descriptor status
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The API comment of rte_eth_tx_descriptor_status() was incorrect. The reference descriptor (when offset = 0) is not where the next packet will be sent, but where the latest packet has been enqueued.
--- lib/librte_ethdev/rte_ethdev.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/librte_ethdev/rte_ethdev.h +2 -2
@@ -4245,8 +4245,8 @@ rte_eth_rx_descriptor_status(uint16_t port_id, uint16_t queue_id,* @param queue_id * A valid Tx queue identifier on this port. * @param offset -* The offset of the descriptor starting from tail (0 is the place where -* the next packet will be send). +* The offset of the descriptor starting from tail (0 is the last written +* descriptor). * * @return * - (RTE_ETH_TX_DESC_FULL) Descriptor is being processed by the hw, i.e.
Signature · 1 line
2.11.0
PATCH v3 3/12 net/e1000: fix Tx descriptor status api (igb)
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
--- drivers/net/e1000/igb_rxtx.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/net/e1000/igb_rxtx.c +5 -4
@@ -1835,14 +1835,15 @@ eth_igb_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct igb_tx_queue *txq = tx_queue;volatile uint32_t *status; -uint32_t desc; +int32_t desc;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; - desc = txq->tx_tail + offset; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; + desc = txq->sw_ring[desc].last_id; status = &txq->tx_ring[desc].wb.status;if (*status &rte_cpu_to_le_32 (E1000_TXD_STAT_DD ))
Signature · 1 line
2.11.0
PATCH v3 4/12 net/e1000: fix Tx descriptor status api (em)
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/e1000/em_rxtx.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-)
drivers/net/e1000/em_rxtx.c +23 -10
@@ -152,6 +152,7 @@ struct em_tx_queue {uint64_t tx_ring_phys_addr;/**< TX ring DMA address. */ struct em_tx_entry *sw_ring;/**< virtual address of SW ring. */ volatile uint32_t *tdt_reg_addr;/**< Address of TDT register. */ +volatile uint32_t *tdh_reg_addr;/**< Address of TDH register. */ uint16_t nb_tx_desc;/**< number of TX descriptors. */ uint16_t tx_tail;/**< Current value of TDT register. */ /**< Start freeing TX buffers if there are less free descriptors than
@@ -1304,6 +1305,7 @@ eth_em_tx_queue_setup(struct rte_eth_dev *dev, txq->port_id = dev->data->port_id; txq->tdt_reg_addr =E1000_PCI_REG_ADDR (hw,E1000_TDT (queue_idx)); + txq->tdh_reg_addr =E1000_PCI_REG_ADDR (hw,E1000_TDH (queue_idx)); txq->tx_ring_phys_addr = tz->iova; txq->tx_ring = (struct e1000_data_desc *) tz->addr;
@@ -1557,22 +1559,33 @@ eth_em_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct em_tx_queue *txq = tx_queue;volatile uint8_t *status; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; + +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =e1000_read_addr (txq->tdh_reg_addr); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].upper.fields.status; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].upper.fields.status;if (*status &E1000_TXD_STAT_DD )return RTE_ETH_TX_DESC_DONE ;
Signature · 1 line
2.11.0
PATCH v3 5/12 net/ixgbe: fix Tx descriptor status api
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/ixgbe/ixgbe_rxtx.c | 45 +++++++++++++++++++++++++++++++----------- drivers/net/ixgbe/ixgbe_rxtx.h | 1 + 2 files changed, 34 insertions(+), 12 deletions(-)
drivers/net/ixgbe/ixgbe_rxtx.c +33 -12
@@ -2627,10 +2627,15 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev *dev, hw->mac.type == ixgbe_mac_X540_vf || hw->mac.type == ixgbe_mac_X550_vf || hw->mac.type == ixgbe_mac_X550EM_x_vf || - hw->mac.type == ixgbe_mac_X550EM_a_vf) + hw->mac.type == ixgbe_mac_X550EM_a_vf) { txq->tdt_reg_addr = IXGBE_PCI_REG_ADDR(hw, IXGBE_VFTDT(queue_idx)); - else + txq->tdh_reg_addr =IXGBE_PCI_REG_ADDR (hw, +IXGBE_VFTDH (queue_idx)); + } else { txq->tdt_reg_addr =IXGBE_PCI_REG_ADDR (hw,IXGBE_TDT (txq->reg_idx)); + txq->tdh_reg_addr =IXGBE_PCI_REG_ADDR (hw, +IXGBE_TDH (txq->reg_idx)); + } txq->tx_ring_phys_addr = tz->iova; txq->tx_ring = (union ixgbe_adv_tx_desc *) tz->addr;
@@ -3163,22 +3168,38 @@ ixgbe_dev_tx_descriptor_status(void *tx_queue, uint16_t offset) {struct ixgbe_tx_queue *txq = tx_queue;volatile uint32_t *status; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; + + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =ixgbe_read_addr (txq->tdh_reg_addr); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].wb.status; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + +/* In full featured mode, RS bit is only set in the last descriptor */ +/* of a multisegments packet */ +if (!(txq->offloads ==0 && + txq->tx_rs_thresh >=RTE_PMD_IXGBE_TX_MAX_BURST )) + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].wb.status;if (*status &rte_cpu_to_le_32 (IXGBE_ADVTXD_STAT_DD ))return RTE_ETH_TX_DESC_DONE ;
drivers/net/ixgbe/ixgbe_rxtx.h +1 -0
@@ -201,6 +201,7 @@ struct ixgbe_tx_queue {struct ixgbe_tx_entry_v *sw_ring_v;/**< address of SW ring for vector PMD */ };volatile uint32_t *tdt_reg_addr;/**< Address of TDT register. */ +volatile uint32_t *tdh_reg_addr;/**< Address of TDH register. */ uint16_t nb_tx_desc;/**< number of TX descriptors. */ uint16_t tx_tail;/**< current value of TDT reg. */ /**< Start freeing TX buffers if there are less free descriptors than
Signature · 1 line
2.11.0
PATCH v3 6/12 net/i40e: fix Tx descriptor status api
In reply to Thierry HerbelotFrom: Olivier Matz <[email protected]>
The Tx descriptor status api was not behaving as expected. This API is used to inspect the content of the descriptors in the Tx ring to determine the length of the Tx queue.
Since the software advances the tail pointer and the hardware advances the head pointer, the Tx queue is located before txq->tx_tail in the ring. Therefore, a call to rte_eth_tx_descriptor_status(..., offset=20) should inspect the 20th descriptor before the tail, not after.
As before, we still need to take care about only checking descriptors that have the RS bit.
Additionally, we can avoid an access to the ring if offset is greater or equal to nb_tx_desc - nb_tx_free.
--- drivers/net/i40e/i40e_rxtx.c | 37 +++++++++++++++++++++++++++---------- 1 file changed, 27 insertions(+), 10 deletions(-)
drivers/net/i40e/i40e_rxtx.c +27 -10
@@ -2031,22 +2031,39 @@ i40e_dev_tx_descriptor_status(void *tx_queue, uint16_t offset)struct i40e_tx_queue *txq = tx_queue;volatile uint64_t *status;uint64_t mask, expect; -uint32_t desc; +int32_t desc, dd;if (unlikely (offset >= txq->nb_tx_desc))return -EINVAL ; +if (offset >= txq->nb_tx_desc - txq->nb_tx_free) +return RTE_ETH_TX_DESC_DONE ; + + desc = txq->tx_tail - offset -1 ; +if (desc <0 ) + desc += txq->nb_tx_desc; - desc = txq->tx_tail + offset; -/* go to next desc that has the RS bit */ - desc = ((desc + txq->tx_rs_thresh -1 ) / txq->tx_rs_thresh) * - txq->tx_rs_thresh; -if (desc >= txq->nb_tx_desc) { - desc -= txq->nb_tx_desc; -if (desc >= txq->nb_tx_desc) - desc -= txq->nb_tx_desc; +/* offset is too small, no other way than reading PCI reg */ +if (unlikely (offset < txq->tx_rs_thresh)) { +int16_t tx_head, queue_size; + tx_head =I40E_READ_REG (I40E_VSI_TO_HW (txq->vsi), +I40E_QTX_HEAD (txq->reg_idx)); + queue_size = txq->tx_tail - tx_head; +if (queue_size <0 ) + queue_size += txq->nb_tx_desc; +return queue_size > offset ?RTE_ETH_TX_DESC_FULL : +RTE_ETH_TX_DESC_DONE ; } - status = &txq->tx_ring[desc].cmd_type_offset_bsz; +/* index of the dd bit to look at */ + dd = (desc / txq->tx_rs_thresh +1 ) * txq->tx_rs_thresh -1 ; + +/* In full featured mode, RS bit is only set in the last descriptor */ +/* of a multisegments packet */ +if (!(txq->offloads ==0 && + txq->tx_rs_thresh >=RTE_PMD_I40E_TX_MAX_BURST )) + dd = txq->sw_ring[dd].last_id; + + status = &txq->tx_ring[dd].cmd_type_offset_bsz; mask =rte_le_to_cpu_64 (I40E_TXD_QW1_DTYPE_MASK ); expect =rte_cpu_to_le_64 (I40E_TX_DESC_DTYPE_DESC_DONE <<I40E_TXD_QW1_DTYPE_SHIFT );
Signature · 1 line
2.11.0
PATCH v3 7/12 net/i40e: set speed to undefined for default case in link update
In reply to Thierry HerbelotFrom: Laurent Hardy <[email protected]>
During PF/VF link update, a default speed value of 100M will be set if get_link_info has failed or speed is unknown.
Consequently if PF is put in no-carrier state, VFs will switch to "in carrier" state due to a link up + a link speed set to 100M (default value if no speed detected).
To be consistent with linux drivers on which PF and VFs are in same carrier state, sets default speed to undefined (instead of 100M) and updates a link status of VF only if link is up and speed is different from undefined.
--- drivers/net/i40e/i40e_ethdev.c | 4 ++-- drivers/net/i40e/i40e_ethdev_vf.c | 8 +++++--- 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/net/i40e/i40e_ethdev.c +2 -2
@@ -2743,7 +2743,7 @@ update_link_aq(struct i40e_hw *hw, struct rte_eth_link *link, status =i40e_aq_get_link_info (hw, enable_lse, &link_status,NULL );if (unlikely (status !=I40E_SUCCESS )) { - link->link_speed =ETH_SPEED_NUM_100M ; + link->link_speed =ETH_SPEED_NUM_NONE ; link->link_duplex =ETH_LINK_FULL_DUPLEX ;PMD_DRV_LOG (ERR ,"Failed to get link info" );return ;
@@ -2777,7 +2777,7 @@ update_link_aq(struct i40e_hw *hw, struct rte_eth_link *link, link->link_speed =ETH_SPEED_NUM_40G ;break ;default : - link->link_speed =ETH_SPEED_NUM_100M ; + link->link_speed =ETH_SPEED_NUM_NONE ;break ; } }
drivers/net/i40e/i40e_ethdev_vf.c +5 -3
@@ -2143,13 +2143,15 @@ i40evf_dev_link_update(struct rte_eth_dev *dev, new_link.link_speed =ETH_SPEED_NUM_40G ;break ;default : - new_link.link_speed =ETH_SPEED_NUM_100M ; + new_link.link_speed =ETH_SPEED_NUM_NONE ;break ; }/* full duplex only */ new_link.link_duplex =ETH_LINK_FULL_DUPLEX ; - new_link.link_status = vf->link_up ?ETH_LINK_UP : -ETH_LINK_DOWN ; + new_link.link_status = vf->link_up && + new_link.link_speed !=ETH_SPEED_NUM_NONE + ?ETH_LINK_UP + :ETH_LINK_DOWN ; new_link.link_autoneg = !(dev->data->dev_conf.link_speeds Ð_LINK_SPEED_FIXED );
Signature · 1 line
2.11.0
PATCH v3 8/12 virtio: fix rx stats with vectorized functions
In reply to Thierry HerbelotFrom: Thibaut Collet <[email protected]>
With vectorized functions, only the rx stats for number of packets is incremented.
Update also the other statistics.
Performance impact is about 2%
--- drivers/net/virtio/virtio_rxtx.c | 2 +- drivers/net/virtio/virtio_rxtx.h | 2 ++ drivers/net/virtio/virtio_rxtx_simple_neon.c | 5 +++++ drivers/net/virtio/virtio_rxtx_simple_sse.c | 5 +++++ 4 files changed, 13 insertions(+), 1 deletion(-)
drivers/net/virtio/virtio_rxtx.c +1 -1
@@ -1083,7 +1083,7 @@ virtio_discard_rxbuf_inorder(struct virtqueue *vq, struct rte_mbuf *m) } } -static inline void +void virtio_update_packet_stats (struct virtnet_stats *stats,struct rte_mbuf *mbuf) {uint32_t s = mbuf->pkt_len;
drivers/net/virtio/virtio_rxtx.h +2 -0
@@ -59,5 +59,7 @@ struct virtnet_ctl { };int virtio_rxq_vec_setup (struct virtnet_rx *rxvq); +void virtio_update_packet_stats (struct virtnet_stats *stats, +struct rte_mbuf *mbuf);#endif /* _VIRTIO_RXTX_H_ */
drivers/net/virtio/virtio_rxtx_simple_neon.c +5 -0
@@ -47,6 +47,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct vring_used_elem *rused;struct rte_mbuf **sw_ring;struct rte_mbuf **sw_ring_end; +struct rte_mbuf **ref_rx_pkts;uint16_t nb_pkts_received =0 ;uint8x16_t shuf_msk1 = {
@@ -105,6 +106,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; nb_pkts_received < nb_used;) {uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];
@@ -204,5 +206,8 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, vq->vq_used_cons_idx += nb_pkts_received; vq->vq_free_cnt += nb_pkts_received; rxvq->stats.packets += nb_pkts_received; +for (nb_used =0 ; nb_used < nb_pkts_received; nb_used++) +virtio_update_packet_stats (&rxvq->stats, ref_rx_pkts[nb_used]); +return nb_pkts_received; }
drivers/net/virtio/virtio_rxtx_simple_sse.c +5 -0
@@ -48,6 +48,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct vring_used_elem *rused;struct rte_mbuf **sw_ring;struct rte_mbuf **sw_ring_end; +struct rte_mbuf **ref_rx_pkts;uint16_t nb_pkts_received =0 ;__m128i shuf_msk1, shuf_msk2, len_adjust;
@@ -107,6 +108,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; nb_pkts_received < nb_used;) {__m128i desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];
@@ -190,5 +192,8 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, vq->vq_used_cons_idx += nb_pkts_received; vq->vq_free_cnt += nb_pkts_received; rxvq->stats.packets += nb_pkts_received; +for (nb_used =0 ; nb_used < nb_pkts_received; nb_used++) +virtio_update_packet_stats (&rxvq->stats, ref_rx_pkts[nb_used]); +return nb_pkts_received; }
Signature · 1 line
2.11.0
PATCH v3 9/12 virtio: get all pending rx packets with vectorized functions
In reply to Thierry HerbelotFrom: Thibaut Collet <[email protected]>
The loop to read packets does not take all packets as the number of available packets (nb_used) is decremented in the loop. Take all available packets provides a performance improvement of 3%.
--- drivers/net/virtio/virtio_rxtx_simple_neon.c | 5 +++-- drivers/net/virtio/virtio_rxtx_simple_sse.c | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-)
drivers/net/virtio/virtio_rxtx_simple_neon.c +3 -2
@@ -42,7 +42,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct virtnet_rx *rxvq = rx_queue;struct virtqueue *vq = rxvq->vq;struct virtio_hw *hw = vq->hw; -uint16_t nb_used; +uint16_t nb_used, nb_total;uint16_t desc_idx;struct vring_used_elem *rused;struct rte_mbuf **sw_ring;
@@ -106,9 +106,10 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + nb_total = nb_used; ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; - nb_pkts_received < nb_used;) { + nb_pkts_received < nb_total;) {uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];uint64x2_t mbp[RTE_VIRTIO_DESC_PER_LOOP /2 ];uint64x2_t pkt_mb[RTE_VIRTIO_DESC_PER_LOOP ];
drivers/net/virtio/virtio_rxtx_simple_sse.c +3 -2
@@ -43,7 +43,7 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,struct virtnet_rx *rxvq = rx_queue;struct virtqueue *vq = rxvq->vq;struct virtio_hw *hw = vq->hw; -uint16_t nb_used; +uint16_t nb_used, nb_total;uint16_t desc_idx;struct vring_used_elem *rused;struct rte_mbuf **sw_ring;
@@ -108,9 +108,10 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,virtqueue_notify (vq); } + nb_total = nb_used; ref_rx_pkts = rx_pkts;for (nb_pkts_received =0 ; - nb_pkts_received < nb_used;) { + nb_pkts_received < nb_total;) {__m128i desc[RTE_VIRTIO_DESC_PER_LOOP /2 ];__m128i mbp[RTE_VIRTIO_DESC_PER_LOOP /2 ];__m128i pkt_mb[RTE_VIRTIO_DESC_PER_LOOP ];
Signature · 1 line
2.11.0
PATCH v3 10/12 drivers/crypto/openssl: use a local copy for the session contexts
In reply to Thierry HerbelotSession contexts are used for temporary storage when processing a packet.
If packets for the same session are to be processed simultaneously on multiple cores, separate contexts must be used.
Note: with openssl 1.1.1 EVP_CIPHER_CTX can no longer be defined as a variable on the stack: it must be allocated. This in turn reduces the performance.
--- drivers/crypto/openssl/rte_openssl_pmd.c | 34 +++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
drivers/crypto/openssl/rte_openssl_pmd.c +25 -9
@@ -1290,6 +1290,7 @@ process_openssl_combined_opint srclen, aadlen, status =-1 ;uint32_t offset;uint8_t taglen; +EVP_CIPHER_CTX *ctx_copy;/* * Segmented destination buffer is not supported for
@@ -1326,6 +1327,8 @@ process_openssl_combined_op } taglen = sess->auth.digest_length; + ctx_copy =EVP_CIPHER_CTX_new (); +EVP_CIPHER_CTX_copy (ctx_copy, sess->cipher.ctx);if (sess->cipher.direction ==RTE_CRYPTO_CIPHER_OP_ENCRYPT ) { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1333,12 +1336,12 @@ process_openssl_combined_op status =process_openssl_auth_encryption_gcm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx_copy); else status =process_openssl_auth_encryption_ccm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx_copy); }else { if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC ||
@@ -1346,14 +1349,15 @@ process_openssl_combined_op status =process_openssl_auth_decryption_gcm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, sess->cipher.ctx); + dst, tag, ctx_copy); else status =process_openssl_auth_decryption_ccm ( mbuf_src, offset, srclen, aad, aadlen, iv, - dst, tag, taglen, sess->cipher.ctx); + dst, tag, taglen, ctx_copy); } +EVP_CIPHER_CTX_free (ctx_copy); if (status != 0) { if (status == (-EFAULT) && sess->auth.operation ==
@@ -1372,6 +1376,7 @@ process_openssl_cipher_op {uint8_t *dst, *iv;int srclen, status; +EVP_CIPHER_CTX *ctx_copy;/* * Segmented destination buffer is not supported for
@@ -1388,22 +1393,25 @@ process_openssl_cipher_op iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); + ctx_copy =EVP_CIPHER_CTX_new (); +EVP_CIPHER_CTX_copy (ctx_copy, sess->cipher.ctx);if (sess->cipher.mode ==OPENSSL_CIPHER_LIB )if (sess->cipher.direction ==RTE_CRYPTO_CIPHER_OP_ENCRYPT ) status =process_openssl_cipher_encrypt (mbuf_src, dst, op->sym->cipher.data.offset, iv, - srclen, sess->cipher.ctx); + srclen, ctx_copy);else status =process_openssl_cipher_decrypt (mbuf_src, dst, op->sym->cipher.data.offset, iv, - srclen, sess->cipher.ctx); + srclen, ctx_copy);else status =process_openssl_cipher_des3ctr (mbuf_src, dst, op->sym->cipher.data.offset, iv, sess->cipher.key.data, srclen, - sess->cipher.ctx); + ctx_copy); +EVP_CIPHER_CTX_free (ctx_copy);if (status !=0 ) op->status =RTE_CRYPTO_OP_STATUS_ERROR ; }
@@ -1507,6 +1515,8 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op, {uint8_t *dst;int srclen, status; +EVP_MD_CTX *ctx_a; +HMAC_CTX *ctx_h; srclen = op->sym->auth.data.length;
@@ -1514,14 +1524,20 @@ process_openssl_auth_op(struct openssl_qp *qp, struct rte_crypto_op *op,switch (sess->auth.mode) {case OPENSSL_AUTH_AS_AUTH : + ctx_a =EVP_MD_CTX_create (); +EVP_MD_CTX_copy_ex (ctx_a, sess->auth.auth.ctx); status =process_openssl_auth (mbuf_src, dst, op->sym->auth.data.offset,NULL ,NULL , srclen, - sess->auth.auth.ctx, sess->auth.auth.evp_algo); + ctx_a, sess->auth.auth.evp_algo); +EVP_MD_CTX_destroy (ctx_a);break ;case OPENSSL_AUTH_AS_HMAC : + ctx_h =HMAC_CTX_new (); +HMAC_CTX_copy (ctx_h, sess->auth.hmac.ctx); status =process_openssl_auth_hmac (mbuf_src, dst, op->sym->auth.data.offset, srclen, - sess->auth.hmac.ctx); + ctx_h); +HMAC_CTX_free (ctx_h);break ;default : status =-1 ;
Signature · 1 line
2.11.0
PATCH v3 11/12 drivers/crypto/dpaa_sec: update DPAA iova table in dpaa_mem_vtop
In reply to Thierry Herbelotdpaa_sec needs translations between physical and virtual addresses. V to P translation is relatively fast, as memory is managed in contiguous segments.
The result of each V to P translation is used to update the DPAA iova table, which should be updated by a Mem event callback, but is not. Then the DPAA iova table has entries for all needed memory ranges.
With this patch, dpaa_mem_ptov will always use dpaax_iova_table_get_va, which ensures optimal performance.
--- drivers/crypto/dpaa_sec/dpaa_sec.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/crypto/dpaa_sec/dpaa_sec.c +4 -1
@@ -38,6 +38,7 @@#include <rte_dpaa_bus.h> #include <dpaa_sec.h> #include <dpaa_sec_log.h> +#include <dpaax_iova_table.h> enum rta_sec_era rta_sec_era;
@@ -100,8 +101,10 @@ dpaa_mem_vtop(void *vaddr)const struct rte_memseg *ms; ms =rte_mem_virt2memseg (vaddr,NULL ); -if (ms) +if (ms) { +dpaax_iova_table_update (ms->iova, (void *)ms->addr_64, ms->len);return ms->iova +RTE_PTR_DIFF (vaddr, ms->addr); + }return (size_t )NULL ; }
Signature · 1 line
2.11.0
PATCH v3 12/12 drivers/crypto/octeontx: enable unbinding for the OcteonTx crypto engines
In reply to Thierry HerbelotLike for Ethernet ports, the OcteonTx crypto engines must first be unbound from their kernel module, then rebound to vfio-pci, before being usable in DPDK.
As this capability is detected at runtime by dpdk-pmdinfo, add the info in the PMD registering directives.
Then an external script can be used for bind and unbind.
--- drivers/crypto/octeontx/otx_cryptodev.c | 1 + 1 file changed, 1 insertion(+)
drivers/crypto/octeontx/otx_cryptodev.c +1 -0
@@ -118,6 +118,7 @@ static struct cryptodev_driver otx_cryptodev_drv;RTE_PMD_REGISTER_PCI (CRYPTODEV_NAME_OCTEONTX_PMD , otx_cryptodev_pmd);RTE_PMD_REGISTER_PCI_TABLE (CRYPTODEV_NAME_OCTEONTX_PMD , pci_id_cpt_table); +RTE_PMD_REGISTER_KMOD_DEP (CRYPTODEV_NAME_OCTEONTX_PMD ,"* igb_uio | uio_pci_generic | vfio-pci" );RTE_PMD_REGISTER_CRYPTO_DRIVER (otx_cryptodev_drv, otx_cryptodev_pmd.driver, otx_cryptodev_driver_id);
Signature · 1 line
2.11.0
Re: [dpdk-dev] [PATCH 19.11 V3 02/12] ethdev: fix description of tx descriptor status
In reply to Thierry HerbelotQuoted 19 lines · Thierry Herbelot
From: Olivier Matz <[email protected]>
The API comment of
rte_eth_tx_descriptor_status()was incorrect. The reference descriptor (when offset = 0) is not where the next packet will be sent, but where the latest packet has been enqueued.Fixes 52f5cdd2e897 ("ethdev: add descriptor status API")Signed-off-by Olivier Matz <[email protected]>--- lib/librte_ethdev/rte_ethdev.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/librte_ethdev/rte_ethdev.h +0
-0
lib/librte_ethdev/rte_ethdev.h rte_eth_rx_descriptor_status(uint16_t port_id, uint16_t queue_id, · line 4252
Quoted 10 lines
lib/librte_ethdev/rte_ethdev.h +2
-2@@ -4245,8 +4245,8 @@ rte_eth_rx_descriptor_status(uint16_t port_id, uint16_t queue_id,* @param queue_id * A valid Tx queue identifier on this port. * @param offset -* The offset of the descriptor starting from tail (0 is the place where -* the next packet will be send). +* The offset of the descriptor starting from tail (0 is the last written +* descriptor). * * @return * - (RTE_ETH_TX_DESC_FULL) Descriptor is being processed by the hw, i.e.
The patch lacks explanation why it was wrong before and why it is correct after the patch.
I think there is a space here for description improvements since it is still confusing. I'd suggest to remove 'tail' term from the description or write down full description to explain what/where is head and tail.
Rx case:
V-->offset
=========***********----------------00000000000==
UNAVAIL DONE AVAIL UNUSED
i.e. V points to the next packet to be received by the driver, i.e. the first * (the descriptor is done by HW and packet is ready to be processed). DONE(*) are descriptors filled in by HW, but not processed by the driver yet. AVAIL(-) are available for HW to fill in with received packets. UNUSED(0) (right now it is UNAVAIL as well) are descriptors which can be done AVAIL on Rx queue refill
UNAVAIL(=) are reserved Rx ring space which cannot be used
The key question for Rx is the length of DONE. Is CPU fast enough to handle incoming traffic?
Tx case (where is tail V and why):
=========************--------------00000000000==
UNAVAIL DONE FULL UNUSED
DONE(*) are descriptors processed by HW and should be processed by the driver to free associated mbuf (rte_eth_tx_done_cleanup())
FULL(-) are available for HW to send these packets out, but not processed yet. UNUSED(0) is available space in Tx ring to make it FULL
UNAVAIL(=) is reserved Tx ring space which cannot be used
I'd say that it should be similar to Rx here, but it looks different in accordance with current description. Above patch fixes it from the first '-' (or the first '0'?) to be the last '-'.
What is the main question for Tx status descriptor user?
Quoted 4 lines · Thierry Herbelot
This series of patches includes fixes for issues seen with 6WIND fast path, built on DPDK.
The patches are in 6WIND version of DPDK, and should have been upstreamed a long time ago.
In fact, I will re-submit another series of patches, as some are not yet ready.
Please disregard the current series.
Thierry
Quoted 49 lines
V3 changes:
fix double increment of stats in virtio vectorized functionsV2 changes:
fix checkpatch issuesGuo Fengtian (1):
net/ixgbevf: fix stats update after a PF resetLaurent Hardy (1):
net/i40e: set speed to undefined for default case in link updateOlivier Matz (5):
ethdev: fix description of tx descriptor status net/e1000: fix Tx descriptor status api (igb) net/e1000: fix Tx descriptor status api (em) net/ixgbe: fix Tx descriptor status api net/i40e: fix Tx descriptor status apiThibaut Collet (2):
virtio: fix rx stats with vectorized functions
virtio: get all pending rx packets with vectorized functionsThierry Herbelot (3):
drivers/crypto/openssl: use a local copy for the session contextsdrivers/crypto/dpaa_sec: update DPAA iova table indpaa_mem_vtopdrivers/crypto/octeontx: enable unbinding for the OcteonTx crypto
enginesdrivers/crypto/dpaa_sec/dpaa_sec.c | 5 +++- drivers/crypto/octeontx/otx_cryptodev.c | 1 + drivers/crypto/openssl/rte_openssl_pmd.c | 34 +++++++++++++++------ drivers/net/e1000/em_rxtx.c | 33 +++++++++++++------- drivers/net/e1000/igb_rxtx.c | 9 +++--- drivers/net/i40e/i40e_ethdev.c | 4 +-- drivers/net/i40e/i40e_ethdev_vf.c | 8 +++-- drivers/net/i40e/i40e_rxtx.c | 37 ++++++++++++++++------- drivers/net/ixgbe/ixgbe_ethdev.c | 6 ++-- drivers/net/ixgbe/ixgbe_rxtx.c | 45 ++++++++++++++++++++-------- drivers/net/ixgbe/ixgbe_rxtx.h | 1 + drivers/net/virtio/virtio_rxtx.c | 2 +- drivers/net/virtio/virtio_rxtx.h | 2 ++ drivers/net/virtio/virtio_rxtx_simple_neon.c | 10 +++++-- drivers/net/virtio/virtio_rxtx_simple_sse.c | 10 +++++-- lib/librte_ethdev/rte_ethdev.h | 4 +-- 16 files changed, 151 insertions(+), 60 deletions(-)
Signature · 2 lines
Thierry Herbelot 6WIND
08/08/2019 15:19, Thierry Herbelot:
Quoted 8 lines · Thierry Herbelot
Quoted 4 lines
This series of patches includes fixes for issues seen with 6WIND fast path, built on DPDK.
The patches are in 6WIND version of DPDK, and should have been upstreamed a long time ago.In fact, I will re-submit another series of patches, as some are not yet ready.
No hurry to send 3 versions of this series while we are still closing 19.08.
Quoted 1 line
Please disregard the current series.
Quoted 8 lines · Thomas Monjalon
08/08/2019 15:19, Thierry Herbelot:
Quoted 6 lines · Thierry Herbelot
Quoted 4 lines
This series of patches includes fixes for issues seen with 6WIND fast path, built on DPDK.
The patches are in 6WIND version of DPDK, and should have been upstreamed a long time ago.
Quoted 4 lines
Quoted 2 lines
In fact, I will re-submit another series of patches, as some are not yet ready.
No hurry to send 3 versions of this series while we are still closing 19.08.
I can see some patches went through the virtio subtree, is this series still valid for 19.11?
Please update these patches status in patchwork.
Thanks.
--
David Marchand
Quoted 9 lines · David Marchand
Quoted 8 lines · Thomas Monjalon
08/08/2019 15:19, Thierry Herbelot:
Quoted 6 lines · Thierry Herbelot
Quoted 4 lines
This series of patches includes fixes for issues seen with 6WIND fast path, built on DPDK.
The patches are in 6WIND version of DPDK, and should have been upstreamed a long time ago.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
In fact, I will re-submit another series of patches, as some are not yet ready.
No hurry to send 3 versions of this series while we are still closing 19.08.
I can see some patches went through the virtio subtree, is this series still valid for 19.11?
Please update these patches status in patchwork.
done: all have been set as superseded.
Th
Quoted 6 lines
Thanks.
--
David Marchand
Signature · 3 lines
Thierry Herbelot 6WIND Senior Software Engineer