Skip to contentSkip to conversation

introduce new iavf driver on vfio-user client

PATCH Superseded Jingjing Wu 4 participants 18 messages Dec 19, 2020 – Jan 14, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover introduce new iavf driver on vfio-user client

This series introduces a new net virtual device called iavf_client which is based on vfio-user client interface. Through vfio-user client interface, PCI liked device could be used similar as Intel® Ethernet Adaptive Virtual Function specification.
The code to enable iavf_client mainly contains two parts:
- Emulated pci interfaces for vfio-user client which is located in common/iavf/vfio_user/.
- A new net driver base on vdev abstraction, i.e. iavf_client_ethdev.c

      ----------------------
      | ------------------ |
      | |  iavf driver   | |----> (iavf_client_ethdev.c)
      | ------------------ |
      | ------------------ |
      | | device emulate | |------>(common/iavf/vfio_user/)
      | ------------------ |
      ----------------------
                |
                |  
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------

This driver can be used to test the device emulation framework mentioned in:
[RFC 0/2] Add device emulation support in DPDK
http://patchwork.dpdk.org/cover/75549/

                    +------------------------------------------------------+
                    |   +---------------+      +---------------+           |
                    |   | iavf_emudev   |      | iavfbe_ethdev |           |
                    |   |    driver     |      |     driver    |           |
                    |   +---------------+      +---------------+           |
                    |           |                       |                  |
                    | ------------------------------------------- VDEV BUS |
                    |           |                       |                  |
                    |   +---------------+       +--------------+           |
+--------------+    |   | vdev:         |       | vdev:        |           |
| +----------+ |    |   | /path/to/vfio |       |iavf_emudev_# |           |
| |   iavf   | |    |   +---------------+       +--------------+           |
| |  client  |      |           |                                          |
| +----------+ |    |           |                                          |
| +----------+ |    |      +----------+                                    |
| | vfio-user| |    |      | vfio-user|                                    |
| | client   | |<---|----->| server   |                                    |
| +----------+ |    |      +----------+                                    |
| QEMU/DPDK    |    | DPDK                                                 |
+--------------+    +------------------------------------------------------+

It can be launched together with patch series for the testing:
[0/9] Introduce vfio-user library:
http://patchwork.dpdk.org/cover/85389/
[0/8]Introduce emudev library and iavf emudev driver
http://patchwork.dpdk.org/cover/85488/

This series depends on patch serieses:
[0/9] Introduce vfio-user library:
http://patchwork.dpdk.org/cover/85389/

Jingjing Wu (2):
common/iavf: emulated pci interfaces on vfio-user client net/iavf: introduce iavf driver on vfio-user client

 config/rte_config.h                           |   3 +
 drivers/common/iavf/iavf_common.c             |   7 +
 drivers/common/iavf/iavf_impl.c               |  45 +-
 drivers/common/iavf/iavf_osdep.h              |  14 +
 drivers/common/iavf/iavf_prototype.h          |   1 +
 drivers/common/iavf/iavf_type.h               |   3 +
 drivers/common/iavf/meson.build               |  11 +-
 drivers/common/iavf/version.map               |   5 +
 drivers/common/iavf/vfio_user/vfio_user_pci.c | 517 ++++++++++++++++++
 drivers/common/iavf/vfio_user/vfio_user_pci.h |  66 +++
 drivers/net/iavf/iavf.h                       |  18 +-
 drivers/net/iavf/iavf_client_ethdev.c         | 298 ++++++++++
 drivers/net/iavf/iavf_ethdev.c                |  26 +-
 drivers/net/iavf/iavf_rxtx.c                  |  23 +-
 drivers/net/iavf/meson.build                  |   1 +
 15 files changed, 1015 insertions(+), 23 deletions(-)
 create mode 100644 drivers/common/iavf/vfio_user/vfio_user_pci.c
 create mode 100644 drivers/common/iavf/vfio_user/vfio_user_pci.h
 create mode 100644 drivers/net/iavf/iavf_client_ethdev.c
Signature · 1 line
2.21.1
#1 lore raw

PATCH 1/2 common/iavf: emulated pci interfaces on vfio-user client

This patch implements Emulated pci interfaces on vfio-user client which is located in common/iavf/vfio_user/.
Four main functions provided to upper layer (driver) to set up or talk to device:
- client_vfio_user_setup
- client_vfio_user_release
- client_vfio_user_get_bar_addr
- client_vfio_user_pci_bar_access

      ----------------------
      | ------------------ |
      | |  iavf driver   | |
      | ------------------ |
      | ------------------ |
      | | device emulate | |------>(common/iavf/vfio_user/)
      | ------------------ |
      ----------------------
                |
                |
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------
---
 config/rte_config.h                           |   3 +
 drivers/common/iavf/iavf_common.c             |   7 +
 drivers/common/iavf/iavf_impl.c               |  45 +-
 drivers/common/iavf/iavf_osdep.h              |  14 +
 drivers/common/iavf/iavf_type.h               |   3 +
 drivers/common/iavf/meson.build               |  11 +-
 drivers/common/iavf/version.map               |   4 +
 drivers/common/iavf/vfio_user/vfio_user_pci.c | 517 ++++++++++++++++++
 drivers/common/iavf/vfio_user/vfio_user_pci.h |  66 +++
 9 files changed, 668 insertions(+), 2 deletions(-)
 create mode 100644 drivers/common/iavf/vfio_user/vfio_user_pci.c
 create mode 100644 drivers/common/iavf/vfio_user/vfio_user_pci.h
config/rte_config.h +3 -0
index a0b5160ff2..76ed1ae83f 100644
@@ -151,4 +151,7 @@
 #define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
 #define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
 
+/* IAVF bsed on vfio_user client */
+#define RTE_LIBRTE_IAVF_CLIENT 1
+
 #endif /* _RTE_CONFIG_H_ */
drivers/common/iavf/iavf_common.c +7 -0
index c951b7d787..2e68f5d08b 100644
@@ -20,6 +20,13 @@ enum iavf_status iavf_set_mac_type(struct iavf_hw *hw)
 
 	DEBUGFUNC("iavf_set_mac_type\n");
 
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user) {
+		hw->mac.type = IAVF_MAC_VF;
+		return status;
+	}
+#endif
+
 	if (hw->vendor_id == IAVF_INTEL_VENDOR_ID) {
 		switch (hw->device_id) {
 		case IAVF_DEV_ID_X722_VF:
drivers/common/iavf/iavf_impl.c +44 -1

57 lines not on this page · raw message

drivers/common/iavf/iavf_osdep.h +14 -0
index 7cba13ff74..025d48a702 100644
@@ -107,8 +107,22 @@ writeq(uint64_t value, volatile void *addr)
 	rte_write64(rte_cpu_to_le_64(value), addr);
 }
 
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+struct iavf_hw;
+
+__rte_internal
+uint32_t iavf_read_addr(struct iavf_hw *hw, uint32_t offset);
+__rte_internal
+void iavf_write_addr(struct iavf_hw *hw, uint32_t offset, uint32_t value);
+
+#define wr32(a, reg, value)  iavf_write_addr((a), (reg), (value))
+#define rd32(a, reg)         iavf_read_addr((a), (reg))
+
+#else
 #define wr32(a, reg, value) writel((value), (a)->hw_addr + (reg))
 #define rd32(a, reg)        readl((a)->hw_addr + (reg))
+#endif /* RTE_LIBRTE_IAVF_CLIENT */
+
 #define wr64(a, reg, value) writeq((value), (a)->hw_addr + (reg))
 #define rd64(a, reg)        readq((a)->hw_addr + (reg))
 
drivers/common/iavf/iavf_type.h +3 -0
index 0990c9aa33..faa4a21eac 100644
@@ -264,6 +264,9 @@ enum iavf_bus_type {
 	iavf_bus_type_pci,
 	iavf_bus_type_pcix,
 	iavf_bus_type_pci_express,
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	iavf_bus_type_vfio_user,
+#endif
 	iavf_bus_type_reserved
 };
 
drivers/common/iavf/meson.build +10 -1
index 1f4d8b898d..e0160ea135 100644
@@ -1,7 +1,16 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
-sources = files('iavf_adminq.c', 'iavf_common.c', 'iavf_impl.c')
+includes += include_directories('vfio_user')
+
+sources = files(
+	'iavf_adminq.c',
+	'iavf_common.c',
+	'iavf_impl.c',
+	'vfio_user/vfio_user_pci.c'
+)
+
+deps += ['vfio_user']
 
 if cc.has_argument('-Wno-pointer-to-int-cast')
         cflags += '-Wno-pointer-to-int-cast'
drivers/common/iavf/version.map +4 -0
index e0f117197c..8808779ab7 100644
@@ -7,6 +7,10 @@ INTERNAL {
 	iavf_set_mac_type;
 	iavf_shutdown_adminq;
 	iavf_vf_parse_hw_config;
+	client_vfio_user_setup;
+	client_vfio_user_get_bar_addr;
+	iavf_write_addr;
+	iavf_read_addr;
 
 	local: *;
 };
drivers/common/iavf/vfio_user/vfio_user_pci.c new file +517 -0

517 lines not on this page · raw message

drivers/common/iavf/vfio_user/vfio_user_pci.h new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.21.1
#2 lore raw

PATCH 2/2 net/iavf: introduce iavf driver on vfio-user client

In reply to Jingjing Wu

This patch add a new net driver based on vdev abstraction, i.e. iavf_client_ethdev.c. It is using common iavf functions to talk with Emulated pci interfaces based on vfio-user.

      ----------------------
      | ------------------ |
      | |  iavf driver   | |----> (iavf_client_ethdev.c)
      | ------------------ |
      | ------------------ |
      | | device emulate | |
      | | vfio-user adapt| |
      | ------------------ |
      ----------------------
                |
                |
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------
---
 drivers/common/iavf/iavf_prototype.h  |   1 +
 drivers/common/iavf/version.map       |   1 +
 drivers/net/iavf/iavf.h               |  18 +-
 drivers/net/iavf/iavf_client_ethdev.c | 298 ++++++++++++++++++++++++++
 drivers/net/iavf/iavf_ethdev.c        |  26 +--
 drivers/net/iavf/iavf_rxtx.c          |  23 +-
 drivers/net/iavf/meson.build          |   1 +
 7 files changed, 347 insertions(+), 21 deletions(-)
 create mode 100644 drivers/net/iavf/iavf_client_ethdev.c
drivers/common/iavf/iavf_prototype.h +1 -0
index f34e77db0f..3998d26dc0 100644
@@ -83,6 +83,7 @@ void iavf_destroy_spinlock(struct iavf_spinlock *sp);
 __rte_internal
 void iavf_vf_parse_hw_config(struct iavf_hw *hw,
 			     struct virtchnl_vf_resource *msg);
+__rte_internal
 enum iavf_status iavf_vf_reset(struct iavf_hw *hw);
 __rte_internal
 enum iavf_status iavf_aq_send_msg_to_pf(struct iavf_hw *hw,
drivers/common/iavf/version.map +1 -0
index 8808779ab7..4dc2d42196 100644
@@ -7,6 +7,7 @@ INTERNAL {
 	iavf_set_mac_type;
 	iavf_shutdown_adminq;
 	iavf_vf_parse_hw_config;
+	iavf_vf_reset;
 	client_vfio_user_setup;
 	client_vfio_user_get_bar_addr;
 	iavf_write_addr;
drivers/net/iavf/iavf.h +17 -1
index 6d5912d8c1..c34f971721 100644
@@ -195,7 +195,10 @@ struct iavf_adapter {
 	struct iavf_hw hw;
 	struct rte_eth_dev *eth_dev;
 	struct iavf_info vf;
-
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	/* used for avf_client driver */
+	struct vfio_device *user_dev;
+#endif
 	bool rx_bulk_alloc_allowed;
 	/* For vector PMD */
 	bool rx_vec_allowed;
@@ -231,6 +234,16 @@ iavf_init_adminq_parameter(struct iavf_hw *hw)
 	hw->aq.asq_buf_size = IAVF_AQ_BUF_SZ;
 }
 
+static inline void
+iavf_disable_irq0(struct iavf_hw *hw)
+{
+	/* Disable all interrupt types */
+	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1, 0);
+	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
+		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+	IAVF_WRITE_FLUSH(hw);
+}
+
 static inline uint16_t
 iavf_calc_itr_interval(int16_t interval)
 {
@@ -284,6 +297,9 @@ _atomic_set_cmd(struct iavf_info *vf, enum virtchnl_ops ops)
 	return !ret;
 }
 
+extern const struct eth_dev_ops iavf_eth_dev_ops;
+
+int iavf_init_vf(struct rte_eth_dev *dev);
 int iavf_check_api_version(struct iavf_adapter *adapter);
 int iavf_get_vf_resource(struct iavf_adapter *adapter);
 void iavf_handle_virtchnl_msg(struct rte_eth_dev *dev);
drivers/net/iavf/iavf_client_ethdev.c new file +298 -0

298 lines not on this page · raw message

drivers/net/iavf/iavf_ethdev.c +8 -18
index 7e3c26a94e..6b5e47adf2 100644
@@ -154,7 +154,7 @@ static const struct rte_iavf_xstats_name_off rte_iavf_stats_strings[] = {
 #define IAVF_NB_XSTATS (sizeof(rte_iavf_stats_strings) / \
 		sizeof(rte_iavf_stats_strings[0]))
 
-static const struct eth_dev_ops iavf_eth_dev_ops = {
+const struct eth_dev_ops iavf_eth_dev_ops = {
 	.dev_configure              = iavf_dev_configure,
 	.dev_start                  = iavf_dev_start,
 	.dev_stop                   = iavf_dev_stop,
@@ -1780,7 +1780,7 @@ iavf_init_proto_xtr(struct rte_eth_dev *dev)
 	}
 }
 
-static int
+int
 iavf_init_vf(struct rte_eth_dev *dev)
 {
 	int err, bufsz;
@@ -1789,12 +1789,6 @@ iavf_init_vf(struct rte_eth_dev *dev)
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private);
 
-	err = iavf_parse_devargs(dev);
-	if (err) {
-		PMD_INIT_LOG(ERR, "Failed to parse devargs");
-		goto err;
-	}
-
 	err = iavf_set_mac_type(hw);
 	if (err) {
 		PMD_INIT_LOG(ERR, "set_mac_type failed: %d", err);
@@ -1891,16 +1885,6 @@ iavf_enable_irq0(struct iavf_hw *hw)
 	IAVF_WRITE_FLUSH(hw);
 }
 
-static inline void
-iavf_disable_irq0(struct iavf_hw *hw)
-{
-	/* Disable all interrupt types */
-	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1, 0);
-	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
-		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
-	IAVF_WRITE_FLUSH(hw);
-}
-
 static void
 iavf_dev_interrupt_handler(void *param)
 {
@@ -1986,6 +1970,12 @@ iavf_dev_init(struct rte_eth_dev *eth_dev)
 	adapter->eth_dev = eth_dev;
 	adapter->stopped = 1;
 
+	ret  = iavf_parse_devargs(eth_dev);
+	if (ret) {
+		PMD_INIT_LOG(ERR, "Failed to parse devargs");
+		return ret;
+	}
+
 	if (iavf_init_vf(eth_dev) != 0) {
 		PMD_INIT_LOG(ERR, "Init vf failed");
 		return -1;
drivers/net/iavf/iavf_rxtx.c +21 -2
index 21d508b3f4..d8192f3675 100644
@@ -24,6 +24,9 @@
 #include <rte_ip.h>
 #include <rte_net.h>
 #include <rte_vect.h>
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+#include <vfio_user/vfio_user_pci.h>
+#endif
 
 #include "iavf.h"
 #include "iavf_rxtx.h"
@@ -595,7 +598,15 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
 	reset_rx_queue(rxq);
 	rxq->q_set = true;
 	dev->data->rx_queues[queue_idx] = rxq;
-	rxq->qrx_tail = hw->hw_addr + IAVF_QRX_TAIL1(rxq->queue_id);
+
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user)
+		rxq->qrx_tail = client_vfio_user_get_bar_addr(
+				(struct vfio_device *)hw->hw_addr, 0,
+				IAVF_QRX_TAIL1(rxq->queue_id), 4);
+	else
+#endif
+		rxq->qrx_tail = hw->hw_addr + IAVF_QRX_TAIL1(rxq->queue_id);
 	rxq->ops = &def_rxq_ops;
 
 	if (check_rx_bulk_allow(rxq) == true) {
@@ -705,7 +716,15 @@ iavf_dev_tx_queue_setup(struct rte_eth_dev *dev,
 	reset_tx_queue(txq);
 	txq->q_set = true;
 	dev->data->tx_queues[queue_idx] = txq;
-	txq->qtx_tail = hw->hw_addr + IAVF_QTX_TAIL1(queue_idx);
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user)
+		txq->qtx_tail = client_vfio_user_get_bar_addr(
+			(struct vfio_device *)hw->hw_addr, 0,
+			IAVF_QTX_TAIL1(queue_idx), 4);
+	else
+#endif
+		txq->qtx_tail = hw->hw_addr + IAVF_QTX_TAIL1(queue_idx);
+
 	txq->ops = &def_txq_ops;
 
 	if (check_tx_vec_allow(txq) == false) {
drivers/net/iavf/meson.build +1 -0
index 26c02c4401..580307c462 100644
@@ -13,6 +13,7 @@ sources = files(
 	'iavf_generic_flow.c',
 	'iavf_fdir.c',
 	'iavf_hash.c',
+	'iavf_client_ethdev.c',
 )
 
 if arch_subdir == 'x86'
Signature · 1 line
2.21.1
#3 lore raw

Re: [dpdk-dev] [PATCH v1 2/2] net/iavf: introduce iavf driver on vfio-user client

Hi Jingjing,

Quoted 234 lines

-----Original Message-----
From: Wu, Jingjing <[email protected]>
Sent: Saturday, December 19, 2020 10:38 PM
To: [email protected]
Cc: Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>; Xia, Chenbo <[email protected]>; Lu, Xiuchun <[email protected]>
Subject: [PATCH v1 2/2] net/iavf: introduce iavf driver on vfio-user client

This patch add a new net driver based on vdev abstraction, i.e. iavf_client_ethdev.c. It is using common iavf functions to talk with Emulated pci interfaces based on vfio-user.

      ----------------------
      | ------------------ |
      | |  iavf driver   | |----> (iavf_client_ethdev.c)
      | ------------------ |
      | ------------------ |
      | | device emulate | |
      | | vfio-user adapt| |
      | ------------------ |
      ----------------------
                |
                |
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------
---
 drivers/common/iavf/iavf_prototype.h  |   1 +
 drivers/common/iavf/version.map       |   1 +
 drivers/net/iavf/iavf.h               |  18 +-
 drivers/net/iavf/iavf_client_ethdev.c | 298 ++++++++++++++++++++++++++
 drivers/net/iavf/iavf_ethdev.c        |  26 +--
 drivers/net/iavf/iavf_rxtx.c          |  23 +-
 drivers/net/iavf/meson.build          |   1 +
 7 files changed, 347 insertions(+), 21 deletions(-)
 create mode 100644 drivers/net/iavf/iavf_client_ethdev.c
drivers/common/iavf/iavf_prototype.h +1 -0
index f34e77db0f..3998d26dc0 100644
@@ -83,6 +83,7 @@ void iavf_destroy_spinlock(struct iavf_spinlock *sp);
 __rte_internal
 void iavf_vf_parse_hw_config(struct iavf_hw *hw,
 			     struct virtchnl_vf_resource *msg);
+__rte_internal
 enum iavf_status iavf_vf_reset(struct iavf_hw *hw);
 __rte_internal
 enum iavf_status iavf_aq_send_msg_to_pf(struct iavf_hw *hw,
drivers/common/iavf/version.map +1 -0
index 8808779ab7..4dc2d42196 100644
@@ -7,6 +7,7 @@ INTERNAL {
 	iavf_set_mac_type;
 	iavf_shutdown_adminq;
 	iavf_vf_parse_hw_config;
+	iavf_vf_reset;
 	client_vfio_user_setup;
 	client_vfio_user_get_bar_addr;
 	iavf_write_addr;
drivers/net/iavf/iavf.h +17 -1
index 6d5912d8c1..c34f971721 100644
@@ -195,7 +195,10 @@ struct iavf_adapter {
 	struct iavf_hw hw;
 	struct rte_eth_dev *eth_dev;
 	struct iavf_info vf;
-
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	/* used for avf_client driver */
+	struct vfio_device *user_dev;
+#endif
 	bool rx_bulk_alloc_allowed;
 	/* For vector PMD */
 	bool rx_vec_allowed;
@@ -231,6 +234,16 @@ iavf_init_adminq_parameter(struct iavf_hw *hw)
 	hw->aq.asq_buf_size = IAVF_AQ_BUF_SZ;
 }
 
+static inline void
+iavf_disable_irq0(struct iavf_hw *hw)
+{
+	/* Disable all interrupt types */
+	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1, 0);
+	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
+		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+	IAVF_WRITE_FLUSH(hw);
+}
+
 static inline uint16_t
 iavf_calc_itr_interval(int16_t interval)
 {
@@ -284,6 +297,9 @@ _atomic_set_cmd(struct iavf_info *vf, enum virtchnl_ops
 ops)
 	return !ret;
 }
 
+extern const struct eth_dev_ops iavf_eth_dev_ops;
+
+int iavf_init_vf(struct rte_eth_dev *dev);
 int iavf_check_api_version(struct iavf_adapter *adapter);
 int iavf_get_vf_resource(struct iavf_adapter *adapter);

void iavf_handle_virtchnl_msg(struct rte_eth_dev *dev);

drivers/net/iavf/iavf_client_ethdev.c new file +117 -0
new file mode 100644
index 0000000000..03f759c761
@@ -0,0 +1,298 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2017 Intel Corporation
+ */
+
+#include <errno.h>
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_ether.h>
+#include <rte_ethdev_driver.h>
+#include <rte_ethdev_vdev.h>
+#include <rte_alarm.h>
+#include <rte_bus_vdev.h>
+#include <rte_malloc.h>
+#include <vfio_user/vfio_user_pci.h>
+
+#include "iavf.h"
+#include "iavf_rxtx.h"
+
+static int iavf_client_dev_close(struct rte_eth_dev *dev);
+static int iavf_client_dev_reset(struct rte_eth_dev *dev);
+
+/* set iavf_client_dev_ops to iavf's by default */
+static struct eth_dev_ops iavf_client_eth_dev_ops;
+
+static const char *valid_args[] = {
+#define AVF_CLIENT_ARG_PATH           "path"
+	AVF_CLIENT_ARG_PATH,
+	NULL
+};
+
+/* set up vfio_device for iavf_client*/
+static int
+iavf_client_vfio_user_setup(struct rte_eth_dev *dev, const char *path)
+{
+	struct iavf_adapter *adapter =
+		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
+	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
+	struct vfio_device *vfio_dev;
+	int max_fds, i;
+
+	vfio_dev = client_vfio_user_setup(path, dev->device->numa_node);
+	if (vfio_dev == NULL) {
+		printf("Error to create vfio_device for iavf_client\n");
+		return -1;
+	}
+	hw->bus.type = iavf_bus_type_vfio_user;
+
+	/* Use hw_addr to record dev ptr */
+	hw->hw_addr = (uint8_t *)vfio_dev;
+
+	hw->back = IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
+
+	if (!dev->intr_handle) {
+		dev->intr_handle = malloc(sizeof(*dev->intr_handle));
+		if (!dev->intr_handle) {
+			PMD_INIT_LOG(ERR, "fail to allocate intr_handle");
+			return -1;
+		}
+		memset(dev->intr_handle, 0, sizeof(*dev->intr_handle));
+	}
+
+	max_fds = RTE_MIN(RTE_MAX_RXTX_INTR_VEC_ID, adapter->vf.num_queue_pairs);
+
+	/* FD for control has been enabled */
+	dev->intr_handle->fd = -1;
+	dev->intr_handle->type = RTE_INTR_HANDLE_VDEV;
+	dev->intr_handle->max_intr = max_fds + 1;
+	dev->intr_handle->nb_efd = max_fds;
+	for (i = 0; i < max_fds; ++i)
+		dev->intr_handle->efds[i] = vfio_dev->irqfds[i];
+	dev->intr_handle->efd_counter_size = 0;
+
+	return 0;
+}
+
+
+static inline void
+avf_client_init_eth_ops(void)
+{
+	iavf_client_eth_dev_ops = iavf_eth_dev_ops;
+	/* keep other unchanged */
+	iavf_client_eth_dev_ops.dev_close  = iavf_client_dev_close,
+	iavf_client_eth_dev_ops.dev_reset  = iavf_client_dev_reset,
+	iavf_client_eth_dev_ops.dev_supported_ptypes_get   = NULL;
+	iavf_client_eth_dev_ops.reta_update                = NULL;
+	iavf_client_eth_dev_ops.reta_query                 = NULL;
+	iavf_client_eth_dev_ops.rss_hash_update            = NULL;
+	iavf_client_eth_dev_ops.rss_hash_conf_get          = NULL;
+	iavf_client_eth_dev_ops.rx_queue_intr_enable       = NULL;
+	iavf_client_eth_dev_ops.rx_queue_intr_disable      = NULL;
+}
+
+#define IAVF_CLIENT_ALARM_INTERVAL 50000 /* us */
+static void
+iavf_client_dev_alarm_handler(void *param)
+{
+	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
+
+	iavf_handle_virtchnl_msg(dev);
+
+	rte_eal_alarm_set(IAVF_CLIENT_ALARM_INTERVAL,
+			  iavf_client_dev_alarm_handler, dev);
+}
+
+/* init ethdev for the avf client device*/
+static int
+iavf_client_eth_init(struct rte_eth_dev *eth_dev)
+{
+	struct iavf_adapter *adapter =
+		IAVF_DEV_PRIVATE_TO_ADAPTER(eth_dev->data->dev_private);
+	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
+
+	/* update eth_dev_op by assigning ops func pointer */
+	avf_client_init_eth_ops();
+	eth_dev->dev_ops = (const struct eth_dev_ops

*)(&iavf_client_eth_dev_ops);

drivers/net/iavf/iavf_client_ethdev.c line 126

Quoted 9 lines
drivers/net/iavf/iavf_client_ethdev.c new file +9 -0
+
+	eth_dev->rx_pkt_burst = &iavf_recv_pkts;
+	eth_dev->tx_pkt_burst = &iavf_xmit_pkts;
+	eth_dev->tx_pkt_prepare = &iavf_prep_pkts;
+
+	hw->back = IAVF_DEV_PRIVATE_TO_ADAPTER(eth_dev->data->dev_private);
+	adapter->eth_dev = eth_dev;
+	adapter->stopped = 1;
+

I think the ptype table (adapter->ptype_tbl) should be set during init.

Maybe using iavf_get_default_ptype_table().

Thanks!
Chenbo

Quoted 301 lines
drivers/net/iavf/iavf_client_ethdev.c new file +172 -0
+	if (iavf_init_vf(eth_dev) != 0) {
+		PMD_INIT_LOG(ERR, "Init vf failed");
+		return -1;
+	}
+
+	/* copy mac addr */
+	eth_dev->data->mac_addrs = rte_zmalloc(
+			"iavf_client_mac",
+			RTE_ETHER_ADDR_LEN * IAVF_NUM_MACADDR_MAX, 0);
+	if (!eth_dev->data->mac_addrs) {
+		PMD_INIT_LOG(ERR, "Failed to allocate %d bytes needed to"
+			     " store MAC addresses",
+			     RTE_ETHER_ADDR_LEN * IAVF_NUM_MACADDR_MAX);
+		return -ENOMEM;
+	}
+	/* If the MAC address is not configured by host,
+	 * generate a random one.
+	 */
+	if (!rte_is_valid_assigned_ether_addr(
+			(struct rte_ether_addr *)hw->mac.addr))
+		rte_eth_random_addr(hw->mac.addr);
+	rte_ether_addr_copy((struct rte_ether_addr *)hw->mac.addr,
+			&eth_dev->data->mac_addrs[0]);
+
+	rte_eal_alarm_set(IAVF_CLIENT_ALARM_INTERVAL,
+			  iavf_client_dev_alarm_handler, eth_dev);
+	return 0;
+}
+
+static int
+iavf_client_dev_reset(struct rte_eth_dev *dev)
+{
+	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	int ret;
+
+	rte_eal_alarm_cancel(iavf_client_dev_alarm_handler, dev);
+
+	iavf_shutdown_adminq(hw);
+	ret = iavf_init_vf(dev);
+
+	/* send reset msg to PF */
+	iavf_vf_reset(hw);
+	rte_eal_alarm_set(IAVF_CLIENT_ALARM_INTERVAL,
+			  iavf_client_dev_alarm_handler, dev);
+
+	return ret;
+}
+
+static int
+iavf_client_dev_close(struct rte_eth_dev *dev)
+{
+	struct iavf_adapter *adapter =
+		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
+	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct rte_intr_handle *intr_handle = dev->intr_handle;
+
+	if (!adapter->stopped) {
+		iavf_stop_queues(dev);
+
+		if (intr_handle) {
+			/* Disable the interrupt for Rx */
+			rte_intr_efd_disable(intr_handle);
+			/* Rx interrupt vector mapping free */
+			if (intr_handle->intr_vec) {
+				rte_free(intr_handle->intr_vec);
+				intr_handle->intr_vec = NULL;
+			}
+		}
+		/* Remove all mac addrs */
+		iavf_add_del_all_mac_addr(adapter, false);
+		adapter->stopped = 1;
+	}
+	iavf_shutdown_adminq(hw);
+	iavf_disable_irq0(hw);
+	rte_eal_alarm_cancel(iavf_client_dev_alarm_handler, dev);
+
+	return 0;
+}
+
+static int
+iavf_client_get_string_arg(const char *key __rte_unused,
+	       const char *value, void *extra_args)
+{
+	if (!value || !extra_args)
+		return -EINVAL;
+
+	*(char **)extra_args = strdup(value);
+
+	if (!*(char **)extra_args)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static int
+avf_client_pmd_probe(struct rte_vdev_device *vdev)
+{
+	struct rte_kvargs *kvlist = NULL;
+	struct rte_eth_dev *eth_dev;
+	struct iavf_adapter *adapter;
+	char *path = NULL;
+	int ret;
+
+	kvlist = rte_kvargs_parse(rte_vdev_device_args(vdev), valid_args);
+	if (!kvlist) {
+		PMD_INIT_LOG(ERR, "error when parsing param");
+		return -EINVAL;
+	}
+
+	if (rte_kvargs_count(kvlist, AVF_CLIENT_ARG_PATH) == 1) {
+		if (rte_kvargs_process(kvlist, AVF_CLIENT_ARG_PATH,
+				       &iavf_client_get_string_arg, &path) < 0) {
+			PMD_INIT_LOG(ERR, "error to parse %s",
+				     AVF_CLIENT_ARG_PATH);
+			return -EINVAL;
+		}
+	} else {
+		PMD_INIT_LOG(ERR, "arg %s is mandatory for virtio_user",
+			     AVF_CLIENT_ARG_PATH);
+		return -EINVAL;
+	}
+
+	eth_dev = rte_eth_vdev_allocate(vdev, sizeof(*adapter));
+
+	ret = iavf_client_vfio_user_setup(eth_dev, path);
+	if (ret) {
+		goto err;
+	}
+
+	ret = iavf_client_eth_init(eth_dev);
+	if (ret) {
+		goto err;
+	}
+
+	rte_eth_dev_probing_finish(eth_dev);
+	rte_kvargs_free(kvlist);
+
+	return 0;
+err:
+	rte_eth_dev_release_port(eth_dev);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+
+static int
+avf_client_pmd_remove(struct rte_vdev_device *vdev)
+{
+	struct rte_eth_dev *eth_dev = NULL;
+
+	if (vdev == NULL)
+		return -1;
+
+	/* find the ethdev entry */
+	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	if (eth_dev == NULL)
+		return 0;
+
+	iavf_client_dev_close(eth_dev);
+	rte_eth_dev_release_port(eth_dev);
+	return 0;
+}
+
+static struct rte_vdev_driver iavf_client_driver = {
+	.probe = avf_client_pmd_probe,
+	.remove = avf_client_pmd_remove,
+};
+
+RTE_PMD_REGISTER_VDEV(net_iavf_client, iavf_client_driver);
+RTE_PMD_REGISTER_ALIAS(net_iavf_client, iavf_client);
+RTE_PMD_REGISTER_PARAM_STRING(net_iavf_client,
+	"path=<path>");
drivers/net/iavf/iavf_ethdev.c +1 -1
index 7e3c26a94e..6b5e47adf2 100644
@@ -154,7 +154,7 @@ static const struct rte_iavf_xstats_name_off
 rte_iavf_stats_strings[] = {
 #define IAVF_NB_XSTATS (sizeof(rte_iavf_stats_strings) / \
 		sizeof(rte_iavf_stats_strings[0]))
 
-static const struct eth_dev_ops iavf_eth_dev_ops = {
+const struct eth_dev_ops iavf_eth_dev_ops = {
 	.dev_configure              = iavf_dev_configure,
 	.dev_start                  = iavf_dev_start,
drivers/net/iavf/iavf_ethdev.c +7 -17
 	.dev_stop                   = iavf_dev_stop,
@@ -1780,7 +1780,7 @@ iavf_init_proto_xtr(struct rte_eth_dev *dev)
 	}
 }
 
-static int
+int
 iavf_init_vf(struct rte_eth_dev *dev)
 {
 	int err, bufsz;
@@ -1789,12 +1789,6 @@ iavf_init_vf(struct rte_eth_dev *dev)
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private);
 
-	err = iavf_parse_devargs(dev);
-	if (err) {
-		PMD_INIT_LOG(ERR, "Failed to parse devargs");
-		goto err;
-	}
-
 	err = iavf_set_mac_type(hw);
 	if (err) {
 		PMD_INIT_LOG(ERR, "set_mac_type failed: %d", err);
@@ -1891,16 +1885,6 @@ iavf_enable_irq0(struct iavf_hw *hw)
 	IAVF_WRITE_FLUSH(hw);
 }
 
-static inline void
-iavf_disable_irq0(struct iavf_hw *hw)
-{
-	/* Disable all interrupt types */
-	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1, 0);
-	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
-		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
-	IAVF_WRITE_FLUSH(hw);
-}
-
 static void
 iavf_dev_interrupt_handler(void *param)
 {
@@ -1986,6 +1970,12 @@ iavf_dev_init(struct rte_eth_dev *eth_dev)
 	adapter->eth_dev = eth_dev;
 	adapter->stopped = 1;
 
+	ret  = iavf_parse_devargs(eth_dev);
+	if (ret) {
+		PMD_INIT_LOG(ERR, "Failed to parse devargs");
+		return ret;
+	}
+
 	if (iavf_init_vf(eth_dev) != 0) {
 		PMD_INIT_LOG(ERR, "Init vf failed");
 		return -1;
drivers/net/iavf/iavf_rxtx.c +12 -1
index 21d508b3f4..d8192f3675 100644
@@ -24,6 +24,9 @@
 #include <rte_ip.h>
 #include <rte_net.h>
 #include <rte_vect.h>
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+#include <vfio_user/vfio_user_pci.h>
+#endif
 
 #include "iavf.h"
 #include "iavf_rxtx.h"
@@ -595,7 +598,15 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t
 queue_idx,
 	reset_rx_queue(rxq);
 	rxq->q_set = true;
 	dev->data->rx_queues[queue_idx] = rxq;
-	rxq->qrx_tail = hw->hw_addr + IAVF_QRX_TAIL1(rxq->queue_id);
+
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user)
+		rxq->qrx_tail = client_vfio_user_get_bar_addr(
+				(struct vfio_device *)hw->hw_addr, 0,
+				IAVF_QRX_TAIL1(rxq->queue_id), 4);
+	else
+#endif
+		rxq->qrx_tail = hw->hw_addr + IAVF_QRX_TAIL1(rxq->queue_id);
 	rxq->ops = &def_rxq_ops;
 
drivers/net/iavf/iavf_rxtx.c +9 -1
 	if (check_rx_bulk_allow(rxq) == true) {
@@ -705,7 +716,15 @@ iavf_dev_tx_queue_setup(struct rte_eth_dev *dev,
 	reset_tx_queue(txq);
 	txq->q_set = true;
 	dev->data->tx_queues[queue_idx] = txq;
-	txq->qtx_tail = hw->hw_addr + IAVF_QTX_TAIL1(queue_idx);
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user)
+		txq->qtx_tail = client_vfio_user_get_bar_addr(
+			(struct vfio_device *)hw->hw_addr, 0,
+			IAVF_QTX_TAIL1(queue_idx), 4);
+	else
+#endif
+		txq->qtx_tail = hw->hw_addr + IAVF_QTX_TAIL1(queue_idx);
+
 	txq->ops = &def_txq_ops;
 
 	if (check_tx_vec_allow(txq) == false) {
drivers/net/iavf/meson.build +1 -1
index 26c02c4401..580307c462 100644
@@ -13,6 +13,7 @@ sources = files(
 	'iavf_generic_flow.c',
 	'iavf_fdir.c',
 	'iavf_hash.c',
+	'iavf_client_ethdev.c',
 )
 
 if arch_subdir == 'x86'
--

2.21.1

#4 lore raw

v2 cover introduce new iavf driver on vfio-user client

In reply to Jingjing Wu

This series introduces a new net virtual device called iavf_client which is based on vfio-user client interface. Through vfio-user client interface, PCI liked device could be used similar as Intel® Ethernet Adaptive Virtual Function specification.
The code to enable iavf_client mainly contains two parts:
- Emulated pci interfaces for vfio-user client which is located in common/iavf/vfio_user/.
- A new net driver base on vdev abstraction, i.e. iavf_client_ethdev.c

      ----------------------
      | ------------------ |
      | |  iavf driver   | |----> (iavf_client_ethdev.c)
      | ------------------ |
      | ------------------ |
      | | device emulate | |------>(common/iavf/vfio_user/)
      | ------------------ |
      ----------------------
                |
                |  
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------

This driver can be used to test the device emulation framework mentioned in:
[RFC 0/2] Add device emulation support in DPDK
http://patchwork.dpdk.org/cover/75549/

                    +------------------------------------------------------+
                    |   +---------------+      +---------------+           |
                    |   | iavf_emudev   |      | iavfbe_ethdev |           |
                    |   |    driver     |      |     driver    |           |
                    |   +---------------+      +---------------+           |
                    |           |                       |                  |
                    | ------------------------------------------- VDEV BUS |
                    |           |                       |                  |
                    |   +---------------+       +--------------+           |
+--------------+    |   | vdev:         |       | vdev:        |           |
| +----------+ |    |   | /path/to/vfio |       |iavf_emudev_# |           |
| |   iavf   | |    |   +---------------+       +--------------+           |
| |  client  |      |           |                                          |
| +----------+ |    |           |                                          |
| +----------+ |    |      +----------+                                    |
| | vfio-user| |    |      | vfio-user|                                    |
| | client   | |<---|----->| server   |                                    |
| +----------+ |    |      +----------+                                    |
| QEMU/DPDK    |    | DPDK                                                 |
+--------------+    +------------------------------------------------------+

It can be launched together with patch series for the testing:
[0/9] Introduce vfio-user library:
http://patchwork.dpdk.org/cover/85389/
[0/8] Introduce emudev library and iavf emudev driver
http://patchwork.dpdk.org/cover/85488/
[0/6] Introduce iavf backend driver
http://patchwork.dpdk.org/cover/86084/

This series depends on patch serieses:
[0/9] Introduce vfio-user library:
http://patchwork.dpdk.org/cover/85389/

v2:
- Enable interrupt for control queue
- Enable interrupt for rx queue
- Rename some Macros
- Fix resource release when close
- Fix ptype_tbl assignment
- Fix typo

Jingjing Wu (4):
net/iavf_client: introduce iavf driver on vfio-user client net/iavf_client: enable interrupt on control queue net/iavf_client: client to enable intr_handle for rx_irq net/iavf: fix vector mapping with queue

 drivers/common/iavf/iavf_prototype.h  |   1 +
 drivers/common/iavf/version.map       |   1 +
 drivers/net/iavf/iavf.h               |  37 ++-
 drivers/net/iavf/iavf_client_ethdev.c | 404 ++++++++++++++++++++++++++
 drivers/net/iavf/iavf_ethdev.c        |  55 +---
 drivers/net/iavf/iavf_rxtx.c          |  23 +-
 drivers/net/iavf/meson.build          |   1 +
 7 files changed, 477 insertions(+), 45 deletions(-)
 create mode 100644 drivers/net/iavf/iavf_client_ethdev.c
Signature · 1 line
2.21.1
#5 lore raw

PATCH v2 1/4 net/iavf_client: introduce iavf driver on vfio-user client

This patch add a new net driver based on vdev abstraction, i.e. iavf_client_ethdev.c. It is using common iavf functions to talk with Emulated pci interfaces based on vfio-user.

      ----------------------
      | ------------------ |
      | |  iavf driver   | |----> (iavf_client_ethdev.c)
      | ------------------ |
      | ------------------ |
      | | device emulate | |
      | | vfio-user adapt| |
      | ------------------ |
      ----------------------
                |
                |
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------
---
 drivers/common/iavf/iavf_prototype.h  |   1 +
 drivers/common/iavf/version.map       |   1 +
 drivers/net/iavf/iavf.h               |  18 +-
 drivers/net/iavf/iavf_client_ethdev.c | 290 ++++++++++++++++++++++++++
 drivers/net/iavf/iavf_ethdev.c        |  26 +--
 drivers/net/iavf/iavf_rxtx.c          |  23 +-
 drivers/net/iavf/meson.build          |   1 +
 7 files changed, 339 insertions(+), 21 deletions(-)
 create mode 100644 drivers/net/iavf/iavf_client_ethdev.c
drivers/common/iavf/iavf_prototype.h +1 -0
index f34e77db0f..3998d26dc0 100644
@@ -83,6 +83,7 @@ void iavf_destroy_spinlock(struct iavf_spinlock *sp);
 __rte_internal
 void iavf_vf_parse_hw_config(struct iavf_hw *hw,
 			     struct virtchnl_vf_resource *msg);
+__rte_internal
 enum iavf_status iavf_vf_reset(struct iavf_hw *hw);
 __rte_internal
 enum iavf_status iavf_aq_send_msg_to_pf(struct iavf_hw *hw,
drivers/common/iavf/version.map +1 -0
index 883d919487..23c3577b3e 100644
@@ -7,6 +7,7 @@ INTERNAL {
 	iavf_set_mac_type;
 	iavf_shutdown_adminq;
 	iavf_vf_parse_hw_config;
+	iavf_vf_reset;
 	client_vfio_user_setup;
 	client_vfio_user_release;
 	client_vfio_user_get_bar_addr;
drivers/net/iavf/iavf.h +17 -1
index 6d5912d8c1..c34f971721 100644
@@ -195,7 +195,10 @@ struct iavf_adapter {
 	struct iavf_hw hw;
 	struct rte_eth_dev *eth_dev;
 	struct iavf_info vf;
-
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	/* used for avf_client driver */
+	struct vfio_device *user_dev;
+#endif
 	bool rx_bulk_alloc_allowed;
 	/* For vector PMD */
 	bool rx_vec_allowed;
@@ -231,6 +234,16 @@ iavf_init_adminq_parameter(struct iavf_hw *hw)
 	hw->aq.asq_buf_size = IAVF_AQ_BUF_SZ;
 }
 
+static inline void
+iavf_disable_irq0(struct iavf_hw *hw)
+{
+	/* Disable all interrupt types */
+	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1, 0);
+	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
+		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+	IAVF_WRITE_FLUSH(hw);
+}
+
 static inline uint16_t
 iavf_calc_itr_interval(int16_t interval)
 {
@@ -284,6 +297,9 @@ _atomic_set_cmd(struct iavf_info *vf, enum virtchnl_ops ops)
 	return !ret;
 }
 
+extern const struct eth_dev_ops iavf_eth_dev_ops;
+
+int iavf_init_vf(struct rte_eth_dev *dev);
 int iavf_check_api_version(struct iavf_adapter *adapter);
 int iavf_get_vf_resource(struct iavf_adapter *adapter);
 void iavf_handle_virtchnl_msg(struct rte_eth_dev *dev);
drivers/net/iavf/iavf_client_ethdev.c new file +290 -0

290 lines not on this page · raw message

drivers/net/iavf/iavf_ethdev.c +8 -18
index 7e3c26a94e..6b5e47adf2 100644
@@ -154,7 +154,7 @@ static const struct rte_iavf_xstats_name_off rte_iavf_stats_strings[] = {
 #define IAVF_NB_XSTATS (sizeof(rte_iavf_stats_strings) / \
 		sizeof(rte_iavf_stats_strings[0]))
 
-static const struct eth_dev_ops iavf_eth_dev_ops = {
+const struct eth_dev_ops iavf_eth_dev_ops = {
 	.dev_configure              = iavf_dev_configure,
 	.dev_start                  = iavf_dev_start,
 	.dev_stop                   = iavf_dev_stop,
@@ -1780,7 +1780,7 @@ iavf_init_proto_xtr(struct rte_eth_dev *dev)
 	}
 }
 
-static int
+int
 iavf_init_vf(struct rte_eth_dev *dev)
 {
 	int err, bufsz;
@@ -1789,12 +1789,6 @@ iavf_init_vf(struct rte_eth_dev *dev)
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private);
 
-	err = iavf_parse_devargs(dev);
-	if (err) {
-		PMD_INIT_LOG(ERR, "Failed to parse devargs");
-		goto err;
-	}
-
 	err = iavf_set_mac_type(hw);
 	if (err) {
 		PMD_INIT_LOG(ERR, "set_mac_type failed: %d", err);
@@ -1891,16 +1885,6 @@ iavf_enable_irq0(struct iavf_hw *hw)
 	IAVF_WRITE_FLUSH(hw);
 }
 
-static inline void
-iavf_disable_irq0(struct iavf_hw *hw)
-{
-	/* Disable all interrupt types */
-	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1, 0);
-	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
-		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
-	IAVF_WRITE_FLUSH(hw);
-}
-
 static void
 iavf_dev_interrupt_handler(void *param)
 {
@@ -1986,6 +1970,12 @@ iavf_dev_init(struct rte_eth_dev *eth_dev)
 	adapter->eth_dev = eth_dev;
 	adapter->stopped = 1;
 
+	ret  = iavf_parse_devargs(eth_dev);
+	if (ret) {
+		PMD_INIT_LOG(ERR, "Failed to parse devargs");
+		return ret;
+	}
+
 	if (iavf_init_vf(eth_dev) != 0) {
 		PMD_INIT_LOG(ERR, "Init vf failed");
 		return -1;
drivers/net/iavf/iavf_rxtx.c +21 -2
index 21d508b3f4..d8192f3675 100644
@@ -24,6 +24,9 @@
 #include <rte_ip.h>
 #include <rte_net.h>
 #include <rte_vect.h>
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+#include <vfio_user/vfio_user_pci.h>
+#endif
 
 #include "iavf.h"
 #include "iavf_rxtx.h"
@@ -595,7 +598,15 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
 	reset_rx_queue(rxq);
 	rxq->q_set = true;
 	dev->data->rx_queues[queue_idx] = rxq;
-	rxq->qrx_tail = hw->hw_addr + IAVF_QRX_TAIL1(rxq->queue_id);
+
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user)
+		rxq->qrx_tail = client_vfio_user_get_bar_addr(
+				(struct vfio_device *)hw->hw_addr, 0,
+				IAVF_QRX_TAIL1(rxq->queue_id), 4);
+	else
+#endif
+		rxq->qrx_tail = hw->hw_addr + IAVF_QRX_TAIL1(rxq->queue_id);
 	rxq->ops = &def_rxq_ops;
 
 	if (check_rx_bulk_allow(rxq) == true) {
@@ -705,7 +716,15 @@ iavf_dev_tx_queue_setup(struct rte_eth_dev *dev,
 	reset_tx_queue(txq);
 	txq->q_set = true;
 	dev->data->tx_queues[queue_idx] = txq;
-	txq->qtx_tail = hw->hw_addr + IAVF_QTX_TAIL1(queue_idx);
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user)
+		txq->qtx_tail = client_vfio_user_get_bar_addr(
+			(struct vfio_device *)hw->hw_addr, 0,
+			IAVF_QTX_TAIL1(queue_idx), 4);
+	else
+#endif
+		txq->qtx_tail = hw->hw_addr + IAVF_QTX_TAIL1(queue_idx);
+
 	txq->ops = &def_txq_ops;
 
 	if (check_tx_vec_allow(txq) == false) {
drivers/net/iavf/meson.build +1 -0
index 26c02c4401..580307c462 100644
@@ -13,6 +13,7 @@ sources = files(
 	'iavf_generic_flow.c',
 	'iavf_fdir.c',
 	'iavf_hash.c',
+	'iavf_client_ethdev.c',
 )
 
 if arch_subdir == 'x86'
Signature · 1 line
2.21.1
#6 lore raw

PATCH v2 2/4 net/iavf_client: enable interrupt on control queue

In reply to Jingjing Wu

New devarg "intr": if intr=1, use interrupt mode on control queue

---
 drivers/net/iavf/iavf.h               |  19 ++++
 drivers/net/iavf/iavf_client_ethdev.c | 131 ++++++++++++++++++++++----
 drivers/net/iavf/iavf_ethdev.c        |  18 +---
 3 files changed, 134 insertions(+), 34 deletions(-)
drivers/net/iavf/iavf.h +19 -0
index c34f971721..5516ecf021 100644
@@ -198,6 +198,7 @@ struct iavf_adapter {
 #ifdef RTE_LIBRTE_IAVF_CLIENT
 	/* used for avf_client driver */
 	struct vfio_device *user_dev;
+	int intr_mode; /* interrupt mode if true */
 #endif
 	bool rx_bulk_alloc_allowed;
 	/* For vector PMD */
@@ -234,6 +235,22 @@ iavf_init_adminq_parameter(struct iavf_hw *hw)
 	hw->aq.asq_buf_size = IAVF_AQ_BUF_SZ;
 }
 
+/* Enable default admin queue interrupt setting */
+static inline void
+iavf_enable_irq0(struct iavf_hw *hw)
+{
+	/* Enable admin queue interrupt trigger */
+	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1,
+		       IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+
+	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
+		       IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+		       IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+	IAVF_WRITE_FLUSH(hw);
+}
+
 static inline void
 iavf_disable_irq0(struct iavf_hw *hw)
 {
@@ -342,4 +359,6 @@ int iavf_add_del_mc_addr_list(struct iavf_adapter *adapter,
 			uint32_t mc_addrs_num, bool add);
 int iavf_request_queues(struct iavf_adapter *adapter, uint16_t num);
 int iavf_get_max_rss_queue_region(struct iavf_adapter *adapter);
+void iavf_dev_interrupt_handler(void *param);
+
 #endif /* _IAVF_ETHDEV_H_ */
drivers/net/iavf/iavf_client_ethdev.c +114 -17

243 lines not on this page · raw message

drivers/net/iavf/iavf_ethdev.c +1 -17
index 6b5e47adf2..bc07ecbed7 100644
@@ -1869,23 +1869,7 @@ iavf_init_vf(struct rte_eth_dev *dev)
 	return -1;
 }
 
-/* Enable default admin queue interrupt setting */
-static inline void
-iavf_enable_irq0(struct iavf_hw *hw)
-{
-	/* Enable admin queue interrupt trigger */
-	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1,
-		       IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
-
-	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
-		       IAVF_VFINT_DYN_CTL01_INTENA_MASK |
-		       IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
-		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
-
-	IAVF_WRITE_FLUSH(hw);
-}
-
-static void
+void
 iavf_dev_interrupt_handler(void *param)
 {
 	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
Signature · 1 line
2.21.1
#7 lore raw

PATCH v2 3/4 net/iavf_client: enable interrupt of control queue

In reply to Jingjing Wu
---
 drivers/net/iavf/iavf_client_ethdev.c | 29 +++++++++++++++++++++------
 drivers/net/iavf/iavf_ethdev.c        |  9 +++------
 2 files changed, 26 insertions(+), 12 deletions(-)
drivers/net/iavf/iavf_client_ethdev.c +23 -6
index 770a7a8200..27daaed0e8 100644
@@ -59,6 +59,7 @@ iavf_client_vfio_user_setup(struct rte_eth_dev *dev, const char *path)
 		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
 	struct vfio_device *vfio_dev;
+	uint32_t max_fds, i;
 
 	vfio_dev = client_vfio_user_setup(path, dev->device->numa_node);
 	if (vfio_dev == NULL) {
@@ -87,11 +88,21 @@ iavf_client_vfio_user_setup(struct rte_eth_dev *dev, const char *path)
 		}
 
 	}
-
 	dev->intr_handle->fd = vfio_dev->irqfds[0];
 	dev->intr_handle->type = RTE_INTR_HANDLE_VDEV;
 	dev->intr_handle->max_intr = 1;
 
+	/* Assign rxq fds */
+	if (vfio_dev->nb_irqs > 1) {
+		max_fds = RTE_MIN((uint32_t)RTE_MAX_RXTX_INTR_VEC_ID,
+				  vfio_dev->nb_irqs - 1);
+		dev->intr_handle->nb_efd = max_fds;
+		for (i = 0; i < max_fds; ++i)
+			dev->intr_handle->efds[i] =
+				vfio_dev->irqfds[i + RTE_INTR_VEC_RXTX_OFFSET];
+		dev->intr_handle->efd_counter_size = 0;
+		dev->intr_handle->max_intr += max_fds;
+	}
 	return 0;
 }
 
@@ -108,8 +119,6 @@ avf_client_init_eth_ops(void)
 	iavf_client_eth_dev_ops.reta_query                 = NULL;
 	iavf_client_eth_dev_ops.rss_hash_update            = NULL;
 	iavf_client_eth_dev_ops.rss_hash_conf_get          = NULL;
-	iavf_client_eth_dev_ops.rx_queue_intr_enable       = NULL;
-	iavf_client_eth_dev_ops.rx_queue_intr_disable      = NULL;
 }
 
 #define IAVF_CLIENT_ALARM_INTERVAL 50000 /* us */
@@ -227,11 +236,12 @@ iavf_client_dev_close(struct rte_eth_dev *dev)
 	struct iavf_adapter *adapter =
 		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct rte_intr_handle *intr_handle = dev->intr_handle;
 
 	if (adapter->intr_mode) {
 		iavf_disable_irq0(hw);
 		/* unregister callback func from eal lib */
-		rte_intr_callback_unregister(dev->intr_handle,
+		rte_intr_callback_unregister(intr_handle,
 					     iavf_client_event_handler, dev);
 	} else {
 		rte_eal_alarm_cancel(iavf_client_dev_alarm_handler, dev);
@@ -240,8 +250,15 @@ iavf_client_dev_close(struct rte_eth_dev *dev)
 	if (!adapter->stopped) {
 		iavf_stop_queues(dev);
 
-		if (dev->intr_handle) {
-			rte_free(dev->intr_handle);
+		if (intr_handle) {
+			/* Disable the interrupt for Rx */
+			rte_intr_efd_disable(intr_handle);
+			/* Rx interrupt vector mapping free */
+			if (intr_handle->intr_vec) {
+				rte_free(intr_handle->intr_vec);
+				intr_handle->intr_vec = NULL;
+			}
+			rte_free(intr_handle);
 			dev->intr_handle = NULL;
 		}
 
drivers/net/iavf/iavf_ethdev.c +3 -6
index bc07ecbed7..06395f852b 100644
@@ -1368,11 +1368,10 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
 {
 	struct iavf_adapter *adapter =
 		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
 	uint16_t msix_intr;
 
-	msix_intr = pci_dev->intr_handle.intr_vec[queue_id];
+	msix_intr = dev->intr_handle->intr_vec[queue_id];
 	if (msix_intr == IAVF_MISC_VEC_ID) {
 		PMD_DRV_LOG(INFO, "MISC is also enabled for control");
 		IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
@@ -1387,10 +1386,9 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
 			       IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
 			       IAVF_VFINT_DYN_CTLN1_ITR_INDX_MASK);
 	}
-
 	IAVF_WRITE_FLUSH(hw);
 
-	rte_intr_ack(&pci_dev->intr_handle);
+	rte_intr_ack(dev->intr_handle);
 
 	return 0;
 }
@@ -1398,11 +1396,10 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
 static int
 iavf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
 {
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	uint16_t msix_intr;
 
-	msix_intr = pci_dev->intr_handle.intr_vec[queue_id];
+	msix_intr = dev->intr_handle->intr_vec[queue_id];
 	if (msix_intr == IAVF_MISC_VEC_ID) {
 		PMD_DRV_LOG(ERR, "MISC is used for control, cannot disable it");
 		return -EIO;
Signature · 1 line
2.21.1
#8 lore raw

PATCH v2 4/4 net/iavf: fix vector mapping with queue

In reply to Jingjing Wu

Fix the vector mapping with queue by changing the recircle when exceeds RX_VEC_START + nb_msix;

---
 drivers/net/iavf/iavf_ethdev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/iavf/iavf_ethdev.c +1 -1
index 06395f852b..6182fc6f4f 100644
@@ -578,7 +578,7 @@ static int iavf_config_rx_queues_irqs(struct rte_eth_dev *dev,
 				qv_map[i].queue_id = i;
 				qv_map[i].vector_id = vec;
 				intr_handle->intr_vec[i] = vec++;
-				if (vec >= vf->nb_msix)
+				if (vec >= vf->nb_msix + IAVF_RX_VEC_START)
 					vec = IAVF_RX_VEC_START;
 			}
 			vf->qv_map = qv_map;
Signature · 1 line
2.21.1
#9 lore raw

v3 cover introduce new iavf driver on vfio-user client

In reply to Jingjing Wu

This series introduces a new net virtual device called iavf_client which is based on vfio-user client interface. Through vfio-user client interface, PCI liked device could be used similar as Intel® Ethernet Adaptive Virtual Function specification.
The code to enable iavf_client mainly contains two parts:
- Emulated pci interfaces for vfio-user client which is located in common/iavf/vfio_user/.
- A new net driver base on vdev abstraction, i.e. iavf_client_ethdev.c

      ----------------------
      | ------------------ |
      | |  iavf driver   | |----> (iavf_client_ethdev.c)
      | ------------------ |
      | ------------------ |
      | | device emulate | |------>(common/iavf/vfio_user/)
      | ------------------ |
      ----------------------
                |
                |  
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------

This driver can be used to test the device emulation framework mentioned in:
[RFC 0/2] Add device emulation support in DPDK
http://patchwork.dpdk.org/cover/75549/

                    +------------------------------------------------------+
                    |   +---------------+      +---------------+           |
                    |   | iavf_emudev   |      | iavfbe_ethdev |           |
                    |   |    driver     |      |     driver    |           |
                    |   +---------------+      +---------------+           |
                    |           |                       |                  |
                    | ------------------------------------------- VDEV BUS |
                    |           |                       |                  |
                    |   +---------------+       +--------------+           |
+--------------+    |   | vdev:         |       | vdev:        |           |
| +----------+ |    |   | /path/to/vfio |       |iavf_emudev_# |           |
| |   iavf   | |    |   +---------------+       +--------------+           |
| |  client  |      |           |                                          |
| +----------+ |    |           |                                          |
| +----------+ |    |      +----------+                                    |
| | vfio-user| |    |      | vfio-user|                                    |
| | client   | |<---|----->| server   |                                    |
| +----------+ |    |      +----------+                                    |
| QEMU/DPDK    |    | DPDK                                                 |
+--------------+    +------------------------------------------------------+

It can be launched together with patch series for the testing:
[0/9] Introduce vfio-user library:
http://patchwork.dpdk.org/cover/85389/
[0/8] Introduce emudev library and iavf emudev driver
http://patchwork.dpdk.org/cover/85488/
[0/6] Introduce iavf backend driver
http://patchwork.dpdk.org/cover/86084/

This series depends on patch serieses:
[0/9] Introduce vfio-user library:
http://patchwork.dpdk.org/cover/85389/

v3:
- Reword commit log
- Add missed patch in v2

v2:
- Enable interrupt for control queue
- Enable interrupt for rx queue
- Rename some Macros
- Fix resource release when close
- Fix ptype_tbl assignment
- Fix typo

Jingjing Wu (5):
common/iavf: emulated pci interfaces on vfio-user client net/iavf_client: introduce iavf driver on vfio-user client net/iavf_client: enable interrupt on control queue net/iavf_client: enable interrupt of Rx queue net/iavf: fix vector mapping with queue

 config/rte_config.h                           |   3 +
 drivers/common/iavf/iavf_common.c             |   7 +
 drivers/common/iavf/iavf_impl.c               |  45 +-
 drivers/common/iavf/iavf_osdep.h              |  14 +
 drivers/common/iavf/iavf_prototype.h          |   1 +
 drivers/common/iavf/iavf_type.h               |   3 +
 drivers/common/iavf/meson.build               |  11 +-
 drivers/common/iavf/version.map               |   6 +
 drivers/common/iavf/vfio_user/vfio_user_pci.c | 525 ++++++++++++++++++
 drivers/common/iavf/vfio_user/vfio_user_pci.h |  68 +++
 drivers/net/iavf/iavf.h                       |  37 +-
 drivers/net/iavf/iavf_client_ethdev.c         | 404 ++++++++++++++
 drivers/net/iavf/iavf_ethdev.c                |  57 +-
 drivers/net/iavf/iavf_rxtx.c                  |  23 +-
 drivers/net/iavf/meson.build                  |   1 +
 15 files changed, 1157 insertions(+), 48 deletions(-)
 create mode 100644 drivers/common/iavf/vfio_user/vfio_user_pci.c
 create mode 100644 drivers/common/iavf/vfio_user/vfio_user_pci.h
 create mode 100644 drivers/net/iavf/iavf_client_ethdev.c
Signature · 1 line
2.21.1
#10 lore raw

PATCH v3 1/5 common/iavf: emulated pci interfaces on vfio-user client

This patch implements Emulated pci interfaces on vfio-user client which is located in common/iavf/vfio_user/.
Four main functions provided to upper layer (driver) to set up or talk to device:
- client_vfio_user_setup
- client_vfio_user_release
- client_vfio_user_get_bar_addr
- client_vfio_user_pci_bar_access

      ----------------------
      | ------------------ |
      | |  iavf driver   | |
      | ------------------ |
      | ------------------ |
      | | device emulate | |------>(common/iavf/vfio_user/)
      | ------------------ |
      ----------------------
                |
                |
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------
---
 config/rte_config.h                           |   3 +
 drivers/common/iavf/iavf_common.c             |   7 +
 drivers/common/iavf/iavf_impl.c               |  45 +-
 drivers/common/iavf/iavf_osdep.h              |  14 +
 drivers/common/iavf/iavf_type.h               |   3 +
 drivers/common/iavf/meson.build               |  11 +-
 drivers/common/iavf/version.map               |   5 +
 drivers/common/iavf/vfio_user/vfio_user_pci.c | 525 ++++++++++++++++++
 drivers/common/iavf/vfio_user/vfio_user_pci.h |  68 +++
 9 files changed, 679 insertions(+), 2 deletions(-)
 create mode 100644 drivers/common/iavf/vfio_user/vfio_user_pci.c
 create mode 100644 drivers/common/iavf/vfio_user/vfio_user_pci.h
config/rte_config.h +3 -0
index a0b5160ff2..76ed1ae83f 100644
@@ -151,4 +151,7 @@
 #define RTE_LIBRTE_PMD_DLB2_SW_CREDIT_QUANTA 32
 #define RTE_PMD_DLB2_DEFAULT_DEPTH_THRESH 256
 
+/* IAVF bsed on vfio_user client */
+#define RTE_LIBRTE_IAVF_CLIENT 1
+
 #endif /* _RTE_CONFIG_H_ */
drivers/common/iavf/iavf_common.c +7 -0
index c951b7d787..2e68f5d08b 100644
@@ -20,6 +20,13 @@ enum iavf_status iavf_set_mac_type(struct iavf_hw *hw)
 
 	DEBUGFUNC("iavf_set_mac_type\n");
 
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user) {
+		hw->mac.type = IAVF_MAC_VF;
+		return status;
+	}
+#endif
+
 	if (hw->vendor_id == IAVF_INTEL_VENDOR_ID) {
 		switch (hw->device_id) {
 		case IAVF_DEV_ID_X722_VF:
drivers/common/iavf/iavf_impl.c +44 -1

57 lines not on this page · raw message

drivers/common/iavf/iavf_osdep.h +14 -0
index 7cba13ff74..025d48a702 100644
@@ -107,8 +107,22 @@ writeq(uint64_t value, volatile void *addr)
 	rte_write64(rte_cpu_to_le_64(value), addr);
 }
 
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+struct iavf_hw;
+
+__rte_internal
+uint32_t iavf_read_addr(struct iavf_hw *hw, uint32_t offset);
+__rte_internal
+void iavf_write_addr(struct iavf_hw *hw, uint32_t offset, uint32_t value);
+
+#define wr32(a, reg, value)  iavf_write_addr((a), (reg), (value))
+#define rd32(a, reg)         iavf_read_addr((a), (reg))
+
+#else
 #define wr32(a, reg, value) writel((value), (a)->hw_addr + (reg))
 #define rd32(a, reg)        readl((a)->hw_addr + (reg))
+#endif /* RTE_LIBRTE_IAVF_CLIENT */
+
 #define wr64(a, reg, value) writeq((value), (a)->hw_addr + (reg))
 #define rd64(a, reg)        readq((a)->hw_addr + (reg))
 
drivers/common/iavf/iavf_type.h +3 -0
index 0990c9aa33..faa4a21eac 100644
@@ -264,6 +264,9 @@ enum iavf_bus_type {
 	iavf_bus_type_pci,
 	iavf_bus_type_pcix,
 	iavf_bus_type_pci_express,
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	iavf_bus_type_vfio_user,
+#endif
 	iavf_bus_type_reserved
 };
 
drivers/common/iavf/meson.build +10 -1
index 1f4d8b898d..e0160ea135 100644
@@ -1,7 +1,16 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2019-2020 Intel Corporation
 
-sources = files('iavf_adminq.c', 'iavf_common.c', 'iavf_impl.c')
+includes += include_directories('vfio_user')
+
+sources = files(
+	'iavf_adminq.c',
+	'iavf_common.c',
+	'iavf_impl.c',
+	'vfio_user/vfio_user_pci.c'
+)
+
+deps += ['vfio_user']
 
 if cc.has_argument('-Wno-pointer-to-int-cast')
         cflags += '-Wno-pointer-to-int-cast'
drivers/common/iavf/version.map +5 -0
index e0f117197c..883d919487 100644
@@ -7,6 +7,11 @@ INTERNAL {
 	iavf_set_mac_type;
 	iavf_shutdown_adminq;
 	iavf_vf_parse_hw_config;
+	client_vfio_user_setup;
+	client_vfio_user_release;
+	client_vfio_user_get_bar_addr;
+	iavf_write_addr;
+	iavf_read_addr;
 
 	local: *;
 };
drivers/common/iavf/vfio_user/vfio_user_pci.c new file +525 -0

525 lines not on this page · raw message

drivers/common/iavf/vfio_user/vfio_user_pci.h new file +68 -0

68 lines not on this page · raw message

Signature · 1 line
2.21.1
#11 lore raw

PATCH v3 2/5 net/iavf_client: introduce iavf driver on vfio-user client

In reply to Jingjing Wu

This patch add a new net driver based on vdev abstraction, i.e. iavf_client_ethdev.c. It is using common iavf functions to talk with Emulated pci interfaces based on vfio-user.

      ----------------------
      | ------------------ |
      | |  iavf driver   | |----> (iavf_client_ethdev.c)
      | ------------------ |
      | ------------------ |
      | | device emulate | |
      | | vfio-user adapt| |
      | ------------------ |
      ----------------------
                |
                |
      ----------------------
      |      vfio-user     |
      |       client       |
      ----------------------
---
 drivers/common/iavf/iavf_prototype.h  |   1 +
 drivers/common/iavf/version.map       |   1 +
 drivers/net/iavf/iavf.h               |  18 +-
 drivers/net/iavf/iavf_client_ethdev.c | 290 ++++++++++++++++++++++++++
 drivers/net/iavf/iavf_ethdev.c        |  26 +--
 drivers/net/iavf/iavf_rxtx.c          |  23 +-
 drivers/net/iavf/meson.build          |   1 +
 7 files changed, 339 insertions(+), 21 deletions(-)
 create mode 100644 drivers/net/iavf/iavf_client_ethdev.c
drivers/common/iavf/iavf_prototype.h +1 -0
index f34e77db0f..3998d26dc0 100644
@@ -83,6 +83,7 @@ void iavf_destroy_spinlock(struct iavf_spinlock *sp);
 __rte_internal
 void iavf_vf_parse_hw_config(struct iavf_hw *hw,
 			     struct virtchnl_vf_resource *msg);
+__rte_internal
 enum iavf_status iavf_vf_reset(struct iavf_hw *hw);
 __rte_internal
 enum iavf_status iavf_aq_send_msg_to_pf(struct iavf_hw *hw,
drivers/common/iavf/version.map +1 -0
index 883d919487..23c3577b3e 100644
@@ -7,6 +7,7 @@ INTERNAL {
 	iavf_set_mac_type;
 	iavf_shutdown_adminq;
 	iavf_vf_parse_hw_config;
+	iavf_vf_reset;
 	client_vfio_user_setup;
 	client_vfio_user_release;
 	client_vfio_user_get_bar_addr;
drivers/net/iavf/iavf.h +17 -1
index 6d5912d8c1..c34f971721 100644
@@ -195,7 +195,10 @@ struct iavf_adapter {
 	struct iavf_hw hw;
 	struct rte_eth_dev *eth_dev;
 	struct iavf_info vf;
-
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	/* used for avf_client driver */
+	struct vfio_device *user_dev;
+#endif
 	bool rx_bulk_alloc_allowed;
 	/* For vector PMD */
 	bool rx_vec_allowed;
@@ -231,6 +234,16 @@ iavf_init_adminq_parameter(struct iavf_hw *hw)
 	hw->aq.asq_buf_size = IAVF_AQ_BUF_SZ;
 }
 
+static inline void
+iavf_disable_irq0(struct iavf_hw *hw)
+{
+	/* Disable all interrupt types */
+	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1, 0);
+	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
+		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+	IAVF_WRITE_FLUSH(hw);
+}
+
 static inline uint16_t
 iavf_calc_itr_interval(int16_t interval)
 {
@@ -284,6 +297,9 @@ _atomic_set_cmd(struct iavf_info *vf, enum virtchnl_ops ops)
 	return !ret;
 }
 
+extern const struct eth_dev_ops iavf_eth_dev_ops;
+
+int iavf_init_vf(struct rte_eth_dev *dev);
 int iavf_check_api_version(struct iavf_adapter *adapter);
 int iavf_get_vf_resource(struct iavf_adapter *adapter);
 void iavf_handle_virtchnl_msg(struct rte_eth_dev *dev);
drivers/net/iavf/iavf_client_ethdev.c new file +290 -0

290 lines not on this page · raw message

drivers/net/iavf/iavf_ethdev.c +8 -18
index 7e3c26a94e..6b5e47adf2 100644
@@ -154,7 +154,7 @@ static const struct rte_iavf_xstats_name_off rte_iavf_stats_strings[] = {
 #define IAVF_NB_XSTATS (sizeof(rte_iavf_stats_strings) / \
 		sizeof(rte_iavf_stats_strings[0]))
 
-static const struct eth_dev_ops iavf_eth_dev_ops = {
+const struct eth_dev_ops iavf_eth_dev_ops = {
 	.dev_configure              = iavf_dev_configure,
 	.dev_start                  = iavf_dev_start,
 	.dev_stop                   = iavf_dev_stop,
@@ -1780,7 +1780,7 @@ iavf_init_proto_xtr(struct rte_eth_dev *dev)
 	}
 }
 
-static int
+int
 iavf_init_vf(struct rte_eth_dev *dev)
 {
 	int err, bufsz;
@@ -1789,12 +1789,6 @@ iavf_init_vf(struct rte_eth_dev *dev)
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private);
 
-	err = iavf_parse_devargs(dev);
-	if (err) {
-		PMD_INIT_LOG(ERR, "Failed to parse devargs");
-		goto err;
-	}
-
 	err = iavf_set_mac_type(hw);
 	if (err) {
 		PMD_INIT_LOG(ERR, "set_mac_type failed: %d", err);
@@ -1891,16 +1885,6 @@ iavf_enable_irq0(struct iavf_hw *hw)
 	IAVF_WRITE_FLUSH(hw);
 }
 
-static inline void
-iavf_disable_irq0(struct iavf_hw *hw)
-{
-	/* Disable all interrupt types */
-	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1, 0);
-	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
-		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
-	IAVF_WRITE_FLUSH(hw);
-}
-
 static void
 iavf_dev_interrupt_handler(void *param)
 {
@@ -1986,6 +1970,12 @@ iavf_dev_init(struct rte_eth_dev *eth_dev)
 	adapter->eth_dev = eth_dev;
 	adapter->stopped = 1;
 
+	ret  = iavf_parse_devargs(eth_dev);
+	if (ret) {
+		PMD_INIT_LOG(ERR, "Failed to parse devargs");
+		return ret;
+	}
+
 	if (iavf_init_vf(eth_dev) != 0) {
 		PMD_INIT_LOG(ERR, "Init vf failed");
 		return -1;
drivers/net/iavf/iavf_rxtx.c +21 -2
index 21d508b3f4..d8192f3675 100644
@@ -24,6 +24,9 @@
 #include <rte_ip.h>
 #include <rte_net.h>
 #include <rte_vect.h>
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+#include <vfio_user/vfio_user_pci.h>
+#endif
 
 #include "iavf.h"
 #include "iavf_rxtx.h"
@@ -595,7 +598,15 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,
 	reset_rx_queue(rxq);
 	rxq->q_set = true;
 	dev->data->rx_queues[queue_idx] = rxq;
-	rxq->qrx_tail = hw->hw_addr + IAVF_QRX_TAIL1(rxq->queue_id);
+
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user)
+		rxq->qrx_tail = client_vfio_user_get_bar_addr(
+				(struct vfio_device *)hw->hw_addr, 0,
+				IAVF_QRX_TAIL1(rxq->queue_id), 4);
+	else
+#endif
+		rxq->qrx_tail = hw->hw_addr + IAVF_QRX_TAIL1(rxq->queue_id);
 	rxq->ops = &def_rxq_ops;
 
 	if (check_rx_bulk_allow(rxq) == true) {
@@ -705,7 +716,15 @@ iavf_dev_tx_queue_setup(struct rte_eth_dev *dev,
 	reset_tx_queue(txq);
 	txq->q_set = true;
 	dev->data->tx_queues[queue_idx] = txq;
-	txq->qtx_tail = hw->hw_addr + IAVF_QTX_TAIL1(queue_idx);
+#ifdef RTE_LIBRTE_IAVF_CLIENT
+	if (hw->bus.type == iavf_bus_type_vfio_user)
+		txq->qtx_tail = client_vfio_user_get_bar_addr(
+			(struct vfio_device *)hw->hw_addr, 0,
+			IAVF_QTX_TAIL1(queue_idx), 4);
+	else
+#endif
+		txq->qtx_tail = hw->hw_addr + IAVF_QTX_TAIL1(queue_idx);
+
 	txq->ops = &def_txq_ops;
 
 	if (check_tx_vec_allow(txq) == false) {
drivers/net/iavf/meson.build +1 -0
index 26c02c4401..580307c462 100644
@@ -13,6 +13,7 @@ sources = files(
 	'iavf_generic_flow.c',
 	'iavf_fdir.c',
 	'iavf_hash.c',
+	'iavf_client_ethdev.c',
 )
 
 if arch_subdir == 'x86'
Signature · 1 line
2.21.1
#12 lore raw

PATCH v3 3/5 net/iavf_client: enable interrupt on control queue

In reply to Jingjing Wu

New devarg "intr": if intr=1, use interrupt mode on control queue

---
 drivers/net/iavf/iavf.h               |  19 ++++
 drivers/net/iavf/iavf_client_ethdev.c | 131 ++++++++++++++++++++++----
 drivers/net/iavf/iavf_ethdev.c        |  18 +---
 3 files changed, 134 insertions(+), 34 deletions(-)
drivers/net/iavf/iavf.h +19 -0
index c34f971721..5516ecf021 100644
@@ -198,6 +198,7 @@ struct iavf_adapter {
 #ifdef RTE_LIBRTE_IAVF_CLIENT
 	/* used for avf_client driver */
 	struct vfio_device *user_dev;
+	int intr_mode; /* interrupt mode if true */
 #endif
 	bool rx_bulk_alloc_allowed;
 	/* For vector PMD */
@@ -234,6 +235,22 @@ iavf_init_adminq_parameter(struct iavf_hw *hw)
 	hw->aq.asq_buf_size = IAVF_AQ_BUF_SZ;
 }
 
+/* Enable default admin queue interrupt setting */
+static inline void
+iavf_enable_irq0(struct iavf_hw *hw)
+{
+	/* Enable admin queue interrupt trigger */
+	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1,
+		       IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
+
+	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
+		       IAVF_VFINT_DYN_CTL01_INTENA_MASK |
+		       IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
+		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
+
+	IAVF_WRITE_FLUSH(hw);
+}
+
 static inline void
 iavf_disable_irq0(struct iavf_hw *hw)
 {
@@ -342,4 +359,6 @@ int iavf_add_del_mc_addr_list(struct iavf_adapter *adapter,
 			uint32_t mc_addrs_num, bool add);
 int iavf_request_queues(struct iavf_adapter *adapter, uint16_t num);
 int iavf_get_max_rss_queue_region(struct iavf_adapter *adapter);
+void iavf_dev_interrupt_handler(void *param);
+
 #endif /* _IAVF_ETHDEV_H_ */
drivers/net/iavf/iavf_client_ethdev.c +114 -17

243 lines not on this page · raw message

drivers/net/iavf/iavf_ethdev.c +1 -17
index 6b5e47adf2..bc07ecbed7 100644
@@ -1869,23 +1869,7 @@ iavf_init_vf(struct rte_eth_dev *dev)
 	return -1;
 }
 
-/* Enable default admin queue interrupt setting */
-static inline void
-iavf_enable_irq0(struct iavf_hw *hw)
-{
-	/* Enable admin queue interrupt trigger */
-	IAVF_WRITE_REG(hw, IAVF_VFINT_ICR0_ENA1,
-		       IAVF_VFINT_ICR0_ENA1_ADMINQ_MASK);
-
-	IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
-		       IAVF_VFINT_DYN_CTL01_INTENA_MASK |
-		       IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
-		       IAVF_VFINT_DYN_CTL01_ITR_INDX_MASK);
-
-	IAVF_WRITE_FLUSH(hw);
-}
-
-static void
+void
 iavf_dev_interrupt_handler(void *param)
 {
 	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
Signature · 1 line
2.21.1
#13 lore raw

PATCH v3 4/5 net/iavf_client: enable interrupt of Rx queue

In reply to Jingjing Wu
---
 drivers/net/iavf/iavf_client_ethdev.c | 29 +++++++++++++++++++++------
 drivers/net/iavf/iavf_ethdev.c        |  9 +++------
 2 files changed, 26 insertions(+), 12 deletions(-)
drivers/net/iavf/iavf_client_ethdev.c +23 -6
index 770a7a8200..27daaed0e8 100644
@@ -59,6 +59,7 @@ iavf_client_vfio_user_setup(struct rte_eth_dev *dev, const char *path)
 		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
 	struct vfio_device *vfio_dev;
+	uint32_t max_fds, i;
 
 	vfio_dev = client_vfio_user_setup(path, dev->device->numa_node);
 	if (vfio_dev == NULL) {
@@ -87,11 +88,21 @@ iavf_client_vfio_user_setup(struct rte_eth_dev *dev, const char *path)
 		}
 
 	}
-
 	dev->intr_handle->fd = vfio_dev->irqfds[0];
 	dev->intr_handle->type = RTE_INTR_HANDLE_VDEV;
 	dev->intr_handle->max_intr = 1;
 
+	/* Assign rxq fds */
+	if (vfio_dev->nb_irqs > 1) {
+		max_fds = RTE_MIN((uint32_t)RTE_MAX_RXTX_INTR_VEC_ID,
+				  vfio_dev->nb_irqs - 1);
+		dev->intr_handle->nb_efd = max_fds;
+		for (i = 0; i < max_fds; ++i)
+			dev->intr_handle->efds[i] =
+				vfio_dev->irqfds[i + RTE_INTR_VEC_RXTX_OFFSET];
+		dev->intr_handle->efd_counter_size = 0;
+		dev->intr_handle->max_intr += max_fds;
+	}
 	return 0;
 }
 
@@ -108,8 +119,6 @@ avf_client_init_eth_ops(void)
 	iavf_client_eth_dev_ops.reta_query                 = NULL;
 	iavf_client_eth_dev_ops.rss_hash_update            = NULL;
 	iavf_client_eth_dev_ops.rss_hash_conf_get          = NULL;
-	iavf_client_eth_dev_ops.rx_queue_intr_enable       = NULL;
-	iavf_client_eth_dev_ops.rx_queue_intr_disable      = NULL;
 }
 
 #define IAVF_CLIENT_ALARM_INTERVAL 50000 /* us */
@@ -227,11 +236,12 @@ iavf_client_dev_close(struct rte_eth_dev *dev)
 	struct iavf_adapter *adapter =
 		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct rte_intr_handle *intr_handle = dev->intr_handle;
 
 	if (adapter->intr_mode) {
 		iavf_disable_irq0(hw);
 		/* unregister callback func from eal lib */
-		rte_intr_callback_unregister(dev->intr_handle,
+		rte_intr_callback_unregister(intr_handle,
 					     iavf_client_event_handler, dev);
 	} else {
 		rte_eal_alarm_cancel(iavf_client_dev_alarm_handler, dev);
@@ -240,8 +250,15 @@ iavf_client_dev_close(struct rte_eth_dev *dev)
 	if (!adapter->stopped) {
 		iavf_stop_queues(dev);
 
-		if (dev->intr_handle) {
-			rte_free(dev->intr_handle);
+		if (intr_handle) {
+			/* Disable the interrupt for Rx */
+			rte_intr_efd_disable(intr_handle);
+			/* Rx interrupt vector mapping free */
+			if (intr_handle->intr_vec) {
+				rte_free(intr_handle->intr_vec);
+				intr_handle->intr_vec = NULL;
+			}
+			rte_free(intr_handle);
 			dev->intr_handle = NULL;
 		}
 
drivers/net/iavf/iavf_ethdev.c +3 -6
index bc07ecbed7..06395f852b 100644
@@ -1368,11 +1368,10 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
 {
 	struct iavf_adapter *adapter =
 		IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private);
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(adapter);
 	uint16_t msix_intr;
 
-	msix_intr = pci_dev->intr_handle.intr_vec[queue_id];
+	msix_intr = dev->intr_handle->intr_vec[queue_id];
 	if (msix_intr == IAVF_MISC_VEC_ID) {
 		PMD_DRV_LOG(INFO, "MISC is also enabled for control");
 		IAVF_WRITE_REG(hw, IAVF_VFINT_DYN_CTL01,
@@ -1387,10 +1386,9 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
 			       IAVF_VFINT_DYN_CTL01_CLEARPBA_MASK |
 			       IAVF_VFINT_DYN_CTLN1_ITR_INDX_MASK);
 	}
-
 	IAVF_WRITE_FLUSH(hw);
 
-	rte_intr_ack(&pci_dev->intr_handle);
+	rte_intr_ack(dev->intr_handle);
 
 	return 0;
 }
@@ -1398,11 +1396,10 @@ iavf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
 static int
 iavf_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id)
 {
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
 	struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private);
 	uint16_t msix_intr;
 
-	msix_intr = pci_dev->intr_handle.intr_vec[queue_id];
+	msix_intr = dev->intr_handle->intr_vec[queue_id];
 	if (msix_intr == IAVF_MISC_VEC_ID) {
 		PMD_DRV_LOG(ERR, "MISC is used for control, cannot disable it");
 		return -EIO;
Signature · 1 line
2.21.1
#14 lore raw

PATCH v3 5/5 net/iavf: fix vector mapping with queue

In reply to Jingjing Wu

Fix the vector mapping with queue by changing the recircle when exceeds RX_VEC_START + nb_msix;

Fixes d6bde6b5eae9 ("net/avf: enable Rx interrupt")
---
 drivers/net/iavf/iavf_ethdev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/iavf/iavf_ethdev.c +2 -2
index 06395f852b..b169b19975 100644
@@ -570,7 +570,7 @@ static int iavf_config_rx_queues_irqs(struct rte_eth_dev *dev,
 			/* If Rx interrupt is reuquired, and we can use
 			 * multi interrupts, then the vec is from 1
 			 */
-			vf->nb_msix = RTE_MIN(vf->vf_res->max_vectors,
+			vf->nb_msix = RTE_MIN(vf->vf_res->max_vectors - 1,
 					      intr_handle->nb_efd);
 			vf->msix_base = IAVF_RX_VEC_START;
 			vec = IAVF_RX_VEC_START;
@@ -578,7 +578,7 @@ static int iavf_config_rx_queues_irqs(struct rte_eth_dev *dev,
 				qv_map[i].queue_id = i;
 				qv_map[i].vector_id = vec;
 				intr_handle->intr_vec[i] = vec++;
-				if (vec >= vf->nb_msix)
+				if (vec >= vf->nb_msix + IAVF_RX_VEC_START)
 					vec = IAVF_RX_VEC_START;
 			}
 			vf->qv_map = qv_map;
Signature · 1 line
2.21.1
#15 lore raw

Re: [dpdk-dev] [PATCH v3 5/5] net/iavf: fix vector mapping with queue

Seems the patch is conflict with patch https://patches.dpdk.org/patch/86202/, please help to review.

Quoted 44 lines

-----Original Message-----
From: Wu, Jingjing <[email protected]>
Sent: Thursday, January 7, 2021 4:27 PM
To: [email protected]
Cc: Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>; Xia, Chenbo <[email protected]>; Lu, Xiuchun <[email protected]>
Subject: [PATCH v3 5/5] net/iavf: fix vector mapping with queue

Fix the vector mapping with queue by changing the recircle when exceeds RX_VEC_START + nb_msix;

Fixes d6bde6b5eae9 ("net/avf: enable Rx interrupt")
---
 drivers/net/iavf/iavf_ethdev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/iavf/iavf_ethdev.c +1 -1
index 06395f852b..b169b19975 100644
@@ -570,7 +570,7 @@ static int iavf_config_rx_queues_irqs(struct
 rte_eth_dev *dev,
 			/* If Rx interrupt is reuquired, and we can use
 			 * multi interrupts, then the vec is from 1
 			 */
-			vf->nb_msix = RTE_MIN(vf->vf_res->max_vectors,
+			vf->nb_msix = RTE_MIN(vf->vf_res->max_vectors - 1,
 					      intr_handle->nb_efd);
 			vf->msix_base = IAVF_RX_VEC_START;
drivers/net/iavf/iavf_ethdev.c +1 -1
 			vec = IAVF_RX_VEC_START;
@@ -578,7 +578,7 @@ static int iavf_config_rx_queues_irqs(struct
 rte_eth_dev *dev,
 				qv_map[i].queue_id = i;
 				qv_map[i].vector_id = vec;
 				intr_handle->intr_vec[i] = vec++;
-				if (vec >= vf->nb_msix)
+				if (vec >= vf->nb_msix + IAVF_RX_VEC_START)
 					vec = IAVF_RX_VEC_START;
 			}

vf->qv_map = qv_map;
--
2.21.1

#16 lore raw

Re: [dpdk-dev] [PATCH v3 5/5] net/iavf: fix vector mapping with queue

Quoted 10 lines

-----Original Message-----
From: Xing, Beilei <[email protected]>
Sent: Tuesday, January 12, 2021 2:11 PM
To: Wu, Jingjing <[email protected]>; [email protected]
Cc: Xia, Chenbo <[email protected]>; Lu, Xiuchun <[email protected]>
Subject: RE: [PATCH v3 5/5] net/iavf: fix vector mapping with queue

Seems the patch is conflict with patch
https://patches.dpdk.org/patch/86202/, please help to review.

Thanks. Looks like the similar fix. Will comment that patch and drop this one if that is merged.

Quoted 48 lines
Quoted 25 lines

-----Original Message-----
From: Wu, Jingjing <[email protected]>
Sent: Thursday, January 7, 2021 4:27 PM
To: [email protected]
Cc: Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>; Xia, Chenbo <[email protected]>; Lu, Xiuchun <[email protected]>
Subject: [PATCH v3 5/5] net/iavf: fix vector mapping with queue

Fix the vector mapping with queue by changing the recircle when exceeds RX_VEC_START + nb_msix;

Fixes d6bde6b5eae9 ("net/avf: enable Rx interrupt")
---
 drivers/net/iavf/iavf_ethdev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/drivers/net/iavf/iavf_ethdev.c index 06395f852b..b169b19975 100644

drivers/net/iavf/iavf_ethdev.c static int iavf_config_rx_queues_irqs(struct · line 574

Quoted 6 lines
drivers/net/iavf/iavf_ethdev.c +1 -1
@@ -570,7 +570,7 @@ static int iavf_config_rx_queues_irqs(struct
 rte_eth_dev *dev,
 			/* If Rx interrupt is reuquired, and we can use
 			 * multi interrupts, then the vec is from 1
 			 */
-			vf->nb_msix = RTE_MIN(vf->vf_res->max_vectors,
+			vf->nb_msix = RTE_MIN(vf->vf_res->max_vectors -

1,

drivers/net/iavf/iavf_ethdev.c static int iavf_config_rx_queues_irqs(struct · line 582

Quoted 10 lines
drivers/net/iavf/iavf_ethdev.c +0 -0
 					      intr_handle->nb_efd);
 			vf->msix_base = IAVF_RX_VEC_START;
drivers/net/iavf/iavf_ethdev.c +1 -1
 			vec = IAVF_RX_VEC_START;
@@ -578,7 +578,7 @@ static int iavf_config_rx_queues_irqs(struct
 rte_eth_dev *dev,
 				qv_map[i].queue_id = i;
 				qv_map[i].vector_id = vec;
 				intr_handle->intr_vec[i] = vec++;
-				if (vec >= vf->nb_msix)
+				if (vec >= vf->nb_msix +

IAVF_RX_VEC_START)

Quoted 5 lines
drivers/net/iavf/iavf_ethdev.c +0 -0
 					vec = IAVF_RX_VEC_START;
 			}

vf->qv_map = qv_map;
--
2.21.1

Tested-by on v3 5/5
#17 lore raw

Re: [dpdk-dev] [PATCH v3 5/5] net/iavf: fix vector mapping with queue

In reply to Jingjing Wu

Regards,
Xie Wei

Quoted 47 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Jingjing Wu
Sent: Thursday, January 7, 2021 4:27 PM
To: [email protected]
Cc: Wu, Jingjing <[email protected]>; Xing, Beilei <[email protected]>; Xia, Chenbo <[email protected]>; Lu, Xiuchun <[email protected]>
Subject: [dpdk-dev] [PATCH v3 5/5] net/iavf: fix vector mapping with queue

Fix the vector mapping with queue by changing the recircle when exceeds RX_VEC_START + nb_msix;

Fixes d6bde6b5eae9 ("net/avf: enable Rx interrupt")
---
 drivers/net/iavf/iavf_ethdev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/iavf/iavf_ethdev.c +1 -1
index 06395f852b..b169b19975 100644
@@ -570,7 +570,7 @@ static int iavf_config_rx_queues_irqs(struct
 rte_eth_dev *dev,
 			/* If Rx interrupt is reuquired, and we can use
 			 * multi interrupts, then the vec is from 1
 			 */
-			vf->nb_msix = RTE_MIN(vf->vf_res->max_vectors,
+			vf->nb_msix = RTE_MIN(vf->vf_res->max_vectors -
 1,
 					      intr_handle->nb_efd);
drivers/net/iavf/iavf_ethdev.c +1 -1
 			vf->msix_base = IAVF_RX_VEC_START;
 			vec = IAVF_RX_VEC_START;
@@ -578,7 +578,7 @@ static int iavf_config_rx_queues_irqs(struct
 rte_eth_dev *dev,
 				qv_map[i].queue_id = i;
 				qv_map[i].vector_id = vec;
 				intr_handle->intr_vec[i] = vec++;
-				if (vec >= vf->nb_msix)
+				if (vec >= vf->nb_msix +
 IAVF_RX_VEC_START)
 					vec = IAVF_RX_VEC_START;
drivers/net/iavf/iavf_ethdev.c +0 -1
 			}
 			vf->qv_map = qv_map;
--

2.21.1