Skip to contentSkip to conversation

net/enic: support VF and fix minor issues

PATCH Superseded Hyong Youb Kim 2 participants 19 messages Aug 8, 2024 – Sep 22, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover net/enic: support VF and fix minor issues

This series contains minor updates for net/enic. The first patch supports SR-IOV VF, which now requires the use of admin channel. The other patches are not related to VF, but included here to ease review.

Hyong Youb Kim (3):
net/enic: support SR-IOV VF using admin channel net/enic: add speed capabilities for newer models net/enic: allow multicast in MAC address add callback

 drivers/net/enic/base/vnic_cq.c       |  27 +
 drivers/net/enic/base/vnic_cq.h       |   3 +
 drivers/net/enic/base/vnic_dev.c      |  48 ++
 drivers/net/enic/base/vnic_dev.h      |   3 +
 drivers/net/enic/base/vnic_devcmd.h   |  49 ++
 drivers/net/enic/base/vnic_resource.h |  32 +-
 drivers/net/enic/base/vnic_rq.c       |  27 +
 drivers/net/enic/base/vnic_rq.h       |   7 +
 drivers/net/enic/base/vnic_wq.c       |  37 +-
 drivers/net/enic/base/vnic_wq.h       |   5 +
 drivers/net/enic/enic.h               |  28 +-
 drivers/net/enic/enic_ethdev.c        |  29 +-
 drivers/net/enic/enic_main.c          |  97 ++--
 drivers/net/enic/enic_res.c           |  12 +
 drivers/net/enic/enic_rxtx.c          |  20 +
 drivers/net/enic/enic_sriov.c         | 804 ++++++++++++++++++++++++++
 drivers/net/enic/enic_sriov.h         | 209 +++++++
 drivers/net/enic/meson.build          |   1 +
 18 files changed, 1394 insertions(+), 44 deletions(-)
 create mode 100644 drivers/net/enic/enic_sriov.c
 create mode 100644 drivers/net/enic/enic_sriov.h
Signature · 1 line
2.35.2
#1 lore raw

PATCH 1/3 net/enic: support SR-IOV VF using admin channel

The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).

The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.

The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.

For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.

Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.

- Backward compatibility mode
Old VIC PF drivers on some operating systems may support only VF_CAPABILITY_REQUEST message or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.

---
 drivers/net/enic/base/vnic_cq.c       |  27 +
 drivers/net/enic/base/vnic_cq.h       |   3 +
 drivers/net/enic/base/vnic_dev.c      |  48 ++
 drivers/net/enic/base/vnic_dev.h      |   3 +
 drivers/net/enic/base/vnic_devcmd.h   |  49 ++
 drivers/net/enic/base/vnic_resource.h |  32 +-
 drivers/net/enic/base/vnic_rq.c       |  27 +
 drivers/net/enic/base/vnic_rq.h       |   7 +
 drivers/net/enic/base/vnic_wq.c       |  37 +-
 drivers/net/enic/base/vnic_wq.h       |   5 +
 drivers/net/enic/enic.h               |  28 +-
 drivers/net/enic/enic_ethdev.c        |   8 +-
 drivers/net/enic/enic_main.c          |  77 ++-
 drivers/net/enic/enic_res.c           |  12 +
 drivers/net/enic/enic_rxtx.c          |  20 +
 drivers/net/enic/enic_sriov.c         | 804 ++++++++++++++++++++++++++
 drivers/net/enic/enic_sriov.h         | 209 +++++++
 drivers/net/enic/meson.build          |   1 +
 18 files changed, 1373 insertions(+), 24 deletions(-)
 create mode 100644 drivers/net/enic/enic_sriov.c
 create mode 100644 drivers/net/enic/enic_sriov.h
drivers/net/enic/base/vnic_cq.c +27 -0
index 51d6fd387b..e966de2f67 100644
@@ -24,6 +24,7 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 
 	cq->index = index;
 	cq->vdev = vdev;
+	cq->admin_chan = false;
 
 	cq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_CQ, index);
 	if (!cq->ctrl) {
@@ -40,6 +41,32 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 	return 0;
 }
 
+int vnic_admin_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
+	unsigned int socket_id, unsigned int desc_count, unsigned int desc_size)
+{
+	int err;
+	char res_name[RTE_MEMZONE_NAMESIZE];
+	static int instance;
+
+	cq->index = index;
+	cq->vdev = vdev;
+	cq->admin_chan = true;
+
+	cq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_ADMIN_CQ, index);
+	if (!cq->ctrl) {
+		pr_err("Failed to get admin CQ[%u] resource\n", index);
+		return -EINVAL;
+	}
+
+	snprintf(res_name, sizeof(res_name), "%d-admin-cq-%u", instance++, index);
+	err = vnic_dev_alloc_desc_ring(vdev, &cq->ring, desc_count, desc_size,
+		socket_id, res_name);
+	if (err)
+		return err;
+
+	return 0;
+}
+
 void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable,
 	unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail,
 	unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_cq.h +3 -0
index 2e48759c49..6469a5c12a 100644
@@ -59,12 +59,15 @@ struct vnic_cq {
 	unsigned int tobe_rx_coal_timeval;
 	ktime_t prev_ts;
 #endif
+	bool admin_chan;
 };
 
 void vnic_cq_free(struct vnic_cq *cq);
 int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 	unsigned int socket_id,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
+	unsigned int socket_id, unsigned int desc_count, unsigned int desc_size);
 void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable,
 	unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail,
 	unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_dev.c +48 -0

81 lines not on this page · raw message

drivers/net/enic/base/vnic_dev.h +3 -0
index 44ca5c739f..0336e07ffc 100644
@@ -199,5 +199,8 @@ int vnic_dev_capable_geneve(struct vnic_dev *vdev);
 uint64_t vnic_dev_capable_cq_entry_size(struct vnic_dev *vdev);
 int vnic_dev_set_cq_entry_size(struct vnic_dev *vdev, uint32_t rq_idx,
 			       uint32_t size_flag);
+int vnic_dev_alloc_sriov_stats_mem(struct vnic_dev *vdev);
+int vnic_dev_sriov_stats(struct vnic_dev *vdev, struct vnic_sriov_stats **stats);
+int vnic_dev_enable_admin_qp(struct vnic_dev *vdev, uint32_t enable);
 
 #endif /* _VNIC_DEV_H_ */
drivers/net/enic/base/vnic_devcmd.h +49 -0

59 lines not on this page · raw message

drivers/net/enic/base/vnic_resource.h +30 -2
index 870a4de6e4..69cc96433e 100644
@@ -38,8 +38,36 @@ enum vnic_res_type {
 	RES_TYPE_MQ_WQ,                 /* MQ Work queues */
 	RES_TYPE_MQ_RQ,                 /* MQ Receive queues */
 	RES_TYPE_MQ_CQ,                 /* MQ Completion queues */
-	RES_TYPE_DEPRECATED1,           /* Old version of devcmd 2 */
-	RES_TYPE_DEVCMD2,               /* Device control region */
+	RES_TYPE_DEPRECATED1,		/* Old version of devcmd 2 */
+	RES_TYPE_DEPRECATED2,		/* Old version of devcmd 2 */
+	RES_TYPE_DEVCMD2,		/* Device control region */
+	RES_TYPE_RDMA_WQ,		/* RDMA WQ */
+	RES_TYPE_RDMA_RQ,		/* RDMA RQ */
+	RES_TYPE_RDMA_CQ,		/* RDMA CQ */
+	RES_TYPE_RDMA_RKEY_TABLE,	/* RDMA RKEY table */
+	RES_TYPE_RDMA_RQ_HEADER_TABLE,	/* RDMA RQ Header Table */
+	RES_TYPE_RDMA_RQ_TABLE,		/* RDMA RQ Table */
+	RES_TYPE_RDMA_RD_RESP_HEADER_TABLE,	/* RDMA Read Response Header Table */
+	RES_TYPE_RDMA_RD_RESP_TABLE,	/* RDMA Read Response Table */
+	RES_TYPE_RDMA_QP_STATS_TABLE,	/* RDMA per QP stats table */
+	RES_TYPE_WQ_MREGS,		/* XXX snic proto only */
+	RES_TYPE_GRPMBR_INTR,		/* Group member interrupt control */
+	RES_TYPE_DPKT,			/* Direct Packet memory region */
+	RES_TYPE_RDMA2_DATA_WQ,		/* RDMA datapath command WQ */
+	RES_TYPE_RDMA2_REG_WQ,		/* RDMA registration command WQ */
+	RES_TYPE_RDMA2_CQ,		/* RDMA datapath CQ */
+	RES_TYPE_MQ_RDMA2_DATA_WQ,	/* RDMA datapath command WQ */
+	RES_TYPE_MQ_RDMA2_REG_WQ,	/* RDMA registration command WQ */
+	RES_TYPE_MQ_RDMA2_CQ,		/* RDMA datapath CQ */
+	RES_TYPE_PTP,			/* PTP registers */
+	RES_TYPE_INTR_CTRL2,		/* Extended INTR CTRL registers */
+	RES_TYPE_SRIOV_INTR,		/* VF intr */
+	RES_TYPE_VF_WQ,			/* VF WQ */
+	RES_TYPE_VF_RQ,			/* VF RQ */
+	RES_TYPE_VF_CQ,			/* VF CQ */
+	RES_TYPE_ADMIN_WQ,		/* admin channel WQ */
+	RES_TYPE_ADMIN_RQ,		/* admin channel RQ */
+	RES_TYPE_ADMIN_CQ,		/* admin channel CQ */
 	RES_TYPE_MAX,			/* Count of resource types */
 };
 
drivers/net/enic/base/vnic_rq.c +27 -0
index 1af96a9413..88def104ad 100644
@@ -27,6 +27,7 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 
 	rq->index = index;
 	rq->vdev = vdev;
+	rq->admin_chan = false;
 
 	rq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_RQ, index);
 	if (!rq->ctrl) {
@@ -42,6 +43,32 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 	return rc;
 }
 
+int vnic_admin_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq,
+	unsigned int desc_count, unsigned int desc_size)
+{
+	int rc;
+	char res_name[RTE_MEMZONE_NAMESIZE];
+	static int instance;
+
+	rq->index = 0;
+	rq->vdev = vdev;
+	rq->admin_chan = true;
+	rq->socket_id = SOCKET_ID_ANY;
+
+	rq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_ADMIN_RQ, 0);
+	if (!rq->ctrl) {
+		pr_err("Failed to get admin RQ resource\n");
+		return -EINVAL;
+	}
+
+	vnic_rq_disable(rq);
+
+	snprintf(res_name, sizeof(res_name), "%d-admin-rq", instance++);
+	rc = vnic_dev_alloc_desc_ring(vdev, &rq->ring, desc_count, desc_size,
+		rq->socket_id, res_name);
+	return rc;
+}
+
 void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_rq.h +7 -0
index cfe65015d5..0f068d2e18 100644
@@ -73,6 +73,11 @@ struct vnic_rq {
 	unsigned int max_mbufs_per_pkt;
 	uint16_t tot_nb_desc;
 	bool need_initial_post;
+	bool admin_chan;
+	const struct rte_memzone *admin_msg_rz;
+	int admin_next_idx;
+	uint64_t soft_stats_pkts;
+	uint64_t soft_stats_bytes;
 };
 
 static inline unsigned int vnic_rq_desc_avail(struct vnic_rq *rq)
@@ -127,6 +132,8 @@ static inline int vnic_rq_fill_count(struct vnic_rq *rq,
 void vnic_rq_free(struct vnic_rq *rq);
 int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq,
+	unsigned int desc_count, unsigned int desc_size);
 void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.c +35 -2
index fc6dde5ae8..04eabefae7 100644
@@ -23,7 +23,8 @@ int vnic_wq_alloc_ring(struct vnic_dev *vdev, struct vnic_wq *wq,
 	char res_name[RTE_MEMZONE_NAMESIZE];
 	static int instance;
 
-	snprintf(res_name, sizeof(res_name), "%d-wq-%u", instance++, wq->index);
+	snprintf(res_name, sizeof(res_name), "%d-%swq-%u",
+		 instance++, wq->admin_chan ? "admin-" : "", wq->index);
 	return vnic_dev_alloc_desc_ring(vdev, &wq->ring, desc_count, desc_size,
 		wq->socket_id, res_name);
 }
@@ -32,7 +33,7 @@ static int vnic_wq_alloc_bufs(struct vnic_wq *wq)
 {
 	unsigned int count = wq->ring.desc_count;
        /* Allocate the mbuf ring */
-	wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket("wq->bufs",
+	wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket(wq->admin_chan ? "admin-wq-bufs" : "wq-bufs",
 		    sizeof(struct rte_mbuf *) * count,
 		    RTE_CACHE_LINE_SIZE, wq->socket_id);
 	wq->head_idx = 0;
@@ -62,6 +63,7 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 
 	wq->index = index;
 	wq->vdev = vdev;
+	wq->admin_chan = false;
 
 	err = vnic_wq_get_ctrl(vdev, wq, index, RES_TYPE_WQ);
 	if (err) {
@@ -84,6 +86,37 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 	return 0;
 }
 
+int vnic_admin_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq,
+	unsigned int desc_count, unsigned int desc_size)
+{
+	int err;
+
+	wq->index = 0;
+	wq->vdev = vdev;
+	wq->admin_chan = true;
+	wq->socket_id = SOCKET_ID_ANY;
+
+	err = vnic_wq_get_ctrl(vdev, wq, 0, RES_TYPE_ADMIN_WQ);
+	if (err) {
+		pr_err("Failed to get admin WQ resource err %d\n", err);
+		return err;
+	}
+
+	vnic_wq_disable(wq);
+
+	err = vnic_wq_alloc_ring(vdev, wq, desc_count, desc_size);
+	if (err)
+		return err;
+
+	err = vnic_wq_alloc_bufs(wq);
+	if (err) {
+		vnic_wq_free(wq);
+		return err;
+	}
+
+	return 0;
+}
+
 void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.h +5 -0
index 789a50a647..381f78431c 100644
@@ -50,6 +50,9 @@ struct vnic_wq {
 	const struct rte_memzone *cqmsg_rz;
 	uint16_t last_completed_index;
 	uint64_t offloads;
+	bool admin_chan;
+	const struct rte_memzone *admin_msg_rz;
+	uint64_t soft_stats_tx;
 };
 
 static inline unsigned int vnic_wq_desc_avail(struct vnic_wq *wq)
@@ -149,6 +152,8 @@ buf_idx_incr(uint32_t n_descriptors, uint32_t idx)
 void vnic_wq_free(struct vnic_wq *wq);
 int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq,
+	unsigned int desc_count, unsigned int desc_size);
 void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/enic.h +26 -2
index f46903ea9e..dc1d16f7d5 100644
@@ -43,7 +43,6 @@
 /*#define ENIC_DESC_COUNT_MAKE_ODD (x) do{if ((~(x)) & 1) { (x)--; } }while(0)*/
 
 #define PCI_DEVICE_ID_CISCO_VIC_ENET         0x0043  /* ethernet vnic */
-#define PCI_DEVICE_ID_CISCO_VIC_ENET_VF      0x0071  /* enet SRIOV VF */
 /* enet SRIOV Standalone vNic VF */
 #define PCI_DEVICE_ID_CISCO_VIC_ENET_SN      0x02B7
 
@@ -51,7 +50,7 @@
 #define ENIC_MAGIC_FILTER_ID 0xffff
 
 /*
- * Interrupt 0: LSC and errors
+ * Interrupt 0: LSC and errors / VF admin channel RQ
  * Interrupt 1: rx queue 0
  * Interrupt 2: rx queue 1
  * ...
@@ -154,11 +153,26 @@ struct enic {
 	/* software counters */
 	struct enic_soft_stats soft_stats;
 
+	struct vnic_wq admin_wq;
+	struct vnic_rq admin_rq;
+	struct vnic_cq admin_cq[2];
+
 	/* configured resources on vic */
 	unsigned int conf_rq_count;
 	unsigned int conf_wq_count;
 	unsigned int conf_cq_count;
 	unsigned int conf_intr_count;
+	/* SR-IOV VF has queues for admin channel to PF */
+	unsigned int conf_admin_rq_count;
+	unsigned int conf_admin_wq_count;
+	unsigned int conf_admin_cq_count;
+	uint64_t admin_chan_msg_num;
+	int admin_chan_vf_id;
+	uint32_t admin_pf_cap_version;
+	bool admin_chan_enabled;
+	bool sriov_vf_soft_rx_stats;
+	bool sriov_vf_compat_mode;
+	pthread_mutex_t admin_chan_lock;
 
 	/* linked list storing memory allocations */
 	LIST_HEAD(enic_memzone_list, enic_memzone_entry) memzone_list;
@@ -230,6 +244,16 @@ struct enic_vf_representor {
 	struct rte_flow *rep2vf_flow[2];
 };
 
+#define ENIC_ADMIN_WQ_CQ 0
+#define ENIC_ADMIN_RQ_CQ 1
+#define ENIC_ADMIN_BUF_SIZE 1024
+
+static inline bool enic_is_vf(struct enic *enic)
+{
+	return enic->pdev->id.device_id == PCI_DEVICE_ID_CISCO_VIC_ENET_SN &&
+		!enic->sriov_vf_compat_mode;
+}
+
 #define VF_ENIC_TO_VF_REP(vf_enic) \
 	container_of(vf_enic, struct enic_vf_representor, enic)
 
drivers/net/enic/enic_ethdev.c +4 -4
index cad8db2f6f..5c967677fb 100644
@@ -21,6 +21,7 @@
 #include "vnic_rq.h"
 #include "vnic_enet.h"
 #include "enic.h"
+#include "enic_sriov.h"
 
 /*
  * The set of PCI devices this driver supports
@@ -28,7 +29,6 @@
 #define CISCO_PCI_VENDOR_ID 0x1137
 static const struct rte_pci_id pci_id_enic_map[] = {
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET)},
-	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET_VF)},
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET_SN)},
 	{.vendor_id = 0, /* sentinel */},
 };
@@ -707,7 +707,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		for (i = 0; i < enic->mc_count; i++) {
 			addr = &enic->mc_addrs[i];
 			debug_log_add_del_addr(addr, false);
-			ret = vnic_dev_del_addr(enic->vdev, addr->addr_bytes);
+			ret = enic_dev_del_addr(enic, addr->addr_bytes);
 			if (ret)
 				return ret;
 		}
@@ -734,7 +734,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		if (j < nb_mc_addr)
 			continue;
 		debug_log_add_del_addr(addr, false);
-		ret = vnic_dev_del_addr(enic->vdev, addr->addr_bytes);
+		ret = enic_dev_del_addr(enic, addr->addr_bytes);
 		if (ret)
 			return ret;
 	}
@@ -748,7 +748,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		if (j < enic->mc_count)
 			continue;
 		debug_log_add_del_addr(addr, true);
-		ret = vnic_dev_add_addr(enic->vdev, addr->addr_bytes);
+		ret = enic_dev_add_addr(enic, addr->addr_bytes);
 		if (ret)
 			return ret;
 	}
drivers/net/enic/enic_main.c +63 -14

191 lines not on this page · raw message

drivers/net/enic/enic_res.c +12 -0
index caf773bab2..f09f4c0abb 100644
@@ -300,4 +300,16 @@ void enic_get_res_counts(struct enic *enic)
 		"vNIC resources avail: wq %d rq %d cq %d intr %d\n",
 		enic->conf_wq_count, enic->conf_rq_count,
 		enic->conf_cq_count, enic->conf_intr_count);
+
+	if (!enic_is_vf(enic))
+		return;
+
+	enic->conf_admin_wq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_WQ);
+	enic->conf_admin_rq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_RQ);
+	enic->conf_admin_cq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_CQ);
+
+	dev_info(enic_get_dev(enic),
+		"vNIC admin channel resources avail: wq %d rq %d cq %d\n",
+		enic->conf_admin_wq_count, enic->conf_admin_rq_count,
+		enic->conf_admin_cq_count);
 }
drivers/net/enic/enic_rxtx.c +20 -0
index 7a66d72275..549a153332 100644
@@ -55,6 +55,7 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 		sizeof(struct cq_enet_rq_desc_64) :
 		sizeof(struct cq_enet_rq_desc);
 	RTE_BUILD_BUG_ON(sizeof(struct cq_enet_rq_desc_64) != 64);
+	uint64_t bytes;
 
 	cq = &enic->cq[enic_cq_rq(enic, sop_rq->index)];
 	cq_idx = cq->to_clean;		/* index of cqd, rqd, mbuf_table */
@@ -67,6 +68,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 	/* Receive until the end of the ring, at most. */
 	max_rx = RTE_MIN(nb_pkts, cq->ring.desc_count - cq_idx);
 
+	bytes = 0;
+
 	while (max_rx) {
 		volatile struct rq_enet_desc *rqd_ptr;
 		struct cq_desc cqd;
@@ -155,6 +158,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 		rxmb->port = enic->port_id;
 		rxmb->data_len = seg_length;
 
+		bytes += seg_length;
+
 		rq->rx_nb_hold++;
 
 		if (!(enic_cq_rx_desc_eop(ciflags))) {
@@ -225,6 +230,10 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 				  &sop_rq->ctrl->posted_index);
 	}
 
+	if (enic->sriov_vf_soft_rx_stats && bytes) {
+		sop_rq->soft_stats_pkts += nb_rx;
+		sop_rq->soft_stats_bytes += bytes;
+	}
 
 	return nb_rx;
 }
@@ -256,6 +265,7 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	uint8_t color;
 	bool overlay;
 	bool tnl;
+	uint64_t bytes;
 
 	rq = rx_queue;
 	enic = vnic_dev_priv(rq->vdev);
@@ -283,6 +293,8 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	port_id = enic->port_id;
 	overlay = enic->overlay_offload;
 
+	bytes = 0;
+
 	rx = rx_pkts;
 	while (max_rx) {
 		max_rx--;
@@ -304,6 +316,9 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 			CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK;
 		mb->pkt_len = mb->data_len;
 		mb->port = port_id;
+
+		bytes += mb->pkt_len;
+
 		tnl = overlay && (cqd->completed_index_flags &
 				  CQ_ENET_RQ_DESC_FLAGS_FCOE) != 0;
 		mb->packet_type =
@@ -352,6 +367,11 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 				  &rq->ctrl->posted_index);
 	}
 
+	if (enic->sriov_vf_soft_rx_stats && bytes) {
+		rq->soft_stats_pkts += (rx - rx_pkts);
+		rq->soft_stats_bytes += bytes;
+	}
+
 	return rx - rx_pkts;
 }
 
drivers/net/enic/enic_sriov.c new file +804 -0

804 lines not on this page · raw message

drivers/net/enic/enic_sriov.h new file +209 -0

209 lines not on this page · raw message

drivers/net/enic/meson.build +1 -0
index 8700ae27f2..00f8348348 100644
@@ -23,6 +23,7 @@ sources = files(
         'enic_main.c',
         'enic_res.c',
         'enic_rxtx.c',
+        'enic_sriov.c',
         'enic_vf_representor.c',
 )
 deps += ['hash']
Signature · 1 line
2.35.2
#2 lore raw

PATCH 2/3 net/enic: add speed capabilities for newer models

In reply to Hyong Youb Kim

Add 1400/14000 and 15000 models to the speed_capa list.

---
 drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
drivers/net/enic/enic_ethdev.c +21 -0
index 5c967677fb..62c8751d09 100644
@@ -62,6 +62,27 @@ static const struct vic_speed_capa {
 	{ 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */
 	{ 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1495 PCIe */
 	{ 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1497 MLOM */
+	{ 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G }, /* 1467 MLOM */
+	{ 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1477 MLOM */
+	{ 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */
+	{ 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */
+	{ 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */
+	{ 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */
+	{ 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */
+	{ 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */
+	{ 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */
+	{ 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */
+	{ 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G }, /* 15230 MLOM */
+	{ 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G }, /* 15422 Mezz */
+	{ 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */
+	{ 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */
+	{ 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */
 	{ 0, 0 }, /* End marker */
 };
 
Signature · 1 line
2.35.2
#3 lore raw

PATCH 3/3 net/enic: allow multicast in MAC address add callback

In reply to Hyong Youb Kim

enic_set_mac_address() (mac_addr_add callback) currently allows only non-zero, unicast address to be added. It is overly restrictive. rte_eth_dev_mac_addr_add() itself allows multicast addresses. And, some applications do use rte_eth_dev_mac_addr_add() to accept multicast addresses. So, remove the unicast check in enic_set_mac_address().

---
 drivers/net/enic/enic_main.c | 20 --------------------
 1 file changed, 20 deletions(-)
drivers/net/enic/enic_main.c +0 -20
index e59b0a5077..b755b15d92 100644
@@ -32,21 +32,6 @@
 #include "vnic_intr.h"
 #include "vnic_nic.h"
 
-static int is_zero_addr(uint8_t *addr)
-{
-	return !(addr[0] |  addr[1] | addr[2] | addr[3] | addr[4] | addr[5]);
-}
-
-static int is_mcast_addr(uint8_t *addr)
-{
-	return addr[0] & 1;
-}
-
-static int is_eth_addr_valid(uint8_t *addr)
-{
-	return !is_mcast_addr(addr) && !is_zero_addr(addr);
-}
-
 void
 enic_rxmbuf_queue_release(__rte_unused struct enic *enic, struct vnic_rq *rq)
 {
@@ -177,11 +162,6 @@ int enic_set_mac_address(struct enic *enic, uint8_t *mac_addr)
 {
 	int err;
 
-	if (!is_eth_addr_valid(mac_addr)) {
-		dev_err(enic, "invalid mac address\n");
-		return -EINVAL;
-	}
-
 	err = enic_dev_add_addr(enic, mac_addr);
 	if (err)
 		dev_err(enic, "add mac addr failed\n");
Signature · 1 line
2.35.2
#4 lore raw

v2 cover net/enic: support VF and fix minor issues

This series contains minor updates for net/enic. The first patch supports SR-IOV VF, which now requires the use of admin channel. The other patches are not related to VF, but included here to ease review.

---

v2:
* fix compiler warnings

Hyong Youb Kim (3):
net/enic: support SR-IOV VF using admin channel net/enic: add speed capabilities for newer models net/enic: allow multicast in MAC address add callback

 drivers/net/enic/base/vnic_cq.c       |  27 +
 drivers/net/enic/base/vnic_cq.h       |   3 +
 drivers/net/enic/base/vnic_dev.c      |  48 ++
 drivers/net/enic/base/vnic_dev.h      |   3 +
 drivers/net/enic/base/vnic_devcmd.h   |  49 ++
 drivers/net/enic/base/vnic_resource.h |  32 +-
 drivers/net/enic/base/vnic_rq.c       |  27 +
 drivers/net/enic/base/vnic_rq.h       |   7 +
 drivers/net/enic/base/vnic_wq.c       |  37 +-
 drivers/net/enic/base/vnic_wq.h       |   5 +
 drivers/net/enic/enic.h               |  28 +-
 drivers/net/enic/enic_ethdev.c        |  29 +-
 drivers/net/enic/enic_main.c          |  97 ++--
 drivers/net/enic/enic_res.c           |  12 +
 drivers/net/enic/enic_rxtx.c          |  20 +
 drivers/net/enic/enic_sriov.c         | 801 ++++++++++++++++++++++++++
 drivers/net/enic/enic_sriov.h         | 209 +++++++
 drivers/net/enic/meson.build          |   1 +
 18 files changed, 1391 insertions(+), 44 deletions(-)
 create mode 100644 drivers/net/enic/enic_sriov.c
 create mode 100644 drivers/net/enic/enic_sriov.h
Signature · 1 line
2.35.2
#5 lore raw

PATCH v2 1/3 net/enic: support SR-IOV VF using admin channel

The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).

The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.

The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.

For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.

Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.

- Backward compatibility mode
Old VIC PF drivers on some operating systems may support only VF_CAPABILITY_REQUEST message or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.

---
 drivers/net/enic/base/vnic_cq.c       |  27 +
 drivers/net/enic/base/vnic_cq.h       |   3 +
 drivers/net/enic/base/vnic_dev.c      |  48 ++
 drivers/net/enic/base/vnic_dev.h      |   3 +
 drivers/net/enic/base/vnic_devcmd.h   |  49 ++
 drivers/net/enic/base/vnic_resource.h |  32 +-
 drivers/net/enic/base/vnic_rq.c       |  27 +
 drivers/net/enic/base/vnic_rq.h       |   7 +
 drivers/net/enic/base/vnic_wq.c       |  37 +-
 drivers/net/enic/base/vnic_wq.h       |   5 +
 drivers/net/enic/enic.h               |  28 +-
 drivers/net/enic/enic_ethdev.c        |   8 +-
 drivers/net/enic/enic_main.c          |  77 ++-
 drivers/net/enic/enic_res.c           |  12 +
 drivers/net/enic/enic_rxtx.c          |  20 +
 drivers/net/enic/enic_sriov.c         | 801 ++++++++++++++++++++++++++
 drivers/net/enic/enic_sriov.h         | 209 +++++++
 drivers/net/enic/meson.build          |   1 +
 18 files changed, 1370 insertions(+), 24 deletions(-)
 create mode 100644 drivers/net/enic/enic_sriov.c
 create mode 100644 drivers/net/enic/enic_sriov.h
drivers/net/enic/base/vnic_cq.c +27 -0
index 51d6fd387b..e966de2f67 100644
@@ -24,6 +24,7 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 
 	cq->index = index;
 	cq->vdev = vdev;
+	cq->admin_chan = false;
 
 	cq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_CQ, index);
 	if (!cq->ctrl) {
@@ -40,6 +41,32 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 	return 0;
 }
 
+int vnic_admin_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
+	unsigned int socket_id, unsigned int desc_count, unsigned int desc_size)
+{
+	int err;
+	char res_name[RTE_MEMZONE_NAMESIZE];
+	static int instance;
+
+	cq->index = index;
+	cq->vdev = vdev;
+	cq->admin_chan = true;
+
+	cq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_ADMIN_CQ, index);
+	if (!cq->ctrl) {
+		pr_err("Failed to get admin CQ[%u] resource\n", index);
+		return -EINVAL;
+	}
+
+	snprintf(res_name, sizeof(res_name), "%d-admin-cq-%u", instance++, index);
+	err = vnic_dev_alloc_desc_ring(vdev, &cq->ring, desc_count, desc_size,
+		socket_id, res_name);
+	if (err)
+		return err;
+
+	return 0;
+}
+
 void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable,
 	unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail,
 	unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_cq.h +3 -0
index 2e48759c49..6469a5c12a 100644
@@ -59,12 +59,15 @@ struct vnic_cq {
 	unsigned int tobe_rx_coal_timeval;
 	ktime_t prev_ts;
 #endif
+	bool admin_chan;
 };
 
 void vnic_cq_free(struct vnic_cq *cq);
 int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 	unsigned int socket_id,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
+	unsigned int socket_id, unsigned int desc_count, unsigned int desc_size);
 void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable,
 	unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail,
 	unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_dev.c +48 -0

81 lines not on this page · raw message

drivers/net/enic/base/vnic_dev.h +3 -0
index 44ca5c739f..0336e07ffc 100644
@@ -199,5 +199,8 @@ int vnic_dev_capable_geneve(struct vnic_dev *vdev);
 uint64_t vnic_dev_capable_cq_entry_size(struct vnic_dev *vdev);
 int vnic_dev_set_cq_entry_size(struct vnic_dev *vdev, uint32_t rq_idx,
 			       uint32_t size_flag);
+int vnic_dev_alloc_sriov_stats_mem(struct vnic_dev *vdev);
+int vnic_dev_sriov_stats(struct vnic_dev *vdev, struct vnic_sriov_stats **stats);
+int vnic_dev_enable_admin_qp(struct vnic_dev *vdev, uint32_t enable);
 
 #endif /* _VNIC_DEV_H_ */
drivers/net/enic/base/vnic_devcmd.h +49 -0

59 lines not on this page · raw message

drivers/net/enic/base/vnic_resource.h +30 -2
index 870a4de6e4..69cc96433e 100644
@@ -38,8 +38,36 @@ enum vnic_res_type {
 	RES_TYPE_MQ_WQ,                 /* MQ Work queues */
 	RES_TYPE_MQ_RQ,                 /* MQ Receive queues */
 	RES_TYPE_MQ_CQ,                 /* MQ Completion queues */
-	RES_TYPE_DEPRECATED1,           /* Old version of devcmd 2 */
-	RES_TYPE_DEVCMD2,               /* Device control region */
+	RES_TYPE_DEPRECATED1,		/* Old version of devcmd 2 */
+	RES_TYPE_DEPRECATED2,		/* Old version of devcmd 2 */
+	RES_TYPE_DEVCMD2,		/* Device control region */
+	RES_TYPE_RDMA_WQ,		/* RDMA WQ */
+	RES_TYPE_RDMA_RQ,		/* RDMA RQ */
+	RES_TYPE_RDMA_CQ,		/* RDMA CQ */
+	RES_TYPE_RDMA_RKEY_TABLE,	/* RDMA RKEY table */
+	RES_TYPE_RDMA_RQ_HEADER_TABLE,	/* RDMA RQ Header Table */
+	RES_TYPE_RDMA_RQ_TABLE,		/* RDMA RQ Table */
+	RES_TYPE_RDMA_RD_RESP_HEADER_TABLE,	/* RDMA Read Response Header Table */
+	RES_TYPE_RDMA_RD_RESP_TABLE,	/* RDMA Read Response Table */
+	RES_TYPE_RDMA_QP_STATS_TABLE,	/* RDMA per QP stats table */
+	RES_TYPE_WQ_MREGS,		/* XXX snic proto only */
+	RES_TYPE_GRPMBR_INTR,		/* Group member interrupt control */
+	RES_TYPE_DPKT,			/* Direct Packet memory region */
+	RES_TYPE_RDMA2_DATA_WQ,		/* RDMA datapath command WQ */
+	RES_TYPE_RDMA2_REG_WQ,		/* RDMA registration command WQ */
+	RES_TYPE_RDMA2_CQ,		/* RDMA datapath CQ */
+	RES_TYPE_MQ_RDMA2_DATA_WQ,	/* RDMA datapath command WQ */
+	RES_TYPE_MQ_RDMA2_REG_WQ,	/* RDMA registration command WQ */
+	RES_TYPE_MQ_RDMA2_CQ,		/* RDMA datapath CQ */
+	RES_TYPE_PTP,			/* PTP registers */
+	RES_TYPE_INTR_CTRL2,		/* Extended INTR CTRL registers */
+	RES_TYPE_SRIOV_INTR,		/* VF intr */
+	RES_TYPE_VF_WQ,			/* VF WQ */
+	RES_TYPE_VF_RQ,			/* VF RQ */
+	RES_TYPE_VF_CQ,			/* VF CQ */
+	RES_TYPE_ADMIN_WQ,		/* admin channel WQ */
+	RES_TYPE_ADMIN_RQ,		/* admin channel RQ */
+	RES_TYPE_ADMIN_CQ,		/* admin channel CQ */
 	RES_TYPE_MAX,			/* Count of resource types */
 };
 
drivers/net/enic/base/vnic_rq.c +27 -0
index 1af96a9413..88def104ad 100644
@@ -27,6 +27,7 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 
 	rq->index = index;
 	rq->vdev = vdev;
+	rq->admin_chan = false;
 
 	rq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_RQ, index);
 	if (!rq->ctrl) {
@@ -42,6 +43,32 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 	return rc;
 }
 
+int vnic_admin_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq,
+	unsigned int desc_count, unsigned int desc_size)
+{
+	int rc;
+	char res_name[RTE_MEMZONE_NAMESIZE];
+	static int instance;
+
+	rq->index = 0;
+	rq->vdev = vdev;
+	rq->admin_chan = true;
+	rq->socket_id = SOCKET_ID_ANY;
+
+	rq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_ADMIN_RQ, 0);
+	if (!rq->ctrl) {
+		pr_err("Failed to get admin RQ resource\n");
+		return -EINVAL;
+	}
+
+	vnic_rq_disable(rq);
+
+	snprintf(res_name, sizeof(res_name), "%d-admin-rq", instance++);
+	rc = vnic_dev_alloc_desc_ring(vdev, &rq->ring, desc_count, desc_size,
+		rq->socket_id, res_name);
+	return rc;
+}
+
 void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_rq.h +7 -0
index cfe65015d5..0f068d2e18 100644
@@ -73,6 +73,11 @@ struct vnic_rq {
 	unsigned int max_mbufs_per_pkt;
 	uint16_t tot_nb_desc;
 	bool need_initial_post;
+	bool admin_chan;
+	const struct rte_memzone *admin_msg_rz;
+	int admin_next_idx;
+	uint64_t soft_stats_pkts;
+	uint64_t soft_stats_bytes;
 };
 
 static inline unsigned int vnic_rq_desc_avail(struct vnic_rq *rq)
@@ -127,6 +132,8 @@ static inline int vnic_rq_fill_count(struct vnic_rq *rq,
 void vnic_rq_free(struct vnic_rq *rq);
 int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq,
+	unsigned int desc_count, unsigned int desc_size);
 void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.c +35 -2
index fc6dde5ae8..04eabefae7 100644
@@ -23,7 +23,8 @@ int vnic_wq_alloc_ring(struct vnic_dev *vdev, struct vnic_wq *wq,
 	char res_name[RTE_MEMZONE_NAMESIZE];
 	static int instance;
 
-	snprintf(res_name, sizeof(res_name), "%d-wq-%u", instance++, wq->index);
+	snprintf(res_name, sizeof(res_name), "%d-%swq-%u",
+		 instance++, wq->admin_chan ? "admin-" : "", wq->index);
 	return vnic_dev_alloc_desc_ring(vdev, &wq->ring, desc_count, desc_size,
 		wq->socket_id, res_name);
 }
@@ -32,7 +33,7 @@ static int vnic_wq_alloc_bufs(struct vnic_wq *wq)
 {
 	unsigned int count = wq->ring.desc_count;
        /* Allocate the mbuf ring */
-	wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket("wq->bufs",
+	wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket(wq->admin_chan ? "admin-wq-bufs" : "wq-bufs",
 		    sizeof(struct rte_mbuf *) * count,
 		    RTE_CACHE_LINE_SIZE, wq->socket_id);
 	wq->head_idx = 0;
@@ -62,6 +63,7 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 
 	wq->index = index;
 	wq->vdev = vdev;
+	wq->admin_chan = false;
 
 	err = vnic_wq_get_ctrl(vdev, wq, index, RES_TYPE_WQ);
 	if (err) {
@@ -84,6 +86,37 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 	return 0;
 }
 
+int vnic_admin_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq,
+	unsigned int desc_count, unsigned int desc_size)
+{
+	int err;
+
+	wq->index = 0;
+	wq->vdev = vdev;
+	wq->admin_chan = true;
+	wq->socket_id = SOCKET_ID_ANY;
+
+	err = vnic_wq_get_ctrl(vdev, wq, 0, RES_TYPE_ADMIN_WQ);
+	if (err) {
+		pr_err("Failed to get admin WQ resource err %d\n", err);
+		return err;
+	}
+
+	vnic_wq_disable(wq);
+
+	err = vnic_wq_alloc_ring(vdev, wq, desc_count, desc_size);
+	if (err)
+		return err;
+
+	err = vnic_wq_alloc_bufs(wq);
+	if (err) {
+		vnic_wq_free(wq);
+		return err;
+	}
+
+	return 0;
+}
+
 void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.h +5 -0
index 789a50a647..381f78431c 100644
@@ -50,6 +50,9 @@ struct vnic_wq {
 	const struct rte_memzone *cqmsg_rz;
 	uint16_t last_completed_index;
 	uint64_t offloads;
+	bool admin_chan;
+	const struct rte_memzone *admin_msg_rz;
+	uint64_t soft_stats_tx;
 };
 
 static inline unsigned int vnic_wq_desc_avail(struct vnic_wq *wq)
@@ -149,6 +152,8 @@ buf_idx_incr(uint32_t n_descriptors, uint32_t idx)
 void vnic_wq_free(struct vnic_wq *wq);
 int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq,
+	unsigned int desc_count, unsigned int desc_size);
 void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/enic.h +26 -2
index f46903ea9e..dc1d16f7d5 100644
@@ -43,7 +43,6 @@
 /*#define ENIC_DESC_COUNT_MAKE_ODD (x) do{if ((~(x)) & 1) { (x)--; } }while(0)*/
 
 #define PCI_DEVICE_ID_CISCO_VIC_ENET         0x0043  /* ethernet vnic */
-#define PCI_DEVICE_ID_CISCO_VIC_ENET_VF      0x0071  /* enet SRIOV VF */
 /* enet SRIOV Standalone vNic VF */
 #define PCI_DEVICE_ID_CISCO_VIC_ENET_SN      0x02B7
 
@@ -51,7 +50,7 @@
 #define ENIC_MAGIC_FILTER_ID 0xffff
 
 /*
- * Interrupt 0: LSC and errors
+ * Interrupt 0: LSC and errors / VF admin channel RQ
  * Interrupt 1: rx queue 0
  * Interrupt 2: rx queue 1
  * ...
@@ -154,11 +153,26 @@ struct enic {
 	/* software counters */
 	struct enic_soft_stats soft_stats;
 
+	struct vnic_wq admin_wq;
+	struct vnic_rq admin_rq;
+	struct vnic_cq admin_cq[2];
+
 	/* configured resources on vic */
 	unsigned int conf_rq_count;
 	unsigned int conf_wq_count;
 	unsigned int conf_cq_count;
 	unsigned int conf_intr_count;
+	/* SR-IOV VF has queues for admin channel to PF */
+	unsigned int conf_admin_rq_count;
+	unsigned int conf_admin_wq_count;
+	unsigned int conf_admin_cq_count;
+	uint64_t admin_chan_msg_num;
+	int admin_chan_vf_id;
+	uint32_t admin_pf_cap_version;
+	bool admin_chan_enabled;
+	bool sriov_vf_soft_rx_stats;
+	bool sriov_vf_compat_mode;
+	pthread_mutex_t admin_chan_lock;
 
 	/* linked list storing memory allocations */
 	LIST_HEAD(enic_memzone_list, enic_memzone_entry) memzone_list;
@@ -230,6 +244,16 @@ struct enic_vf_representor {
 	struct rte_flow *rep2vf_flow[2];
 };
 
+#define ENIC_ADMIN_WQ_CQ 0
+#define ENIC_ADMIN_RQ_CQ 1
+#define ENIC_ADMIN_BUF_SIZE 1024
+
+static inline bool enic_is_vf(struct enic *enic)
+{
+	return enic->pdev->id.device_id == PCI_DEVICE_ID_CISCO_VIC_ENET_SN &&
+		!enic->sriov_vf_compat_mode;
+}
+
 #define VF_ENIC_TO_VF_REP(vf_enic) \
 	container_of(vf_enic, struct enic_vf_representor, enic)
 
drivers/net/enic/enic_ethdev.c +4 -4
index cad8db2f6f..5c967677fb 100644
@@ -21,6 +21,7 @@
 #include "vnic_rq.h"
 #include "vnic_enet.h"
 #include "enic.h"
+#include "enic_sriov.h"
 
 /*
  * The set of PCI devices this driver supports
@@ -28,7 +29,6 @@
 #define CISCO_PCI_VENDOR_ID 0x1137
 static const struct rte_pci_id pci_id_enic_map[] = {
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET)},
-	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET_VF)},
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET_SN)},
 	{.vendor_id = 0, /* sentinel */},
 };
@@ -707,7 +707,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		for (i = 0; i < enic->mc_count; i++) {
 			addr = &enic->mc_addrs[i];
 			debug_log_add_del_addr(addr, false);
-			ret = vnic_dev_del_addr(enic->vdev, addr->addr_bytes);
+			ret = enic_dev_del_addr(enic, addr->addr_bytes);
 			if (ret)
 				return ret;
 		}
@@ -734,7 +734,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		if (j < nb_mc_addr)
 			continue;
 		debug_log_add_del_addr(addr, false);
-		ret = vnic_dev_del_addr(enic->vdev, addr->addr_bytes);
+		ret = enic_dev_del_addr(enic, addr->addr_bytes);
 		if (ret)
 			return ret;
 	}
@@ -748,7 +748,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		if (j < enic->mc_count)
 			continue;
 		debug_log_add_del_addr(addr, true);
-		ret = vnic_dev_add_addr(enic->vdev, addr->addr_bytes);
+		ret = enic_dev_add_addr(enic, addr->addr_bytes);
 		if (ret)
 			return ret;
 	}
drivers/net/enic/enic_main.c +63 -14

191 lines not on this page · raw message

drivers/net/enic/enic_res.c +12 -0
index caf773bab2..f09f4c0abb 100644
@@ -300,4 +300,16 @@ void enic_get_res_counts(struct enic *enic)
 		"vNIC resources avail: wq %d rq %d cq %d intr %d\n",
 		enic->conf_wq_count, enic->conf_rq_count,
 		enic->conf_cq_count, enic->conf_intr_count);
+
+	if (!enic_is_vf(enic))
+		return;
+
+	enic->conf_admin_wq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_WQ);
+	enic->conf_admin_rq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_RQ);
+	enic->conf_admin_cq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_CQ);
+
+	dev_info(enic_get_dev(enic),
+		"vNIC admin channel resources avail: wq %d rq %d cq %d\n",
+		enic->conf_admin_wq_count, enic->conf_admin_rq_count,
+		enic->conf_admin_cq_count);
 }
drivers/net/enic/enic_rxtx.c +20 -0
index 7a66d72275..549a153332 100644
@@ -55,6 +55,7 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 		sizeof(struct cq_enet_rq_desc_64) :
 		sizeof(struct cq_enet_rq_desc);
 	RTE_BUILD_BUG_ON(sizeof(struct cq_enet_rq_desc_64) != 64);
+	uint64_t bytes;
 
 	cq = &enic->cq[enic_cq_rq(enic, sop_rq->index)];
 	cq_idx = cq->to_clean;		/* index of cqd, rqd, mbuf_table */
@@ -67,6 +68,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 	/* Receive until the end of the ring, at most. */
 	max_rx = RTE_MIN(nb_pkts, cq->ring.desc_count - cq_idx);
 
+	bytes = 0;
+
 	while (max_rx) {
 		volatile struct rq_enet_desc *rqd_ptr;
 		struct cq_desc cqd;
@@ -155,6 +158,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 		rxmb->port = enic->port_id;
 		rxmb->data_len = seg_length;
 
+		bytes += seg_length;
+
 		rq->rx_nb_hold++;
 
 		if (!(enic_cq_rx_desc_eop(ciflags))) {
@@ -225,6 +230,10 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 				  &sop_rq->ctrl->posted_index);
 	}
 
+	if (enic->sriov_vf_soft_rx_stats && bytes) {
+		sop_rq->soft_stats_pkts += nb_rx;
+		sop_rq->soft_stats_bytes += bytes;
+	}
 
 	return nb_rx;
 }
@@ -256,6 +265,7 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	uint8_t color;
 	bool overlay;
 	bool tnl;
+	uint64_t bytes;
 
 	rq = rx_queue;
 	enic = vnic_dev_priv(rq->vdev);
@@ -283,6 +293,8 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	port_id = enic->port_id;
 	overlay = enic->overlay_offload;
 
+	bytes = 0;
+
 	rx = rx_pkts;
 	while (max_rx) {
 		max_rx--;
@@ -304,6 +316,9 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 			CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK;
 		mb->pkt_len = mb->data_len;
 		mb->port = port_id;
+
+		bytes += mb->pkt_len;
+
 		tnl = overlay && (cqd->completed_index_flags &
 				  CQ_ENET_RQ_DESC_FLAGS_FCOE) != 0;
 		mb->packet_type =
@@ -352,6 +367,11 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 				  &rq->ctrl->posted_index);
 	}
 
+	if (enic->sriov_vf_soft_rx_stats && bytes) {
+		rq->soft_stats_pkts += (rx - rx_pkts);
+		rq->soft_stats_bytes += bytes;
+	}
+
 	return rx - rx_pkts;
 }
 
drivers/net/enic/enic_sriov.c new file +801 -0

801 lines not on this page · raw message

drivers/net/enic/enic_sriov.h new file +209 -0

209 lines not on this page · raw message

drivers/net/enic/meson.build +1 -0
index 8700ae27f2..00f8348348 100644
@@ -23,6 +23,7 @@ sources = files(
         'enic_main.c',
         'enic_res.c',
         'enic_rxtx.c',
+        'enic_sriov.c',
         'enic_vf_representor.c',
 )
 deps += ['hash']
Signature · 1 line
2.35.2
#6 lore raw

Re: [PATCH v2 1/3] net/enic: support SR-IOV VF using admin channel

Quoted 24 lines · Hyong Youb Kim

The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).

The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.

The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.

For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.

Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.

Quoted 7 lines

- Backward compatibility mode
Old VIC PF drivers on some operating systems may support only VF_CAPABILITY_REQUEST message or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.

Do you think does it worth to document above restrictions in the driver guide?

Quoted 12 lines
---
 drivers/net/enic/base/vnic_cq.c       |  27 +
 drivers/net/enic/base/vnic_cq.h       |   3 +
 drivers/net/enic/base/vnic_dev.c      |  48 ++
 drivers/net/enic/base/vnic_dev.h      |   3 +
 drivers/net/enic/base/vnic_devcmd.h   |  49 ++
 drivers/net/enic/base/vnic_resource.h |  32 +-
 drivers/net/enic/base/vnic_rq.c       |  27 +
 drivers/net/enic/base/vnic_rq.h       |   7 +
 drivers/net/enic/base/vnic_wq.c       |  37 +-
Quoted 9 lines
 drivers/net/enic/base/vnic_wq.h       |   5 +
 drivers/net/enic/enic.h               |  28 +-
 drivers/net/enic/enic_ethdev.c        |   8 +-
 drivers/net/enic/enic_main.c          |  77 ++-
 drivers/net/enic/enic_res.c           |  12 +
 drivers/net/enic/enic_rxtx.c          |  20 +
 drivers/net/enic/enic_sriov.c         | 801 ++++++++++++++++++++++++++
 drivers/net/enic/enic_sriov.h         | 209 +++++++

This new file is missing coopyright and license information. (Detected by './devtools/check-spdx-tag.sh' script)

<...>

Quoted 13 lines
drivers/net/enic/enic_ethdev.c +1 -0
index cad8db2f6f..5c967677fb 100644
@@ -21,6 +21,7 @@
 #include "vnic_rq.h"
 #include "vnic_enet.h"
 #include "enic.h"
+#include "enic_sriov.h"
 
 /*
  * The set of PCI devices this driver supports

drivers/net/enic/enic_ethdev.c line 33

Quoted 6 lines
drivers/net/enic/enic_ethdev.c +0 -1
@@ -28,7 +29,6 @@
 #define CISCO_PCI_VENDOR_ID 0x1137
 static const struct rte_pci_id pci_id_enic_map[] = {
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET)},
-	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET_VF)},
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET_SN)},
 

Won't there be a specific PCIe device ID for the VF? Can there be some old devices around that requires this support?

What is the difference between PCI_DEVICE_ID_CISCO_VIC_ENET_VF & PCI_DEVICE_ID_CISCO_VIC_ENET_SN?
Current code seems detecting device as VF when the PCIe device ID is PCI_DEVICE_ID_CISCO_VIC_ENET_SN.

#7 lore raw

RE: [PATCH v2 1/3] net/enic: support SR-IOV VF using admin channel

Quoted 42 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, August 8, 2024 5:50 PM
To: Hyong Youb Kim (hyonkim) <[email protected]>
Cc: [email protected]; John Daley (johndale) <[email protected]>
Subject: Re: [PATCH v2 1/3] net/enic: support SR-IOV VF using admin channel

Quoted 24 lines · Hyong Youb Kim

The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).

The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.

The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.

For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.

Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.

Quoted 7 lines

- Backward compatibility mode
Old VIC PF drivers on some operating systems may support only VF_CAPABILITY_REQUEST message or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.

Quoted 3 lines

Do you think does it worth to document above restrictions in the driver guide?

I updated doc and release notes. Thanks for the reminder.

Quoted 22 lines
Quoted 12 lines
---
 drivers/net/enic/base/vnic_cq.c       |  27 +
 drivers/net/enic/base/vnic_cq.h       |   3 +
 drivers/net/enic/base/vnic_dev.c      |  48 ++
 drivers/net/enic/base/vnic_dev.h      |   3 +
 drivers/net/enic/base/vnic_devcmd.h   |  49 ++
 drivers/net/enic/base/vnic_resource.h |  32 +-
 drivers/net/enic/base/vnic_rq.c       |  27 +
 drivers/net/enic/base/vnic_rq.h       |   7 +
 drivers/net/enic/base/vnic_wq.c       |  37 +-
Quoted 9 lines
 drivers/net/enic/base/vnic_wq.h       |   5 +
 drivers/net/enic/enic.h               |  28 +-
 drivers/net/enic/enic_ethdev.c        |   8 +-
 drivers/net/enic/enic_main.c          |  77 ++-
 drivers/net/enic/enic_res.c           |  12 +
 drivers/net/enic/enic_rxtx.c          |  20 +
 drivers/net/enic/enic_sriov.c         | 801 ++++++++++++++++++++++++++
 drivers/net/enic/enic_sriov.h         | 209 +++++++
Quoted 3 lines

This new file is missing coopyright and license information. (Detected by './devtools/check-spdx-tag.sh' script)

Added SPDX.

Quoted 25 lines

<...>

Quoted 13 lines
drivers/net/enic/enic_ethdev.c +1 -0
index cad8db2f6f..5c967677fb 100644
@@ -21,6 +21,7 @@
 #include "vnic_rq.h"
 #include "vnic_enet.h"
 #include "enic.h"
+#include "enic_sriov.h"
 
 /*
  * The set of PCI devices this driver supports

drivers/net/enic/enic_ethdev.c line 31

Quoted 3 lines
drivers/net/enic/enic_ethdev.c +0 -0
@@ -28,7 +29,6 @@
 #define CISCO_PCI_VENDOR_ID 0x1137
 static const struct rte_pci_id pci_id_enic_map[] = {
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID,

PCI_DEVICE_ID_CISCO_VIC_ENET)},

drivers/net/enic/enic_ethdev.c line 31

Quoted 1 line
drivers/net/enic/enic_ethdev.c +0 -1
-	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID,

PCI_DEVICE_ID_CISCO_VIC_ENET_VF)},

drivers/net/enic/enic_ethdev.c line 32

Quoted 1 line
drivers/net/enic/enic_ethdev.c +0 -0
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID,

PCI_DEVICE_ID_CISCO_VIC_ENET_SN)},

Quoted 1 line
Quoted 7 lines

Won't there be a specific PCIe device ID for the VF? Can there be some old devices around that requires this support?

What is the difference between PCI_DEVICE_ID_CISCO_VIC_ENET_VF & PCI_DEVICE_ID_CISCO_VIC_ENET_SN?
Current code seems detecting device as VF when the PCIe device ID is PCI_DEVICE_ID_CISCO_VIC_ENET_SN.

The previous SR-IOV implementation was tied to Cisco VM-FEX. ENET_VF is the device ID for that. That has been discontinued for some time. There is no need to support that anymore.

The current SR-IOV implementation is not tied to VM-FEX. ENET_SN is the new device ID for this implementation. VF/SN naming is a bit confusing. But SN is the name used internally..

Thanks.
-Hyong

#8 lore raw

PATCH v2 2/3 net/enic: add speed capabilities for newer models

In reply to Hyong Youb Kim

Add 1400/14000 and 15000 models to the speed_capa list.

---
 drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
drivers/net/enic/enic_ethdev.c +21 -0
index 5c967677fb..62c8751d09 100644
@@ -62,6 +62,27 @@ static const struct vic_speed_capa {
 	{ 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */
 	{ 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1495 PCIe */
 	{ 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1497 MLOM */
+	{ 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G }, /* 1467 MLOM */
+	{ 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1477 MLOM */
+	{ 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */
+	{ 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */
+	{ 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */
+	{ 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */
+	{ 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */
+	{ 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */
+	{ 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */
+	{ 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */
+	{ 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G }, /* 15230 MLOM */
+	{ 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G }, /* 15422 Mezz */
+	{ 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */
+	{ 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */
+	{ 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */
 	{ 0, 0 }, /* End marker */
 };
 
Signature · 1 line
2.35.2
#9 lore raw

Re: [PATCH v2 2/3] net/enic: add speed capabilities for newer models

Quoted 2 lines · Hyong Youb Kim

Add 1400/14000 and 15000 models to the speed_capa list.

Do you want to introduce this new device variants in the driver guide?

Also does it makes sense to document new device model support in the release notes?

Quoted 38 lines
---
 drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
drivers/net/enic/enic_ethdev.c +21 -0
index 5c967677fb..62c8751d09 100644
@@ -62,6 +62,27 @@ static const struct vic_speed_capa {
 	{ 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */
 	{ 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1495 PCIe */
 	{ 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1497 MLOM */
+	{ 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G }, /* 1467 MLOM */
+	{ 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1477 MLOM */
+	{ 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */
+	{ 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */
+	{ 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */
+	{ 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */
+	{ 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */
+	{ 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */
+	{ 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */
+	{ 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */
+	{ 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G }, /* 15230 MLOM */
+	{ 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G }, /* 15422 Mezz */
+	{ 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */
+	{ 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */
+	{ 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */
 	{ 0, 0 }, /* End marker */
 };
 
#10 lore raw

RE: [PATCH v2 2/3] net/enic: add speed capabilities for newer models

Quoted 9 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, August 8, 2024 5:51 PM
To: Hyong Youb Kim (hyonkim) <[email protected]>
Cc: [email protected]; John Daley (johndale) <[email protected]>
Subject: Re: [PATCH v2 2/3] net/enic: add speed capabilities for newer models

Quoted 9 lines
Quoted 2 lines · Hyong Youb Kim

Add 1400/14000 and 15000 models to the speed_capa list.

Do you want to introduce this new device variants in the driver guide?

Also does it makes sense to document new device model support in the release notes?

Updated doc and release notes.

Thanks.
-Hyong

Quoted 44 lines

drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 63

Quoted 13 lines
---
 drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
drivers/net/enic/enic_ethdev.c +0 -0
index 5c967677fb..62c8751d09 100644
@@ -62,6 +62,27 @@ static const struct vic_speed_capa {
 	{ 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */
 	{ 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G },

/* 1495 PCIe */

drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 64

Quoted 1 line
drivers/net/enic/enic_ethdev.c +0 -0
 	{ 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G },

/* 1497 MLOM */

drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 65

Quoted 1 line
drivers/net/enic/enic_ethdev.c +1 -0
+	{ 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G },

/* 1467 MLOM */

drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 66

Quoted 1 line
drivers/net/enic/enic_ethdev.c +1 -0
+	{ 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G },

/* 1477 MLOM */

drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa {

Quoted 12 lines
drivers/net/enic/enic_ethdev.c +12 -0
+	{ 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */
+	{ 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */
+	{ 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */
+	{ 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */
+	{ 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */
+	{ 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */
+	{ 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */
+	{ 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */
+	{ 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G },

/* 15230 MLOM */

drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 79

Quoted 1 line
drivers/net/enic/enic_ethdev.c +1 -0
+	{ 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G },

/* 15422 Mezz */

Quoted 9 lines
drivers/net/enic/enic_ethdev.c +6 -0
+	{ 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */
+	{ 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */
+	{ 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */
 	{ 0, 0 }, /* End marker */
 };
 
#11 lore raw

PATCH v2 3/3 net/enic: allow multicast in MAC address add callback

In reply to Hyong Youb Kim

enic_set_mac_address() (mac_addr_add callback) currently allows only non-zero, unicast address to be added. It is overly restrictive. rte_eth_dev_mac_addr_add() itself allows multicast addresses. And, some applications do use rte_eth_dev_mac_addr_add() to accept multicast addresses. So, remove the unicast check in enic_set_mac_address().

---
 drivers/net/enic/enic_main.c | 20 --------------------
 1 file changed, 20 deletions(-)
drivers/net/enic/enic_main.c +0 -20
index e59b0a5077..b755b15d92 100644
@@ -32,21 +32,6 @@
 #include "vnic_intr.h"
 #include "vnic_nic.h"
 
-static int is_zero_addr(uint8_t *addr)
-{
-	return !(addr[0] |  addr[1] | addr[2] | addr[3] | addr[4] | addr[5]);
-}
-
-static int is_mcast_addr(uint8_t *addr)
-{
-	return addr[0] & 1;
-}
-
-static int is_eth_addr_valid(uint8_t *addr)
-{
-	return !is_mcast_addr(addr) && !is_zero_addr(addr);
-}
-
 void
 enic_rxmbuf_queue_release(__rte_unused struct enic *enic, struct vnic_rq *rq)
 {
@@ -177,11 +162,6 @@ int enic_set_mac_address(struct enic *enic, uint8_t *mac_addr)
 {
 	int err;
 
-	if (!is_eth_addr_valid(mac_addr)) {
-		dev_err(enic, "invalid mac address\n");
-		return -EINVAL;
-	}
-
 	err = enic_dev_add_addr(enic, mac_addr);
 	if (err)
 		dev_err(enic, "add mac addr failed\n");
Signature · 1 line
2.35.2
#12 lore raw

Re: [PATCH v2 3/3] net/enic: allow multicast in MAC address add callback

Quoted 7 lines · Hyong Youb Kim

enic_set_mac_address() (mac_addr_add callback) currently allows only non-zero, unicast address to be added. It is overly restrictive. rte_eth_dev_mac_addr_add() itself allows multicast addresses. And, some applications do use rte_eth_dev_mac_addr_add() to accept multicast addresses. So, remove the unicast check in enic_set_mac_address().

What is the usecase to set Multicast MAC address as device default MAC address?

Also, just a reminder that we have 'rte_eth_dev_set_mc_addr_list()' API to set multicast MAC addresses.

#13 lore raw

RE: [PATCH v2 3/3] net/enic: allow multicast in MAC address add callback

Quoted 19 lines

-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, August 8, 2024 5:51 PM
To: Hyong Youb Kim (hyonkim) <[email protected]>
Cc: [email protected]; John Daley (johndale) <[email protected]>
Subject: Re: [PATCH v2 3/3] net/enic: allow multicast in MAC address add callback

Quoted 7 lines · Hyong Youb Kim

enic_set_mac_address() (mac_addr_add callback) currently allows only non-zero, unicast address to be added. It is overly restrictive. rte_eth_dev_mac_addr_add() itself allows multicast addresses. And, some applications do use rte_eth_dev_mac_addr_add() to accept multicast addresses. So, remove the unicast check in enic_set_mac_address().

Quoted 5 lines

What is the usecase to set Multicast MAC address as device default MAC address?

Also, just a reminder that we have 'rte_eth_dev_set_mc_addr_list()' API to set multicast MAC addresses.

I am aware of mc_addr_list().. Some people seem to use mac_addr_add() as 'add a MAC filter'.

For the VIC adapter, there is no difference between the default MAC and other MAC addresses added to allow/accept. They are all MAC filters in HW. I believe some other NICs also do not distinguish 'default' vs. additional allowed MAC addresses (can be unicast or multicast).

The app in question is VPP. There might be others. It uses rte_eth_dev_mac_addr_add() to add multicast MAC filters like all-hosts address. I am not defending such usage. But given that (1) rte_eth_dev_mac_addr_add() accepts multicast, and (2) adding multicast works fine on multiple NICs, such usage is probably acceptable.

I am guessing some app developers find it easier to use mac_addr_add() than mc_addr_list()..

We could tighten the semantics of rte_eth_dev_mac_addr_add() and check if the given MAC is a valid unicast address in that API function.

Thanks.
-Hyong

#14 lore raw

v3 cover net/enic: support VF and fix minor issues

In reply to Hyong Youb Kim

This series contains minor updates for net/enic. The first patch supports SR-IOV VF, which now requires the use of admin channel. The other patches are not related to VF, but included here to ease review.

---

v3:
* add spdx, doc, rel_notes (Ferruh's comments)

v2:
* fix compiler warnings

Hyong Youb Kim (3):
net/enic: support SR-IOV VF using admin channel net/enic: add speed capabilities for newer models net/enic: allow multicast in MAC address add callback

 doc/guides/nics/enic.rst               | 123 ++--
 doc/guides/rel_notes/release_24_11.rst |   5 +
 drivers/net/enic/base/vnic_cq.c        |  27 +
 drivers/net/enic/base/vnic_cq.h        |   3 +
 drivers/net/enic/base/vnic_dev.c       |  48 ++
 drivers/net/enic/base/vnic_dev.h       |   3 +
 drivers/net/enic/base/vnic_devcmd.h    |  49 ++
 drivers/net/enic/base/vnic_resource.h  |  32 +-
 drivers/net/enic/base/vnic_rq.c        |  27 +
 drivers/net/enic/base/vnic_rq.h        |   7 +
 drivers/net/enic/base/vnic_wq.c        |  37 +-
 drivers/net/enic/base/vnic_wq.h        |   5 +
 drivers/net/enic/enic.h                |  28 +-
 drivers/net/enic/enic_ethdev.c         |  29 +-
 drivers/net/enic/enic_main.c           |  97 +--
 drivers/net/enic/enic_res.c            |  12 +
 drivers/net/enic/enic_rxtx.c           |  20 +
 drivers/net/enic/enic_sriov.c          | 801 +++++++++++++++++++++++++
 drivers/net/enic/enic_sriov.h          | 211 +++++++
 drivers/net/enic/meson.build           |   1 +
 20 files changed, 1452 insertions(+), 113 deletions(-)
 create mode 100644 drivers/net/enic/enic_sriov.c
 create mode 100644 drivers/net/enic/enic_sriov.h
Signature · 1 line
2.35.2
#15 lore raw

PATCH v3 1/3 net/enic: support SR-IOV VF using admin channel

The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).

The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.

The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.

For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.

Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.

- Backward compatibility mode
Old VIC PF drivers on some operating systems may support only VF_CAPABILITY_REQUEST message or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.

---
 doc/guides/nics/enic.rst               | 120 ++--
 doc/guides/rel_notes/release_24_11.rst |   4 +
 drivers/net/enic/base/vnic_cq.c        |  27 +
 drivers/net/enic/base/vnic_cq.h        |   3 +
 drivers/net/enic/base/vnic_dev.c       |  48 ++
 drivers/net/enic/base/vnic_dev.h       |   3 +
 drivers/net/enic/base/vnic_devcmd.h    |  49 ++
 drivers/net/enic/base/vnic_resource.h  |  32 +-
 drivers/net/enic/base/vnic_rq.c        |  27 +
 drivers/net/enic/base/vnic_rq.h        |   7 +
 drivers/net/enic/base/vnic_wq.c        |  37 +-
 drivers/net/enic/base/vnic_wq.h        |   5 +
 drivers/net/enic/enic.h                |  28 +-
 drivers/net/enic/enic_ethdev.c         |   8 +-
 drivers/net/enic/enic_main.c           |  77 ++-
 drivers/net/enic/enic_res.c            |  12 +
 drivers/net/enic/enic_rxtx.c           |  20 +
 drivers/net/enic/enic_sriov.c          | 801 +++++++++++++++++++++++++
 drivers/net/enic/enic_sriov.h          | 211 +++++++
 drivers/net/enic/meson.build           |   1 +
 20 files changed, 1428 insertions(+), 92 deletions(-)
 create mode 100644 drivers/net/enic/enic_sriov.c
 create mode 100644 drivers/net/enic/enic_sriov.h
doc/guides/nics/enic.rst +52 -68
index 5bf5cc9ab9..639d3f5939 100644
@@ -37,7 +37,7 @@ Supported features
 - UDP RSS hashing (1400 series and later adapters)
 - Scattered Rx
 - MTU update
-- SR-IOV on UCS managed servers connected to Fabric Interconnects
+- SR-IOV virtual function
 - Flow API
 - Overlay offload
 
@@ -135,103 +135,87 @@ Configuration information
     TCP, IPv4, TCP-IPv4, IPv6, TCP-IPv6, IPv6 Extension, TCP-IPv6 Extension.
 
 
-SR-IOV mode utilization
+SR-IOV Virtual Function
 -----------------------
 
-UCS blade servers configured with dynamic vNIC connection policies in UCSM
-are capable of supporting SR-IOV. SR-IOV virtual functions (VFs) are
-specialized vNICs, distinct from regular Ethernet vNICs. These VFs can be
-directly assigned to virtual machines (VMs) as 'passthrough' devices.
+VIC 1400 and later series supports SR-IOV. It can be enabled via both
+UCSM and CIMC. Please refer to the following guides to enable SR-IOV
+virtual functions (VFs).
 
-In UCS, SR-IOV VFs require the use of the Cisco Virtual Machine Fabric Extender
-(VM-FEX), which gives the VM a dedicated
-interface on the Fabric Interconnect (FI). Layer 2 switching is done at
-the FI. This may eliminate the requirement for software switching on the
-host to route intra-host VM traffic.
+  - CIMC: `Managing vNICs <https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/sw/gui/config/guide/4_3/b_cisco_ucs_c-series_gui_configuration_guide_43/b_Cisco_UCS_C-series_GUI_Configuration_Guide_41_chapter_01011.html#d77871e5874a1635>`_
 
-Please refer to `Creating a Dynamic vNIC Connection Policy
-<http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vm_fex/vmware/gui/config_guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide_chapter_010.html#task_433E01651F69464783A68E66DA8A47A5>`_
-for information on configuring SR-IOV adapter policies and port profiles
-using UCSM.
+  - UCSM: `Configuring SRIOV HPN Connection Policies <https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Network-Mgmt/4-3/b_UCSM_Network_Mgmt_Guide_4_3/b_UCSM_Network_Mgmt_Guide_chapter_01010.html#d21438e9555a1635>`_
 
-Once the policies are in place and the host OS is rebooted, VFs should be
-visible on the host, E.g.:
+Note that the previous SR-IOV implementation that is tied to VM-FEX
+(Cisco Virtual Machine Fabric Extender) has been discontinued, and
+ENIC PMD no longer supports it. The current SR-IOV implementation does
+not require the Fabric Interconnect (FI), as layer 2 switching is done
+within the VIC adapter.
+
+Once SR-IOV is enabled, reboot the host OS and follow OS specific
+steps to create VFs and assign them to virtual machines (VMs) or
+containers as necessary. The VIC physical function (PF) drivers for ESXi
+and Linux support SR-IOV. The following shows simplified steps for
+Linux.
 
 .. code-block:: console
 
+     # echo 4 > /sys/class/net/<pf-interface>/device/sriov_numvfs
+
      # lspci | grep Cisco | grep Ethernet
-     0d:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
-     0d:00.1 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
-     0d:00.2 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
-     0d:00.3 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
-     0d:00.4 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
-     0d:00.5 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
-     0d:00.6 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
-     0d:00.7 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
-
-Enable Intel IOMMU on the host and install KVM and libvirt, and reboot again as
-required. Then, using libvirt, create a VM instance with an assigned device.
-Below is an example ``interface`` block (part of the domain configuration XML)
-that adds the host VF 0d:00:01 to the VM. ``profileid='pp-vlan-25'`` indicates
-the port profile that has been configured in UCSM.
+     12:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
+     12:00.1 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2)
+     12:00.2 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2)
+     12:00.3 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2)
+     12:00.4 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2)
+
+Writing 4 to ``sriov_numvfs`` creates 4 VFs. ``lspci`` shows VFs and
+their PCI locations.  Interfaces with device ID ``02b7`` are the
+VFs. The following snippet for libvirt XML assigns VF at ``12:00.1``
+to VM.
 
 .. code-block:: console
 
-    <interface type='hostdev' managed='yes'>
-      <mac address='52:54:00:ac:ff:b6'/>
+    <interface type="hostdev" managed="yes">
+      <mac address="fa:16:3e:46:39:c5"/>
       <driver name='vfio'/>
       <source>
-        <address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/>
+        <address type="pci" domain="0x0000" bus="0x12" slot="0x00" function="0x1"/>
       </source>
-      <virtualport type='802.1Qbh'>
-        <parameters profileid='pp-vlan-25'/>
-      </virtualport>
+      <vlan>
+        <tag id="1000"/>
+      </vlan>
     </interface>
 
-
-Alternatively, the configuration can be done in a separate file using the
-``network`` keyword. These methods are described in the libvirt documentation for
-`Network XML format <https://libvirt.org/formatnetwork.html>`_.
-
 When the VM instance is started, libvirt will bind the host VF to
-vfio, complete provisioning on the FI and bring up the link.
-
-.. note::
-
-    It is not possible to use a VF directly from the host because it is not
-    fully provisioned until libvirt brings up the VM that it is assigned
-    to.
-
-In the VM instance, the VF will now be visible. E.g., here the VF 00:04.0 is
-seen on the VM instance and should be available for binding to a DPDK.
+vfio-pci. In the VM instance, the VF will now be visible. In this
+example, VF at ``07:00.0`` is seen on the VM instance and is available
+for binding to DPDK.
 
 .. code-block:: console
 
-     # lspci | grep Ether
-     00:04.0 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2)
+     # lspci | grep Cisco
+     07:00.0 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2)
 
-Follow the normal DPDK install procedure, binding the VF to either ``igb_uio``
-or ``vfio`` in non-IOMMU mode.
+There are two known limitations of the current SR-IOV implementation.
 
-In the VM, the kernel enic driver may be automatically bound to the VF during
-boot. Unbinding it currently hangs due to a known issue with the driver. To
-work around the issue, block the enic module as follows.
-Please see :ref:`Limitations <enic_limitations>` for limitations in
-the use of SR-IOV.
+  - Software Rx statistics
 
-.. code-block:: console
+    VF on old VIC models does not have hardware Rx counters. In this case,
+    ENIC PMD counts packets/bytes and reports them as device statistics.
 
-     # cat /etc/modprobe.d/enic.conf
-     blacklist enic
+  - Backward compatibility mode
 
-     # dracut --force
+    Old PF drivers on ESXi may lack full admin channel support. ENIC PMD
+    detects such PF driver during initialization and reverts to the
+    compatibility mode. In this mode, ENIC PMD does not use the admin channel,
+    and trust mode (e.g. enabling promiscuous mode on VF) is not supported.
 
 .. note::
 
-    Passthrough does not require SR-IOV. If VM-FEX is not desired, the user
+    Passthrough does not require SR-IOV. If SR-IOV is not desired, the user
     may create as many regular vNICs as necessary and assign them to VMs as
-    passthrough devices. Since these vNICs are not SR-IOV VFs, using them as
-    passthrough devices do not require libvirt, port profiles, and VM-FEX.
+    passthrough devices.
 
 
 .. _enic-generic-flow-api:
doc/guides/rel_notes/release_24_11.rst +4 -0
index 0ff70d9057..f79b1bee58 100644
@@ -55,6 +55,10 @@ New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Updated Cisco enic driver.**
+
+  * Added SR-IOV VF support.
+
 
 Removed Items
 -------------
drivers/net/enic/base/vnic_cq.c +27 -0
index 51d6fd387b..e966de2f67 100644
@@ -24,6 +24,7 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 
 	cq->index = index;
 	cq->vdev = vdev;
+	cq->admin_chan = false;
 
 	cq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_CQ, index);
 	if (!cq->ctrl) {
@@ -40,6 +41,32 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 	return 0;
 }
 
+int vnic_admin_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
+	unsigned int socket_id, unsigned int desc_count, unsigned int desc_size)
+{
+	int err;
+	char res_name[RTE_MEMZONE_NAMESIZE];
+	static int instance;
+
+	cq->index = index;
+	cq->vdev = vdev;
+	cq->admin_chan = true;
+
+	cq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_ADMIN_CQ, index);
+	if (!cq->ctrl) {
+		pr_err("Failed to get admin CQ[%u] resource\n", index);
+		return -EINVAL;
+	}
+
+	snprintf(res_name, sizeof(res_name), "%d-admin-cq-%u", instance++, index);
+	err = vnic_dev_alloc_desc_ring(vdev, &cq->ring, desc_count, desc_size,
+		socket_id, res_name);
+	if (err)
+		return err;
+
+	return 0;
+}
+
 void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable,
 	unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail,
 	unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_cq.h +3 -0
index 2e48759c49..6469a5c12a 100644
@@ -59,12 +59,15 @@ struct vnic_cq {
 	unsigned int tobe_rx_coal_timeval;
 	ktime_t prev_ts;
 #endif
+	bool admin_chan;
 };
 
 void vnic_cq_free(struct vnic_cq *cq);
 int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
 	unsigned int socket_id,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,
+	unsigned int socket_id, unsigned int desc_count, unsigned int desc_size);
 void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable,
 	unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail,
 	unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_dev.c +48 -0

81 lines not on this page · raw message

drivers/net/enic/base/vnic_dev.h +3 -0
index 44ca5c739f..0336e07ffc 100644
@@ -199,5 +199,8 @@ int vnic_dev_capable_geneve(struct vnic_dev *vdev);
 uint64_t vnic_dev_capable_cq_entry_size(struct vnic_dev *vdev);
 int vnic_dev_set_cq_entry_size(struct vnic_dev *vdev, uint32_t rq_idx,
 			       uint32_t size_flag);
+int vnic_dev_alloc_sriov_stats_mem(struct vnic_dev *vdev);
+int vnic_dev_sriov_stats(struct vnic_dev *vdev, struct vnic_sriov_stats **stats);
+int vnic_dev_enable_admin_qp(struct vnic_dev *vdev, uint32_t enable);
 
 #endif /* _VNIC_DEV_H_ */
drivers/net/enic/base/vnic_devcmd.h +49 -0

59 lines not on this page · raw message

drivers/net/enic/base/vnic_resource.h +30 -2
index 870a4de6e4..69cc96433e 100644
@@ -38,8 +38,36 @@ enum vnic_res_type {
 	RES_TYPE_MQ_WQ,                 /* MQ Work queues */
 	RES_TYPE_MQ_RQ,                 /* MQ Receive queues */
 	RES_TYPE_MQ_CQ,                 /* MQ Completion queues */
-	RES_TYPE_DEPRECATED1,           /* Old version of devcmd 2 */
-	RES_TYPE_DEVCMD2,               /* Device control region */
+	RES_TYPE_DEPRECATED1,		/* Old version of devcmd 2 */
+	RES_TYPE_DEPRECATED2,		/* Old version of devcmd 2 */
+	RES_TYPE_DEVCMD2,		/* Device control region */
+	RES_TYPE_RDMA_WQ,		/* RDMA WQ */
+	RES_TYPE_RDMA_RQ,		/* RDMA RQ */
+	RES_TYPE_RDMA_CQ,		/* RDMA CQ */
+	RES_TYPE_RDMA_RKEY_TABLE,	/* RDMA RKEY table */
+	RES_TYPE_RDMA_RQ_HEADER_TABLE,	/* RDMA RQ Header Table */
+	RES_TYPE_RDMA_RQ_TABLE,		/* RDMA RQ Table */
+	RES_TYPE_RDMA_RD_RESP_HEADER_TABLE,	/* RDMA Read Response Header Table */
+	RES_TYPE_RDMA_RD_RESP_TABLE,	/* RDMA Read Response Table */
+	RES_TYPE_RDMA_QP_STATS_TABLE,	/* RDMA per QP stats table */
+	RES_TYPE_WQ_MREGS,		/* XXX snic proto only */
+	RES_TYPE_GRPMBR_INTR,		/* Group member interrupt control */
+	RES_TYPE_DPKT,			/* Direct Packet memory region */
+	RES_TYPE_RDMA2_DATA_WQ,		/* RDMA datapath command WQ */
+	RES_TYPE_RDMA2_REG_WQ,		/* RDMA registration command WQ */
+	RES_TYPE_RDMA2_CQ,		/* RDMA datapath CQ */
+	RES_TYPE_MQ_RDMA2_DATA_WQ,	/* RDMA datapath command WQ */
+	RES_TYPE_MQ_RDMA2_REG_WQ,	/* RDMA registration command WQ */
+	RES_TYPE_MQ_RDMA2_CQ,		/* RDMA datapath CQ */
+	RES_TYPE_PTP,			/* PTP registers */
+	RES_TYPE_INTR_CTRL2,		/* Extended INTR CTRL registers */
+	RES_TYPE_SRIOV_INTR,		/* VF intr */
+	RES_TYPE_VF_WQ,			/* VF WQ */
+	RES_TYPE_VF_RQ,			/* VF RQ */
+	RES_TYPE_VF_CQ,			/* VF CQ */
+	RES_TYPE_ADMIN_WQ,		/* admin channel WQ */
+	RES_TYPE_ADMIN_RQ,		/* admin channel RQ */
+	RES_TYPE_ADMIN_CQ,		/* admin channel CQ */
 	RES_TYPE_MAX,			/* Count of resource types */
 };
 
drivers/net/enic/base/vnic_rq.c +27 -0
index 1af96a9413..88def104ad 100644
@@ -27,6 +27,7 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 
 	rq->index = index;
 	rq->vdev = vdev;
+	rq->admin_chan = false;
 
 	rq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_RQ, index);
 	if (!rq->ctrl) {
@@ -42,6 +43,32 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 	return rc;
 }
 
+int vnic_admin_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq,
+	unsigned int desc_count, unsigned int desc_size)
+{
+	int rc;
+	char res_name[RTE_MEMZONE_NAMESIZE];
+	static int instance;
+
+	rq->index = 0;
+	rq->vdev = vdev;
+	rq->admin_chan = true;
+	rq->socket_id = SOCKET_ID_ANY;
+
+	rq->ctrl = vnic_dev_get_res(vdev, RES_TYPE_ADMIN_RQ, 0);
+	if (!rq->ctrl) {
+		pr_err("Failed to get admin RQ resource\n");
+		return -EINVAL;
+	}
+
+	vnic_rq_disable(rq);
+
+	snprintf(res_name, sizeof(res_name), "%d-admin-rq", instance++);
+	rc = vnic_dev_alloc_desc_ring(vdev, &rq->ring, desc_count, desc_size,
+		rq->socket_id, res_name);
+	return rc;
+}
+
 void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_rq.h +7 -0
index cfe65015d5..0f068d2e18 100644
@@ -73,6 +73,11 @@ struct vnic_rq {
 	unsigned int max_mbufs_per_pkt;
 	uint16_t tot_nb_desc;
 	bool need_initial_post;
+	bool admin_chan;
+	const struct rte_memzone *admin_msg_rz;
+	int admin_next_idx;
+	uint64_t soft_stats_pkts;
+	uint64_t soft_stats_bytes;
 };
 
 static inline unsigned int vnic_rq_desc_avail(struct vnic_rq *rq)
@@ -127,6 +132,8 @@ static inline int vnic_rq_fill_count(struct vnic_rq *rq,
 void vnic_rq_free(struct vnic_rq *rq);
 int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq,
+	unsigned int desc_count, unsigned int desc_size);
 void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.c +35 -2
index fc6dde5ae8..04eabefae7 100644
@@ -23,7 +23,8 @@ int vnic_wq_alloc_ring(struct vnic_dev *vdev, struct vnic_wq *wq,
 	char res_name[RTE_MEMZONE_NAMESIZE];
 	static int instance;
 
-	snprintf(res_name, sizeof(res_name), "%d-wq-%u", instance++, wq->index);
+	snprintf(res_name, sizeof(res_name), "%d-%swq-%u",
+		 instance++, wq->admin_chan ? "admin-" : "", wq->index);
 	return vnic_dev_alloc_desc_ring(vdev, &wq->ring, desc_count, desc_size,
 		wq->socket_id, res_name);
 }
@@ -32,7 +33,7 @@ static int vnic_wq_alloc_bufs(struct vnic_wq *wq)
 {
 	unsigned int count = wq->ring.desc_count;
        /* Allocate the mbuf ring */
-	wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket("wq->bufs",
+	wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket(wq->admin_chan ? "admin-wq-bufs" : "wq-bufs",
 		    sizeof(struct rte_mbuf *) * count,
 		    RTE_CACHE_LINE_SIZE, wq->socket_id);
 	wq->head_idx = 0;
@@ -62,6 +63,7 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 
 	wq->index = index;
 	wq->vdev = vdev;
+	wq->admin_chan = false;
 
 	err = vnic_wq_get_ctrl(vdev, wq, index, RES_TYPE_WQ);
 	if (err) {
@@ -84,6 +86,37 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 	return 0;
 }
 
+int vnic_admin_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq,
+	unsigned int desc_count, unsigned int desc_size)
+{
+	int err;
+
+	wq->index = 0;
+	wq->vdev = vdev;
+	wq->admin_chan = true;
+	wq->socket_id = SOCKET_ID_ANY;
+
+	err = vnic_wq_get_ctrl(vdev, wq, 0, RES_TYPE_ADMIN_WQ);
+	if (err) {
+		pr_err("Failed to get admin WQ resource err %d\n", err);
+		return err;
+	}
+
+	vnic_wq_disable(wq);
+
+	err = vnic_wq_alloc_ring(vdev, wq, desc_count, desc_size);
+	if (err)
+		return err;
+
+	err = vnic_wq_alloc_bufs(wq);
+	if (err) {
+		vnic_wq_free(wq);
+		return err;
+	}
+
+	return 0;
+}
+
 void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.h +5 -0
index 789a50a647..381f78431c 100644
@@ -50,6 +50,9 @@ struct vnic_wq {
 	const struct rte_memzone *cqmsg_rz;
 	uint16_t last_completed_index;
 	uint64_t offloads;
+	bool admin_chan;
+	const struct rte_memzone *admin_msg_rz;
+	uint64_t soft_stats_tx;
 };
 
 static inline unsigned int vnic_wq_desc_avail(struct vnic_wq *wq)
@@ -149,6 +152,8 @@ buf_idx_incr(uint32_t n_descriptors, uint32_t idx)
 void vnic_wq_free(struct vnic_wq *wq);
 int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,
 	unsigned int desc_count, unsigned int desc_size);
+int vnic_admin_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq,
+	unsigned int desc_count, unsigned int desc_size);
 void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index,
 	unsigned int fetch_index, unsigned int posted_index,
 	unsigned int error_interrupt_enable,
drivers/net/enic/enic.h +26 -2
index f46903ea9e..dc1d16f7d5 100644
@@ -43,7 +43,6 @@
 /*#define ENIC_DESC_COUNT_MAKE_ODD (x) do{if ((~(x)) & 1) { (x)--; } }while(0)*/
 
 #define PCI_DEVICE_ID_CISCO_VIC_ENET         0x0043  /* ethernet vnic */
-#define PCI_DEVICE_ID_CISCO_VIC_ENET_VF      0x0071  /* enet SRIOV VF */
 /* enet SRIOV Standalone vNic VF */
 #define PCI_DEVICE_ID_CISCO_VIC_ENET_SN      0x02B7
 
@@ -51,7 +50,7 @@
 #define ENIC_MAGIC_FILTER_ID 0xffff
 
 /*
- * Interrupt 0: LSC and errors
+ * Interrupt 0: LSC and errors / VF admin channel RQ
  * Interrupt 1: rx queue 0
  * Interrupt 2: rx queue 1
  * ...
@@ -154,11 +153,26 @@ struct enic {
 	/* software counters */
 	struct enic_soft_stats soft_stats;
 
+	struct vnic_wq admin_wq;
+	struct vnic_rq admin_rq;
+	struct vnic_cq admin_cq[2];
+
 	/* configured resources on vic */
 	unsigned int conf_rq_count;
 	unsigned int conf_wq_count;
 	unsigned int conf_cq_count;
 	unsigned int conf_intr_count;
+	/* SR-IOV VF has queues for admin channel to PF */
+	unsigned int conf_admin_rq_count;
+	unsigned int conf_admin_wq_count;
+	unsigned int conf_admin_cq_count;
+	uint64_t admin_chan_msg_num;
+	int admin_chan_vf_id;
+	uint32_t admin_pf_cap_version;
+	bool admin_chan_enabled;
+	bool sriov_vf_soft_rx_stats;
+	bool sriov_vf_compat_mode;
+	pthread_mutex_t admin_chan_lock;
 
 	/* linked list storing memory allocations */
 	LIST_HEAD(enic_memzone_list, enic_memzone_entry) memzone_list;
@@ -230,6 +244,16 @@ struct enic_vf_representor {
 	struct rte_flow *rep2vf_flow[2];
 };
 
+#define ENIC_ADMIN_WQ_CQ 0
+#define ENIC_ADMIN_RQ_CQ 1
+#define ENIC_ADMIN_BUF_SIZE 1024
+
+static inline bool enic_is_vf(struct enic *enic)
+{
+	return enic->pdev->id.device_id == PCI_DEVICE_ID_CISCO_VIC_ENET_SN &&
+		!enic->sriov_vf_compat_mode;
+}
+
 #define VF_ENIC_TO_VF_REP(vf_enic) \
 	container_of(vf_enic, struct enic_vf_representor, enic)
 
drivers/net/enic/enic_ethdev.c +4 -4
index cad8db2f6f..5c967677fb 100644
@@ -21,6 +21,7 @@
 #include "vnic_rq.h"
 #include "vnic_enet.h"
 #include "enic.h"
+#include "enic_sriov.h"
 
 /*
  * The set of PCI devices this driver supports
@@ -28,7 +29,6 @@
 #define CISCO_PCI_VENDOR_ID 0x1137
 static const struct rte_pci_id pci_id_enic_map[] = {
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET)},
-	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET_VF)},
 	{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID, PCI_DEVICE_ID_CISCO_VIC_ENET_SN)},
 	{.vendor_id = 0, /* sentinel */},
 };
@@ -707,7 +707,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		for (i = 0; i < enic->mc_count; i++) {
 			addr = &enic->mc_addrs[i];
 			debug_log_add_del_addr(addr, false);
-			ret = vnic_dev_del_addr(enic->vdev, addr->addr_bytes);
+			ret = enic_dev_del_addr(enic, addr->addr_bytes);
 			if (ret)
 				return ret;
 		}
@@ -734,7 +734,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		if (j < nb_mc_addr)
 			continue;
 		debug_log_add_del_addr(addr, false);
-		ret = vnic_dev_del_addr(enic->vdev, addr->addr_bytes);
+		ret = enic_dev_del_addr(enic, addr->addr_bytes);
 		if (ret)
 			return ret;
 	}
@@ -748,7 +748,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,
 		if (j < enic->mc_count)
 			continue;
 		debug_log_add_del_addr(addr, true);
-		ret = vnic_dev_add_addr(enic->vdev, addr->addr_bytes);
+		ret = enic_dev_add_addr(enic, addr->addr_bytes);
 		if (ret)
 			return ret;
 	}
drivers/net/enic/enic_main.c +63 -14

191 lines not on this page · raw message

drivers/net/enic/enic_res.c +12 -0
index caf773bab2..f09f4c0abb 100644
@@ -300,4 +300,16 @@ void enic_get_res_counts(struct enic *enic)
 		"vNIC resources avail: wq %d rq %d cq %d intr %d\n",
 		enic->conf_wq_count, enic->conf_rq_count,
 		enic->conf_cq_count, enic->conf_intr_count);
+
+	if (!enic_is_vf(enic))
+		return;
+
+	enic->conf_admin_wq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_WQ);
+	enic->conf_admin_rq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_RQ);
+	enic->conf_admin_cq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_ADMIN_CQ);
+
+	dev_info(enic_get_dev(enic),
+		"vNIC admin channel resources avail: wq %d rq %d cq %d\n",
+		enic->conf_admin_wq_count, enic->conf_admin_rq_count,
+		enic->conf_admin_cq_count);
 }
drivers/net/enic/enic_rxtx.c +20 -0
index 7a66d72275..549a153332 100644
@@ -55,6 +55,7 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 		sizeof(struct cq_enet_rq_desc_64) :
 		sizeof(struct cq_enet_rq_desc);
 	RTE_BUILD_BUG_ON(sizeof(struct cq_enet_rq_desc_64) != 64);
+	uint64_t bytes;
 
 	cq = &enic->cq[enic_cq_rq(enic, sop_rq->index)];
 	cq_idx = cq->to_clean;		/* index of cqd, rqd, mbuf_table */
@@ -67,6 +68,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 	/* Receive until the end of the ring, at most. */
 	max_rx = RTE_MIN(nb_pkts, cq->ring.desc_count - cq_idx);
 
+	bytes = 0;
+
 	while (max_rx) {
 		volatile struct rq_enet_desc *rqd_ptr;
 		struct cq_desc cqd;
@@ -155,6 +158,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 		rxmb->port = enic->port_id;
 		rxmb->data_len = seg_length;
 
+		bytes += seg_length;
+
 		rq->rx_nb_hold++;
 
 		if (!(enic_cq_rx_desc_eop(ciflags))) {
@@ -225,6 +230,10 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,
 				  &sop_rq->ctrl->posted_index);
 	}
 
+	if (enic->sriov_vf_soft_rx_stats && bytes) {
+		sop_rq->soft_stats_pkts += nb_rx;
+		sop_rq->soft_stats_bytes += bytes;
+	}
 
 	return nb_rx;
 }
@@ -256,6 +265,7 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	uint8_t color;
 	bool overlay;
 	bool tnl;
+	uint64_t bytes;
 
 	rq = rx_queue;
 	enic = vnic_dev_priv(rq->vdev);
@@ -283,6 +293,8 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	port_id = enic->port_id;
 	overlay = enic->overlay_offload;
 
+	bytes = 0;
+
 	rx = rx_pkts;
 	while (max_rx) {
 		max_rx--;
@@ -304,6 +316,9 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 			CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK;
 		mb->pkt_len = mb->data_len;
 		mb->port = port_id;
+
+		bytes += mb->pkt_len;
+
 		tnl = overlay && (cqd->completed_index_flags &
 				  CQ_ENET_RQ_DESC_FLAGS_FCOE) != 0;
 		mb->packet_type =
@@ -352,6 +367,11 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 				  &rq->ctrl->posted_index);
 	}
 
+	if (enic->sriov_vf_soft_rx_stats && bytes) {
+		rq->soft_stats_pkts += (rx - rx_pkts);
+		rq->soft_stats_bytes += bytes;
+	}
+
 	return rx - rx_pkts;
 }
 
drivers/net/enic/enic_sriov.c new file +801 -0

801 lines not on this page · raw message

drivers/net/enic/enic_sriov.h new file +211 -0

211 lines not on this page · raw message

drivers/net/enic/meson.build +1 -0
index 8700ae27f2..00f8348348 100644
@@ -23,6 +23,7 @@ sources = files(
         'enic_main.c',
         'enic_res.c',
         'enic_rxtx.c',
+        'enic_sriov.c',
         'enic_vf_representor.c',
 )
 deps += ['hash']
Signature · 1 line
2.35.2
#16 lore raw

PATCH v3 2/3 net/enic: add speed capabilities for newer models

In reply to Hyong Youb Kim

Add 1400/14000 and 15000 models to the speed_capa list.

---
 doc/guides/nics/enic.rst               |  3 ++-
 doc/guides/rel_notes/release_24_11.rst |  1 +
 drivers/net/enic/enic_ethdev.c         | 21 +++++++++++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)
doc/guides/nics/enic.rst +2 -1
index 639d3f5939..1295545c73 100644
@@ -17,7 +17,8 @@ ENIC PMD supports all recent generations of Cisco VIC adapters including:
 
 - VIC 1200 series
 - VIC 1300 series
-- VIC 1400 series
+- VIC 1400/14000 series
+- VIC 15000 series
 
 Supported features
 ------------------
doc/guides/rel_notes/release_24_11.rst +1 -0
index f79b1bee58..53dbf99324 100644
@@ -58,6 +58,7 @@ New Features
 * **Updated Cisco enic driver.**
 
   * Added SR-IOV VF support.
+  * Added recent 1400/14000 and 15000 models to the supported list.
 
 
 Removed Items
drivers/net/enic/enic_ethdev.c +21 -0
index 5c967677fb..62c8751d09 100644
@@ -62,6 +62,27 @@ static const struct vic_speed_capa {
 	{ 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */
 	{ 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1495 PCIe */
 	{ 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1497 MLOM */
+	{ 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G }, /* 1467 MLOM */
+	{ 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1477 MLOM */
+	{ 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */
+	{ 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */
+	{ 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */
+	{ 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */
+	{ 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */
+	{ 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */
+	{ 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */
+	{ 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */
+	{ 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G }, /* 15230 MLOM */
+	{ 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G }, /* 15422 Mezz */
+	{ 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */
+	{ 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G |
+		  RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */
+	{ 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G |
+		  RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */
 	{ 0, 0 }, /* End marker */
 };
 
Signature · 1 line
2.35.2
#17 lore raw

PATCH v3 3/3 net/enic: allow multicast in MAC address add callback

In reply to Hyong Youb Kim

enic_set_mac_address() (mac_addr_add callback) currently allows only non-zero, unicast address to be added. It is overly restrictive. rte_eth_dev_mac_addr_add() itself allows multicast addresses. And, some applications do use rte_eth_dev_mac_addr_add() to accept multicast addresses. So, remove the unicast check in enic_set_mac_address().

---
 drivers/net/enic/enic_main.c | 20 --------------------
 1 file changed, 20 deletions(-)
drivers/net/enic/enic_main.c +0 -20
index e59b0a5077..b755b15d92 100644
@@ -32,21 +32,6 @@
 #include "vnic_intr.h"
 #include "vnic_nic.h"
 
-static int is_zero_addr(uint8_t *addr)
-{
-	return !(addr[0] |  addr[1] | addr[2] | addr[3] | addr[4] | addr[5]);
-}
-
-static int is_mcast_addr(uint8_t *addr)
-{
-	return addr[0] & 1;
-}
-
-static int is_eth_addr_valid(uint8_t *addr)
-{
-	return !is_mcast_addr(addr) && !is_zero_addr(addr);
-}
-
 void
 enic_rxmbuf_queue_release(__rte_unused struct enic *enic, struct vnic_rq *rq)
 {
@@ -177,11 +162,6 @@ int enic_set_mac_address(struct enic *enic, uint8_t *mac_addr)
 {
 	int err;
 
-	if (!is_eth_addr_valid(mac_addr)) {
-		dev_err(enic, "invalid mac address\n");
-		return -EINVAL;
-	}
-
 	err = enic_dev_add_addr(enic, mac_addr);
 	if (err)
 		dev_err(enic, "add mac addr failed\n");
Signature · 1 line
2.35.2
#18 lore raw
In reply to Hyong Youb Kim
Quoted 9 lines · Hyong Youb Kim

This series contains minor updates for net/enic. The first patch supports SR-IOV VF, which now requires the use of admin channel. The other patches are not related to VF, but included here to ease review.

---

v3:
* add spdx, doc, rel_notes (Ferruh's comments)

Quoted 8 lines

v2:
* fix compiler warnings

Hyong Youb Kim (3):
net/enic: support SR-IOV VF using admin channel net/enic: add speed capabilities for newer models net/enic: allow multicast in MAC address add callback

Series applied to dpdk-next-net/main, thanks.