net/enic: support VF and fix minor issues
Subsystems (2)
cover net/enic: support VF and fix minor issues
This series contains minor updates for net/enic. The first patch supports SR-IOV VF, which now requires the use of admin channel. The other patches are not related to VF, but included here to ease review.
Hyong Youb Kim (3):
net/enic: support SR-IOV VF using admin channel net/enic: add speed capabilities for newer models net/enic: allow multicast in MAC address add callback
drivers/net/enic/base/vnic_cq.c | 27 + drivers/net/enic/base/vnic_cq.h | 3 + drivers/net/enic/base/vnic_dev.c | 48 ++ drivers/net/enic/base/vnic_dev.h | 3 + drivers/net/enic/base/vnic_devcmd.h | 49 ++ drivers/net/enic/base/vnic_resource.h | 32 +- drivers/net/enic/base/vnic_rq.c | 27 + drivers/net/enic/base/vnic_rq.h | 7 + drivers/net/enic/base/vnic_wq.c | 37 +- drivers/net/enic/base/vnic_wq.h | 5 + drivers/net/enic/enic.h | 28 +- drivers/net/enic/enic_ethdev.c | 29 +- drivers/net/enic/enic_main.c | 97 ++-- drivers/net/enic/enic_res.c | 12 + drivers/net/enic/enic_rxtx.c | 20 + drivers/net/enic/enic_sriov.c | 804 ++++++++++++++++++++++++++ drivers/net/enic/enic_sriov.h | 209 +++++++ drivers/net/enic/meson.build | 1 + 18 files changed, 1394 insertions(+), 44 deletions(-) create mode 100644 drivers/net/enic/enic_sriov.c create mode 100644 drivers/net/enic/enic_sriov.h
Signature · 1 line
2.35.2
PATCH 1/3 net/enic: support SR-IOV VF using admin channel
The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).
The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.
The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.
For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.
Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.
- Backward compatibility mode
Old VIC PF drivers on some operating systems may support only VF_CAPABILITY_REQUEST message or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.
--- drivers/net/enic/base/vnic_cq.c | 27 + drivers/net/enic/base/vnic_cq.h | 3 + drivers/net/enic/base/vnic_dev.c | 48 ++ drivers/net/enic/base/vnic_dev.h | 3 + drivers/net/enic/base/vnic_devcmd.h | 49 ++ drivers/net/enic/base/vnic_resource.h | 32 +- drivers/net/enic/base/vnic_rq.c | 27 + drivers/net/enic/base/vnic_rq.h | 7 + drivers/net/enic/base/vnic_wq.c | 37 +- drivers/net/enic/base/vnic_wq.h | 5 + drivers/net/enic/enic.h | 28 +- drivers/net/enic/enic_ethdev.c | 8 +- drivers/net/enic/enic_main.c | 77 ++- drivers/net/enic/enic_res.c | 12 + drivers/net/enic/enic_rxtx.c | 20 + drivers/net/enic/enic_sriov.c | 804 ++++++++++++++++++++++++++ drivers/net/enic/enic_sriov.h | 209 +++++++ drivers/net/enic/meson.build | 1 + 18 files changed, 1373 insertions(+), 24 deletions(-) create mode 100644 drivers/net/enic/enic_sriov.c create mode 100644 drivers/net/enic/enic_sriov.h
drivers/net/enic/base/vnic_cq.c +27 -0
@@ -24,6 +24,7 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index, cq->index = index; cq->vdev = vdev; + cq->admin_chan = false; cq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_CQ , index);if (!cq->ctrl) {
@@ -40,6 +41,32 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,return 0 ; } +int vnic_admin_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index, +unsigned int socket_id,unsigned int desc_count,unsigned int desc_size) +{ +int err; +char res_name[RTE_MEMZONE_NAMESIZE ]; +static int instance; + + cq->index = index; + cq->vdev = vdev; + cq->admin_chan = true; + + cq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_ADMIN_CQ , index); +if (!cq->ctrl) { +pr_err ("Failed to get admin CQ[%u] resource\n" , index); +return -EINVAL ; + } + +snprintf (res_name,sizeof (res_name),"%d-admin-cq-%u" , instance++, index); + err =vnic_dev_alloc_desc_ring (vdev, &cq->ring, desc_count, desc_size, + socket_id, res_name); +if (err) +return err; + +return 0 ; +} + void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable, unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail, unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_cq.h +3 -0
@@ -59,12 +59,15 @@ struct vnic_cq {unsigned int tobe_rx_coal_timeval;ktime_t prev_ts;#endif +bool admin_chan; };void vnic_cq_free (struct vnic_cq *cq);int vnic_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index,unsigned int socket_id,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index, +unsigned int socket_id,unsigned int desc_count,unsigned int desc_size); void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable, unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail, unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_dev.c +48 -0
81 lines not on this page · raw message
drivers/net/enic/base/vnic_dev.h +3 -0
@@ -199,5 +199,8 @@ int vnic_dev_capable_geneve(struct vnic_dev *vdev);uint64_t vnic_dev_capable_cq_entry_size (struct vnic_dev *vdev);int vnic_dev_set_cq_entry_size (struct vnic_dev *vdev,uint32_t rq_idx,uint32_t size_flag); +int vnic_dev_alloc_sriov_stats_mem (struct vnic_dev *vdev); +int vnic_dev_sriov_stats (struct vnic_dev *vdev,struct vnic_sriov_stats **stats); +int vnic_dev_enable_admin_qp (struct vnic_dev *vdev,uint32_t enable);#endif /* _VNIC_DEV_H_ */
drivers/net/enic/base/vnic_devcmd.h +49 -0
59 lines not on this page · raw message
drivers/net/enic/base/vnic_resource.h +30 -2
@@ -38,8 +38,36 @@ enum vnic_res_type {RES_TYPE_MQ_WQ ,/* MQ Work queues */ RES_TYPE_MQ_RQ ,/* MQ Receive queues */ RES_TYPE_MQ_CQ ,/* MQ Completion queues */ -RES_TYPE_DEPRECATED1 ,/* Old version of devcmd 2 */ -RES_TYPE_DEVCMD2 ,/* Device control region */ +RES_TYPE_DEPRECATED1 ,/* Old version of devcmd 2 */ +RES_TYPE_DEPRECATED2 ,/* Old version of devcmd 2 */ +RES_TYPE_DEVCMD2 ,/* Device control region */ +RES_TYPE_RDMA_WQ ,/* RDMA WQ */ +RES_TYPE_RDMA_RQ ,/* RDMA RQ */ +RES_TYPE_RDMA_CQ ,/* RDMA CQ */ +RES_TYPE_RDMA_RKEY_TABLE ,/* RDMA RKEY table */ +RES_TYPE_RDMA_RQ_HEADER_TABLE ,/* RDMA RQ Header Table */ +RES_TYPE_RDMA_RQ_TABLE ,/* RDMA RQ Table */ +RES_TYPE_RDMA_RD_RESP_HEADER_TABLE ,/* RDMA Read Response Header Table */ +RES_TYPE_RDMA_RD_RESP_TABLE ,/* RDMA Read Response Table */ +RES_TYPE_RDMA_QP_STATS_TABLE ,/* RDMA per QP stats table */ +RES_TYPE_WQ_MREGS ,/* XXX snic proto only */ +RES_TYPE_GRPMBR_INTR ,/* Group member interrupt control */ +RES_TYPE_DPKT ,/* Direct Packet memory region */ +RES_TYPE_RDMA2_DATA_WQ ,/* RDMA datapath command WQ */ +RES_TYPE_RDMA2_REG_WQ ,/* RDMA registration command WQ */ +RES_TYPE_RDMA2_CQ ,/* RDMA datapath CQ */ +RES_TYPE_MQ_RDMA2_DATA_WQ ,/* RDMA datapath command WQ */ +RES_TYPE_MQ_RDMA2_REG_WQ ,/* RDMA registration command WQ */ +RES_TYPE_MQ_RDMA2_CQ ,/* RDMA datapath CQ */ +RES_TYPE_PTP ,/* PTP registers */ +RES_TYPE_INTR_CTRL2 ,/* Extended INTR CTRL registers */ +RES_TYPE_SRIOV_INTR ,/* VF intr */ +RES_TYPE_VF_WQ ,/* VF WQ */ +RES_TYPE_VF_RQ ,/* VF RQ */ +RES_TYPE_VF_CQ ,/* VF CQ */ +RES_TYPE_ADMIN_WQ ,/* admin channel WQ */ +RES_TYPE_ADMIN_RQ ,/* admin channel RQ */ +RES_TYPE_ADMIN_CQ ,/* admin channel CQ */ RES_TYPE_MAX, /* Count of resource types */ };
drivers/net/enic/base/vnic_rq.c +27 -0
@@ -27,6 +27,7 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index, rq->index = index; rq->vdev = vdev; + rq->admin_chan = false; rq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_RQ , index);if (!rq->ctrl) {
@@ -42,6 +43,32 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,return rc; } +int vnic_admin_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq, +unsigned int desc_count,unsigned int desc_size) +{ +int rc; +char res_name[RTE_MEMZONE_NAMESIZE ]; +static int instance; + + rq->index =0 ; + rq->vdev = vdev; + rq->admin_chan = true; + rq->socket_id =SOCKET_ID_ANY ; + + rq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_ADMIN_RQ ,0 ); +if (!rq->ctrl) { +pr_err ("Failed to get admin RQ resource\n" ); +return -EINVAL ; + } + +vnic_rq_disable (rq); + +snprintf (res_name,sizeof (res_name),"%d-admin-rq" , instance++); + rc =vnic_dev_alloc_desc_ring (vdev, &rq->ring, desc_count, desc_size, + rq->socket_id, res_name); +return rc; +} + void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_rq.h +7 -0
@@ -73,6 +73,11 @@ struct vnic_rq {unsigned int max_mbufs_per_pkt;uint16_t tot_nb_desc;bool need_initial_post; +bool admin_chan; +const struct rte_memzone *admin_msg_rz; +int admin_next_idx; +uint64_t soft_stats_pkts; +uint64_t soft_stats_bytes; };static inline unsigned int vnic_rq_desc_avail (struct vnic_rq *rq)
@@ -127,6 +132,8 @@ static inline int vnic_rq_fill_count(struct vnic_rq *rq,void vnic_rq_free (struct vnic_rq *rq);int vnic_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq,unsigned int index,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq, +unsigned int desc_count,unsigned int desc_size); void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.c +35 -2
@@ -23,7 +23,8 @@ int vnic_wq_alloc_ring(struct vnic_dev *vdev, struct vnic_wq *wq,char res_name[RTE_MEMZONE_NAMESIZE ];static int instance; -snprintf (res_name,sizeof (res_name),"%d-wq-%u" , instance++, wq->index); +snprintf (res_name,sizeof (res_name),"%d-%swq-%u" , + instance++, wq->admin_chan ?"admin-" :"" , wq->index);return vnic_dev_alloc_desc_ring (vdev, &wq->ring, desc_count, desc_size, wq->socket_id, res_name); }
@@ -32,7 +33,7 @@ static int vnic_wq_alloc_bufs(struct vnic_wq *wq) {unsigned int count = wq->ring.desc_count;/* Allocate the mbuf ring */ - wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket ("wq->bufs" , + wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket (wq->admin_chan ?"admin-wq-bufs" :"wq-bufs" ,sizeof (struct rte_mbuf *) * count,RTE_CACHE_LINE_SIZE , wq->socket_id); wq->head_idx =0 ;
@@ -62,6 +63,7 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index, wq->index = index; wq->vdev = vdev; + wq->admin_chan = false; err =vnic_wq_get_ctrl (vdev, wq, index,RES_TYPE_WQ );if (err) {
@@ -84,6 +86,37 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,return 0 ; } +int vnic_admin_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq, +unsigned int desc_count,unsigned int desc_size) +{ +int err; + + wq->index =0 ; + wq->vdev = vdev; + wq->admin_chan = true; + wq->socket_id =SOCKET_ID_ANY ; + + err =vnic_wq_get_ctrl (vdev, wq,0 ,RES_TYPE_ADMIN_WQ ); +if (err) { +pr_err ("Failed to get admin WQ resource err %d\n" , err); +return err; + } + +vnic_wq_disable (wq); + + err =vnic_wq_alloc_ring (vdev, wq, desc_count, desc_size); +if (err) +return err; + + err =vnic_wq_alloc_bufs (wq); +if (err) { +vnic_wq_free (wq); +return err; + } + +return 0 ; +} + void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.h +5 -0
@@ -50,6 +50,9 @@ struct vnic_wq {const struct rte_memzone *cqmsg_rz;uint16_t last_completed_index;uint64_t offloads; +bool admin_chan; +const struct rte_memzone *admin_msg_rz; +uint64_t soft_stats_tx; };static inline unsigned int vnic_wq_desc_avail (struct vnic_wq *wq)
@@ -149,6 +152,8 @@ buf_idx_incr(uint32_t n_descriptors, uint32_t idx)void vnic_wq_free (struct vnic_wq *wq);int vnic_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq,unsigned int index,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq, +unsigned int desc_count,unsigned int desc_size); void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/enic.h +26 -2
@@ -43,7 +43,6 @@/*#define ENIC_DESC_COUNT_MAKE_ODD (x) do{if ((~(x)) & 1) { (x)--; } }while(0)*/ #define PCI_DEVICE_ID_CISCO_VIC_ENET 0x0043/* ethernet vnic */ -#define PCI_DEVICE_ID_CISCO_VIC_ENET_VF 0x0071/* enet SRIOV VF */ /* enet SRIOV Standalone vNic VF */ #define PCI_DEVICE_ID_CISCO_VIC_ENET_SN 0x02B7
@@ -51,7 +50,7 @@#define ENIC_MAGIC_FILTER_ID 0xffff/* -* Interrupt 0: LSC and errors +* Interrupt 0: LSC and errors / VF admin channel RQ * Interrupt 1: rx queue 0 * Interrupt 2: rx queue 1 * ...
@@ -154,11 +153,26 @@ struct enic {/* software counters */ struct enic_soft_stats soft_stats; +struct vnic_wq admin_wq; +struct vnic_rq admin_rq; +struct vnic_cq admin_cq[2 ]; +/* configured resources on vic */ unsigned int conf_rq_count;unsigned int conf_wq_count;unsigned int conf_cq_count;unsigned int conf_intr_count; +/* SR-IOV VF has queues for admin channel to PF */ +unsigned int conf_admin_rq_count; +unsigned int conf_admin_wq_count; +unsigned int conf_admin_cq_count; +uint64_t admin_chan_msg_num; +int admin_chan_vf_id; +uint32_t admin_pf_cap_version; +bool admin_chan_enabled; +bool sriov_vf_soft_rx_stats; +bool sriov_vf_compat_mode; +pthread_mutex_t admin_chan_lock;/* linked list storing memory allocations */ LIST_HEAD(enic_memzone_list, enic_memzone_entry) memzone_list;
@@ -230,6 +244,16 @@ struct enic_vf_representor {struct rte_flow *rep2vf_flow[2 ]; }; +#define ENIC_ADMIN_WQ_CQ 0 +#define ENIC_ADMIN_RQ_CQ 1 +#define ENIC_ADMIN_BUF_SIZE 1024 + +static inline bool enic_is_vf (struct enic *enic) +{ +return enic->pdev->id.device_id ==PCI_DEVICE_ID_CISCO_VIC_ENET_SN && + !enic->sriov_vf_compat_mode; +} +#define VF_ENIC_TO_VF_REP (vf_enic) \ container_of(vf_enic, struct enic_vf_representor, enic)
drivers/net/enic/enic_ethdev.c +4 -4
@@ -21,6 +21,7 @@#include "vnic_rq.h" #include "vnic_enet.h" #include "enic.h" +#include "enic_sriov.h" /* * The set of PCI devices this driver supports
@@ -28,7 +29,6 @@#define CISCO_PCI_VENDOR_ID 0x1137static const struct rte_pci_id pci_id_enic_map[] = { {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET )}, - {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET_VF )}, {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET_SN )}, {.vendor_id =0 ,/* sentinel */ }, };
@@ -707,7 +707,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,for (i =0 ; i < enic->mc_count; i++) { addr = &enic->mc_addrs[i];debug_log_add_del_addr (addr, false); - ret =vnic_dev_del_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_del_addr (enic, addr->addr_bytes);if (ret)return ret; }
@@ -734,7 +734,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,if (j < nb_mc_addr)continue ;debug_log_add_del_addr (addr, false); - ret =vnic_dev_del_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_del_addr (enic, addr->addr_bytes);if (ret)return ret; }
@@ -748,7 +748,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,if (j < enic->mc_count)continue ;debug_log_add_del_addr (addr, true); - ret =vnic_dev_add_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_add_addr (enic, addr->addr_bytes);if (ret)return ret; }
drivers/net/enic/enic_main.c +63 -14
191 lines not on this page · raw message
drivers/net/enic/enic_res.c +12 -0
@@ -300,4 +300,16 @@ void enic_get_res_counts(struct enic *enic)"vNIC resources avail: wq %d rq %d cq %d intr %d\n" , enic->conf_wq_count, enic->conf_rq_count, enic->conf_cq_count, enic->conf_intr_count); + +if (!enic_is_vf (enic)) +return ; + + enic->conf_admin_wq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_WQ ); + enic->conf_admin_rq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_RQ ); + enic->conf_admin_cq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_CQ ); + +dev_info (enic_get_dev (enic), +"vNIC admin channel resources avail: wq %d rq %d cq %d\n" , + enic->conf_admin_wq_count, enic->conf_admin_rq_count, + enic->conf_admin_cq_count); }
drivers/net/enic/enic_rxtx.c +20 -0
@@ -55,6 +55,7 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, sizeof(struct cq_enet_rq_desc_64) :sizeof (struct cq_enet_rq_desc );RTE_BUILD_BUG_ON (sizeof (struct cq_enet_rq_desc_64 ) !=64 ); +uint64_t bytes; cq = &enic->cq[enic_cq_rq (enic, sop_rq->index)]; cq_idx = cq->to_clean;/* index of cqd, rqd, mbuf_table */
@@ -67,6 +68,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,/* Receive until the end of the ring, at most. */ max_rx =RTE_MIN (nb_pkts, cq->ring.desc_count - cq_idx); + bytes =0 ; +while (max_rx) {volatile struct rq_enet_desc *rqd_ptr;struct cq_desc cqd;
@@ -155,6 +158,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, rxmb->port = enic->port_id; rxmb->data_len = seg_length; + bytes += seg_length; + rq->rx_nb_hold++;if (!(enic_cq_rx_desc_eop (ciflags))) {
@@ -225,6 +230,10 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, &sop_rq->ctrl->posted_index); } +if (enic->sriov_vf_soft_rx_stats && bytes) { + sop_rq->soft_stats_pkts += nb_rx; + sop_rq->soft_stats_bytes += bytes; + }return nb_rx; }
@@ -256,6 +265,7 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,uint8_t color;bool overlay;bool tnl; +uint64_t bytes; rq = rx_queue; enic =vnic_dev_priv (rq->vdev);
@@ -283,6 +293,8 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, port_id = enic->port_id; overlay = enic->overlay_offload; + bytes =0 ; + rx = rx_pkts;while (max_rx) { max_rx--;
@@ -304,6 +316,9 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK ; mb->pkt_len = mb->data_len; mb->port = port_id; + + bytes += mb->pkt_len; + tnl = overlay && (cqd->completed_index_flags &CQ_ENET_RQ_DESC_FLAGS_FCOE ) !=0 ; mb->packet_type =
@@ -352,6 +367,11 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, &rq->ctrl->posted_index); } +if (enic->sriov_vf_soft_rx_stats && bytes) { + rq->soft_stats_pkts += (rx - rx_pkts); + rq->soft_stats_bytes += bytes; + } +return rx - rx_pkts; }
drivers/net/enic/enic_sriov.c new file +804 -0
804 lines not on this page · raw message
drivers/net/enic/enic_sriov.h new file +209 -0
209 lines not on this page · raw message
drivers/net/enic/meson.build +1 -0
@@ -23,6 +23,7 @@ sources = files( 'enic_main.c', 'enic_res.c', 'enic_rxtx.c', + 'enic_sriov.c', 'enic_vf_representor.c', ) deps += ['hash']
Signature · 1 line
2.35.2
PATCH 2/3 net/enic: add speed capabilities for newer models
In reply to Hyong Youb KimAdd 1400/14000 and 15000 models to the speed_capa list.
--- drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
drivers/net/enic/enic_ethdev.c +21 -0
@@ -62,6 +62,27 @@ static const struct vic_speed_capa { { 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */ { 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1495 PCIe */ { 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1497 MLOM */ + { 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G }, /* 1467 MLOM */ + { 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1477 MLOM */ + { 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */ + { 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */ + { 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */ + { 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */ + { 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */ + { 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */ + { 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */ + { 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */ + { 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G }, /* 15230 MLOM */ + { 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G }, /* 15422 Mezz */ + { 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */ + { 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */ + { 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */ { 0, 0 }, /* End marker */ };
Signature · 1 line
2.35.2
PATCH 3/3 net/enic: allow multicast in MAC address add callback
In reply to Hyong Youb Kimenic_set_mac_address() (mac_addr_add callback) currently allows only non-zero, unicast address to be added. It is overly restrictive. rte_eth_dev_mac_addr_add() itself allows multicast addresses. And, some applications do use rte_eth_dev_mac_addr_add() to accept multicast addresses. So, remove the unicast check in enic_set_mac_address().
--- drivers/net/enic/enic_main.c | 20 -------------------- 1 file changed, 20 deletions(-)
drivers/net/enic/enic_main.c +0 -20
@@ -32,21 +32,6 @@#include "vnic_intr.h" #include "vnic_nic.h" -static int is_zero_addr (uint8_t *addr) -{ -return !(addr[0 ] | addr[1 ] | addr[2 ] | addr[3 ] | addr[4 ] | addr[5 ]); -} - -static int is_mcast_addr (uint8_t *addr) -{ -return addr[0 ] &1 ; -} - -static int is_eth_addr_valid (uint8_t *addr) -{ -return !is_mcast_addr (addr) && !is_zero_addr (addr); -} -void enic_rxmbuf_queue_release(__rte_unused struct enic *enic, struct vnic_rq *rq) {
@@ -177,11 +162,6 @@ int enic_set_mac_address(struct enic *enic, uint8_t *mac_addr) {int err; -if (!is_eth_addr_valid (mac_addr)) { -dev_err (enic,"invalid mac address\n" ); -return -EINVAL ; - } - err =enic_dev_add_addr (enic, mac_addr);if (err)dev_err (enic,"add mac addr failed\n" );
Signature · 1 line
2.35.2
v2 cover net/enic: support VF and fix minor issues
This series contains minor updates for net/enic. The first patch supports SR-IOV VF, which now requires the use of admin channel. The other patches are not related to VF, but included here to ease review.
---
v2:
* fix compiler warnings
Hyong Youb Kim (3):
net/enic: support SR-IOV VF using admin channel net/enic: add speed capabilities for newer models net/enic: allow multicast in MAC address add callback
drivers/net/enic/base/vnic_cq.c | 27 + drivers/net/enic/base/vnic_cq.h | 3 + drivers/net/enic/base/vnic_dev.c | 48 ++ drivers/net/enic/base/vnic_dev.h | 3 + drivers/net/enic/base/vnic_devcmd.h | 49 ++ drivers/net/enic/base/vnic_resource.h | 32 +- drivers/net/enic/base/vnic_rq.c | 27 + drivers/net/enic/base/vnic_rq.h | 7 + drivers/net/enic/base/vnic_wq.c | 37 +- drivers/net/enic/base/vnic_wq.h | 5 + drivers/net/enic/enic.h | 28 +- drivers/net/enic/enic_ethdev.c | 29 +- drivers/net/enic/enic_main.c | 97 ++-- drivers/net/enic/enic_res.c | 12 + drivers/net/enic/enic_rxtx.c | 20 + drivers/net/enic/enic_sriov.c | 801 ++++++++++++++++++++++++++ drivers/net/enic/enic_sriov.h | 209 +++++++ drivers/net/enic/meson.build | 1 + 18 files changed, 1391 insertions(+), 44 deletions(-) create mode 100644 drivers/net/enic/enic_sriov.c create mode 100644 drivers/net/enic/enic_sriov.h
Signature · 1 line
2.35.2
PATCH v2 1/3 net/enic: support SR-IOV VF using admin channel
The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).
The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.
The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.
For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.
Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.
- Backward compatibility mode
Old VIC PF drivers on some operating systems may support only VF_CAPABILITY_REQUEST message or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.
--- drivers/net/enic/base/vnic_cq.c | 27 + drivers/net/enic/base/vnic_cq.h | 3 + drivers/net/enic/base/vnic_dev.c | 48 ++ drivers/net/enic/base/vnic_dev.h | 3 + drivers/net/enic/base/vnic_devcmd.h | 49 ++ drivers/net/enic/base/vnic_resource.h | 32 +- drivers/net/enic/base/vnic_rq.c | 27 + drivers/net/enic/base/vnic_rq.h | 7 + drivers/net/enic/base/vnic_wq.c | 37 +- drivers/net/enic/base/vnic_wq.h | 5 + drivers/net/enic/enic.h | 28 +- drivers/net/enic/enic_ethdev.c | 8 +- drivers/net/enic/enic_main.c | 77 ++- drivers/net/enic/enic_res.c | 12 + drivers/net/enic/enic_rxtx.c | 20 + drivers/net/enic/enic_sriov.c | 801 ++++++++++++++++++++++++++ drivers/net/enic/enic_sriov.h | 209 +++++++ drivers/net/enic/meson.build | 1 + 18 files changed, 1370 insertions(+), 24 deletions(-) create mode 100644 drivers/net/enic/enic_sriov.c create mode 100644 drivers/net/enic/enic_sriov.h
drivers/net/enic/base/vnic_cq.c +27 -0
@@ -24,6 +24,7 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index, cq->index = index; cq->vdev = vdev; + cq->admin_chan = false; cq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_CQ , index);if (!cq->ctrl) {
@@ -40,6 +41,32 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,return 0 ; } +int vnic_admin_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index, +unsigned int socket_id,unsigned int desc_count,unsigned int desc_size) +{ +int err; +char res_name[RTE_MEMZONE_NAMESIZE ]; +static int instance; + + cq->index = index; + cq->vdev = vdev; + cq->admin_chan = true; + + cq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_ADMIN_CQ , index); +if (!cq->ctrl) { +pr_err ("Failed to get admin CQ[%u] resource\n" , index); +return -EINVAL ; + } + +snprintf (res_name,sizeof (res_name),"%d-admin-cq-%u" , instance++, index); + err =vnic_dev_alloc_desc_ring (vdev, &cq->ring, desc_count, desc_size, + socket_id, res_name); +if (err) +return err; + +return 0 ; +} + void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable, unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail, unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_cq.h +3 -0
@@ -59,12 +59,15 @@ struct vnic_cq {unsigned int tobe_rx_coal_timeval;ktime_t prev_ts;#endif +bool admin_chan; };void vnic_cq_free (struct vnic_cq *cq);int vnic_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index,unsigned int socket_id,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index, +unsigned int socket_id,unsigned int desc_count,unsigned int desc_size); void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable, unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail, unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_dev.c +48 -0
81 lines not on this page · raw message
drivers/net/enic/base/vnic_dev.h +3 -0
@@ -199,5 +199,8 @@ int vnic_dev_capable_geneve(struct vnic_dev *vdev);uint64_t vnic_dev_capable_cq_entry_size (struct vnic_dev *vdev);int vnic_dev_set_cq_entry_size (struct vnic_dev *vdev,uint32_t rq_idx,uint32_t size_flag); +int vnic_dev_alloc_sriov_stats_mem (struct vnic_dev *vdev); +int vnic_dev_sriov_stats (struct vnic_dev *vdev,struct vnic_sriov_stats **stats); +int vnic_dev_enable_admin_qp (struct vnic_dev *vdev,uint32_t enable);#endif /* _VNIC_DEV_H_ */
drivers/net/enic/base/vnic_devcmd.h +49 -0
59 lines not on this page · raw message
drivers/net/enic/base/vnic_resource.h +30 -2
@@ -38,8 +38,36 @@ enum vnic_res_type {RES_TYPE_MQ_WQ ,/* MQ Work queues */ RES_TYPE_MQ_RQ ,/* MQ Receive queues */ RES_TYPE_MQ_CQ ,/* MQ Completion queues */ -RES_TYPE_DEPRECATED1 ,/* Old version of devcmd 2 */ -RES_TYPE_DEVCMD2 ,/* Device control region */ +RES_TYPE_DEPRECATED1 ,/* Old version of devcmd 2 */ +RES_TYPE_DEPRECATED2 ,/* Old version of devcmd 2 */ +RES_TYPE_DEVCMD2 ,/* Device control region */ +RES_TYPE_RDMA_WQ ,/* RDMA WQ */ +RES_TYPE_RDMA_RQ ,/* RDMA RQ */ +RES_TYPE_RDMA_CQ ,/* RDMA CQ */ +RES_TYPE_RDMA_RKEY_TABLE ,/* RDMA RKEY table */ +RES_TYPE_RDMA_RQ_HEADER_TABLE ,/* RDMA RQ Header Table */ +RES_TYPE_RDMA_RQ_TABLE ,/* RDMA RQ Table */ +RES_TYPE_RDMA_RD_RESP_HEADER_TABLE ,/* RDMA Read Response Header Table */ +RES_TYPE_RDMA_RD_RESP_TABLE ,/* RDMA Read Response Table */ +RES_TYPE_RDMA_QP_STATS_TABLE ,/* RDMA per QP stats table */ +RES_TYPE_WQ_MREGS ,/* XXX snic proto only */ +RES_TYPE_GRPMBR_INTR ,/* Group member interrupt control */ +RES_TYPE_DPKT ,/* Direct Packet memory region */ +RES_TYPE_RDMA2_DATA_WQ ,/* RDMA datapath command WQ */ +RES_TYPE_RDMA2_REG_WQ ,/* RDMA registration command WQ */ +RES_TYPE_RDMA2_CQ ,/* RDMA datapath CQ */ +RES_TYPE_MQ_RDMA2_DATA_WQ ,/* RDMA datapath command WQ */ +RES_TYPE_MQ_RDMA2_REG_WQ ,/* RDMA registration command WQ */ +RES_TYPE_MQ_RDMA2_CQ ,/* RDMA datapath CQ */ +RES_TYPE_PTP ,/* PTP registers */ +RES_TYPE_INTR_CTRL2 ,/* Extended INTR CTRL registers */ +RES_TYPE_SRIOV_INTR ,/* VF intr */ +RES_TYPE_VF_WQ ,/* VF WQ */ +RES_TYPE_VF_RQ ,/* VF RQ */ +RES_TYPE_VF_CQ ,/* VF CQ */ +RES_TYPE_ADMIN_WQ ,/* admin channel WQ */ +RES_TYPE_ADMIN_RQ ,/* admin channel RQ */ +RES_TYPE_ADMIN_CQ ,/* admin channel CQ */ RES_TYPE_MAX, /* Count of resource types */ };
drivers/net/enic/base/vnic_rq.c +27 -0
@@ -27,6 +27,7 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index, rq->index = index; rq->vdev = vdev; + rq->admin_chan = false; rq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_RQ , index);if (!rq->ctrl) {
@@ -42,6 +43,32 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,return rc; } +int vnic_admin_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq, +unsigned int desc_count,unsigned int desc_size) +{ +int rc; +char res_name[RTE_MEMZONE_NAMESIZE ]; +static int instance; + + rq->index =0 ; + rq->vdev = vdev; + rq->admin_chan = true; + rq->socket_id =SOCKET_ID_ANY ; + + rq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_ADMIN_RQ ,0 ); +if (!rq->ctrl) { +pr_err ("Failed to get admin RQ resource\n" ); +return -EINVAL ; + } + +vnic_rq_disable (rq); + +snprintf (res_name,sizeof (res_name),"%d-admin-rq" , instance++); + rc =vnic_dev_alloc_desc_ring (vdev, &rq->ring, desc_count, desc_size, + rq->socket_id, res_name); +return rc; +} + void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_rq.h +7 -0
@@ -73,6 +73,11 @@ struct vnic_rq {unsigned int max_mbufs_per_pkt;uint16_t tot_nb_desc;bool need_initial_post; +bool admin_chan; +const struct rte_memzone *admin_msg_rz; +int admin_next_idx; +uint64_t soft_stats_pkts; +uint64_t soft_stats_bytes; };static inline unsigned int vnic_rq_desc_avail (struct vnic_rq *rq)
@@ -127,6 +132,8 @@ static inline int vnic_rq_fill_count(struct vnic_rq *rq,void vnic_rq_free (struct vnic_rq *rq);int vnic_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq,unsigned int index,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq, +unsigned int desc_count,unsigned int desc_size); void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.c +35 -2
@@ -23,7 +23,8 @@ int vnic_wq_alloc_ring(struct vnic_dev *vdev, struct vnic_wq *wq,char res_name[RTE_MEMZONE_NAMESIZE ];static int instance; -snprintf (res_name,sizeof (res_name),"%d-wq-%u" , instance++, wq->index); +snprintf (res_name,sizeof (res_name),"%d-%swq-%u" , + instance++, wq->admin_chan ?"admin-" :"" , wq->index);return vnic_dev_alloc_desc_ring (vdev, &wq->ring, desc_count, desc_size, wq->socket_id, res_name); }
@@ -32,7 +33,7 @@ static int vnic_wq_alloc_bufs(struct vnic_wq *wq) {unsigned int count = wq->ring.desc_count;/* Allocate the mbuf ring */ - wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket ("wq->bufs" , + wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket (wq->admin_chan ?"admin-wq-bufs" :"wq-bufs" ,sizeof (struct rte_mbuf *) * count,RTE_CACHE_LINE_SIZE , wq->socket_id); wq->head_idx =0 ;
@@ -62,6 +63,7 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index, wq->index = index; wq->vdev = vdev; + wq->admin_chan = false; err =vnic_wq_get_ctrl (vdev, wq, index,RES_TYPE_WQ );if (err) {
@@ -84,6 +86,37 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,return 0 ; } +int vnic_admin_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq, +unsigned int desc_count,unsigned int desc_size) +{ +int err; + + wq->index =0 ; + wq->vdev = vdev; + wq->admin_chan = true; + wq->socket_id =SOCKET_ID_ANY ; + + err =vnic_wq_get_ctrl (vdev, wq,0 ,RES_TYPE_ADMIN_WQ ); +if (err) { +pr_err ("Failed to get admin WQ resource err %d\n" , err); +return err; + } + +vnic_wq_disable (wq); + + err =vnic_wq_alloc_ring (vdev, wq, desc_count, desc_size); +if (err) +return err; + + err =vnic_wq_alloc_bufs (wq); +if (err) { +vnic_wq_free (wq); +return err; + } + +return 0 ; +} + void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.h +5 -0
@@ -50,6 +50,9 @@ struct vnic_wq {const struct rte_memzone *cqmsg_rz;uint16_t last_completed_index;uint64_t offloads; +bool admin_chan; +const struct rte_memzone *admin_msg_rz; +uint64_t soft_stats_tx; };static inline unsigned int vnic_wq_desc_avail (struct vnic_wq *wq)
@@ -149,6 +152,8 @@ buf_idx_incr(uint32_t n_descriptors, uint32_t idx)void vnic_wq_free (struct vnic_wq *wq);int vnic_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq,unsigned int index,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq, +unsigned int desc_count,unsigned int desc_size); void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/enic.h +26 -2
@@ -43,7 +43,6 @@/*#define ENIC_DESC_COUNT_MAKE_ODD (x) do{if ((~(x)) & 1) { (x)--; } }while(0)*/ #define PCI_DEVICE_ID_CISCO_VIC_ENET 0x0043/* ethernet vnic */ -#define PCI_DEVICE_ID_CISCO_VIC_ENET_VF 0x0071/* enet SRIOV VF */ /* enet SRIOV Standalone vNic VF */ #define PCI_DEVICE_ID_CISCO_VIC_ENET_SN 0x02B7
@@ -51,7 +50,7 @@#define ENIC_MAGIC_FILTER_ID 0xffff/* -* Interrupt 0: LSC and errors +* Interrupt 0: LSC and errors / VF admin channel RQ * Interrupt 1: rx queue 0 * Interrupt 2: rx queue 1 * ...
@@ -154,11 +153,26 @@ struct enic {/* software counters */ struct enic_soft_stats soft_stats; +struct vnic_wq admin_wq; +struct vnic_rq admin_rq; +struct vnic_cq admin_cq[2 ]; +/* configured resources on vic */ unsigned int conf_rq_count;unsigned int conf_wq_count;unsigned int conf_cq_count;unsigned int conf_intr_count; +/* SR-IOV VF has queues for admin channel to PF */ +unsigned int conf_admin_rq_count; +unsigned int conf_admin_wq_count; +unsigned int conf_admin_cq_count; +uint64_t admin_chan_msg_num; +int admin_chan_vf_id; +uint32_t admin_pf_cap_version; +bool admin_chan_enabled; +bool sriov_vf_soft_rx_stats; +bool sriov_vf_compat_mode; +pthread_mutex_t admin_chan_lock;/* linked list storing memory allocations */ LIST_HEAD(enic_memzone_list, enic_memzone_entry) memzone_list;
@@ -230,6 +244,16 @@ struct enic_vf_representor {struct rte_flow *rep2vf_flow[2 ]; }; +#define ENIC_ADMIN_WQ_CQ 0 +#define ENIC_ADMIN_RQ_CQ 1 +#define ENIC_ADMIN_BUF_SIZE 1024 + +static inline bool enic_is_vf (struct enic *enic) +{ +return enic->pdev->id.device_id ==PCI_DEVICE_ID_CISCO_VIC_ENET_SN && + !enic->sriov_vf_compat_mode; +} +#define VF_ENIC_TO_VF_REP (vf_enic) \ container_of(vf_enic, struct enic_vf_representor, enic)
drivers/net/enic/enic_ethdev.c +4 -4
@@ -21,6 +21,7 @@#include "vnic_rq.h" #include "vnic_enet.h" #include "enic.h" +#include "enic_sriov.h" /* * The set of PCI devices this driver supports
@@ -28,7 +29,6 @@#define CISCO_PCI_VENDOR_ID 0x1137static const struct rte_pci_id pci_id_enic_map[] = { {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET )}, - {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET_VF )}, {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET_SN )}, {.vendor_id =0 ,/* sentinel */ }, };
@@ -707,7 +707,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,for (i =0 ; i < enic->mc_count; i++) { addr = &enic->mc_addrs[i];debug_log_add_del_addr (addr, false); - ret =vnic_dev_del_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_del_addr (enic, addr->addr_bytes);if (ret)return ret; }
@@ -734,7 +734,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,if (j < nb_mc_addr)continue ;debug_log_add_del_addr (addr, false); - ret =vnic_dev_del_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_del_addr (enic, addr->addr_bytes);if (ret)return ret; }
@@ -748,7 +748,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,if (j < enic->mc_count)continue ;debug_log_add_del_addr (addr, true); - ret =vnic_dev_add_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_add_addr (enic, addr->addr_bytes);if (ret)return ret; }
drivers/net/enic/enic_main.c +63 -14
191 lines not on this page · raw message
drivers/net/enic/enic_res.c +12 -0
@@ -300,4 +300,16 @@ void enic_get_res_counts(struct enic *enic)"vNIC resources avail: wq %d rq %d cq %d intr %d\n" , enic->conf_wq_count, enic->conf_rq_count, enic->conf_cq_count, enic->conf_intr_count); + +if (!enic_is_vf (enic)) +return ; + + enic->conf_admin_wq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_WQ ); + enic->conf_admin_rq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_RQ ); + enic->conf_admin_cq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_CQ ); + +dev_info (enic_get_dev (enic), +"vNIC admin channel resources avail: wq %d rq %d cq %d\n" , + enic->conf_admin_wq_count, enic->conf_admin_rq_count, + enic->conf_admin_cq_count); }
drivers/net/enic/enic_rxtx.c +20 -0
@@ -55,6 +55,7 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, sizeof(struct cq_enet_rq_desc_64) :sizeof (struct cq_enet_rq_desc );RTE_BUILD_BUG_ON (sizeof (struct cq_enet_rq_desc_64 ) !=64 ); +uint64_t bytes; cq = &enic->cq[enic_cq_rq (enic, sop_rq->index)]; cq_idx = cq->to_clean;/* index of cqd, rqd, mbuf_table */
@@ -67,6 +68,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,/* Receive until the end of the ring, at most. */ max_rx =RTE_MIN (nb_pkts, cq->ring.desc_count - cq_idx); + bytes =0 ; +while (max_rx) {volatile struct rq_enet_desc *rqd_ptr;struct cq_desc cqd;
@@ -155,6 +158,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, rxmb->port = enic->port_id; rxmb->data_len = seg_length; + bytes += seg_length; + rq->rx_nb_hold++;if (!(enic_cq_rx_desc_eop (ciflags))) {
@@ -225,6 +230,10 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, &sop_rq->ctrl->posted_index); } +if (enic->sriov_vf_soft_rx_stats && bytes) { + sop_rq->soft_stats_pkts += nb_rx; + sop_rq->soft_stats_bytes += bytes; + }return nb_rx; }
@@ -256,6 +265,7 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,uint8_t color;bool overlay;bool tnl; +uint64_t bytes; rq = rx_queue; enic =vnic_dev_priv (rq->vdev);
@@ -283,6 +293,8 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, port_id = enic->port_id; overlay = enic->overlay_offload; + bytes =0 ; + rx = rx_pkts;while (max_rx) { max_rx--;
@@ -304,6 +316,9 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK ; mb->pkt_len = mb->data_len; mb->port = port_id; + + bytes += mb->pkt_len; + tnl = overlay && (cqd->completed_index_flags &CQ_ENET_RQ_DESC_FLAGS_FCOE ) !=0 ; mb->packet_type =
@@ -352,6 +367,11 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, &rq->ctrl->posted_index); } +if (enic->sriov_vf_soft_rx_stats && bytes) { + rq->soft_stats_pkts += (rx - rx_pkts); + rq->soft_stats_bytes += bytes; + } +return rx - rx_pkts; }
drivers/net/enic/enic_sriov.c new file +801 -0
801 lines not on this page · raw message
drivers/net/enic/enic_sriov.h new file +209 -0
209 lines not on this page · raw message
drivers/net/enic/meson.build +1 -0
@@ -23,6 +23,7 @@ sources = files( 'enic_main.c', 'enic_res.c', 'enic_rxtx.c', + 'enic_sriov.c', 'enic_vf_representor.c', ) deps += ['hash']
Signature · 1 line
2.35.2
Re: [PATCH v2 1/3] net/enic: support SR-IOV VF using admin channel
Quoted 24 lines · Hyong Youb Kim
The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).
The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.
The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.
For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.
Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.
Quoted 7 lines
- Backward compatibility mode
Old VIC PF drivers on some operating systems may support onlyVF_CAPABILITY_REQUESTmessage or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.
Do you think does it worth to document above restrictions in the driver guide?
Quoted 12 lines
Signed-off-by Hyong Youb Kim <[email protected]>Reviewed-by John Daley <[email protected]>--- drivers/net/enic/base/vnic_cq.c | 27 + drivers/net/enic/base/vnic_cq.h | 3 + drivers/net/enic/base/vnic_dev.c | 48 ++ drivers/net/enic/base/vnic_dev.h | 3 + drivers/net/enic/base/vnic_devcmd.h | 49 ++ drivers/net/enic/base/vnic_resource.h | 32 +- drivers/net/enic/base/vnic_rq.c | 27 + drivers/net/enic/base/vnic_rq.h | 7 + drivers/net/enic/base/vnic_wq.c | 37 +-
Quoted 9 lines
drivers/net/enic/base/vnic_wq.h | 5 + drivers/net/enic/enic.h | 28 +- drivers/net/enic/enic_ethdev.c | 8 +- drivers/net/enic/enic_main.c | 77 ++- drivers/net/enic/enic_res.c | 12 + drivers/net/enic/enic_rxtx.c | 20 + drivers/net/enic/enic_sriov.c | 801 ++++++++++++++++++++++++++ drivers/net/enic/enic_sriov.h | 209 +++++++
This new file is missing coopyright and license information. (Detected by './devtools/check-spdx-tag.sh' script)
<...>
Quoted 13 lines
drivers/net/enic/enic_ethdev.c +1
-0@@ -21,6 +21,7 @@#include "vnic_rq.h" #include "vnic_enet.h" #include "enic.h" +#include "enic_sriov.h" /* * The set of PCI devices this driver supports
drivers/net/enic/enic_ethdev.c line 33
Quoted 6 lines
drivers/net/enic/enic_ethdev.c +0
-1@@ -28,7 +29,6 @@#define CISCO_PCI_VENDOR_ID 0x1137static const struct rte_pci_id pci_id_enic_map[] = { {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET )}, - {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET_VF )}, {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET_SN )},
Won't there be a specific PCIe device ID for the VF? Can there be some old devices around that requires this support?
What is the difference between PCI_DEVICE_ID_CISCO_VIC_ENET_VF & PCI_DEVICE_ID_CISCO_VIC_ENET_SN?
Current code seems detecting device as VF when the PCIe device ID is PCI_DEVICE_ID_CISCO_VIC_ENET_SN.
RE: [PATCH v2 1/3] net/enic: support SR-IOV VF using admin channel
Quoted 42 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, August 8, 2024 5:50 PM
To: Hyong Youb Kim (hyonkim) <[email protected]>
Cc: [email protected]; John Daley (johndale) <[email protected]>
Subject: Re: [PATCH v2 1/3] net/enic: support SR-IOV VF using admin channelQuoted 24 lines · Hyong Youb Kim
The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).
The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.
The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.
For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.
Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.Quoted 7 lines
- Backward compatibility mode
Old VIC PF drivers on some operating systems may support onlyVF_CAPABILITY_REQUESTmessage or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.
Quoted 3 lines
Do you think does it worth to document above restrictions in the driver guide?
I updated doc and release notes. Thanks for the reminder.
Quoted 22 lines
Quoted 12 lines
Signed-off-by Hyong Youb Kim <[email protected]>Reviewed-by John Daley <[email protected]>--- drivers/net/enic/base/vnic_cq.c | 27 + drivers/net/enic/base/vnic_cq.h | 3 + drivers/net/enic/base/vnic_dev.c | 48 ++ drivers/net/enic/base/vnic_dev.h | 3 + drivers/net/enic/base/vnic_devcmd.h | 49 ++ drivers/net/enic/base/vnic_resource.h | 32 +- drivers/net/enic/base/vnic_rq.c | 27 + drivers/net/enic/base/vnic_rq.h | 7 + drivers/net/enic/base/vnic_wq.c | 37 +-Quoted 9 lines
drivers/net/enic/base/vnic_wq.h | 5 + drivers/net/enic/enic.h | 28 +- drivers/net/enic/enic_ethdev.c | 8 +- drivers/net/enic/enic_main.c | 77 ++- drivers/net/enic/enic_res.c | 12 + drivers/net/enic/enic_rxtx.c | 20 + drivers/net/enic/enic_sriov.c | 801 ++++++++++++++++++++++++++ drivers/net/enic/enic_sriov.h | 209 +++++++
Quoted 3 lines
This new file is missing coopyright and license information. (Detected by '
./devtools/check-spdx-tag.sh' script)
Added SPDX.
Quoted 25 lines
<...>
Quoted 13 lines
drivers/net/enic/enic_ethdev.c +1
-0@@ -21,6 +21,7 @@#include "vnic_rq.h" #include "vnic_enet.h" #include "enic.h" +#include "enic_sriov.h" /* * The set of PCI devices this driver supports drivers/net/enic/enic_ethdev.c line 31
Quoted 3 lines
drivers/net/enic/enic_ethdev.c +0
-0@@ -28,7 +29,6 @@#define CISCO_PCI_VENDOR_ID 0x1137static const struct rte_pci_id pci_id_enic_map[] = { {RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID,
PCI_DEVICE_ID_CISCO_VIC_ENET)},drivers/net/enic/enic_ethdev.c line 31
Quoted 1 line
drivers/net/enic/enic_ethdev.c +0
-1- {RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID,
PCI_DEVICE_ID_CISCO_VIC_ENET_VF)},drivers/net/enic/enic_ethdev.c line 32
Quoted 1 line
drivers/net/enic/enic_ethdev.c +0
-0{RTE_PCI_DEVICE(CISCO_PCI_VENDOR_ID,
PCI_DEVICE_ID_CISCO_VIC_ENET_SN)},Quoted 1 line
Quoted 7 lines
Won't there be a specific PCIe device ID for the VF? Can there be some old devices around that requires this support?
What is the difference between
PCI_DEVICE_ID_CISCO_VIC_ENET_VF&PCI_DEVICE_ID_CISCO_VIC_ENET_SN?
Current code seems detecting device as VF when the PCIe device ID isPCI_DEVICE_ID_CISCO_VIC_ENET_SN.
The previous SR-IOV implementation was tied to Cisco VM-FEX. ENET_VF is the device ID for that. That has been discontinued for some time. There is no need to support that anymore.
The current SR-IOV implementation is not tied to VM-FEX. ENET_SN is the new device ID for this implementation. VF/SN naming is a bit confusing. But SN is the name used internally..
Thanks.
-Hyong
PATCH v2 2/3 net/enic: add speed capabilities for newer models
In reply to Hyong Youb KimAdd 1400/14000 and 15000 models to the speed_capa list.
--- drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
drivers/net/enic/enic_ethdev.c +21 -0
@@ -62,6 +62,27 @@ static const struct vic_speed_capa { { 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */ { 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1495 PCIe */ { 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1497 MLOM */ + { 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G }, /* 1467 MLOM */ + { 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1477 MLOM */ + { 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */ + { 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */ + { 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */ + { 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */ + { 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */ + { 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */ + { 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */ + { 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */ + { 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G }, /* 15230 MLOM */ + { 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G }, /* 15422 Mezz */ + { 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */ + { 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */ + { 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */ { 0, 0 }, /* End marker */ };
Signature · 1 line
2.35.2
Re: [PATCH v2 2/3] net/enic: add speed capabilities for newer models
Quoted 2 lines · Hyong Youb Kim
Add 1400/14000 and 15000 models to the
speed_capalist.
Do you want to introduce this new device variants in the driver guide?
Also does it makes sense to document new device model support in the release notes?
Quoted 38 lines
Signed-off-by Hyong Youb Kim <[email protected]>Reviewed-by John Daley <[email protected]>--- drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)drivers/net/enic/enic_ethdev.c +21
-0@@ -62,6 +62,27 @@ static const struct vic_speed_capa { { 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */ { 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1495 PCIe */ { 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1497 MLOM */ + { 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G }, /* 1467 MLOM */ + { 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1477 MLOM */ + { 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */ + { 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */ + { 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */ + { 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */ + { 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */ + { 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */ + { 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */ + { 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */ + { 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G }, /* 15230 MLOM */ + { 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G }, /* 15422 Mezz */ + { 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */ + { 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */ + { 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */ { 0, 0 }, /* End marker */ };
RE: [PATCH v2 2/3] net/enic: add speed capabilities for newer models
Quoted 9 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, August 8, 2024 5:51 PM
To: Hyong Youb Kim (hyonkim) <[email protected]>
Cc: [email protected]; John Daley (johndale) <[email protected]>
Subject: Re: [PATCH v2 2/3] net/enic: add speed capabilities for newer models
Quoted 9 lines
Quoted 2 lines · Hyong Youb Kim
Add 1400/14000 and 15000 models to the
speed_capalist.Do you want to introduce this new device variants in the driver guide?
Also does it makes sense to document new device model support in the release notes?
Updated doc and release notes.
Thanks.
-Hyong
Quoted 44 lines
drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 63
Quoted 13 lines
Signed-off-by Hyong Youb Kim <[email protected]>Reviewed-by John Daley <[email protected]>--- drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)drivers/net/enic/enic_ethdev.c +0
-0@@ -62,6 +62,27 @@ static const struct vic_speed_capa { { 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */ { 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G },/* 1495 PCIe */
drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 64
Quoted 1 line
drivers/net/enic/enic_ethdev.c +0
-0{ 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G },/* 1497 MLOM */
drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 65
Quoted 1 line
drivers/net/enic/enic_ethdev.c +1
-0+ { 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G },/* 1467 MLOM */
drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 66
Quoted 1 line
drivers/net/enic/enic_ethdev.c +1
-0+ { 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G },/* 1477 MLOM */
drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa {
Quoted 12 lines
drivers/net/enic/enic_ethdev.c +12
-0+ { 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */ + { 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */ + { 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */ + { 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */ + { 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */ + { 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */ + { 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */ + { 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */ + { 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G },/* 15230 MLOM */
drivers/net/enic/enic_ethdev.c static const struct vic_speed_capa { · line 79
Quoted 1 line
drivers/net/enic/enic_ethdev.c +1
-0+ { 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G },/* 15422 Mezz */
Quoted 9 lines
drivers/net/enic/enic_ethdev.c +6
-0+ {0x02e4 ,RTE_ETH_LINK_SPEED_40G |RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */ + { 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */ + { 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */ { 0, 0 }, /* End marker */ };
PATCH v2 3/3 net/enic: allow multicast in MAC address add callback
In reply to Hyong Youb Kimenic_set_mac_address() (mac_addr_add callback) currently allows only non-zero, unicast address to be added. It is overly restrictive. rte_eth_dev_mac_addr_add() itself allows multicast addresses. And, some applications do use rte_eth_dev_mac_addr_add() to accept multicast addresses. So, remove the unicast check in enic_set_mac_address().
--- drivers/net/enic/enic_main.c | 20 -------------------- 1 file changed, 20 deletions(-)
drivers/net/enic/enic_main.c +0 -20
@@ -32,21 +32,6 @@#include "vnic_intr.h" #include "vnic_nic.h" -static int is_zero_addr (uint8_t *addr) -{ -return !(addr[0 ] | addr[1 ] | addr[2 ] | addr[3 ] | addr[4 ] | addr[5 ]); -} - -static int is_mcast_addr (uint8_t *addr) -{ -return addr[0 ] &1 ; -} - -static int is_eth_addr_valid (uint8_t *addr) -{ -return !is_mcast_addr (addr) && !is_zero_addr (addr); -} -void enic_rxmbuf_queue_release(__rte_unused struct enic *enic, struct vnic_rq *rq) {
@@ -177,11 +162,6 @@ int enic_set_mac_address(struct enic *enic, uint8_t *mac_addr) {int err; -if (!is_eth_addr_valid (mac_addr)) { -dev_err (enic,"invalid mac address\n" ); -return -EINVAL ; - } - err =enic_dev_add_addr (enic, mac_addr);if (err)dev_err (enic,"add mac addr failed\n" );
Signature · 1 line
2.35.2
Re: [PATCH v2 3/3] net/enic: allow multicast in MAC address add callback
Quoted 7 lines · Hyong Youb Kim
enic_set_mac_address()(mac_addr_addcallback) currently allows only non-zero, unicast address to be added. It is overly restrictive.rte_eth_dev_mac_addr_add()itself allows multicast addresses. And, some applications do userte_eth_dev_mac_addr_add()to accept multicast addresses. So, remove the unicast check inenic_set_mac_address().
What is the usecase to set Multicast MAC address as device default MAC address?
Also, just a reminder that we have 'rte_eth_dev_set_mc_addr_list()' API to set multicast MAC addresses.
RE: [PATCH v2 3/3] net/enic: allow multicast in MAC address add callback
Quoted 19 lines
-----Original Message-----
From: Ferruh Yigit <[email protected]>
Sent: Thursday, August 8, 2024 5:51 PM
To: Hyong Youb Kim (hyonkim) <[email protected]>
Cc: [email protected]; John Daley (johndale) <[email protected]>
Subject: Re: [PATCH v2 3/3] net/enic: allow multicast in MAC address add callbackQuoted 7 lines · Hyong Youb Kim
enic_set_mac_address()(mac_addr_addcallback) currently allows only non-zero, unicast address to be added. It is overly restrictive.rte_eth_dev_mac_addr_add()itself allows multicast addresses. And, some applications do userte_eth_dev_mac_addr_add()to accept multicast addresses. So, remove the unicast check inenic_set_mac_address().
Quoted 5 lines
What is the usecase to set Multicast MAC address as device default MAC address?
Also, just a reminder that we have '
rte_eth_dev_set_mc_addr_list()' API to set multicast MAC addresses.
I am aware of mc_addr_list().. Some people seem to use mac_addr_add() as 'add a MAC filter'.
For the VIC adapter, there is no difference between the default MAC and other MAC addresses added to allow/accept. They are all MAC filters in HW. I believe some other NICs also do not distinguish 'default' vs. additional allowed MAC addresses (can be unicast or multicast).
The app in question is VPP. There might be others. It uses rte_eth_dev_mac_addr_add() to add multicast MAC filters like all-hosts address. I am not defending such usage. But given that (1) rte_eth_dev_mac_addr_add() accepts multicast, and (2) adding multicast works fine on multiple NICs, such usage is probably acceptable.
I am guessing some app developers find it easier to use mac_addr_add() than mc_addr_list()..
We could tighten the semantics of rte_eth_dev_mac_addr_add() and check if the given MAC is a valid unicast address in that API function.
Thanks.
-Hyong
v3 cover net/enic: support VF and fix minor issues
In reply to Hyong Youb KimThis series contains minor updates for net/enic. The first patch supports SR-IOV VF, which now requires the use of admin channel. The other patches are not related to VF, but included here to ease review.
---
v3:
* add spdx, doc, rel_notes (Ferruh's comments)
v2:
* fix compiler warnings
Hyong Youb Kim (3):
net/enic: support SR-IOV VF using admin channel net/enic: add speed capabilities for newer models net/enic: allow multicast in MAC address add callback
doc/guides/nics/enic.rst | 123 ++-- doc/guides/rel_notes/release_24_11.rst | 5 + drivers/net/enic/base/vnic_cq.c | 27 + drivers/net/enic/base/vnic_cq.h | 3 + drivers/net/enic/base/vnic_dev.c | 48 ++ drivers/net/enic/base/vnic_dev.h | 3 + drivers/net/enic/base/vnic_devcmd.h | 49 ++ drivers/net/enic/base/vnic_resource.h | 32 +- drivers/net/enic/base/vnic_rq.c | 27 + drivers/net/enic/base/vnic_rq.h | 7 + drivers/net/enic/base/vnic_wq.c | 37 +- drivers/net/enic/base/vnic_wq.h | 5 + drivers/net/enic/enic.h | 28 +- drivers/net/enic/enic_ethdev.c | 29 +- drivers/net/enic/enic_main.c | 97 +-- drivers/net/enic/enic_res.c | 12 + drivers/net/enic/enic_rxtx.c | 20 + drivers/net/enic/enic_sriov.c | 801 +++++++++++++++++++++++++ drivers/net/enic/enic_sriov.h | 211 +++++++ drivers/net/enic/meson.build | 1 + 20 files changed, 1452 insertions(+), 113 deletions(-) create mode 100644 drivers/net/enic/enic_sriov.c create mode 100644 drivers/net/enic/enic_sriov.h
Signature · 1 line
2.35.2
PATCH v3 1/3 net/enic: support SR-IOV VF using admin channel
The Linux VIC PF driver now requires the use of the admin channel between PF and VF drivers. Certain devcmds are disabled for VF. The VF driver is supposed to send control messages through the admin channel to the PF driver to perform those devcmds. This commit adds the admin channel to the VF driver (net/enic).
The VF's admin channel consists of normal Tx/Rx queues. VIC firmware hardwires those queues to PF. Control messages are specially crafted but otherwise normal packets.
The Rx queue uses LSC interrupt (interrupt vector 0) to notify the driver of new Rx control messages. The PF driver may send unsolicited request messages (e.g. asking for VF stats) to VF. Such messages cause LSC interrupts and are processed on the global interrupt thread.
For devcmds that must be sent through the admin channel, use wrapper functions. They check if the device is a VF. If VF, use the admin channel. Otherwise, perform devcmd directly.
Two complications:
- Soft Rx stats
VF on old VIC models does not have HW Rx counters. In this case, the VF driver counts packets/bytes and reports them as device stats.
- Backward compatibility mode
Old VIC PF drivers on some operating systems may support only VF_CAPABILITY_REQUEST message or not support the admin channel at all. When the VF driver detects such PF driver, it reverts to the compatibility mode and does not use the admin channel. In this mode, trust mode (e.g. enabling promiscuous mode) does not work.
--- doc/guides/nics/enic.rst | 120 ++-- doc/guides/rel_notes/release_24_11.rst | 4 + drivers/net/enic/base/vnic_cq.c | 27 + drivers/net/enic/base/vnic_cq.h | 3 + drivers/net/enic/base/vnic_dev.c | 48 ++ drivers/net/enic/base/vnic_dev.h | 3 + drivers/net/enic/base/vnic_devcmd.h | 49 ++ drivers/net/enic/base/vnic_resource.h | 32 +- drivers/net/enic/base/vnic_rq.c | 27 + drivers/net/enic/base/vnic_rq.h | 7 + drivers/net/enic/base/vnic_wq.c | 37 +- drivers/net/enic/base/vnic_wq.h | 5 + drivers/net/enic/enic.h | 28 +- drivers/net/enic/enic_ethdev.c | 8 +- drivers/net/enic/enic_main.c | 77 ++- drivers/net/enic/enic_res.c | 12 + drivers/net/enic/enic_rxtx.c | 20 + drivers/net/enic/enic_sriov.c | 801 +++++++++++++++++++++++++ drivers/net/enic/enic_sriov.h | 211 +++++++ drivers/net/enic/meson.build | 1 + 20 files changed, 1428 insertions(+), 92 deletions(-) create mode 100644 drivers/net/enic/enic_sriov.c create mode 100644 drivers/net/enic/enic_sriov.h
doc/guides/nics/enic.rst +52 -68
@@ -37,7 +37,7 @@ Supported features - UDP RSS hashing (1400 series and later adapters) - Scattered Rx - MTU update -- SR-IOV on UCS managed servers connected to Fabric Interconnects +- SR-IOV virtual function - Flow API - Overlay offload
@@ -135,103 +135,87 @@ Configuration information TCP, IPv4, TCP-IPv4, IPv6, TCP-IPv6, IPv6 Extension, TCP-IPv6 Extension. -SR-IOV mode utilization +SR-IOV Virtual Function ----------------------- -UCS blade servers configured with dynamic vNIC connection policies in UCSM -are capable of supporting SR-IOV. SR-IOV virtual functions (VFs) are -specialized vNICs, distinct from regular Ethernet vNICs. These VFs can be -directly assigned to virtual machines (VMs) as 'passthrough' devices. +VIC 1400 and later series supports SR-IOV. It can be enabled via both +UCSM and CIMC. Please refer to the following guides to enable SR-IOV +virtual functions (VFs). -In UCS, SR-IOV VFs require the use of the Cisco Virtual Machine Fabric Extender -(VM-FEX), which gives the VM a dedicated -interface on the Fabric Interconnect (FI). Layer 2 switching is done at -the FI. This may eliminate the requirement for software switching on the -host to route intra-host VM traffic. + - CIMC: `Managing vNICs <https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/sw/gui/config/guide/4_3/b_cisco_ucs_c-series_gui_configuration_guide_43/b_Cisco_UCS_C-series_GUI_Configuration_Guide_41_chapter_01011.html#d77871e5874a1635>`_ -Please refer to `Creating a Dynamic vNIC Connection Policy -<http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vm_fex/vmware/gui/config_guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide_chapter_010.html#task_433E01651F69464783A68E66DA8A47A5>`_ -for information on configuring SR-IOV adapter policies and port profiles -using UCSM. + - UCSM: `Configuring SRIOV HPN Connection Policies <https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Network-Mgmt/4-3/b_UCSM_Network_Mgmt_Guide_4_3/b_UCSM_Network_Mgmt_Guide_chapter_01010.html#d21438e9555a1635>`_ -Once the policies are in place and the host OS is rebooted, VFs should be -visible on the host, E.g.: +Note that the previous SR-IOV implementation that is tied to VM-FEX +(Cisco Virtual Machine Fabric Extender) has been discontinued, and +ENIC PMD no longer supports it. The current SR-IOV implementation does +not require the Fabric Interconnect (FI), as layer 2 switching is done +within the VIC adapter. + +Once SR-IOV is enabled, reboot the host OS and follow OS specific +steps to create VFs and assign them to virtual machines (VMs) or +containers as necessary. The VIC physical function (PF) drivers for ESXi +and Linux support SR-IOV. The following shows simplified steps for +Linux. .. code-block:: console + # echo 4 > /sys/class/net/<pf-interface>/device/sriov_numvfs + # lspci | grep Cisco | grep Ethernet - 0d:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2) - 0d:00.1 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) - 0d:00.2 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) - 0d:00.3 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) - 0d:00.4 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) - 0d:00.5 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) - 0d:00.6 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) - 0d:00.7 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) - -Enable Intel IOMMU on the host and install KVM and libvirt, and reboot again as -required. Then, using libvirt, create a VM instance with an assigned device. -Below is an example ``interface`` block (part of the domain configuration XML) -that adds the host VF 0d:00:01 to the VM. ``profileid='pp-vlan-25'`` indicates -the port profile that has been configured in UCSM. + 12:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2) + 12:00.1 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2) + 12:00.2 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2) + 12:00.3 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2) + 12:00.4 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2) + +Writing 4 to ``sriov_numvfs`` creates 4 VFs. ``lspci`` shows VFs and +their PCI locations. Interfaces with device ID ``02b7`` are the +VFs. The following snippet for libvirt XML assigns VF at ``12:00.1`` +to VM. .. code-block:: console - <interface type='hostdev' managed='yes'> - <mac address='52:54:00:ac:ff:b6'/> + <interface type="hostdev" managed="yes"> + <mac address="fa:16:3e:46:39:c5"/> <driver name='vfio'/> <source> - <address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/> + <address type="pci" domain="0x0000" bus="0x12" slot="0x00" function="0x1"/> </source> - <virtualport type='802.1Qbh'> - <parameters profileid='pp-vlan-25'/> - </virtualport> + <vlan> + <tag id="1000"/> + </vlan> </interface> - -Alternatively, the configuration can be done in a separate file using the -``network`` keyword. These methods are described in the libvirt documentation for -`Network XML format <https://libvirt.org/formatnetwork.html>`_. - When the VM instance is started, libvirt will bind the host VF to -vfio, complete provisioning on the FI and bring up the link. - -.. note:: - - It is not possible to use a VF directly from the host because it is not - fully provisioned until libvirt brings up the VM that it is assigned - to. - -In the VM instance, the VF will now be visible. E.g., here the VF 00:04.0 is -seen on the VM instance and should be available for binding to a DPDK. +vfio-pci. In the VM instance, the VF will now be visible. In this +example, VF at ``07:00.0`` is seen on the VM instance and is available +for binding to DPDK. .. code-block:: console - # lspci | grep Ether - 00:04.0 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + # lspci | grep Cisco + 07:00.0 Ethernet controller: Cisco Systems Inc Device 02b7 (rev a2) -Follow the normal DPDK install procedure, binding the VF to either ``igb_uio`` -or ``vfio`` in non-IOMMU mode. +There are two known limitations of the current SR-IOV implementation. -In the VM, the kernel enic driver may be automatically bound to the VF during -boot. Unbinding it currently hangs due to a known issue with the driver. To -work around the issue, block the enic module as follows. -Please see :ref:`Limitations <enic_limitations>` for limitations in -the use of SR-IOV. + - Software Rx statistics -.. code-block:: console + VF on old VIC models does not have hardware Rx counters. In this case, + ENIC PMD counts packets/bytes and reports them as device statistics. - # cat /etc/modprobe.d/enic.conf - blacklist enic + - Backward compatibility mode - # dracut --force + Old PF drivers on ESXi may lack full admin channel support. ENIC PMD + detects such PF driver during initialization and reverts to the + compatibility mode. In this mode, ENIC PMD does not use the admin channel, + and trust mode (e.g. enabling promiscuous mode on VF) is not supported. .. note:: - Passthrough does not require SR-IOV. If VM-FEX is not desired, the user + Passthrough does not require SR-IOV. If SR-IOV is not desired, the user may create as many regular vNICs as necessary and assign them to VMs as - passthrough devices. Since these vNICs are not SR-IOV VFs, using them as - passthrough devices do not require libvirt, port profiles, and VM-FEX. + passthrough devices. .. _enic-generic-flow-api:
doc/guides/rel_notes/release_24_11.rst +4 -0
@@ -55,6 +55,10 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Updated Cisco enic driver.** + + * Added SR-IOV VF support. + Removed Items -------------
drivers/net/enic/base/vnic_cq.c +27 -0
@@ -24,6 +24,7 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index, cq->index = index; cq->vdev = vdev; + cq->admin_chan = false; cq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_CQ , index);if (!cq->ctrl) {
@@ -40,6 +41,32 @@ int vnic_cq_alloc(struct vnic_dev *vdev, struct vnic_cq *cq, unsigned int index,return 0 ; } +int vnic_admin_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index, +unsigned int socket_id,unsigned int desc_count,unsigned int desc_size) +{ +int err; +char res_name[RTE_MEMZONE_NAMESIZE ]; +static int instance; + + cq->index = index; + cq->vdev = vdev; + cq->admin_chan = true; + + cq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_ADMIN_CQ , index); +if (!cq->ctrl) { +pr_err ("Failed to get admin CQ[%u] resource\n" , index); +return -EINVAL ; + } + +snprintf (res_name,sizeof (res_name),"%d-admin-cq-%u" , instance++, index); + err =vnic_dev_alloc_desc_ring (vdev, &cq->ring, desc_count, desc_size, + socket_id, res_name); +if (err) +return err; + +return 0 ; +} + void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable, unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail, unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_cq.h +3 -0
@@ -59,12 +59,15 @@ struct vnic_cq {unsigned int tobe_rx_coal_timeval;ktime_t prev_ts;#endif +bool admin_chan; };void vnic_cq_free (struct vnic_cq *cq);int vnic_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index,unsigned int socket_id,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_cq_alloc (struct vnic_dev *vdev,struct vnic_cq *cq,unsigned int index, +unsigned int socket_id,unsigned int desc_count,unsigned int desc_size); void vnic_cq_init(struct vnic_cq *cq, unsigned int flow_control_enable, unsigned int color_enable, unsigned int cq_head, unsigned int cq_tail, unsigned int cq_tail_color, unsigned int interrupt_enable,
drivers/net/enic/base/vnic_dev.c +48 -0
81 lines not on this page · raw message
drivers/net/enic/base/vnic_dev.h +3 -0
@@ -199,5 +199,8 @@ int vnic_dev_capable_geneve(struct vnic_dev *vdev);uint64_t vnic_dev_capable_cq_entry_size (struct vnic_dev *vdev);int vnic_dev_set_cq_entry_size (struct vnic_dev *vdev,uint32_t rq_idx,uint32_t size_flag); +int vnic_dev_alloc_sriov_stats_mem (struct vnic_dev *vdev); +int vnic_dev_sriov_stats (struct vnic_dev *vdev,struct vnic_sriov_stats **stats); +int vnic_dev_enable_admin_qp (struct vnic_dev *vdev,uint32_t enable);#endif /* _VNIC_DEV_H_ */
drivers/net/enic/base/vnic_devcmd.h +49 -0
59 lines not on this page · raw message
drivers/net/enic/base/vnic_resource.h +30 -2
@@ -38,8 +38,36 @@ enum vnic_res_type {RES_TYPE_MQ_WQ ,/* MQ Work queues */ RES_TYPE_MQ_RQ ,/* MQ Receive queues */ RES_TYPE_MQ_CQ ,/* MQ Completion queues */ -RES_TYPE_DEPRECATED1 ,/* Old version of devcmd 2 */ -RES_TYPE_DEVCMD2 ,/* Device control region */ +RES_TYPE_DEPRECATED1 ,/* Old version of devcmd 2 */ +RES_TYPE_DEPRECATED2 ,/* Old version of devcmd 2 */ +RES_TYPE_DEVCMD2 ,/* Device control region */ +RES_TYPE_RDMA_WQ ,/* RDMA WQ */ +RES_TYPE_RDMA_RQ ,/* RDMA RQ */ +RES_TYPE_RDMA_CQ ,/* RDMA CQ */ +RES_TYPE_RDMA_RKEY_TABLE ,/* RDMA RKEY table */ +RES_TYPE_RDMA_RQ_HEADER_TABLE ,/* RDMA RQ Header Table */ +RES_TYPE_RDMA_RQ_TABLE ,/* RDMA RQ Table */ +RES_TYPE_RDMA_RD_RESP_HEADER_TABLE ,/* RDMA Read Response Header Table */ +RES_TYPE_RDMA_RD_RESP_TABLE ,/* RDMA Read Response Table */ +RES_TYPE_RDMA_QP_STATS_TABLE ,/* RDMA per QP stats table */ +RES_TYPE_WQ_MREGS ,/* XXX snic proto only */ +RES_TYPE_GRPMBR_INTR ,/* Group member interrupt control */ +RES_TYPE_DPKT ,/* Direct Packet memory region */ +RES_TYPE_RDMA2_DATA_WQ ,/* RDMA datapath command WQ */ +RES_TYPE_RDMA2_REG_WQ ,/* RDMA registration command WQ */ +RES_TYPE_RDMA2_CQ ,/* RDMA datapath CQ */ +RES_TYPE_MQ_RDMA2_DATA_WQ ,/* RDMA datapath command WQ */ +RES_TYPE_MQ_RDMA2_REG_WQ ,/* RDMA registration command WQ */ +RES_TYPE_MQ_RDMA2_CQ ,/* RDMA datapath CQ */ +RES_TYPE_PTP ,/* PTP registers */ +RES_TYPE_INTR_CTRL2 ,/* Extended INTR CTRL registers */ +RES_TYPE_SRIOV_INTR ,/* VF intr */ +RES_TYPE_VF_WQ ,/* VF WQ */ +RES_TYPE_VF_RQ ,/* VF RQ */ +RES_TYPE_VF_CQ ,/* VF CQ */ +RES_TYPE_ADMIN_WQ ,/* admin channel WQ */ +RES_TYPE_ADMIN_RQ ,/* admin channel RQ */ +RES_TYPE_ADMIN_CQ ,/* admin channel CQ */ RES_TYPE_MAX, /* Count of resource types */ };
drivers/net/enic/base/vnic_rq.c +27 -0
@@ -27,6 +27,7 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index, rq->index = index; rq->vdev = vdev; + rq->admin_chan = false; rq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_RQ , index);if (!rq->ctrl) {
@@ -42,6 +43,32 @@ int vnic_rq_alloc(struct vnic_dev *vdev, struct vnic_rq *rq, unsigned int index,return rc; } +int vnic_admin_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq, +unsigned int desc_count,unsigned int desc_size) +{ +int rc; +char res_name[RTE_MEMZONE_NAMESIZE ]; +static int instance; + + rq->index =0 ; + rq->vdev = vdev; + rq->admin_chan = true; + rq->socket_id =SOCKET_ID_ANY ; + + rq->ctrl =vnic_dev_get_res (vdev,RES_TYPE_ADMIN_RQ ,0 ); +if (!rq->ctrl) { +pr_err ("Failed to get admin RQ resource\n" ); +return -EINVAL ; + } + +vnic_rq_disable (rq); + +snprintf (res_name,sizeof (res_name),"%d-admin-rq" , instance++); + rc =vnic_dev_alloc_desc_ring (vdev, &rq->ring, desc_count, desc_size, + rq->socket_id, res_name); +return rc; +} + void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_rq.h +7 -0
@@ -73,6 +73,11 @@ struct vnic_rq {unsigned int max_mbufs_per_pkt;uint16_t tot_nb_desc;bool need_initial_post; +bool admin_chan; +const struct rte_memzone *admin_msg_rz; +int admin_next_idx; +uint64_t soft_stats_pkts; +uint64_t soft_stats_bytes; };static inline unsigned int vnic_rq_desc_avail (struct vnic_rq *rq)
@@ -127,6 +132,8 @@ static inline int vnic_rq_fill_count(struct vnic_rq *rq,void vnic_rq_free (struct vnic_rq *rq);int vnic_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq,unsigned int index,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_rq_alloc (struct vnic_dev *vdev,struct vnic_rq *rq, +unsigned int desc_count,unsigned int desc_size); void vnic_rq_init_start(struct vnic_rq *rq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.c +35 -2
@@ -23,7 +23,8 @@ int vnic_wq_alloc_ring(struct vnic_dev *vdev, struct vnic_wq *wq,char res_name[RTE_MEMZONE_NAMESIZE ];static int instance; -snprintf (res_name,sizeof (res_name),"%d-wq-%u" , instance++, wq->index); +snprintf (res_name,sizeof (res_name),"%d-%swq-%u" , + instance++, wq->admin_chan ?"admin-" :"" , wq->index);return vnic_dev_alloc_desc_ring (vdev, &wq->ring, desc_count, desc_size, wq->socket_id, res_name); }
@@ -32,7 +33,7 @@ static int vnic_wq_alloc_bufs(struct vnic_wq *wq) {unsigned int count = wq->ring.desc_count;/* Allocate the mbuf ring */ - wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket ("wq->bufs" , + wq->bufs = (struct rte_mbuf **)rte_zmalloc_socket (wq->admin_chan ?"admin-wq-bufs" :"wq-bufs" ,sizeof (struct rte_mbuf *) * count,RTE_CACHE_LINE_SIZE , wq->socket_id); wq->head_idx =0 ;
@@ -62,6 +63,7 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index, wq->index = index; wq->vdev = vdev; + wq->admin_chan = false; err =vnic_wq_get_ctrl (vdev, wq, index,RES_TYPE_WQ );if (err) {
@@ -84,6 +86,37 @@ int vnic_wq_alloc(struct vnic_dev *vdev, struct vnic_wq *wq, unsigned int index,return 0 ; } +int vnic_admin_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq, +unsigned int desc_count,unsigned int desc_size) +{ +int err; + + wq->index =0 ; + wq->vdev = vdev; + wq->admin_chan = true; + wq->socket_id =SOCKET_ID_ANY ; + + err =vnic_wq_get_ctrl (vdev, wq,0 ,RES_TYPE_ADMIN_WQ ); +if (err) { +pr_err ("Failed to get admin WQ resource err %d\n" , err); +return err; + } + +vnic_wq_disable (wq); + + err =vnic_wq_alloc_ring (vdev, wq, desc_count, desc_size); +if (err) +return err; + + err =vnic_wq_alloc_bufs (wq); +if (err) { +vnic_wq_free (wq); +return err; + } + +return 0 ; +} + void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/base/vnic_wq.h +5 -0
@@ -50,6 +50,9 @@ struct vnic_wq {const struct rte_memzone *cqmsg_rz;uint16_t last_completed_index;uint64_t offloads; +bool admin_chan; +const struct rte_memzone *admin_msg_rz; +uint64_t soft_stats_tx; };static inline unsigned int vnic_wq_desc_avail (struct vnic_wq *wq)
@@ -149,6 +152,8 @@ buf_idx_incr(uint32_t n_descriptors, uint32_t idx)void vnic_wq_free (struct vnic_wq *wq);int vnic_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq,unsigned int index,unsigned int desc_count,unsigned int desc_size); +int vnic_admin_wq_alloc (struct vnic_dev *vdev,struct vnic_wq *wq, +unsigned int desc_count,unsigned int desc_size); void vnic_wq_init_start(struct vnic_wq *wq, unsigned int cq_index, unsigned int fetch_index, unsigned int posted_index, unsigned int error_interrupt_enable,
drivers/net/enic/enic.h +26 -2
@@ -43,7 +43,6 @@/*#define ENIC_DESC_COUNT_MAKE_ODD (x) do{if ((~(x)) & 1) { (x)--; } }while(0)*/ #define PCI_DEVICE_ID_CISCO_VIC_ENET 0x0043/* ethernet vnic */ -#define PCI_DEVICE_ID_CISCO_VIC_ENET_VF 0x0071/* enet SRIOV VF */ /* enet SRIOV Standalone vNic VF */ #define PCI_DEVICE_ID_CISCO_VIC_ENET_SN 0x02B7
@@ -51,7 +50,7 @@#define ENIC_MAGIC_FILTER_ID 0xffff/* -* Interrupt 0: LSC and errors +* Interrupt 0: LSC and errors / VF admin channel RQ * Interrupt 1: rx queue 0 * Interrupt 2: rx queue 1 * ...
@@ -154,11 +153,26 @@ struct enic {/* software counters */ struct enic_soft_stats soft_stats; +struct vnic_wq admin_wq; +struct vnic_rq admin_rq; +struct vnic_cq admin_cq[2 ]; +/* configured resources on vic */ unsigned int conf_rq_count;unsigned int conf_wq_count;unsigned int conf_cq_count;unsigned int conf_intr_count; +/* SR-IOV VF has queues for admin channel to PF */ +unsigned int conf_admin_rq_count; +unsigned int conf_admin_wq_count; +unsigned int conf_admin_cq_count; +uint64_t admin_chan_msg_num; +int admin_chan_vf_id; +uint32_t admin_pf_cap_version; +bool admin_chan_enabled; +bool sriov_vf_soft_rx_stats; +bool sriov_vf_compat_mode; +pthread_mutex_t admin_chan_lock;/* linked list storing memory allocations */ LIST_HEAD(enic_memzone_list, enic_memzone_entry) memzone_list;
@@ -230,6 +244,16 @@ struct enic_vf_representor {struct rte_flow *rep2vf_flow[2 ]; }; +#define ENIC_ADMIN_WQ_CQ 0 +#define ENIC_ADMIN_RQ_CQ 1 +#define ENIC_ADMIN_BUF_SIZE 1024 + +static inline bool enic_is_vf (struct enic *enic) +{ +return enic->pdev->id.device_id ==PCI_DEVICE_ID_CISCO_VIC_ENET_SN && + !enic->sriov_vf_compat_mode; +} +#define VF_ENIC_TO_VF_REP (vf_enic) \ container_of(vf_enic, struct enic_vf_representor, enic)
drivers/net/enic/enic_ethdev.c +4 -4
@@ -21,6 +21,7 @@#include "vnic_rq.h" #include "vnic_enet.h" #include "enic.h" +#include "enic_sriov.h" /* * The set of PCI devices this driver supports
@@ -28,7 +29,6 @@#define CISCO_PCI_VENDOR_ID 0x1137static const struct rte_pci_id pci_id_enic_map[] = { {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET )}, - {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET_VF )}, {RTE_PCI_DEVICE (CISCO_PCI_VENDOR_ID ,PCI_DEVICE_ID_CISCO_VIC_ENET_SN )}, {.vendor_id =0 ,/* sentinel */ }, };
@@ -707,7 +707,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,for (i =0 ; i < enic->mc_count; i++) { addr = &enic->mc_addrs[i];debug_log_add_del_addr (addr, false); - ret =vnic_dev_del_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_del_addr (enic, addr->addr_bytes);if (ret)return ret; }
@@ -734,7 +734,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,if (j < nb_mc_addr)continue ;debug_log_add_del_addr (addr, false); - ret =vnic_dev_del_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_del_addr (enic, addr->addr_bytes);if (ret)return ret; }
@@ -748,7 +748,7 @@ static int enicpmd_set_mc_addr_list(struct rte_eth_dev *eth_dev,if (j < enic->mc_count)continue ;debug_log_add_del_addr (addr, true); - ret =vnic_dev_add_addr (enic->vdev, addr->addr_bytes); + ret =enic_dev_add_addr (enic, addr->addr_bytes);if (ret)return ret; }
drivers/net/enic/enic_main.c +63 -14
191 lines not on this page · raw message
drivers/net/enic/enic_res.c +12 -0
@@ -300,4 +300,16 @@ void enic_get_res_counts(struct enic *enic)"vNIC resources avail: wq %d rq %d cq %d intr %d\n" , enic->conf_wq_count, enic->conf_rq_count, enic->conf_cq_count, enic->conf_intr_count); + +if (!enic_is_vf (enic)) +return ; + + enic->conf_admin_wq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_WQ ); + enic->conf_admin_rq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_RQ ); + enic->conf_admin_cq_count =vnic_dev_get_res_count (enic->vdev,RES_TYPE_ADMIN_CQ ); + +dev_info (enic_get_dev (enic), +"vNIC admin channel resources avail: wq %d rq %d cq %d\n" , + enic->conf_admin_wq_count, enic->conf_admin_rq_count, + enic->conf_admin_cq_count); }
drivers/net/enic/enic_rxtx.c +20 -0
@@ -55,6 +55,7 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, sizeof(struct cq_enet_rq_desc_64) :sizeof (struct cq_enet_rq_desc );RTE_BUILD_BUG_ON (sizeof (struct cq_enet_rq_desc_64 ) !=64 ); +uint64_t bytes; cq = &enic->cq[enic_cq_rq (enic, sop_rq->index)]; cq_idx = cq->to_clean;/* index of cqd, rqd, mbuf_table */
@@ -67,6 +68,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts,/* Receive until the end of the ring, at most. */ max_rx =RTE_MIN (nb_pkts, cq->ring.desc_count - cq_idx); + bytes =0 ; +while (max_rx) {volatile struct rq_enet_desc *rqd_ptr;struct cq_desc cqd;
@@ -155,6 +158,8 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, rxmb->port = enic->port_id; rxmb->data_len = seg_length; + bytes += seg_length; + rq->rx_nb_hold++;if (!(enic_cq_rx_desc_eop (ciflags))) {
@@ -225,6 +230,10 @@ enic_recv_pkts_common(void *rx_queue, struct rte_mbuf **rx_pkts, &sop_rq->ctrl->posted_index); } +if (enic->sriov_vf_soft_rx_stats && bytes) { + sop_rq->soft_stats_pkts += nb_rx; + sop_rq->soft_stats_bytes += bytes; + }return nb_rx; }
@@ -256,6 +265,7 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,uint8_t color;bool overlay;bool tnl; +uint64_t bytes; rq = rx_queue; enic =vnic_dev_priv (rq->vdev);
@@ -283,6 +293,8 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, port_id = enic->port_id; overlay = enic->overlay_offload; + bytes =0 ; + rx = rx_pkts;while (max_rx) { max_rx--;
@@ -304,6 +316,9 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,CQ_ENET_RQ_DESC_BYTES_WRITTEN_MASK ; mb->pkt_len = mb->data_len; mb->port = port_id; + + bytes += mb->pkt_len; + tnl = overlay && (cqd->completed_index_flags &CQ_ENET_RQ_DESC_FLAGS_FCOE ) !=0 ; mb->packet_type =
@@ -352,6 +367,11 @@ enic_noscatter_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, &rq->ctrl->posted_index); } +if (enic->sriov_vf_soft_rx_stats && bytes) { + rq->soft_stats_pkts += (rx - rx_pkts); + rq->soft_stats_bytes += bytes; + } +return rx - rx_pkts; }
drivers/net/enic/enic_sriov.c new file +801 -0
801 lines not on this page · raw message
drivers/net/enic/enic_sriov.h new file +211 -0
211 lines not on this page · raw message
drivers/net/enic/meson.build +1 -0
@@ -23,6 +23,7 @@ sources = files( 'enic_main.c', 'enic_res.c', 'enic_rxtx.c', + 'enic_sriov.c', 'enic_vf_representor.c', ) deps += ['hash']
Signature · 1 line
2.35.2
PATCH v3 2/3 net/enic: add speed capabilities for newer models
In reply to Hyong Youb KimAdd 1400/14000 and 15000 models to the speed_capa list.
--- doc/guides/nics/enic.rst | 3 ++- doc/guides/rel_notes/release_24_11.rst | 1 + drivers/net/enic/enic_ethdev.c | 21 +++++++++++++++++++++ 3 files changed, 24 insertions(+), 1 deletion(-)
doc/guides/nics/enic.rst +2 -1
@@ -17,7 +17,8 @@ ENIC PMD supports all recent generations of Cisco VIC adapters including: - VIC 1200 series - VIC 1300 series -- VIC 1400 series +- VIC 1400/14000 series +- VIC 15000 series Supported features ------------------
doc/guides/rel_notes/release_24_11.rst +1 -0
@@ -58,6 +58,7 @@ New Features * **Updated Cisco enic driver.** * Added SR-IOV VF support. + * Added recent 1400/14000 and 15000 models to the supported list. Removed Items
drivers/net/enic/enic_ethdev.c +21 -0
@@ -62,6 +62,27 @@ static const struct vic_speed_capa { { 0x021a, RTE_ETH_LINK_SPEED_40G }, /* 1487 MLOM */ { 0x024a, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1495 PCIe */ { 0x024b, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1497 MLOM */ + { 0x02af, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G }, /* 1467 MLOM */ + { 0x02b0, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G }, /* 1477 MLOM */ + { 0x02cf, RTE_ETH_LINK_SPEED_25G }, /* 14425 MLOM */ + { 0x02d0, RTE_ETH_LINK_SPEED_25G }, /* 14825 Mezz */ + { 0x02db, RTE_ETH_LINK_SPEED_100G }, /* 15231 MLOM */ + { 0x02dc, RTE_ETH_LINK_SPEED_10G }, /* 15411 MLOM */ + { 0x02dd, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15428 MLOM */ + { 0x02de, RTE_ETH_LINK_SPEED_25G }, /* 15420 MLOM */ + { 0x02e8, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G}, /* 15238 MLOM */ + { 0x02e0, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15427 MLOM */ + { 0x02df, RTE_ETH_LINK_SPEED_50G | RTE_ETH_LINK_SPEED_100G }, /* 15230 MLOM */ + { 0x02e1, RTE_ETH_LINK_SPEED_25G | RTE_ETH_LINK_SPEED_50G }, /* 15422 Mezz */ + { 0x02e4, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15235 PCIe */ + { 0x02f2, RTE_ETH_LINK_SPEED_10G | RTE_ETH_LINK_SPEED_25G | + RTE_ETH_LINK_SPEED_50G }, /* 15425 PCIe */ + { 0x02f3, RTE_ETH_LINK_SPEED_40G | RTE_ETH_LINK_SPEED_100G | + RTE_ETH_LINK_SPEED_200G }, /* 15237 MLOM */ { 0, 0 }, /* End marker */ };
Signature · 1 line
2.35.2
PATCH v3 3/3 net/enic: allow multicast in MAC address add callback
In reply to Hyong Youb Kimenic_set_mac_address() (mac_addr_add callback) currently allows only non-zero, unicast address to be added. It is overly restrictive. rte_eth_dev_mac_addr_add() itself allows multicast addresses. And, some applications do use rte_eth_dev_mac_addr_add() to accept multicast addresses. So, remove the unicast check in enic_set_mac_address().
--- drivers/net/enic/enic_main.c | 20 -------------------- 1 file changed, 20 deletions(-)
drivers/net/enic/enic_main.c +0 -20
@@ -32,21 +32,6 @@#include "vnic_intr.h" #include "vnic_nic.h" -static int is_zero_addr (uint8_t *addr) -{ -return !(addr[0 ] | addr[1 ] | addr[2 ] | addr[3 ] | addr[4 ] | addr[5 ]); -} - -static int is_mcast_addr (uint8_t *addr) -{ -return addr[0 ] &1 ; -} - -static int is_eth_addr_valid (uint8_t *addr) -{ -return !is_mcast_addr (addr) && !is_zero_addr (addr); -} -void enic_rxmbuf_queue_release(__rte_unused struct enic *enic, struct vnic_rq *rq) {
@@ -177,11 +162,6 @@ int enic_set_mac_address(struct enic *enic, uint8_t *mac_addr) {int err; -if (!is_eth_addr_valid (mac_addr)) { -dev_err (enic,"invalid mac address\n" ); -return -EINVAL ; - } - err =enic_dev_add_addr (enic, mac_addr);if (err)dev_err (enic,"add mac addr failed\n" );
Signature · 1 line
2.35.2
Quoted 9 lines · Hyong Youb Kim
This series contains minor updates for net/enic. The first patch supports SR-IOV VF, which now requires the use of admin channel. The other patches are not related to VF, but included here to ease review.
---
v3:
* add spdx, doc,rel_notes(Ferruh's comments)
Quoted 8 lines
v2:
* fix compiler warningsHyong Youb Kim (3):
net/enic: support SR-IOV VF using admin channel net/enic: add speed capabilities for newer models net/enic: allow multicast in MAC address add callback
Series applied to dpdk-next-net/main, thanks.