Skip to contentSkip to conversation

VDPA support for Mellanox ConnectX devices

PATCH In review Eli Cohen 5 participants 28 messages Aug 4, 2020 – Aug 5, 2020

Subsystems (4)
lore
Share & tools
lore

Eli Cohen

author
#0 lore raw

v4 cover VDPA support for Mellanox ConnectX devices

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301

Jason, I had to resolve some conflicts so I would appreciate of you can verify that it is ok.

The following series of patches provide VDPA support for Mellanox devices. The supported devices are ConnectX6 DX and newer.

Currently, only a network driver is implemented; future patches will introduce a block device driver. iperf performance on a single queue is around 12 Gbps. Future patches will introduce multi queue support.

The files are organized in such a way that code that can be used by different VDPA implementations will be placed in a common are resides in drivers/vdpa/mlx5/core.

Only virtual functions are currently supported. Also, certain firmware capabilities must be set to enable the driver. Physical functions (PFs) are skipped by the driver.

To make use of the VDPA net driver, one must load mlx5_vdpa. In such case, VFs will be operated by the VDPA driver. Although one can see a regular instance of a network driver on the VF, the VDPA driver takes precedence over the NIC driver, steering-wize.

Currently, the device/interface infrastructure in mlx5_core is used to probe drivers. Future patches will introduce virtbus as a means to register devices and drivers and VDPA will be adapted to it.

The mlx5 mode of operation required to support VDPA is switchdev mode. Once can use Linux or OVS bridge to take care of layer 2 switching.

In order to provide virtio networking to a guest, an updated version of qemu is required. This version has been tested by the following quemu version:

url: https://github.com/jasowang/qemu.git
branch: vdpa
Commit ID: 6f4e59b807db

V2->V3
Fix makefile to use include path relative to the root of the kernel

V3-V4
Rebase Jason's patches on linux-next branch
Fix krobot error on mips arch
Make use of the free callback to destroy resoruces on unload Use VIRTIO_F_ACCESS_PLATFORM instead of legacy VIRTIO_F_IOMMU_PLATFORM Add empty implementations for get_vq_notification() and get_vq_irq()

Eli Cohen (6):
net/vdpa: Use struct for set/get vq state
vdpa: Modify get_vq_state() to return error code vdpa/mlx5: Add hardware descriptive header file vdpa/mlx5: Add support library for mlx5 VDPA implementation vdpa/mlx5: Add shared memory registration code vdpa/mlx5: Add VDPA driver for supported mlx5 devices

Jason Wang (5):
vhost-vdpa: refine ioctl pre-processing
vhost: generialize backend features setting/getting
vhost-vdpa: support get/set backend features
vhost-vdpa: support IOTLB batching hints
vdpasim: support batch updating

Max Gurtovoy (1):
vdpa: remove hard coded virtq num

 drivers/vdpa/Kconfig                   |   19 +
 drivers/vdpa/Makefile                  |    1 +
 drivers/vdpa/ifcvf/ifcvf_base.c        |    4 +-
 drivers/vdpa/ifcvf/ifcvf_base.h        |    4 +-
 drivers/vdpa/ifcvf/ifcvf_main.c        |   13 +-
 drivers/vdpa/mlx5/Makefile             |    4 +
 drivers/vdpa/mlx5/core/mlx5_vdpa.h     |   91 ++
 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h |  168 ++
 drivers/vdpa/mlx5/core/mr.c            |  484 ++++++
 drivers/vdpa/mlx5/core/resources.c     |  284 ++++
 drivers/vdpa/mlx5/net/main.c           |   76 +
 drivers/vdpa/mlx5/net/mlx5_vnet.c      | 1965 ++++++++++++++++++++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.h      |   24 +
 drivers/vdpa/vdpa.c                    |    3 +
 drivers/vdpa/vdpa_sim/vdpa_sim.c       |   53 +-
 drivers/vhost/net.c                    |   18 +-
 drivers/vhost/vdpa.c                   |   76 +-
 drivers/vhost/vhost.c                  |   15 +
 drivers/vhost/vhost.h                  |    2 +
 include/linux/vdpa.h                   |   24 +-
 include/uapi/linux/vhost.h             |    2 +
 include/uapi/linux/vhost_types.h       |   11 +
 22 files changed, 3284 insertions(+), 57 deletions(-)
 create mode 100644 drivers/vdpa/mlx5/Makefile
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa.h
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h
 create mode 100644 drivers/vdpa/mlx5/core/mr.c
 create mode 100644 drivers/vdpa/mlx5/core/resources.c
 create mode 100644 drivers/vdpa/mlx5/net/main.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
Signature · 1 line
2.26.0
author
#1 lore raw

PATCH v4 1/12 vhost-vdpa: refine ioctl pre-processing

From: Jason Wang <[email protected]>

Switch to use 'switch' to make the codes more easier to be extended.

---
 drivers/vhost/vdpa.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/vhost/vdpa.c +5 -4
index 16e0ffb115be..563ae6204052 100644
@@ -377,15 +377,16 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 	idx = array_index_nospec(idx, v->nvqs);
 	vq = &v->vqs[idx];
 
-	if (cmd == VHOST_VDPA_SET_VRING_ENABLE) {
+	switch (cmd) {
+	case VHOST_VDPA_SET_VRING_ENABLE:
 		if (copy_from_user(&s, argp, sizeof(s)))
 			return -EFAULT;
 		ops->set_vq_ready(vdpa, idx, s.num);
 		return 0;
-	}
-
-	if (cmd == VHOST_GET_VRING_BASE)
+	case VHOST_GET_VRING_BASE:
 		vq->last_avail_idx = ops->get_vq_state(v->vdpa, idx);
+		break;
+	}
 
 	r = vhost_vring_ioctl(&v->vdev, cmd, argp);
 	if (r)
Signature · 1 line
2.26.0
author
#2 lore raw

PATCH v4 2/12 vhost: generialize backend features setting/getting

In reply to Eli Cohen

From: Jason Wang <[email protected]>

Move the backend features setting/getting from net.c to vhost.c to be reused by vhost-vdpa.

---
 drivers/vhost/net.c   | 18 ++----------------
 drivers/vhost/vhost.c | 15 +++++++++++++++
 drivers/vhost/vhost.h |  2 ++
 3 files changed, 19 insertions(+), 16 deletions(-)
drivers/vhost/net.c +2 -16
index 8e0921d3805d..bfbbe5c876f9 100644
@@ -1615,21 +1615,6 @@ static long vhost_net_reset_owner(struct vhost_net *n)
 	return err;
 }
 
-static int vhost_net_set_backend_features(struct vhost_net *n, u64 features)
-{
-	int i;
-
-	mutex_lock(&n->dev.mutex);
-	for (i = 0; i < VHOST_NET_VQ_MAX; ++i) {
-		mutex_lock(&n->vqs[i].vq.mutex);
-		n->vqs[i].vq.acked_backend_features = features;
-		mutex_unlock(&n->vqs[i].vq.mutex);
-	}
-	mutex_unlock(&n->dev.mutex);
-
-	return 0;
-}
-
 static int vhost_net_set_features(struct vhost_net *n, u64 features)
 {
 	size_t vhost_hlen, sock_hlen, hdr_len;
@@ -1730,7 +1715,8 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl,
 			return -EFAULT;
 		if (features & ~VHOST_NET_BACKEND_FEATURES)
 			return -EOPNOTSUPP;
-		return vhost_net_set_backend_features(n, features);
+		vhost_set_backend_features(&n->dev, features);
+		return 0;
 	case VHOST_RESET_OWNER:
 		return vhost_net_reset_owner(n);
 	case VHOST_SET_OWNER:
drivers/vhost/vhost.c +15 -0
index 8f622064b3e8..5e5cc3dd983e 100644
@@ -2591,6 +2591,21 @@ struct vhost_msg_node *vhost_dequeue_msg(struct vhost_dev *dev,
 }
 EXPORT_SYMBOL_GPL(vhost_dequeue_msg);
 
+void vhost_set_backend_features(struct vhost_dev *dev, u64 features)
+{
+	struct vhost_virtqueue *vq;
+	int i;
+
+	mutex_lock(&dev->mutex);
+	for (i = 0; i < dev->nvqs; ++i) {
+		vq = dev->vqs[i];
+		mutex_lock(&vq->mutex);
+		vq->acked_backend_features = features;
+		mutex_unlock(&vq->mutex);
+	}
+	mutex_unlock(&dev->mutex);
+}
+EXPORT_SYMBOL_GPL(vhost_set_backend_features);
 
 static int __init vhost_init(void)
 {
drivers/vhost/vhost.h +2 -0
index 38eb1aa3b68d..9032d3c2a9f4 100644
@@ -214,6 +214,8 @@ void vhost_enqueue_msg(struct vhost_dev *dev,
 		       struct vhost_msg_node *node);
 struct vhost_msg_node *vhost_dequeue_msg(struct vhost_dev *dev,
 					 struct list_head *head);
+void vhost_set_backend_features(struct vhost_dev *dev, u64 features);
+
 __poll_t vhost_chr_poll(struct file *file, struct vhost_dev *dev,
 			    poll_table *wait);
 ssize_t vhost_chr_read_iter(struct vhost_dev *dev, struct iov_iter *to,
Signature · 1 line
2.26.0
author
#3 lore raw

PATCH v4 3/12 vhost-vdpa: support get/set backend features

In reply to Eli Cohen

From: Jason Wang <[email protected]>

This patch makes userspace can get and set backend features to vhost-vdpa.

---
 drivers/vhost/vdpa.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
drivers/vhost/vdpa.c +18 -0
index 563ae6204052..2f0310d1b914 100644
@@ -26,6 +26,10 @@
 
 #include "vhost.h"
 
+enum {
+	VHOST_VDPA_BACKEND_FEATURES = (1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2)
+};
+
 /* Currently, only network backend w/o multiqueue is supported. */
 #define VHOST_VDPA_VQ_MAX	2
 
@@ -364,6 +368,8 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 	struct vdpa_callback cb;
 	struct vhost_virtqueue *vq;
 	struct vhost_vring_state s;
+	u64 __user *featurep = argp;
+	u64 features;
 	u32 idx;
 	long r;
 
@@ -386,6 +392,18 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 	case VHOST_GET_VRING_BASE:
 		vq->last_avail_idx = ops->get_vq_state(v->vdpa, idx);
 		break;
+	case VHOST_GET_BACKEND_FEATURES:
+		features = VHOST_VDPA_BACKEND_FEATURES;
+		if (copy_to_user(featurep, &features, sizeof(features)))
+			return -EFAULT;
+		return 0;
+	case VHOST_SET_BACKEND_FEATURES:
+		if (copy_from_user(&features, featurep, sizeof(features)))
+			return -EFAULT;
+		if (features & ~VHOST_VDPA_BACKEND_FEATURES)
+			return -EOPNOTSUPP;
+		vhost_set_backend_features(&v->vdev, features);
+		return 0;
 	}
 
 	r = vhost_vring_ioctl(&v->vdev, cmd, argp);
Signature · 1 line
2.26.0
author
#4 lore raw

PATCH v4 4/12 vhost-vdpa: support IOTLB batching hints

In reply to Eli Cohen

From: Jason Wang <[email protected]>

This patches extend the vhost IOTLB API to accept batch updating hints form userspace. When userspace wants update the device IOTLB in a batch, it may do:

1) Write vhost_iotlb_msg with VHOST_IOTLB_BATCH_BEGIN flag
2) Perform a batch of IOTLB updating via VHOST_IOTLB_UPDATE/INVALIDATE
3) Write vhost_iotlb_msg with VHOST_IOTLB_BATCH_END flag

Vhost-vdpa may decide to batch the IOMMU/IOTLB updating in step 3 when vDPA device support set_map() ops. This is useful for the vDPA device that want to know all the mappings to tweak their own DMA translation logic.

For vDPA device that doesn't require set_map(), no behavior changes.

This capability is advertised via VHOST_BACKEND_F_IOTLB_BATCH capability.

---
 drivers/vhost/vdpa.c             | 36 ++++++++++++++++++++++++--------
 include/uapi/linux/vhost.h       |  2 ++
 include/uapi/linux/vhost_types.h | 11 ++++++++++
 3 files changed, 40 insertions(+), 9 deletions(-)
drivers/vhost/vdpa.c +27 -9
index 2f0310d1b914..0a143615e69d 100644
@@ -27,7 +27,9 @@
 #include "vhost.h"
 
 enum {
-	VHOST_VDPA_BACKEND_FEATURES = (1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2)
+	VHOST_VDPA_BACKEND_FEATURES =
+	(1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2) |
+	(1ULL << VHOST_BACKEND_F_IOTLB_BATCH),
 };
 
 /* Currently, only network backend w/o multiqueue is supported. */
@@ -48,6 +50,7 @@ struct vhost_vdpa {
 	int virtio_id;
 	int minor;
 	struct eventfd_ctx *config_ctx;
+	int in_batch;
 };
 
 static DEFINE_IDA(vhost_vdpa_ida);
@@ -140,6 +143,7 @@ static void vhost_vdpa_reset(struct vhost_vdpa *v)
 	struct vdpa_device *vdpa = v->vdpa;
 
 	vdpa_reset(vdpa);
+	v->in_batch = 0;
 }
 
 static long vhost_vdpa_get_device_id(struct vhost_vdpa *v, u8 __user *argp)
@@ -563,13 +567,15 @@ static int vhost_vdpa_map(struct vhost_vdpa *v,
 	if (r)
 		return r;
 
-	if (ops->dma_map)
+	if (ops->dma_map) {
 		r = ops->dma_map(vdpa, iova, size, pa, perm);
-	else if (ops->set_map)
-		r = ops->set_map(vdpa, dev->iotlb);
-	else
+	} else if (ops->set_map) {
+		if (!v->in_batch)
+			r = ops->set_map(vdpa, dev->iotlb);
+	} else {
 		r = iommu_map(v->domain, iova, pa, size,
 			      perm_to_iommu_flags(perm));
+	}
 
 	return r;
 }
@@ -582,12 +588,14 @@ static void vhost_vdpa_unmap(struct vhost_vdpa *v, u64 iova, u64 size)
 
 	vhost_vdpa_iotlb_unmap(v, iova, iova + size - 1);
 
-	if (ops->dma_map)
+	if (ops->dma_map) {
 		ops->dma_unmap(vdpa, iova, size);
-	else if (ops->set_map)
-		ops->set_map(vdpa, dev->iotlb);
-	else
+	} else if (ops->set_map) {
+		if (!v->in_batch)
+			ops->set_map(vdpa, dev->iotlb);
+	} else {
 		iommu_unmap(v->domain, iova, size);
+	}
 }
 
 static int vhost_vdpa_process_iotlb_update(struct vhost_vdpa *v,
@@ -680,6 +688,8 @@ static int vhost_vdpa_process_iotlb_msg(struct vhost_dev *dev,
 					struct vhost_iotlb_msg *msg)
 {
 	struct vhost_vdpa *v = container_of(dev, struct vhost_vdpa, vdev);
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
 	int r = 0;
 
 	r = vhost_dev_check_owner(dev);
@@ -693,6 +703,14 @@ static int vhost_vdpa_process_iotlb_msg(struct vhost_dev *dev,
 	case VHOST_IOTLB_INVALIDATE:
 		vhost_vdpa_unmap(v, msg->iova, msg->size);
 		break;
+	case VHOST_IOTLB_BATCH_BEGIN:
+		v->in_batch = true;
+		break;
+	case VHOST_IOTLB_BATCH_END:
+		if (v->in_batch && ops->set_map)
+			ops->set_map(vdpa, dev->iotlb);
+		v->in_batch = false;
+		break;
 	default:
 		r = -EINVAL;
 		break;
include/uapi/linux/vhost.h +2 -0
index 5d9254e2a6b6..11a4948b6216 100644
@@ -91,6 +91,8 @@
 
 /* Use message type V2 */
 #define VHOST_BACKEND_F_IOTLB_MSG_V2 0x1
+/* IOTLB can accept batching hints */
+#define VHOST_BACKEND_F_IOTLB_BATCH  0x2
 
 #define VHOST_SET_BACKEND_FEATURES _IOW(VHOST_VIRTIO, 0x25, __u64)
 #define VHOST_GET_BACKEND_FEATURES _IOR(VHOST_VIRTIO, 0x26, __u64)
include/uapi/linux/vhost_types.h +11 -0
index 669457ce5c48..9a269a88a6ff 100644
@@ -60,6 +60,17 @@ struct vhost_iotlb_msg {
 #define VHOST_IOTLB_UPDATE         2
 #define VHOST_IOTLB_INVALIDATE     3
 #define VHOST_IOTLB_ACCESS_FAIL    4
+/*
+ * VHOST_IOTLB_BATCH_BEGIN and VHOST_IOTLB_BATCH_END allow modifying
+ * multiple mappings in one go: beginning with
+ * VHOST_IOTLB_BATCH_BEGIN, followed by any number of
+ * VHOST_IOTLB_UPDATE messages, and ending with VHOST_IOTLB_BATCH_END.
+ * When one of these two values is used as the message type, the rest
+ * of the fields in the message are ignored. There's no guarantee that
+ * these changes take place automatically in the device.
+ */
+#define VHOST_IOTLB_BATCH_BEGIN    5
+#define VHOST_IOTLB_BATCH_END      6
 	__u8 type;
 };
 
Signature · 1 line
2.26.0
author
#5 lore raw

PATCH v4 5/12 vdpasim: support batch updating

In reply to Eli Cohen

From: Jason Wang <[email protected]>

The vDPA simulator support both set_map() and dma_map()/dma_unmap() operations. But vhost-vdpa can only use one of them. So this patch introduce a module parameter (batch_mapping) that let vpda_sim to support only one of those dma operations. The batched mapping via set_map() is enabled by default.

---
 drivers/vdpa/vdpa_sim/vdpa_sim.c | 40 +++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)
drivers/vdpa/vdpa_sim/vdpa_sim.c +37 -3
index 13febedd73ee..07ded545197d 100644
@@ -34,6 +34,10 @@
 #define DRV_DESC     "vDPA Device Simulator"
 #define DRV_LICENSE  "GPL v2"
 
+static int batch_mapping = 1;
+module_param(batch_mapping, int, 0444);
+MODULE_PARM_DESC(batch_mapping, "Batched mapping 1 -Enable; 0 - Disable");
+
 struct vdpasim_virtqueue {
 	struct vringh vring;
 	struct vringh_kiov iov;
@@ -334,15 +338,21 @@ static const struct dma_map_ops vdpasim_dma_ops = {
 };
 
 static const struct vdpa_config_ops vdpasim_net_config_ops;
+static const struct vdpa_config_ops vdpasim_net_batch_config_ops;
 
 static struct vdpasim *vdpasim_create(void)
 {
+	const struct vdpa_config_ops *ops;
 	struct vdpasim *vdpasim;
 	struct device *dev;
 	int ret = -ENOMEM;
 
-	vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL,
-				    &vdpasim_net_config_ops);
+	if (batch_mapping)
+		ops = &vdpasim_net_batch_config_ops;
+	else
+		ops = &vdpasim_net_config_ops;
+
+	vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops);
 	if (!vdpasim)
 		goto err_alloc;
 
@@ -641,12 +651,36 @@ static const struct vdpa_config_ops vdpasim_net_config_ops = {
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
 	.get_generation         = vdpasim_get_generation,
-	.set_map                = vdpasim_set_map,
 	.dma_map                = vdpasim_dma_map,
 	.dma_unmap              = vdpasim_dma_unmap,
 	.free                   = vdpasim_free,
 };
 
+static const struct vdpa_config_ops vdpasim_net_batch_config_ops = {
+	.set_vq_address         = vdpasim_set_vq_address,
+	.set_vq_num             = vdpasim_set_vq_num,
+	.kick_vq                = vdpasim_kick_vq,
+	.set_vq_cb              = vdpasim_set_vq_cb,
+	.set_vq_ready           = vdpasim_set_vq_ready,
+	.get_vq_ready           = vdpasim_get_vq_ready,
+	.set_vq_state           = vdpasim_set_vq_state,
+	.get_vq_state           = vdpasim_get_vq_state,
+	.get_vq_align           = vdpasim_get_vq_align,
+	.get_features           = vdpasim_get_features,
+	.set_features           = vdpasim_set_features,
+	.set_config_cb          = vdpasim_set_config_cb,
+	.get_vq_num_max         = vdpasim_get_vq_num_max,
+	.get_device_id          = vdpasim_get_device_id,
+	.get_vendor_id          = vdpasim_get_vendor_id,
+	.get_status             = vdpasim_get_status,
+	.set_status             = vdpasim_set_status,
+	.get_config             = vdpasim_get_config,
+	.set_config             = vdpasim_set_config,
+	.get_generation         = vdpasim_get_generation,
+	.set_map                = vdpasim_set_map,
+	.free                   = vdpasim_free,
+};
+
 static int __init vdpasim_dev_init(void)
 {
 	vdpasim_dev = vdpasim_create();
Signature · 1 line
2.26.0
author
#6 lore raw

PATCH v4 6/12 vdpa: remove hard coded virtq num

In reply to Eli Cohen

From: Max Gurtovoy <[email protected]>

This will enable vdpa providers to add support for multi queue feature and publish it to upper layers (vhost and virtio).

---
 drivers/vdpa/ifcvf/ifcvf_main.c  | 3 ++-
 drivers/vdpa/vdpa.c              | 3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim.c | 4 ++--
 drivers/vhost/vdpa.c             | 9 +++------
 include/linux/vdpa.h             | 6 ++++--
 5 files changed, 14 insertions(+), 11 deletions(-)
drivers/vdpa/ifcvf/ifcvf_main.c +2 -1
index a902b29b0d29..7c93225367db 100644
@@ -431,7 +431,8 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	}
 
 	adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa,
-				    dev, &ifc_vdpa_ops);
+				    dev, &ifc_vdpa_ops,
+				    IFCVF_MAX_QUEUE_PAIRS * 2);
 	if (adapter == NULL) {
 		IFCVF_ERR(pdev, "Failed to allocate vDPA structure");
 		return -ENOMEM;
drivers/vdpa/vdpa.c +3 -0
index 7105265e4793..a69ffc991e13 100644
@@ -61,6 +61,7 @@ static void vdpa_release_dev(struct device *d)
  * initialized but before registered.
  * @parent: the parent device
  * @config: the bus operations that is supported by this device
+ * @nvqs: number of virtqueues supported by this device
  * @size: size of the parent structure that contains private data
  *
  * Driver should use vdpa_alloc_device() wrapper macro instead of
@@ -71,6 +72,7 @@ static void vdpa_release_dev(struct device *d)
  */
 struct vdpa_device *__vdpa_alloc_device(struct device *parent,
 					const struct vdpa_config_ops *config,
+					int nvqs,
 					size_t size)
 {
 	struct vdpa_device *vdev;
@@ -97,6 +99,7 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent,
 	vdev->index = err;
 	vdev->config = config;
 	vdev->features_valid = false;
+	vdev->nvqs = nvqs;
 
 	err = dev_set_name(&vdev->dev, "vdpa%u", vdev->index);
 	if (err)
drivers/vdpa/vdpa_sim/vdpa_sim.c +2 -2
index 07ded545197d..676cb2f53202 100644
@@ -65,7 +65,7 @@ static u64 vdpasim_features = (1ULL << VIRTIO_F_ANY_LAYOUT) |
 /* State of each vdpasim device */
 struct vdpasim {
 	struct vdpa_device vdpa;
-	struct vdpasim_virtqueue vqs[2];
+	struct vdpasim_virtqueue vqs[VDPASIM_VQ_NUM];
 	struct work_struct work;
 	/* spinlock to synchronize virtqueue state */
 	spinlock_t lock;
@@ -352,7 +352,7 @@ static struct vdpasim *vdpasim_create(void)
 	else
 		ops = &vdpasim_net_config_ops;
 
-	vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops);
+	vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops, VDPASIM_VQ_NUM);
 	if (!vdpasim)
 		goto err_alloc;
 
drivers/vhost/vdpa.c +3 -6
index 0a143615e69d..0049e34f7ca6 100644
@@ -32,9 +32,6 @@ enum {
 	(1ULL << VHOST_BACKEND_F_IOTLB_BATCH),
 };
 
-/* Currently, only network backend w/o multiqueue is supported. */
-#define VHOST_VDPA_VQ_MAX	2
-
 #define VHOST_VDPA_DEV_MAX (1U << MINORBITS)
 
 struct vhost_vdpa {
@@ -947,7 +944,7 @@ static int vhost_vdpa_probe(struct vdpa_device *vdpa)
 {
 	const struct vdpa_config_ops *ops = vdpa->config;
 	struct vhost_vdpa *v;
-	int minor, nvqs = VHOST_VDPA_VQ_MAX;
+	int minor;
 	int r;
 
 	/* Currently, we only accept the network devices. */
@@ -968,14 +965,14 @@ static int vhost_vdpa_probe(struct vdpa_device *vdpa)
 	atomic_set(&v->opened, 0);
 	v->minor = minor;
 	v->vdpa = vdpa;
-	v->nvqs = nvqs;
+	v->nvqs = vdpa->nvqs;
 	v->virtio_id = ops->get_device_id(vdpa);
 
 	device_initialize(&v->dev);
 	v->dev.release = vhost_vdpa_release_dev;
 	v->dev.parent = &vdpa->dev;
 	v->dev.devt = MKDEV(MAJOR(vhost_vdpa_major), minor);
-	v->vqs = kmalloc_array(nvqs, sizeof(struct vhost_virtqueue),
+	v->vqs = kmalloc_array(v->nvqs, sizeof(struct vhost_virtqueue),
 			       GFP_KERNEL);
 	if (!v->vqs) {
 		r = -ENOMEM;
include/linux/vdpa.h +4 -2
index 9f50af7dc4a1..7b1e9df8fc11 100644
@@ -41,6 +41,7 @@ struct vdpa_device {
 	const struct vdpa_config_ops *config;
 	unsigned int index;
 	bool features_valid;
+	int nvqs;
 };
 
 /**
@@ -216,11 +217,12 @@ struct vdpa_config_ops {
 
 struct vdpa_device *__vdpa_alloc_device(struct device *parent,
 					const struct vdpa_config_ops *config,
+					int nvqs,
 					size_t size);
 
-#define vdpa_alloc_device(dev_struct, member, parent, config)   \
+#define vdpa_alloc_device(dev_struct, member, parent, config, nvqs)   \
 			  container_of(__vdpa_alloc_device( \
-				       parent, config, \
+				       parent, config, nvqs, \
 				       sizeof(dev_struct) + \
 				       BUILD_BUG_ON_ZERO(offsetof( \
 				       dev_struct, member))), \
Signature · 1 line
2.26.0
author
#7 lore raw

PATCH v4 7/12 net/vdpa: Use struct for set/get vq state

In reply to Eli Cohen

For now VQ state involves 16 bit available index value encoded in u64 variable. In the future it will be extended to contain more fields. Use struct to contain the state, now containing only a single u16 for the available index. In the future we can add fields to this struct.

---
 drivers/vdpa/ifcvf/ifcvf_base.c  |  4 ++--
 drivers/vdpa/ifcvf/ifcvf_base.h  |  4 ++--
 drivers/vdpa/ifcvf/ifcvf_main.c  |  9 +++++----
 drivers/vdpa/vdpa_sim/vdpa_sim.c | 10 ++++++----
 drivers/vhost/vdpa.c             |  7 +++++--
 include/linux/vdpa.h             | 18 ++++++++++++++----
 6 files changed, 34 insertions(+), 18 deletions(-)
drivers/vdpa/ifcvf/ifcvf_base.c +2 -2
index 94bf0328b68d..f2a128e56de5 100644
@@ -272,7 +272,7 @@ static int ifcvf_config_features(struct ifcvf_hw *hw)
 	return 0;
 }
 
-u64 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
+u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
 {
 	struct ifcvf_lm_cfg __iomem *ifcvf_lm;
 	void __iomem *avail_idx_addr;
@@ -287,7 +287,7 @@ u64 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
 	return last_avail_idx;
 }
 
-int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u64 num)
+int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16 num)
 {
 	struct ifcvf_lm_cfg __iomem *ifcvf_lm;
 	void __iomem *avail_idx_addr;
drivers/vdpa/ifcvf/ifcvf_base.h +2 -2
index 24af422b5a3e..08f267a2aafe 100644
@@ -116,7 +116,7 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8 status);
 void io_write64_twopart(u64 val, u32 *lo, u32 *hi);
 void ifcvf_reset(struct ifcvf_hw *hw);
 u64 ifcvf_get_features(struct ifcvf_hw *hw);
-u64 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid);
-int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u64 num);
+u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid);
+int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16 num);
 struct ifcvf_adapter *vf_to_adapter(struct ifcvf_hw *hw);
 #endif /* _IFCVF_H_ */
drivers/vdpa/ifcvf/ifcvf_main.c +5 -4
index 7c93225367db..dc311e972b9e 100644
@@ -237,19 +237,20 @@ static u16 ifcvf_vdpa_get_vq_num_max(struct vdpa_device *vdpa_dev)
 	return IFCVF_QUEUE_MAX;
 }
 
-static u64 ifcvf_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid)
+static void ifcvf_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
+				    struct vdpa_vq_state *state)
 {
 	struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
 
-	return ifcvf_get_vq_state(vf, qid);
+	state->avail_index = ifcvf_get_vq_state(vf, qid);
 }
 
 static int ifcvf_vdpa_set_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
-				   u64 num)
+				   const struct vdpa_vq_state *state)
 {
 	struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
 
-	return ifcvf_set_vq_state(vf, qid, num);
+	return ifcvf_set_vq_state(vf, qid, state->avail_index);
 }
 
 static void ifcvf_vdpa_set_vq_cb(struct vdpa_device *vdpa_dev, u16 qid,
drivers/vdpa/vdpa_sim/vdpa_sim.c +6 -4
index 676cb2f53202..f1c351d5959b 100644
@@ -450,26 +450,28 @@ static bool vdpasim_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
 	return vq->ready;
 }
 
-static int vdpasim_set_vq_state(struct vdpa_device *vdpa, u16 idx, u64 state)
+static int vdpasim_set_vq_state(struct vdpa_device *vdpa, u16 idx,
+				const struct vdpa_vq_state *state)
 {
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
 	struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
 	struct vringh *vrh = &vq->vring;
 
 	spin_lock(&vdpasim->lock);
-	vrh->last_avail_idx = state;
+	vrh->last_avail_idx = state->avail_index;
 	spin_unlock(&vdpasim->lock);
 
 	return 0;
 }
 
-static u64 vdpasim_get_vq_state(struct vdpa_device *vdpa, u16 idx)
+static void vdpasim_get_vq_state(struct vdpa_device *vdpa, u16 idx,
+				 struct vdpa_vq_state *state)
 {
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
 	struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
 	struct vringh *vrh = &vq->vring;
 
-	return vrh->last_avail_idx;
+	state->avail_index = vrh->last_avail_idx;
 }
 
 static u32 vdpasim_get_vq_align(struct vdpa_device *vdpa)
drivers/vhost/vdpa.c +5 -2
index 0049e34f7ca6..c2e1e2d55084 100644
@@ -366,6 +366,7 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 {
 	struct vdpa_device *vdpa = v->vdpa;
 	const struct vdpa_config_ops *ops = vdpa->config;
+	struct vdpa_vq_state vq_state;
 	struct vdpa_callback cb;
 	struct vhost_virtqueue *vq;
 	struct vhost_vring_state s;
@@ -391,7 +392,8 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 		ops->set_vq_ready(vdpa, idx, s.num);
 		return 0;
 	case VHOST_GET_VRING_BASE:
-		vq->last_avail_idx = ops->get_vq_state(v->vdpa, idx);
+		ops->get_vq_state(v->vdpa, idx, &vq_state);
+		vq->last_avail_idx = vq_state.avail_index;
 		break;
 	case VHOST_GET_BACKEND_FEATURES:
 		features = VHOST_VDPA_BACKEND_FEATURES;
@@ -421,7 +423,8 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 		break;
 
 	case VHOST_SET_VRING_BASE:
-		if (ops->set_vq_state(vdpa, idx, vq->last_avail_idx))
+		vq_state.avail_index = vq->last_avail_idx;
+		if (ops->set_vq_state(vdpa, idx, &vq_state))
 			r = -EINVAL;
 		break;
 
include/linux/vdpa.h +14 -4
index 7b1e9df8fc11..8f412620071d 100644
@@ -27,6 +27,14 @@ struct vdpa_notification_area {
 	resource_size_t size;
 };
 
+/**
+ * vDPA vq_state definition
+ * @avail_index: available index
+ */
+struct vdpa_vq_state {
+	u16	avail_index;
+};
+
 /**
  * vDPA device - representation of a vDPA device
  * @dev: underlying device
@@ -80,12 +88,12 @@ struct vdpa_device {
  * @set_vq_state:		Set the state for a virtqueue
  *				@vdev: vdpa device
  *				@idx: virtqueue index
- *				@state: virtqueue state (last_avail_idx)
+ *				@state: pointer to set virtqueue state (last_avail_idx)
  *				Returns integer: success (0) or error (< 0)
  * @get_vq_state:		Get the state for a virtqueue
  *				@vdev: vdpa device
  *				@idx: virtqueue index
- *				Returns virtqueue state (last_avail_idx)
+ *				@state: pointer to returned state (last_avail_idx)
  * @get_vq_notification: 	Get the notification area for a virtqueue
  *				@vdev: vdpa device
  *				@idx: virtqueue index
@@ -182,8 +190,10 @@ struct vdpa_config_ops {
 			  struct vdpa_callback *cb);
 	void (*set_vq_ready)(struct vdpa_device *vdev, u16 idx, bool ready);
 	bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx);
-	int (*set_vq_state)(struct vdpa_device *vdev, u16 idx, u64 state);
-	u64 (*get_vq_state)(struct vdpa_device *vdev, u16 idx);
+	int (*set_vq_state)(struct vdpa_device *vdev, u16 idx,
+			    const struct vdpa_vq_state *state);
+	void (*get_vq_state)(struct vdpa_device *vdev, u16 idx,
+			     struct vdpa_vq_state *state);
 	struct vdpa_notification_area
 	(*get_vq_notification)(struct vdpa_device *vdev, u16 idx);
 	int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);
Signature · 1 line
2.26.0
author
#8 lore raw

PATCH v4 8/12 vdpa: Modify get_vq_state() to return error code

In reply to Eli Cohen

Modify get_vq_state() so it returns an error code. In case of hardware acceleration, the available index may be retrieved from the device, an operation that can possibly fail.

---
 drivers/vdpa/ifcvf/ifcvf_main.c  | 5 +++--
 drivers/vdpa/vdpa_sim/vdpa_sim.c | 5 +++--
 drivers/vhost/vdpa.c             | 5 ++++-
 include/linux/vdpa.h             | 4 ++--
 4 files changed, 12 insertions(+), 7 deletions(-)
drivers/vdpa/ifcvf/ifcvf_main.c +3 -2
index dc311e972b9e..076d7ac5e723 100644
@@ -237,12 +237,13 @@ static u16 ifcvf_vdpa_get_vq_num_max(struct vdpa_device *vdpa_dev)
 	return IFCVF_QUEUE_MAX;
 }
 
-static void ifcvf_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
-				    struct vdpa_vq_state *state)
+static int ifcvf_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
+				   struct vdpa_vq_state *state)
 {
 	struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
 
 	state->avail_index = ifcvf_get_vq_state(vf, qid);
+	return 0;
 }
 
 static int ifcvf_vdpa_set_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
drivers/vdpa/vdpa_sim/vdpa_sim.c +3 -2
index f1c351d5959b..c68741363643 100644
@@ -464,14 +464,15 @@ static int vdpasim_set_vq_state(struct vdpa_device *vdpa, u16 idx,
 	return 0;
 }
 
-static void vdpasim_get_vq_state(struct vdpa_device *vdpa, u16 idx,
-				 struct vdpa_vq_state *state)
+static int vdpasim_get_vq_state(struct vdpa_device *vdpa, u16 idx,
+				struct vdpa_vq_state *state)
 {
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
 	struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
 	struct vringh *vrh = &vq->vring;
 
 	state->avail_index = vrh->last_avail_idx;
+	return 0;
 }
 
 static u32 vdpasim_get_vq_align(struct vdpa_device *vdpa)
drivers/vhost/vdpa.c +4 -1
index c2e1e2d55084..a0b7c91948e1 100644
@@ -392,7 +392,10 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 		ops->set_vq_ready(vdpa, idx, s.num);
 		return 0;
 	case VHOST_GET_VRING_BASE:
-		ops->get_vq_state(v->vdpa, idx, &vq_state);
+		r = ops->get_vq_state(v->vdpa, idx, &vq_state);
+		if (r)
+			return r;
+
 		vq->last_avail_idx = vq_state.avail_index;
 		break;
 	case VHOST_GET_BACKEND_FEATURES:
include/linux/vdpa.h +2 -2
index 8f412620071d..fd6e560d70f9 100644
@@ -192,8 +192,8 @@ struct vdpa_config_ops {
 	bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx);
 	int (*set_vq_state)(struct vdpa_device *vdev, u16 idx,
 			    const struct vdpa_vq_state *state);
-	void (*get_vq_state)(struct vdpa_device *vdev, u16 idx,
-			     struct vdpa_vq_state *state);
+	int (*get_vq_state)(struct vdpa_device *vdev, u16 idx,
+			    struct vdpa_vq_state *state);
 	struct vdpa_notification_area
 	(*get_vq_notification)(struct vdpa_device *vdev, u16 idx);
 	int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);
Signature · 1 line
2.26.0
Acked-by on v4 8/12
#9 lore raw

Re: [PATCH V4 linux-next 08/12] vdpa: Modify get_vq_state() to return error code

Quoted 6 lines · Eli Cohen

Modify get_vq_state() so it returns an error code. In case of hardware acceleration, the available index may be retrieved from the device, an operation that can possibly fail.

Quoted 70 lines
---
  drivers/vdpa/ifcvf/ifcvf_main.c  | 5 +++--
  drivers/vdpa/vdpa_sim/vdpa_sim.c | 5 +++--
  drivers/vhost/vdpa.c             | 5 ++++-
  include/linux/vdpa.h             | 4 ++--
  4 files changed, 12 insertions(+), 7 deletions(-)
drivers/vdpa/ifcvf/ifcvf_main.c +3 -2
index dc311e972b9e..076d7ac5e723 100644
@@ -237,12 +237,13 @@ static u16 ifcvf_vdpa_get_vq_num_max(struct vdpa_device *vdpa_dev)
  	return IFCVF_QUEUE_MAX;
  }
 
-static void ifcvf_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
-				    struct vdpa_vq_state *state)
+static int ifcvf_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
+				   struct vdpa_vq_state *state)
  {
  	struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
 
  	state->avail_index = ifcvf_get_vq_state(vf, qid);
+	return 0;
  }
 
  static int ifcvf_vdpa_set_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
drivers/vdpa/vdpa_sim/vdpa_sim.c +3 -2
index f1c351d5959b..c68741363643 100644
@@ -464,14 +464,15 @@ static int vdpasim_set_vq_state(struct vdpa_device *vdpa, u16 idx,
  	return 0;
  }
 
-static void vdpasim_get_vq_state(struct vdpa_device *vdpa, u16 idx,
-				 struct vdpa_vq_state *state)
+static int vdpasim_get_vq_state(struct vdpa_device *vdpa, u16 idx,
+				struct vdpa_vq_state *state)
  {
  	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
  	struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
  	struct vringh *vrh = &vq->vring;
 
  	state->avail_index = vrh->last_avail_idx;
+	return 0;
  }
 
  static u32 vdpasim_get_vq_align(struct vdpa_device *vdpa)
drivers/vhost/vdpa.c +4 -1
index c2e1e2d55084..a0b7c91948e1 100644
@@ -392,7 +392,10 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
  		ops->set_vq_ready(vdpa, idx, s.num);
  		return 0;
  	case VHOST_GET_VRING_BASE:
-		ops->get_vq_state(v->vdpa, idx, &vq_state);
+		r = ops->get_vq_state(v->vdpa, idx, &vq_state);
+		if (r)
+			return r;
+
  		vq->last_avail_idx = vq_state.avail_index;
  		break;
  	case VHOST_GET_BACKEND_FEATURES:
include/linux/vdpa.h +0 -0
index 8f412620071d..fd6e560d70f9 100644

include/linux/vdpa.h struct vdpa_config_ops { · line 199

Quoted 10 lines
include/linux/vdpa.h +2 -2
@@ -192,8 +192,8 @@ struct vdpa_config_ops {
  	bool (*get_vq_ready)(struct vdpa_device *vdev, u16 idx);
  	int (*set_vq_state)(struct vdpa_device *vdev, u16 idx,
  			    const struct vdpa_vq_state *state);
-	void (*get_vq_state)(struct vdpa_device *vdev, u16 idx,
-			     struct vdpa_vq_state *state);
+	int (*get_vq_state)(struct vdpa_device *vdev, u16 idx,
+			    struct vdpa_vq_state *state);
  	struct vdpa_notification_area
  	(*get_vq_notification)(struct vdpa_device *vdev, u16 idx);
  	int (*get_vq_irq)(struct vdpa_device *vdv, u16 idx);

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#10 lore raw

PATCH v4 9/12 vdpa/mlx5: Add hardware descriptive header file

In reply to Eli Cohen

Keep all vdpa related hardware definitions in this file.

---
 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h | 168 +++++++++++++++++++++++++
 1 file changed, 168 insertions(+)
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h
drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h new file +168 -0
new file mode 100644
index 000000000000..f6f57a29b38e
@@ -0,0 +1,168 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2020 Mellanox Technologies Ltd. */
+
+#ifndef __MLX5_VDPA_IFC_H_
+#define __MLX5_VDPA_IFC_H_
+
+#include <linux/mlx5/mlx5_ifc.h>
+
+enum {
+	MLX5_VIRTIO_Q_EVENT_MODE_NO_MSIX_MODE  = 0x0,
+	MLX5_VIRTIO_Q_EVENT_MODE_QP_MODE       = 0x1,
+	MLX5_VIRTIO_Q_EVENT_MODE_MSIX_MODE     = 0x2,
+};
+
+enum {
+	MLX5_VIRTIO_EMULATION_CAP_VIRTIO_QUEUE_TYPE_SPLIT   = 0x1, // do I check this caps?
+	MLX5_VIRTIO_EMULATION_CAP_VIRTIO_QUEUE_TYPE_PACKED  = 0x2,
+};
+
+enum {
+	MLX5_VIRTIO_EMULATION_VIRTIO_QUEUE_TYPE_SPLIT   = 0,
+	MLX5_VIRTIO_EMULATION_VIRTIO_QUEUE_TYPE_PACKED  = 1,
+};
+
+struct mlx5_ifc_virtio_q_bits {
+	u8    virtio_q_type[0x8];
+	u8    reserved_at_8[0x5];
+	u8    event_mode[0x3];
+	u8    queue_index[0x10];
+
+	u8    full_emulation[0x1];
+	u8    virtio_version_1_0[0x1];
+	u8    reserved_at_22[0x2];
+	u8    offload_type[0x4];
+	u8    event_qpn_or_msix[0x18];
+
+	u8    doorbell_stride_index[0x10];
+	u8    queue_size[0x10];
+
+	u8    device_emulation_id[0x20];
+
+	u8    desc_addr[0x40];
+
+	u8    used_addr[0x40];
+
+	u8    available_addr[0x40];
+
+	u8    virtio_q_mkey[0x20];
+
+	u8    max_tunnel_desc[0x10];
+	u8    reserved_at_170[0x8];
+	u8    error_type[0x8];
+
+	u8    umem_1_id[0x20];
+
+	u8    umem_1_size[0x20];
+
+	u8    umem_1_offset[0x40];
+
+	u8    umem_2_id[0x20];
+
+	u8    umem_2_size[0x20];
+
+	u8    umem_2_offset[0x40];
+
+	u8    umem_3_id[0x20];
+
+	u8    umem_3_size[0x20];
+
+	u8    umem_3_offset[0x40];
+
+	u8    counter_set_id[0x20];
+
+	u8    reserved_at_320[0x8];
+	u8    pd[0x18];
+
+	u8    reserved_at_340[0xc0];
+};
+
+struct mlx5_ifc_virtio_net_q_object_bits {
+	u8    modify_field_select[0x40];
+
+	u8    reserved_at_40[0x20];
+
+	u8    vhca_id[0x10];
+	u8    reserved_at_70[0x10];
+
+	u8    queue_feature_bit_mask_12_3[0xa];
+	u8    dirty_bitmap_dump_enable[0x1];
+	u8    vhost_log_page[0x5];
+	u8    reserved_at_90[0xc];
+	u8    state[0x4];
+
+	u8    reserved_at_a0[0x5];
+	u8    queue_feature_bit_mask_2_0[0x3];
+	u8    tisn_or_qpn[0x18];
+
+	u8    dirty_bitmap_mkey[0x20];
+
+	u8    dirty_bitmap_size[0x20];
+
+	u8    dirty_bitmap_addr[0x40];
+
+	u8    hw_available_index[0x10];
+	u8    hw_used_index[0x10];
+
+	u8    reserved_at_160[0xa0];
+
+	struct mlx5_ifc_virtio_q_bits virtio_q_context;
+};
+
+struct mlx5_ifc_create_virtio_net_q_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits general_obj_in_cmd_hdr;
+
+	struct mlx5_ifc_virtio_net_q_object_bits obj_context;
+};
+
+struct mlx5_ifc_create_virtio_net_q_out_bits {
+	struct mlx5_ifc_general_obj_out_cmd_hdr_bits general_obj_out_cmd_hdr;
+};
+
+struct mlx5_ifc_destroy_virtio_net_q_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits general_obj_out_cmd_hdr;
+};
+
+struct mlx5_ifc_destroy_virtio_net_q_out_bits {
+	struct mlx5_ifc_general_obj_out_cmd_hdr_bits general_obj_out_cmd_hdr;
+};
+
+struct mlx5_ifc_query_virtio_net_q_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits general_obj_in_cmd_hdr;
+};
+
+struct mlx5_ifc_query_virtio_net_q_out_bits {
+	struct mlx5_ifc_general_obj_out_cmd_hdr_bits general_obj_out_cmd_hdr;
+
+	struct mlx5_ifc_virtio_net_q_object_bits obj_context;
+};
+
+enum {
+	MLX5_VIRTQ_MODIFY_MASK_STATE                    = (u64)1 << 0,
+	MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS      = (u64)1 << 3,
+	MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4,
+};
+
+enum {
+	MLX5_VIRTIO_NET_Q_OBJECT_STATE_INIT     = 0x0,
+	MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY      = 0x1,
+	MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND  = 0x2,
+	MLX5_VIRTIO_NET_Q_OBJECT_STATE_ERR      = 0x3,
+};
+
+enum {
+	MLX5_RQTC_LIST_Q_TYPE_RQ            = 0x0,
+	MLX5_RQTC_LIST_Q_TYPE_VIRTIO_NET_Q  = 0x1,
+};
+
+struct mlx5_ifc_modify_virtio_net_q_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits general_obj_in_cmd_hdr;
+
+	struct mlx5_ifc_virtio_net_q_object_bits obj_context;
+};
+
+struct mlx5_ifc_modify_virtio_net_q_out_bits {
+	struct mlx5_ifc_general_obj_out_cmd_hdr_bits general_obj_out_cmd_hdr;
+};
+
+#endif /* __MLX5_VDPA_IFC_H_ */
Signature · 1 line
2.26.0
author
#11 lore raw

PATCH v4 10/12 vdpa/mlx5: Add support library for mlx5 VDPA implementation

In reply to Eli Cohen

Following patches introduce VDPA network driver for Mellanox Connectx6 devices. This patch provides functionality that will be used by those patches.

---
 drivers/vdpa/Kconfig               |   9 +
 drivers/vdpa/Makefile              |   1 +
 drivers/vdpa/mlx5/Makefile         |   1 +
 drivers/vdpa/mlx5/core/Makefile    |   1 +
 drivers/vdpa/mlx5/core/mlx5_vdpa.h |  57 ++++++
 drivers/vdpa/mlx5/core/resources.c | 281 +++++++++++++++++++++++++++++
 6 files changed, 350 insertions(+)
 create mode 100644 drivers/vdpa/mlx5/Makefile
 create mode 100644 drivers/vdpa/mlx5/core/Makefile
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa.h
 create mode 100644 drivers/vdpa/mlx5/core/resources.c
drivers/vdpa/Kconfig +9 -0
index 3e1ceb8e9f2b..7cb84f82feba 100644
@@ -28,4 +28,13 @@ config IFCVF
 	  To compile this driver as a module, choose M here: the module will
 	  be called ifcvf.
 
+config MLX5_VDPA
+	bool "MLX5 VDPA support library for ConnectX devices"
+	depends on MLX5_CORE
+	default n
+	help
+	  Support library for Mellanox VDPA drivers. Provides code that is
+	  common for all types of VDPA drivers. The following drivers are planned:
+	  net, block.
+
 endif # VDPA
drivers/vdpa/Makefile +1 -0
index 8bbb686ca7a2..d160e9b63a66 100644
@@ -2,3 +2,4 @@
 obj-$(CONFIG_VDPA) += vdpa.o
 obj-$(CONFIG_VDPA_SIM) += vdpa_sim/
 obj-$(CONFIG_IFCVF)    += ifcvf/
+obj-$(CONFIG_MLX5_VDPA) += mlx5/
drivers/vdpa/mlx5/Makefile new file +1 -0
new file mode 100644
index 000000000000..d552abb1d126
@@ -0,0 +1 @@
+obj-$(CONFIG_MLX5_VDPA) += core/resources.o
drivers/vdpa/mlx5/core/Makefile new file +1 -0
new file mode 100644
index 000000000000..7070f8c8680d
@@ -0,0 +1 @@
+obj-y += resources.o
drivers/vdpa/mlx5/core/mlx5_vdpa.h new file +57 -0

57 lines not on this page · raw message

drivers/vdpa/mlx5/core/resources.c new file +281 -0

281 lines not on this page · raw message

Signature · 1 line
2.26.0
author
#12 lore raw

PATCH v4 11/12 vdpa/mlx5: Add shared memory registration code

In reply to Eli Cohen

Add code to support registering address space region for the device. The virtio driver can run as either:
1. Guest virtio driver
2. Userspace virtio driver on the host
3. Kernel virtio driver on the host

In any case a memory key object is required to provide access to memory for the device.

This code will be shared by network or block driver implementations.

---
 drivers/vdpa/mlx5/Makefile         |   2 +-
 drivers/vdpa/mlx5/core/Makefile    |   1 -
 drivers/vdpa/mlx5/core/mlx5_vdpa.h |  34 ++
 drivers/vdpa/mlx5/core/mr.c        | 484 +++++++++++++++++++++++++++++
 drivers/vdpa/mlx5/core/resources.c |   3 +
 5 files changed, 522 insertions(+), 2 deletions(-)
 delete mode 100644 drivers/vdpa/mlx5/core/Makefile
 create mode 100644 drivers/vdpa/mlx5/core/mr.c
drivers/vdpa/mlx5/Makefile +1 -1
index d552abb1d126..b347c62032ea 100644
@@ -1 +1 @@
-obj-$(CONFIG_MLX5_VDPA) += core/resources.o
+obj-$(CONFIG_MLX5_VDPA) += core/resources.o core/mr.o
drivers/vdpa/mlx5/core/Makefile deleted +0 -1
deleted file mode 100644
index 7070f8c8680d..000000000000
@@ -1 +0,0 @@
-obj-y += resources.o
drivers/vdpa/mlx5/core/mlx5_vdpa.h +34 -0
index f3571c8b257e..5c92a576edae 100644
@@ -7,6 +7,31 @@
 #include <linux/vdpa.h>
 #include <linux/mlx5/driver.h>
 
+struct mlx5_vdpa_direct_mr {
+	u64 start;
+	u64 end;
+	u32 perm;
+	struct mlx5_core_mkey mr;
+	struct sg_table sg_head;
+	int log_size;
+	int nsg;
+	struct list_head list;
+	u64 offset;
+};
+
+struct mlx5_vdpa_mr {
+	struct mlx5_core_mkey mkey;
+
+	/* list of direct MRs descendants of this indirect mr */
+	struct list_head head;
+	unsigned long num_directs;
+	unsigned long num_klms;
+	bool initialized;
+
+	/* serialize mkey creation and destruction */
+	struct mutex mkey_mtx;
+};
+
 struct mlx5_vdpa_resources {
 	u32 pdn;
 	struct mlx5_uars_page *uar;
@@ -26,6 +51,8 @@ struct mlx5_vdpa_dev {
 	u8 status;
 	u32 max_vqs;
 	u32 generation;
+
+	struct mlx5_vdpa_mr mr;
 };
 
 int mlx5_vdpa_alloc_pd(struct mlx5_vdpa_dev *dev, u32 *pdn, u16 uid);
@@ -41,6 +68,13 @@ int mlx5_vdpa_alloc_transport_domain(struct mlx5_vdpa_dev *mvdev, u32 *tdn);
 void mlx5_vdpa_dealloc_transport_domain(struct mlx5_vdpa_dev *mvdev, u32 tdn);
 int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev);
 void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev);
+int mlx5_vdpa_create_mkey(struct mlx5_vdpa_dev *mvdev, struct mlx5_core_mkey *mkey, u32 *in,
+			  int inlen);
+int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, struct mlx5_core_mkey *mkey);
+int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb,
+			     bool *change_map);
+int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb);
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev);
 
 #define mlx5_vdpa_warn(__dev, format, ...)                                                         \
 	dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func__, __LINE__,     \
drivers/vdpa/mlx5/core/mr.c new file +484 -0

484 lines not on this page · raw message

drivers/vdpa/mlx5/core/resources.c +3 -0
index 6c6552b7e9b5..96e6421c5d1c 100644
@@ -227,6 +227,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev)
 		mlx5_vdpa_warn(mvdev, "resources already allocated\n");
 		return -EINVAL;
 	}
+	mutex_init(&mvdev->mr.mkey_mtx);
 	res->uar = mlx5_get_uars_page(mdev);
 	if (IS_ERR(res->uar)) {
 		err = PTR_ERR(res->uar);
@@ -262,6 +263,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev)
 err_uctx:
 	mlx5_put_uars_page(mdev, res->uar);
 err_uars:
+	mutex_destroy(&mvdev->mr.mkey_mtx);
 	return err;
 }
 
@@ -277,5 +279,6 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev)
 	dealloc_pd(mvdev, res->pdn, res->uid);
 	destroy_uctx(mvdev, res->uid);
 	mlx5_put_uars_page(mvdev->mdev, res->uar);
+	mutex_destroy(&mvdev->mr.mkey_mtx);
 	res->valid = false;
 }
Signature · 1 line
2.26.0
author
#13 lore raw

PATCH v4 12/12 vdpa/mlx5: Add VDPA driver for supported mlx5 devices

In reply to Eli Cohen

Add a front end VDPA driver that registers in the VDPA bus and provides networking to a guest. The VDPA driver creates the necessary resources on the VF it is driving such that data path will be offloaded.

Notifications are being communicated through the driver.

Currently, only VFs are supported. In subsequent patches we will have devlink support to control which VF is used for VDPA and which function is used for regular networking.

---
 drivers/vdpa/Kconfig              |   10 +
 drivers/vdpa/mlx5/Makefile        |    5 +-
 drivers/vdpa/mlx5/core/mr.c       |    2 +-
 drivers/vdpa/mlx5/net/main.c      |   76 ++
 drivers/vdpa/mlx5/net/mlx5_vnet.c | 1965 +++++++++++++++++++++++++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.h |   24 +
 6 files changed, 2080 insertions(+), 2 deletions(-)
 create mode 100644 drivers/vdpa/mlx5/net/main.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
drivers/vdpa/Kconfig +10 -0
index 7cb84f82feba..a8c7607fdc90 100644
@@ -37,4 +37,14 @@ config MLX5_VDPA
 	  common for all types of VDPA drivers. The following drivers are planned:
 	  net, block.
 
+config MLX5_VDPA_NET
+	tristate "vDPA driver for ConnectX devices"
+	depends on MLX5_VDPA
+	default n
+	help
+	  VDPA network driver for ConnectX6 and newer. Provides offloading
+	  of virtio net datapath such that descriptors put on the ring will
+	  be executed by the hardware. It also supports a variety of stateless
+	  offloads depending on the actual device used and firmware version.
+
 endif # VDPA
drivers/vdpa/mlx5/Makefile +4 -1
index b347c62032ea..89a5bededc9f 100644
@@ -1 +1,4 @@
-obj-$(CONFIG_MLX5_VDPA) += core/resources.o core/mr.o
+subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
+
+obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
+mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/main.o net/mlx5_vnet.o core/resources.o core/mr.o
drivers/vdpa/mlx5/core/mr.c +1 -1
index 084698975c47..f5dec0274133 100644
@@ -464,7 +464,7 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io
 			     bool *change_map)
 {
 	struct mlx5_vdpa_mr *mr = &mvdev->mr;
-	int err;
+	int err = 0;
 
 	*change_map = false;
 	if (map_empty(iotlb)) {
drivers/vdpa/mlx5/net/main.c new file +76 -0

76 lines not on this page · raw message

drivers/vdpa/mlx5/net/mlx5_vnet.c new file +1965 -0

1,965 lines not on this page · raw message

drivers/vdpa/mlx5/net/mlx5_vnet.h new file +24 -0
new file mode 100644
index 000000000000..f2d6d68b020e
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2020 Mellanox Technologies Ltd. */
+
+#ifndef __MLX5_VNET_H_
+#define __MLX5_VNET_H_
+
+#include <linux/vdpa.h>
+#include <linux/virtio_net.h>
+#include <linux/vringh.h>
+#include <linux/mlx5/driver.h>
+#include <linux/mlx5/cq.h>
+#include <linux/mlx5/qp.h>
+#include "mlx5_vdpa.h"
+
+static inline u32 mlx5_vdpa_max_qps(int max_vqs)
+{
+	return max_vqs / 2;
+}
+
+#define to_mlx5_vdpa_ndev(__mvdev) container_of(__mvdev, struct mlx5_vdpa_net, mvdev)
+void *mlx5_vdpa_add_dev(struct mlx5_core_dev *mdev);
+void mlx5_vdpa_remove_dev(struct mlx5_vdpa_dev *mvdev);
+
+#endif /* __MLX5_VNET_H_ */
Signature · 1 line
2.26.0
Acked-by on v4 12/12
#14 lore raw

Re: [PATCH V4 linux-next 12/12] vdpa/mlx5: Add VDPA driver for supported mlx5 devices

Quoted 6 lines · Eli Cohen

Add a front end VDPA driver that registers in the VDPA bus and provides networking to a guest. The VDPA driver creates the necessary resources on the VF it is driving such that data path will be offloaded.

Notifications are being communicated through the driver.

Quoted 7 lines

Currently, only VFs are supported. In subsequent patches we will have devlink support to control which VF is used for VDPA and which function is used for regular networking.

---

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#15 lore raw
In reply to Eli Cohen
Quoted 4 lines · Eli Cohen

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

Thanks! OK so what's the plan for merging this?
Do patches at least build well enough that I can push them upstream? Or do they have to go on top of the mellanox tree?

Quoted 3 lines

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301

The ones you included, right?

Quoted 102 lines

Jason, I had to resolve some conflicts so I would appreciate of you can verify that it is ok.

The following series of patches provide VDPA support for Mellanox devices. The supported devices are ConnectX6 DX and newer.

Currently, only a network driver is implemented; future patches will introduce a block device driver. iperf performance on a single queue is around 12 Gbps. Future patches will introduce multi queue support.

The files are organized in such a way that code that can be used by different VDPA implementations will be placed in a common are resides in drivers/vdpa/mlx5/core.

Only virtual functions are currently supported. Also, certain firmware capabilities must be set to enable the driver. Physical functions (PFs) are skipped by the driver.

To make use of the VDPA net driver, one must load mlx5_vdpa. In such case, VFs will be operated by the VDPA driver. Although one can see a regular instance of a network driver on the VF, the VDPA driver takes precedence over the NIC driver, steering-wize.

Currently, the device/interface infrastructure in mlx5_core is used to probe drivers. Future patches will introduce virtbus as a means to register devices and drivers and VDPA will be adapted to it.

The mlx5 mode of operation required to support VDPA is switchdev mode. Once can use Linux or OVS bridge to take care of layer 2 switching.

In order to provide virtio networking to a guest, an updated version of qemu is required. This version has been tested by the following quemu version:

url: https://github.com/jasowang/qemu.git
branch: vdpa
Commit ID: 6f4e59b807db

V2->V3
Fix makefile to use include path relative to the root of the kernel

V3-V4
Rebase Jason's patches on linux-next branch
Fix krobot error on mips arch
Make use of the free callback to destroy resoruces on unload Use VIRTIO_F_ACCESS_PLATFORM instead of legacy VIRTIO_F_IOMMU_PLATFORM Add empty implementations for get_vq_notification() and get_vq_irq()

Eli Cohen (6):
net/vdpa: Use struct for set/get vq state
vdpa: Modify get_vq_state() to return error code vdpa/mlx5: Add hardware descriptive header file vdpa/mlx5: Add support library for mlx5 VDPA implementation vdpa/mlx5: Add shared memory registration code vdpa/mlx5: Add VDPA driver for supported mlx5 devices

Jason Wang (5):
vhost-vdpa: refine ioctl pre-processing
vhost: generialize backend features setting/getting
vhost-vdpa: support get/set backend features
vhost-vdpa: support IOTLB batching hints
vdpasim: support batch updating

Max Gurtovoy (1):
vdpa: remove hard coded virtq num

 drivers/vdpa/Kconfig                   |   19 +
 drivers/vdpa/Makefile                  |    1 +
 drivers/vdpa/ifcvf/ifcvf_base.c        |    4 +-
 drivers/vdpa/ifcvf/ifcvf_base.h        |    4 +-
 drivers/vdpa/ifcvf/ifcvf_main.c        |   13 +-
 drivers/vdpa/mlx5/Makefile             |    4 +
 drivers/vdpa/mlx5/core/mlx5_vdpa.h     |   91 ++
 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h |  168 ++
 drivers/vdpa/mlx5/core/mr.c            |  484 ++++++
 drivers/vdpa/mlx5/core/resources.c     |  284 ++++
 drivers/vdpa/mlx5/net/main.c           |   76 +
 drivers/vdpa/mlx5/net/mlx5_vnet.c      | 1965 ++++++++++++++++++++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.h      |   24 +
 drivers/vdpa/vdpa.c                    |    3 +
 drivers/vdpa/vdpa_sim/vdpa_sim.c       |   53 +-
 drivers/vhost/net.c                    |   18 +-
 drivers/vhost/vdpa.c                   |   76 +-
 drivers/vhost/vhost.c                  |   15 +
 drivers/vhost/vhost.h                  |    2 +
 include/linux/vdpa.h                   |   24 +-
 include/uapi/linux/vhost.h             |    2 +
 include/uapi/linux/vhost_types.h       |   11 +
 22 files changed, 3284 insertions(+), 57 deletions(-)
 create mode 100644 drivers/vdpa/mlx5/Makefile
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa.h
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h
 create mode 100644 drivers/vdpa/mlx5/core/mr.c
 create mode 100644 drivers/vdpa/mlx5/core/resources.c
 create mode 100644 drivers/vdpa/mlx5/net/main.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
Signature · 1 line
2.26.0
author
#16 lore raw
Quoted 10 lines · Michael S. Tsirkin
Quoted 4 lines · Eli Cohen

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

Thanks! OK so what's the plan for merging this?
Do patches at least build well enough that I can push them upstream? Or do they have to go on top of the mellanox tree?

The patches are built on your linux-next branch which I updated yesterday.

I am based on this commit:
776b7b25f10b (origin/linux-next) vhost: add an RPMsg API

On top of that I merged
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git

and after that I have Jason's patches (five patches), than one patch from Max and then my patches (seven).

It builds fine on x84_64.
I fixed some conflicts on Jason's patches.

I also tested it to verify it's working.

BTW, for some reason I did not get all the patches into my mailbox and I suspect they were not all sent. Did you get all the series 0-13?

Please let me know, and if needed I'll resend.

Quoted 7 lines
Quoted 4 lines

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301

The ones you included, right?

Right.

Quoted 103 lines
Quoted 102 lines

Jason, I had to resolve some conflicts so I would appreciate of you can verify that it is ok.

The following series of patches provide VDPA support for Mellanox devices. The supported devices are ConnectX6 DX and newer.

Currently, only a network driver is implemented; future patches will introduce a block device driver. iperf performance on a single queue is around 12 Gbps. Future patches will introduce multi queue support.

The files are organized in such a way that code that can be used by different VDPA implementations will be placed in a common are resides in drivers/vdpa/mlx5/core.

Only virtual functions are currently supported. Also, certain firmware capabilities must be set to enable the driver. Physical functions (PFs) are skipped by the driver.

To make use of the VDPA net driver, one must load mlx5_vdpa. In such case, VFs will be operated by the VDPA driver. Although one can see a regular instance of a network driver on the VF, the VDPA driver takes precedence over the NIC driver, steering-wize.

Currently, the device/interface infrastructure in mlx5_core is used to probe drivers. Future patches will introduce virtbus as a means to register devices and drivers and VDPA will be adapted to it.

The mlx5 mode of operation required to support VDPA is switchdev mode. Once can use Linux or OVS bridge to take care of layer 2 switching.

In order to provide virtio networking to a guest, an updated version of qemu is required. This version has been tested by the following quemu version:

url: https://github.com/jasowang/qemu.git
branch: vdpa
Commit ID: 6f4e59b807db

V2->V3
Fix makefile to use include path relative to the root of the kernel

V3-V4
Rebase Jason's patches on linux-next branch
Fix krobot error on mips arch
Make use of the free callback to destroy resoruces on unload Use VIRTIO_F_ACCESS_PLATFORM instead of legacy VIRTIO_F_IOMMU_PLATFORM Add empty implementations for get_vq_notification() and get_vq_irq()

Eli Cohen (6):
net/vdpa: Use struct for set/get vq state
vdpa: Modify get_vq_state() to return error code vdpa/mlx5: Add hardware descriptive header file vdpa/mlx5: Add support library for mlx5 VDPA implementation vdpa/mlx5: Add shared memory registration code vdpa/mlx5: Add VDPA driver for supported mlx5 devices

Jason Wang (5):
vhost-vdpa: refine ioctl pre-processing
vhost: generialize backend features setting/getting
vhost-vdpa: support get/set backend features
vhost-vdpa: support IOTLB batching hints
vdpasim: support batch updating

Max Gurtovoy (1):
vdpa: remove hard coded virtq num

 drivers/vdpa/Kconfig                   |   19 +
 drivers/vdpa/Makefile                  |    1 +
 drivers/vdpa/ifcvf/ifcvf_base.c        |    4 +-
 drivers/vdpa/ifcvf/ifcvf_base.h        |    4 +-
 drivers/vdpa/ifcvf/ifcvf_main.c        |   13 +-
 drivers/vdpa/mlx5/Makefile             |    4 +
 drivers/vdpa/mlx5/core/mlx5_vdpa.h     |   91 ++
 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h |  168 ++
 drivers/vdpa/mlx5/core/mr.c            |  484 ++++++
 drivers/vdpa/mlx5/core/resources.c     |  284 ++++
 drivers/vdpa/mlx5/net/main.c           |   76 +
 drivers/vdpa/mlx5/net/mlx5_vnet.c      | 1965 ++++++++++++++++++++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.h      |   24 +
 drivers/vdpa/vdpa.c                    |    3 +
 drivers/vdpa/vdpa_sim/vdpa_sim.c       |   53 +-
 drivers/vhost/net.c                    |   18 +-
 drivers/vhost/vdpa.c                   |   76 +-
 drivers/vhost/vhost.c                  |   15 +
 drivers/vhost/vhost.h                  |    2 +
 include/linux/vdpa.h                   |   24 +-
 include/uapi/linux/vhost.h             |    2 +
 include/uapi/linux/vhost_types.h       |   11 +
 22 files changed, 3284 insertions(+), 57 deletions(-)
 create mode 100644 drivers/vdpa/mlx5/Makefile
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa.h
 create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h
 create mode 100644 drivers/vdpa/mlx5/core/mr.c
 create mode 100644 drivers/vdpa/mlx5/core/resources.c
 create mode 100644 drivers/vdpa/mlx5/net/main.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
Signature · 1 line
2.26.0
#17 lore raw
Quoted 22 lines · Eli Cohen
Quoted 9 lines · Michael S. Tsirkin
Quoted 4 lines · Eli Cohen

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

Thanks! OK so what's the plan for merging this?
Do patches at least build well enough that I can push them upstream? Or do they have to go on top of the mellanox tree?

The patches are built on your linux-next branch which I updated yesterday.

I am based on this commit:
776b7b25f10b (origin/linux-next) vhost: add an RPMsg API

On top of that I merged
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git

and after that I have Jason's patches (five patches), than one patch from Max and then my patches (seven).

Quoted 7 lines

It builds fine on x84_64.
I fixed some conflicts on Jason's patches.

I also tested it to verify it's working.

BTW, for some reason I did not get all the patches into my mailbox and I suspect they were not all sent. Did you get all the series 0-13?

I can see patch 0 to patch 12 but not patch 13 (I guess 12 is all).

Thanks

Quoted 112 lines

Please let me know, and if needed I'll resend.

Quoted 5 lines
Quoted 3 lines

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301

The ones you included, right?

Right.

Quoted 102 lines
Quoted 102 lines

Jason, I had to resolve some conflicts so I would appreciate of you can verify that it is ok.

The following series of patches provide VDPA support for Mellanox devices. The supported devices are ConnectX6 DX and newer.

Currently, only a network driver is implemented; future patches will introduce a block device driver. iperf performance on a single queue is around 12 Gbps. Future patches will introduce multi queue support.

The files are organized in such a way that code that can be used by different VDPA implementations will be placed in a common are resides in drivers/vdpa/mlx5/core.

Only virtual functions are currently supported. Also, certain firmware capabilities must be set to enable the driver. Physical functions (PFs) are skipped by the driver.

To make use of the VDPA net driver, one must load mlx5_vdpa. In such case, VFs will be operated by the VDPA driver. Although one can see a regular instance of a network driver on the VF, the VDPA driver takes precedence over the NIC driver, steering-wize.

Currently, the device/interface infrastructure in mlx5_core is used to probe drivers. Future patches will introduce virtbus as a means to register devices and drivers and VDPA will be adapted to it.

The mlx5 mode of operation required to support VDPA is switchdev mode. Once can use Linux or OVS bridge to take care of layer 2 switching.

In order to provide virtio networking to a guest, an updated version of qemu is required. This version has been tested by the following quemu version:

url: https://github.com/jasowang/qemu.git
branch: vdpa
Commit ID: 6f4e59b807db

V2->V3
Fix makefile to use include path relative to the root of the kernel

V3-V4
Rebase Jason's patches on linux-next branch
Fix krobot error on mips arch
Make use of the free callback to destroy resoruces on unload Use VIRTIO_F_ACCESS_PLATFORM instead of legacy VIRTIO_F_IOMMU_PLATFORM Add empty implementations for get_vq_notification() and get_vq_irq()

Eli Cohen (6):
net/vdpa: Use struct for set/get vq state
vdpa: Modify get_vq_state() to return error code vdpa/mlx5: Add hardware descriptive header file vdpa/mlx5: Add support library for mlx5 VDPA implementation vdpa/mlx5: Add shared memory registration code vdpa/mlx5: Add VDPA driver for supported mlx5 devices

Jason Wang (5):
vhost-vdpa: refine ioctl pre-processing
vhost: generialize backend features setting/getting
vhost-vdpa: support get/set backend features
vhost-vdpa: support IOTLB batching hints
vdpasim: support batch updating

Max Gurtovoy (1):
vdpa: remove hard coded virtq num

  drivers/vdpa/Kconfig                   |   19 +
  drivers/vdpa/Makefile                  |    1 +
  drivers/vdpa/ifcvf/ifcvf_base.c        |    4 +-
  drivers/vdpa/ifcvf/ifcvf_base.h        |    4 +-
  drivers/vdpa/ifcvf/ifcvf_main.c        |   13 +-
  drivers/vdpa/mlx5/Makefile             |    4 +
  drivers/vdpa/mlx5/core/mlx5_vdpa.h     |   91 ++
  drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h |  168 ++
  drivers/vdpa/mlx5/core/mr.c            |  484 ++++++
  drivers/vdpa/mlx5/core/resources.c     |  284 ++++
  drivers/vdpa/mlx5/net/main.c           |   76 +
  drivers/vdpa/mlx5/net/mlx5_vnet.c      | 1965 ++++++++++++++++++++++++
  drivers/vdpa/mlx5/net/mlx5_vnet.h      |   24 +
  drivers/vdpa/vdpa.c                    |    3 +
  drivers/vdpa/vdpa_sim/vdpa_sim.c       |   53 +-
  drivers/vhost/net.c                    |   18 +-
  drivers/vhost/vdpa.c                   |   76 +-
  drivers/vhost/vhost.c                  |   15 +
  drivers/vhost/vhost.h                  |    2 +
  include/linux/vdpa.h                   |   24 +-
  include/uapi/linux/vhost.h             |    2 +
  include/uapi/linux/vhost_types.h       |   11 +
  22 files changed, 3284 insertions(+), 57 deletions(-)

create mode 100644 drivers/vdpa/mlx5/Makefile create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa.h create mode 100644 drivers/vdpa/mlx5/core/mlx5_vdpa_ifc.h create mode 100644 drivers/vdpa/mlx5/core/mr.c create mode 100644 drivers/vdpa/mlx5/core/resources.c create mode 100644 drivers/vdpa/mlx5/net/main.c create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.c create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h

Signature · 1 line
2.26.0

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#18 lore raw
Quoted 9 lines · Jason Wang
Quoted 3 lines

BTW, for some reason I did not get all the patches into my mailbox and I suspect they were not all sent. Did you get all the series 0-13?

I can see patch 0 to patch 12 but not patch 13 (I guess 12 is all).

Thanks

Yes, I meant 13 patches 0-12.

Thanks

#19 lore raw
In reply to Eli Cohen
Quoted 11 lines · Eli Cohen

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

They also depend Jason Wang's patches:https://lkml.org/lkml/2020/7/1/301

Jason, I had to resolve some conflicts so I would appreciate of you can verify that it is ok.

Looks good to me.

Thanks

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#20 lore raw
In reply to Eli Cohen
Quoted 8 lines · Eli Cohen

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301

So if I apply this to linux-next branch of my tree, I get:

CALL scripts/checksyscalls.sh
CALL scripts/atomic/check-atomics.sh
DESCEND objtool
CHK include/generated/compile.h
CC [M] drivers/vdpa/mlx5/net/main.o
In file included from ./include/linux/swab.h:5,

                 from ./include/uapi/linux/byteorder/little_endian.h:13,
                 from ./include/linux/byteorder/little_endian.h:5,
                 from ./arch/x86/include/uapi/asm/byteorder.h:5,
                 from ./include/asm-generic/bitops/le.h:6,
                 from ./arch/x86/include/asm/bitops.h:395,
                 from ./include/linux/bitops.h:29,
                 from ./include/linux/kernel.h:12,
                 from ./include/linux/list.h:9,
                 from ./include/linux/module.h:12,
                 from drivers/vdpa/mlx5/net/main.c:4:
drivers/vdpa/mlx5/net/main.c: In function ‘required_caps_supported’:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~

./include/uapi/linux/swab.h:115:54: note: in definition of macro ‘__swab32’
115 | #define __swab32(x) (__u32)__builtin_bswap32((__u32)(x))
| ^
./include/linux/byteorder/generic.h:95:21: note: in expansion of macro ‘__be32_to_cpu’
95 | #define be32_to_cpu __be32_to_cpu

      |                     ^~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:53:34: note: in expansion of macro ‘__mlx5_bit_off’
53 | #define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)

      |                                  ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:1: note: in expansion of macro ‘__mlx5_dw_off’
   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      | ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:56:43: note: in expansion of macro ‘__mlx5_bit_sz’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from <command-line>:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:56:70: note: in expansion of macro ‘__mlx5_bit_off’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:57:47: note: in expansion of macro ‘__mlx5_bit_sz’
57 | #define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:97:1: note: in expansion of macro ‘__mlx5_mask’
97 | __mlx5_mask(typ, fld))

      | ^~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./include/linux/swab.h:5,
                 from ./include/uapi/linux/byteorder/little_endian.h:13,
                 from ./include/linux/byteorder/little_endian.h:5,
                 from ./arch/x86/include/uapi/asm/byteorder.h:5,
                 from ./include/asm-generic/bitops/le.h:6,
                 from ./arch/x86/include/asm/bitops.h:395,
                 from ./include/linux/bitops.h:29,
                 from ./include/linux/kernel.h:12,
                 from ./include/linux/list.h:9,
                 from ./include/linux/module.h:12,
                 from drivers/vdpa/mlx5/net/main.c:4:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~

./include/uapi/linux/swab.h:115:54: note: in definition of macro ‘__swab32’
115 | #define __swab32(x) (__u32)__builtin_bswap32((__u32)(x))
| ^
./include/linux/byteorder/generic.h:95:21: note: in expansion of macro ‘__be32_to_cpu’
95 | #define be32_to_cpu __be32_to_cpu

      |                     ^~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:53:34: note: in expansion of macro ‘__mlx5_bit_off’
53 | #define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)

      |                                  ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:1: note: in expansion of macro ‘__mlx5_dw_off’
   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      | ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:56:43: note: in expansion of macro ‘__mlx5_bit_sz’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from <command-line>:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:56:70: note: in expansion of macro ‘__mlx5_bit_off’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:57:47: note: in expansion of macro ‘__mlx5_bit_sz’
57 | #define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:97:1: note: in expansion of macro ‘__mlx5_mask’
97 | __mlx5_mask(typ, fld))

      | ^~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/vdpa/mlx5/net/main.c: At top level:
drivers/vdpa/mlx5/net/main.c:62:14: error: ‘MLX5_INTERFACE_PROTOCOL_VDPA’ undeclared here (not in a function); did you mean ‘MLX5_INTERFACE_PROTOCOL_ETH’?
   62 |  .protocol = MLX5_INTERFACE_PROTOCOL_VDPA,
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |              MLX5_INTERFACE_PROTOCOL_ETH

make[3]: *** [scripts/Makefile.build:281: drivers/vdpa/mlx5/net/main.o] Error 1 make[2]: *** [scripts/Makefile.build:497: drivers/vdpa/mlx5] Error 2 make[1]: *** [scripts/Makefile.build:497: drivers/vdpa] Error 2
make: *** [Makefile:1756: drivers] Error 2

I am guessing this is because of the missing dependency, right? So what's the plan for merging this?

Signature · 6 lines
MST

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
author
#21 lore raw
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Eli Cohen

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

Quoted 6 lines
Quoted 3 lines

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301

So if I apply this to linux-next branch of my tree, I get:

Did you merge this?:
git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

Quoted 213 lines

CALL scripts/checksyscalls.sh
CALL scripts/atomic/check-atomics.sh
DESCEND objtool
CHK include/generated/compile.h
CC [M] drivers/vdpa/mlx5/net/main.o
In file included from ./include/linux/swab.h:5,

                 from ./include/uapi/linux/byteorder/little_endian.h:13,
                 from ./include/linux/byteorder/little_endian.h:5,
                 from ./arch/x86/include/uapi/asm/byteorder.h:5,
                 from ./include/asm-generic/bitops/le.h:6,
                 from ./arch/x86/include/asm/bitops.h:395,
                 from ./include/linux/bitops.h:29,
                 from ./include/linux/kernel.h:12,
                 from ./include/linux/list.h:9,
                 from ./include/linux/module.h:12,
                 from drivers/vdpa/mlx5/net/main.c:4:
drivers/vdpa/mlx5/net/main.c: In function ‘required_caps_supported’:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~

./include/uapi/linux/swab.h:115:54: note: in definition of macro ‘__swab32’
115 | #define __swab32(x) (__u32)__builtin_bswap32((__u32)(x))
| ^
./include/linux/byteorder/generic.h:95:21: note: in expansion of macro ‘__be32_to_cpu’
95 | #define be32_to_cpu __be32_to_cpu

      |                     ^~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:53:34: note: in expansion of macro ‘__mlx5_bit_off’
53 | #define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)

      |                                  ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:1: note: in expansion of macro ‘__mlx5_dw_off’
   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      | ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:56:43: note: in expansion of macro ‘__mlx5_bit_sz’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from <command-line>:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:56:70: note: in expansion of macro ‘__mlx5_bit_off’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:57:47: note: in expansion of macro ‘__mlx5_bit_sz’
57 | #define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:97:1: note: in expansion of macro ‘__mlx5_mask’
97 | __mlx5_mask(typ, fld))

      | ^~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./include/linux/swab.h:5,
                 from ./include/uapi/linux/byteorder/little_endian.h:13,
                 from ./include/linux/byteorder/little_endian.h:5,
                 from ./arch/x86/include/uapi/asm/byteorder.h:5,
                 from ./include/asm-generic/bitops/le.h:6,
                 from ./arch/x86/include/asm/bitops.h:395,
                 from ./include/linux/bitops.h:29,
                 from ./include/linux/kernel.h:12,
                 from ./include/linux/list.h:9,
                 from ./include/linux/module.h:12,
                 from drivers/vdpa/mlx5/net/main.c:4:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~

./include/uapi/linux/swab.h:115:54: note: in definition of macro ‘__swab32’
115 | #define __swab32(x) (__u32)__builtin_bswap32((__u32)(x))
| ^
./include/linux/byteorder/generic.h:95:21: note: in expansion of macro ‘__be32_to_cpu’
95 | #define be32_to_cpu __be32_to_cpu

      |                     ^~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:53:34: note: in expansion of macro ‘__mlx5_bit_off’
53 | #define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)

      |                                  ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:1: note: in expansion of macro ‘__mlx5_dw_off’
   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      | ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:56:43: note: in expansion of macro ‘__mlx5_bit_sz’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from <command-line>:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:56:70: note: in expansion of macro ‘__mlx5_bit_off’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:57:47: note: in expansion of macro ‘__mlx5_bit_sz’
57 | #define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:97:1: note: in expansion of macro ‘__mlx5_mask’
97 | __mlx5_mask(typ, fld))

      | ^~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/vdpa/mlx5/net/main.c: At top level:
drivers/vdpa/mlx5/net/main.c:62:14: error: ‘MLX5_INTERFACE_PROTOCOL_VDPA’ undeclared here (not in a function); did you mean ‘MLX5_INTERFACE_PROTOCOL_ETH’?
   62 |  .protocol = MLX5_INTERFACE_PROTOCOL_VDPA,
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |              MLX5_INTERFACE_PROTOCOL_ETH

make[3]: *** [scripts/Makefile.build:281: drivers/vdpa/mlx5/net/main.o] Error 1 make[2]: *** [scripts/Makefile.build:497: drivers/vdpa/mlx5] Error 2 make[1]: *** [scripts/Makefile.build:497: drivers/vdpa] Error 2
make: *** [Makefile:1756: drivers] Error 2

I am guessing this is because of the missing dependency, right? So what's the plan for merging this?

Signature · 1 line
MST
#22 lore raw
Quoted 9 lines · Eli Cohen
Quoted 8 lines · Michael S. Tsirkin
Quoted 7 lines · Eli Cohen

Hi Michael,
please note that this series depends on mlx5 core device driver patches in mlx5-next branch in
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git.

git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

They also depend Jason Wang's patches: https://lkml.org/lkml/2020/7/1/301

So if I apply this to linux-next branch of my tree, I get:

Did you merge this?:
git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

I can only merge this tree if no one else will. Linus does not like getting same patches through two trees.

Is this the case? Is mlx5-next going to be merged through my tree in this cycle?

Quoted 206 lines
Quoted 206 lines

CALL scripts/checksyscalls.sh
CALL scripts/atomic/check-atomics.sh
DESCEND objtool
CHK include/generated/compile.h
CC [M] drivers/vdpa/mlx5/net/main.o
In file included from ./include/linux/swab.h:5,

                 from ./include/uapi/linux/byteorder/little_endian.h:13,
                 from ./include/linux/byteorder/little_endian.h:5,
                 from ./arch/x86/include/uapi/asm/byteorder.h:5,
                 from ./include/asm-generic/bitops/le.h:6,
                 from ./arch/x86/include/asm/bitops.h:395,
                 from ./include/linux/bitops.h:29,
                 from ./include/linux/kernel.h:12,
                 from ./include/linux/list.h:9,
                 from ./include/linux/module.h:12,
                 from drivers/vdpa/mlx5/net/main.c:4:
drivers/vdpa/mlx5/net/main.c: In function ‘required_caps_supported’:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~

./include/uapi/linux/swab.h:115:54: note: in definition of macro ‘__swab32’
115 | #define __swab32(x) (__u32)__builtin_bswap32((__u32)(x))
| ^
./include/linux/byteorder/generic.h:95:21: note: in expansion of macro ‘__be32_to_cpu’
95 | #define be32_to_cpu __be32_to_cpu

      |                     ^~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:53:34: note: in expansion of macro ‘__mlx5_bit_off’
53 | #define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)

      |                                  ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:1: note: in expansion of macro ‘__mlx5_dw_off’
   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      | ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:56:43: note: in expansion of macro ‘__mlx5_bit_sz’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from <command-line>:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:56:70: note: in expansion of macro ‘__mlx5_bit_off’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘event_mode’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:57:47: note: in expansion of macro ‘__mlx5_bit_sz’
57 | #define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:97:1: note: in expansion of macro ‘__mlx5_mask’
97 | __mlx5_mask(typ, fld))

      | ^~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:24:15: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   24 |  event_mode = MLX5_CAP_DEV_VDPA_EMULATION(mdev, event_mode);
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./include/linux/swab.h:5,
                 from ./include/uapi/linux/byteorder/little_endian.h:13,
                 from ./include/linux/byteorder/little_endian.h:5,
                 from ./arch/x86/include/uapi/asm/byteorder.h:5,
                 from ./include/asm-generic/bitops/le.h:6,
                 from ./arch/x86/include/asm/bitops.h:395,
                 from ./include/linux/bitops.h:29,
                 from ./include/linux/kernel.h:12,
                 from ./include/linux/list.h:9,
                 from ./include/linux/module.h:12,
                 from drivers/vdpa/mlx5/net/main.c:4:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~

./include/uapi/linux/swab.h:115:54: note: in definition of macro ‘__swab32’
115 | #define __swab32(x) (__u32)__builtin_bswap32((__u32)(x))
| ^
./include/linux/byteorder/generic.h:95:21: note: in expansion of macro ‘__be32_to_cpu’
95 | #define be32_to_cpu __be32_to_cpu

      |                     ^~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:53:34: note: in expansion of macro ‘__mlx5_bit_off’
53 | #define __mlx5_dw_off(typ, fld) (__mlx5_bit_off(typ, fld) / 32)

      |                                  ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:1: note: in expansion of macro ‘__mlx5_dw_off’
   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      | ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:56:43: note: in expansion of macro ‘__mlx5_bit_sz’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from <command-line>:
././include/linux/compiler_types.h:129:35: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
  129 | #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
      |                                   ^~~~~~~~~~~~~~~~~~
./include/linux/stddef.h:17:32: note: in expansion of macro ‘__compiler_offsetof’
   17 | #define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
      |                                ^~~~~~~~~~~~~~~~~~~

./include/linux/mlx5/device.h:51:35: note: in expansion of macro ‘offsetof’
51 | #define __mlx5_bit_off(typ, fld) (offsetof(struct mlx5_ifc_##typ##_bits, fld))
| ^~~~~~~~
./include/linux/mlx5/device.h:56:70: note: in expansion of macro ‘__mlx5_bit_off’
56 | #define __mlx5_dw_bit_off(typ, fld) (32 - __mlx5_bit_sz(typ, fld) - (__mlx5_bit_off(typ, fld) & 0x1f))
| ^~~~~~~~~~~~~~
./include/linux/mlx5/device.h:96:30: note: in expansion of macro ‘__mlx5_dw_bit_off’

   96 | __mlx5_dw_off(typ, fld))) >> __mlx5_dw_bit_off(typ, fld)) & \
      |                              ^~~~~~~~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~

In file included from ./include/linux/mlx5/driver.h:52,
from drivers/vdpa/mlx5/net/main.c:5:
./include/linux/mlx5/device.h:50:57: error: ‘struct mlx5_ifc_device_virtio_emulation_cap_bits’ has no member named ‘eth_frame_offload_type’
50 | #define __mlx5_bit_sz(typ, fld) sizeof(__mlx5_nullp(typ)->fld)
| ^~
./include/linux/mlx5/device.h:57:47: note: in expansion of macro ‘__mlx5_bit_sz’
57 | #define __mlx5_mask(typ, fld) ((u32)((1ull << __mlx5_bit_sz(typ, fld)) - 1))
| ^~~~~~~~~~~~~
./include/linux/mlx5/device.h:97:1: note: in expansion of macro ‘__mlx5_mask’
97 | __mlx5_mask(typ, fld))

      | ^~~~~~~~~~~
./include/linux/mlx5/device.h:1355:2: note: in expansion of macro ‘MLX5_GET’
 1355 |  MLX5_GET(device_virtio_emulation_cap, \
      |  ^~~~~~~~
drivers/vdpa/mlx5/net/main.c:28:7: note: in expansion of macro ‘MLX5_CAP_DEV_VDPA_EMULATION’
   28 |  if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, eth_frame_offload_type))
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/vdpa/mlx5/net/main.c: At top level:
drivers/vdpa/mlx5/net/main.c:62:14: error: ‘MLX5_INTERFACE_PROTOCOL_VDPA’ undeclared here (not in a function); did you mean ‘MLX5_INTERFACE_PROTOCOL_ETH’?
   62 |  .protocol = MLX5_INTERFACE_PROTOCOL_VDPA,
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |              MLX5_INTERFACE_PROTOCOL_ETH

make[3]: *** [scripts/Makefile.build:281: drivers/vdpa/mlx5/net/main.o] Error 1 make[2]: *** [scripts/Makefile.build:497: drivers/vdpa/mlx5] Error 2 make[1]: *** [scripts/Makefile.build:497: drivers/vdpa] Error 2
make: *** [Makefile:1756: drivers] Error 2

Quoted 7 lines
Quoted 7 lines

I am guessing this is because of the missing dependency, right? So what's the plan for merging this?

Signature · 1 line
MST

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#23 lore raw
Quoted 11 lines · Michael S. Tsirkin
Quoted 3 lines

Did you merge this?:
git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

I can only merge this tree if no one else will. Linus does not like getting same patches through two trees.

Is this the case? Is mlx5-next going to be merged through my tree in this cycle?

Saeed Mahameed from Mellanox (located in California) usuaally sends out net patches. So he's supposed to send that to Dave Miller.

I think Saeed should answer this. Let's wait a few more hours till he wakes up.

#24 lore raw
Quoted 13 lines · Eli Cohen
Quoted 11 lines · Michael S. Tsirkin
Quoted 3 lines

Did you merge this?:
git pull git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git mlx5-next

I can only merge this tree if no one else will. Linus does not like getting same patches through two trees.

Is this the case? Is mlx5-next going to be merged through my tree in this cycle?

Quoted 5 lines

Saeed Mahameed from Mellanox (located in California) usuaally sends out net patches. So he's supposed to send that to Dave Miller.

I think Saeed should answer this. Let's wait a few more hours till he wakes up.

Alternatives:
- merge vdpa through Saeed's tree. I can ack that, we'll need to resolve any conflicts by merging the two trees and show the result to Linus so he can resolve the merge in the same way.
- extract just the necessary patches that are needed for vdpa and merge through my tree.
- if Saeed sends his pull today, it's likely it will be merged early next week. Then I can rebase and send a pull with your patches on top. A bit risky.
- do some tricks with build. E.g. disable build of your code, and enable in Saeed's tree when everything is merged together. Can be somewhat hard.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#25 lore raw
Quoted 23 lines · Michael S. Tsirkin
Quoted 14 lines · Eli Cohen
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines

Did you merge this?:
git pull
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.gi
t mlx5-next

Quoted 7 lines

I can only merge this tree if no one else will. Linus does not like
getting same patches through two trees.

Is this the case? Is mlx5-next going to be merged through my tree in this cycle?

Quoted 7 lines

Saeed Mahameed from Mellanox (located in California) usuaally sends out
net patches. So he's supposed to send that to Dave Miller.

I think Saeed should answer this. Let's wait a few more hours till he
wakes up.

Quoted 14 lines

Alternatives:
- merge vdpa through Saeed's tree. I can ack that, we'll need to resolve any conflicts by merging the two trees and show the result to Linus so he can resolve the merge in the same way.
- extract just the necessary patches that are needed for vdpa and merge through my tree.
- if Saeed sends his pull today, it's likely it will be merged early next week. Then I can rebase and send a pull with your
patches
on top. A bit risky.
- do some tricks with build. E.g. disable build of your code, and enable in Saeed's tree when everything is merged together. Can be somewhat hard.

Hi Michael,

We do this all the time with net-next and rdma,
mlx5-next is a very small branch based on a very early rc that includes mlx5 shared stuff between rdma and net-next, and now virtio as well.

we send pull requests of mlx5-next to both rdma and net-next with the respective features, exactly as we did here, and it works nicely, since we reduce the number of conflicts to 0 between different subsystems that rely on mlx5 core.

all the alternative you suggested have never been tried before :), net-next is Closed, so i can't do further submissions.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#26 lore raw
Quoted 10 lines · Saeed Mahameed
Quoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Eli Cohen
Quoted 7 lines · Michael S. Tsirkin
Quoted 4 lines

Did you merge this?:
git pull
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.gi
t mlx5-next

I can only merge this tree if no one else will. Linus does not like getting same patches through two trees.

This is not quite the case

Linus does not like multiple *copies* of the same patches. The same actual git commits can be OK.

Linus also does not like unnecessarily cross linking trees, mlx5-next is designed to be small enough and approved enough that it is not controversial.

Linus really doesn't like it when people jams stuff together in rc7 or the weeks of the merge window. He wants to see stuff be in linux-next for at least a bit. So it may be too late regardless.

Quoted 3 lines

We do this all the time with net-next and rdma,
mlx5-next is a very small branch based on a very early rc that includes mlx5 shared stuff between rdma and net-next, and now virtio as well.

Yes, going on two years now? Been working well

Jason
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#27 lore raw
Quoted 17 lines · Jason Gunthorpe
Quoted 10 lines · Saeed Mahameed
Quoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Eli Cohen
Quoted 7 lines · Michael S. Tsirkin
Quoted 4 lines

Did you merge this?:
git pull
git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.gi
t mlx5-next

I can only merge this tree if no one else will. Linus does not like getting same patches through two trees.

This is not quite the case

Linus does not like multiple *copies* of the same patches. The same actual git commits can be OK.

Quoted 7 lines

Linus also does not like unnecessarily cross linking trees, mlx5-next is designed to be small enough and approved enough that it is not controversial.

Linus really doesn't like it when people jams stuff together in rc7 or the weeks of the merge window. He wants to see stuff be in linux-next for at least a bit. So it may be too late regardless.

I'll try, let's see what happens.

Quoted 7 lines
Quoted 3 lines

We do this all the time with net-next and rdma,
mlx5-next is a very small branch based on a very early rc that includes mlx5 shared stuff between rdma and net-next, and now virtio as well.

Yes, going on two years now? Been working well

Jason

OK, I'll merge it then. Thanks!

Signature · 1 line
MST