Skip to contentSkip to conversation

vDPA support

PATCH Superseded Jason Wang 3 participants 18 messages Feb 20, 2020 – Feb 26, 2020

Subsystems (5)

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Jason Wang

author
#0 lore raw

v4 cover vDPA support

Hi all:

This is an update version of vDPA support in kernel.

vDPA device is a device that uses a datapath which complies with the virtio specifications with vendor specific control path. vDPA devices can be both physically located on the hardware or emulated by software. vDPA hardware devices are usually implemented through PCIE with the following types:

- PF (Physical Function) - A single Physical Function
- VF (Virtual Function) - Device that supports single root I/O virtualization (SR-IOV). Its Virtual Function (VF) represents a virtualized instance of the device that can be assigned to different partitions
- ADI (Assignable Device Interface) and its equivalents - With technologies such as Intel Scalable IOV, a virtual device (VDEV) composed by host OS utilizing one or more ADIs. Or its equivalent like SF (Sub function) from Mellanox.

From a driver's perspective, depends on how and where the DMA translation is done, vDPA devices are split into two types:

- Platform specific DMA translation - From the driver's perspective, the device can be used on a platform where device access to data in memory is limited and/or translated. An example is a PCIE vDPA whose DMA request was tagged via a bus (e.g PCIE) specific way. DMA translation and protection are done at PCIE bus IOMMU level.
- Device specific DMA translation - The device implements DMA isolation and protection through its own logic. An example is a vDPA device which uses on-chip IOMMU.

To hide the differences and complexity of the above types for a vDPA device/IOMMU options and in order to present a generic virtio device to the upper layer, a device agnostic framework is required.

This series introduces a software vDPA bus which abstracts the common attributes of vDPA device, vDPA bus driver and the communication method, the bus operations (vdpa_config_ops) between the vDPA device abstraction and the vDPA bus driver. This allows multiple types of drivers to be used for vDPA device like the virtio_vdpa and vhost_vdpa driver to operate on the bus and allow vDPA device could be used by either kernel virtio driver or userspace vhost drivers as:

virtio drivers vhost drivers

          |             |
    [virtio bus]   [vhost uAPI]
          |             |
   virtio device   vhost device
   virtio_vdpa drv vhost_vdpa drv
             \       /
            [vDPA bus]
                 |
            vDPA device
            hardware drv
                 |
            [hardware bus]
                 |
            vDPA hardware

virtio_vdpa driver is a transport implementation for kernel virtio drivers on top of vDPA bus operations. An alternative is to refactor virtio bus which is sub-optimal since the bus and drivers are designed to be use by kernel subsystem, a non-trivial major refactoring is needed which may impact a brunches of drivers and devices implementation inside the kernel. Using a new transport may grealy simply both the design and changes.

vhost_vdpa driver is a new type of vhost device which allows userspace vhost drivers to use vDPA devices via vhost uAPI (with minor extension). This help to minimize the changes of existed vhost drivers for using vDPA devices.

With the abstraction of vDPA bus and vDPA bus operations, the difference and complexity of the under layer hardware is hidden from upper layer. The vDPA bus drivers on top can use a unified vdpa_config_ops to control different types of vDPA device.

This series contains the bus and virtio_vdpa implementation. We are working on the vhost part and IFCVF (vDPA driver from Intel) which will be posted in future few days.

Thanks

Changes from V3:

- various Kconfig fixes (Randy)

Changes from V2:

- release idr in the release function for put_device() unwind (Jason)
- don't panic when fail to register vdpa bus (Jason)
- use unsigned int instead of int for ida (Jason)
- fix the wrong commit log in virito_vdpa patches (Jason)
- make vdpa_sim depends on RUNTIME_TESTING_MENU (Michael)
- provide a bus release function for vDPA device (Jason)
- fix the wrong unwind when creating devices for vDPA simulator (Jason)
- move vDPA simulator to a dedicated directory (Lingshan)
- cancel the work before release vDPA simulator

Changes from V1:

- drop sysfs API, leave the management interface to future development (Michael)
- introduce incremental DMA ops (dma_map/dma_unmap) (Michael)
- introduce dma_device and use it instead of parent device for doing IOMMU or DMA from bus driver (Michael, Jason, Ling Shan, Tiwei)
- accept parent device and dma device when register vdpa device
- coding style and compile fixes (Randy)
- using vdpa_xxx instead of xxx_vdpa (Jason)
- ove vDPA accessors to header and make it static inline (Jason)
- split vdp_register_device() into two helpers vdpa_init_device() and vdpa_register_device() which allows intermediate step to be done (Jason)
- warn on invalidate queue state when fail to creating virtqueue (Jason)
- make to_virtio_vdpa_device() static (Jason)
- use kmalloc/kfree instead of devres for virtio vdpa device (Jason)
- avoid using cast in vdpa bus function (Jason)
- introduce module_vdpa_driver and fix module refcnt (Jason)
- fix returning freed address in vdapsim coherent DMA addr allocation (Dan)
- various other fixes and tweaks

V3: https://lkml.org/lkml/2020/2/19/1347
V2: https://lkml.org/lkml/2020/2/9/275
V1: https://lkml.org/lkml/2020/1/16/353

Jason Wang (5):
vhost: factor out IOTLB
vringh: IOTLB support
vDPA: introduce vDPA bus
virtio: introduce a vDPA based transport
vdpasim: vDPA device simulator

 MAINTAINERS                             |   2 +
 drivers/vhost/Kconfig                   |   6 +
 drivers/vhost/Kconfig.vringh            |   1 +
 drivers/vhost/Makefile                  |   2 +
 drivers/vhost/net.c                     |   2 +-
 drivers/vhost/vhost.c                   | 221 +++-----
 drivers/vhost/vhost.h                   |  36 +-
 drivers/vhost/vhost_iotlb.c             | 171 ++++++
 drivers/vhost/vringh.c                  | 421 ++++++++++++++-
 drivers/virtio/Kconfig                  |  15 +
 drivers/virtio/Makefile                 |   2 +
 drivers/virtio/vdpa/Kconfig             |  26 +
 drivers/virtio/vdpa/Makefile            |   3 +
 drivers/virtio/vdpa/vdpa.c              | 167 ++++++
 drivers/virtio/vdpa/vdpa_sim/Makefile   |   2 +
 drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c | 660 ++++++++++++++++++++++++
 drivers/virtio/virtio_vdpa.c            | 392 ++++++++++++++
 include/linux/vdpa.h                    | 232 +++++++++
 include/linux/vhost_iotlb.h             |  45 ++
 include/linux/vringh.h                  |  36 ++
 20 files changed, 2238 insertions(+), 204 deletions(-)
 create mode 100644 drivers/vhost/vhost_iotlb.c
 create mode 100644 drivers/virtio/vdpa/Kconfig
 create mode 100644 drivers/virtio/vdpa/Makefile
 create mode 100644 drivers/virtio/vdpa/vdpa.c
 create mode 100644 drivers/virtio/vdpa/vdpa_sim/Makefile
 create mode 100644 drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c
 create mode 100644 drivers/virtio/virtio_vdpa.c
 create mode 100644 include/linux/vdpa.h
 create mode 100644 include/linux/vhost_iotlb.h
Signature · 1 line
2.19.1
author
#1 lore raw

PATCH v4 1/5 vhost: factor out IOTLB

This patch factors out IOTLB into a dedicated module in order to be reused by other modules like vringh. User may choose to enable the automatic retiring by specifying VHOST_IOTLB_FLAG_RETIRE flag to fit for the case of vhost device IOTLB implementation.

---
 MAINTAINERS                 |   1 +
 drivers/vhost/Kconfig       |   6 +
 drivers/vhost/Makefile      |   2 +
 drivers/vhost/net.c         |   2 +-
 drivers/vhost/vhost.c       | 221 +++++++++++-------------------------
 drivers/vhost/vhost.h       |  36 ++----
 drivers/vhost/vhost_iotlb.c | 171 ++++++++++++++++++++++++++++
 include/linux/vhost_iotlb.h |  45 ++++++++
 8 files changed, 303 insertions(+), 181 deletions(-)
 create mode 100644 drivers/vhost/vhost_iotlb.c
 create mode 100644 include/linux/vhost_iotlb.h
MAINTAINERS +1 -0
index c74e4ea714a5..0fb645b5a7df 100644
@@ -17768,6 +17768,7 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git
 S:	Maintained
 F:	drivers/vhost/
 F:	include/uapi/linux/vhost.h
+F:	include/linux/vhost_iotlb.h
 
 VIRTIO INPUT DRIVER
 M:	Gerd Hoffmann <[email protected]>
drivers/vhost/Kconfig +6 -0
index 3d03ccbd1adc..e76a72490563 100644
@@ -36,6 +36,7 @@ config VHOST_VSOCK
 
 config VHOST
 	tristate
+	select VHOST_IOTLB
 	---help---
 	  This option is selected by any driver which needs to access
 	  the core of vhost.
@@ -54,3 +55,8 @@ config VHOST_CROSS_ENDIAN_LEGACY
 	  adds some overhead, it is disabled by default.
 
 	  If unsure, say "N".
+
+config VHOST_IOTLB
+	tristate
+	help
+	  Generic IOTLB implementation for vhost and vringh.
drivers/vhost/Makefile +2 -0
index 6c6df24f770c..df99756fbb26 100644
@@ -11,3 +11,5 @@ vhost_vsock-y := vsock.o
 obj-$(CONFIG_VHOST_RING) += vringh.o
 
 obj-$(CONFIG_VHOST)	+= vhost.o
+
+obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o
drivers/vhost/net.c +1 -1
index e158159671fa..e4a20d7a2921 100644
@@ -1594,7 +1594,7 @@ static long vhost_net_reset_owner(struct vhost_net *n)
 	struct socket *tx_sock = NULL;
 	struct socket *rx_sock = NULL;
 	long err;
-	struct vhost_umem *umem;
+	struct vhost_iotlb *umem;
 
 	mutex_lock(&n->dev.mutex);
 	err = vhost_dev_check_owner(&n->dev);
drivers/vhost/vhost.c +69 -152

432 lines not on this page · raw message

drivers/vhost/vhost.h +8 -28
index a123fd70847e..b99c6ffb6be1 100644
@@ -12,6 +12,7 @@
 #include <linux/virtio_config.h>
 #include <linux/virtio_ring.h>
 #include <linux/atomic.h>
+#include <linux/vhost_iotlb.h>
 
 struct vhost_work;
 typedef void (*vhost_work_fn_t)(struct vhost_work *work);
@@ -52,27 +53,6 @@ struct vhost_log {
 	u64 len;
 };
 
-#define START(node) ((node)->start)
-#define LAST(node) ((node)->last)
-
-struct vhost_umem_node {
-	struct rb_node rb;
-	struct list_head link;
-	__u64 start;
-	__u64 last;
-	__u64 size;
-	__u64 userspace_addr;
-	__u32 perm;
-	__u32 flags_padding;
-	__u64 __subtree_last;
-};
-
-struct vhost_umem {
-	struct rb_root_cached umem_tree;
-	struct list_head umem_list;
-	int numem;
-};
-
 enum vhost_uaddr_type {
 	VHOST_ADDR_DESC = 0,
 	VHOST_ADDR_AVAIL = 1,
@@ -90,7 +70,7 @@ struct vhost_virtqueue {
 	struct vring_desc __user *desc;
 	struct vring_avail __user *avail;
 	struct vring_used __user *used;
-	const struct vhost_umem_node *meta_iotlb[VHOST_NUM_ADDRS];
+	const struct vhost_iotlb_map *meta_iotlb[VHOST_NUM_ADDRS];
 	struct file *kick;
 	struct eventfd_ctx *call_ctx;
 	struct eventfd_ctx *error_ctx;
@@ -128,8 +108,8 @@ struct vhost_virtqueue {
 	struct iovec *indirect;
 	struct vring_used_elem *heads;
 	/* Protected by virtqueue mutex. */
-	struct vhost_umem *umem;
-	struct vhost_umem *iotlb;
+	struct vhost_iotlb *umem;
+	struct vhost_iotlb *iotlb;
 	void *private_data;
 	u64 acked_features;
 	u64 acked_backend_features;
@@ -164,8 +144,8 @@ struct vhost_dev {
 	struct eventfd_ctx *log_ctx;
 	struct llist_head work_list;
 	struct task_struct *worker;
-	struct vhost_umem *umem;
-	struct vhost_umem *iotlb;
+	struct vhost_iotlb *umem;
+	struct vhost_iotlb *iotlb;
 	spinlock_t iotlb_lock;
 	struct list_head read_list;
 	struct list_head pending_list;
@@ -182,8 +162,8 @@ void vhost_dev_init(struct vhost_dev *, struct vhost_virtqueue **vqs,
 long vhost_dev_set_owner(struct vhost_dev *dev);
 bool vhost_dev_has_owner(struct vhost_dev *dev);
 long vhost_dev_check_owner(struct vhost_dev *);
-struct vhost_umem *vhost_dev_reset_owner_prepare(void);
-void vhost_dev_reset_owner(struct vhost_dev *, struct vhost_umem *);
+struct vhost_iotlb *vhost_dev_reset_owner_prepare(void);
+void vhost_dev_reset_owner(struct vhost_dev *dev, struct vhost_iotlb *iotlb);
 void vhost_dev_cleanup(struct vhost_dev *);
 void vhost_dev_stop(struct vhost_dev *);
 long vhost_dev_ioctl(struct vhost_dev *, unsigned int ioctl, void __user *argp);
drivers/vhost/vhost_iotlb.c new file +171 -0

171 lines not on this page · raw message

include/linux/vhost_iotlb.h new file +45 -0

45 lines not on this page · raw message

Signature · 1 line
2.19.1
author
#2 lore raw

PATCH v4 2/5 vringh: IOTLB support

In reply to Jason Wang

This patch implements the third memory accessor for vringh besides current kernel and userspace accessors. This idea is to allow vringh to do the address translation through an IOTLB which is implemented via vhost_map interval tree. Users should setup and IOVA to PA mapping in this IOTLB.

This allows us to:

- Using vringh to access virtqueues with vIOMMU
- Using vringh to implement software virtqueues for vDPA devices

---
 drivers/vhost/Kconfig.vringh |   1 +
 drivers/vhost/vringh.c       | 421 +++++++++++++++++++++++++++++++++--
 include/linux/vringh.h       |  36 +++
 3 files changed, 435 insertions(+), 23 deletions(-)
drivers/vhost/Kconfig.vringh +1 -0
index c1fe36a9b8d4..a8d4dd0cb06e 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 config VHOST_RING
 	tristate
+	select VHOST_IOTLB
 	---help---
 	  This option is selected by any driver which needs to access
 	  the host side of a virtio ring.
drivers/vhost/vringh.c +398 -23

557 lines not on this page · raw message

include/linux/vringh.h +36 -0
index d237087eb257..bd0503ca6f8f 100644
@@ -14,6 +14,8 @@
 #include <linux/virtio_byteorder.h>
 #include <linux/uio.h>
 #include <linux/slab.h>
+#include <linux/dma-direction.h>
+#include <linux/vhost_iotlb.h>
 #include <asm/barrier.h>
 
 /* virtio_ring with information needed for host access. */
@@ -39,6 +41,9 @@ struct vringh {
 	/* The vring (note: it may contain user pointers!) */
 	struct vring vring;
 
+	/* IOTLB for this vring */
+	struct vhost_iotlb *iotlb;
+
 	/* The function to call to notify the guest about added buffers */
 	void (*notify)(struct vringh *);
 };
@@ -248,4 +253,35 @@ static inline __virtio64 cpu_to_vringh64(const struct vringh *vrh, u64 val)
 {
 	return __cpu_to_virtio64(vringh_is_little_endian(vrh), val);
 }
+
+void vringh_set_iotlb(struct vringh *vrh, struct vhost_iotlb *iotlb);
+
+int vringh_init_iotlb(struct vringh *vrh, u64 features,
+		      unsigned int num, bool weak_barriers,
+		      struct vring_desc *desc,
+		      struct vring_avail *avail,
+		      struct vring_used *used);
+
+int vringh_getdesc_iotlb(struct vringh *vrh,
+			 struct vringh_kiov *riov,
+			 struct vringh_kiov *wiov,
+			 u16 *head,
+			 gfp_t gfp);
+
+ssize_t vringh_iov_pull_iotlb(struct vringh *vrh,
+			      struct vringh_kiov *riov,
+			      void *dst, size_t len);
+ssize_t vringh_iov_push_iotlb(struct vringh *vrh,
+			      struct vringh_kiov *wiov,
+			      const void *src, size_t len);
+
+void vringh_abandon_iotlb(struct vringh *vrh, unsigned int num);
+
+int vringh_complete_iotlb(struct vringh *vrh, u16 head, u32 len);
+
+bool vringh_notify_enable_iotlb(struct vringh *vrh);
+void vringh_notify_disable_iotlb(struct vringh *vrh);
+
+int vringh_need_notify_iotlb(struct vringh *vrh);
+
 #endif /* _LINUX_VRINGH_H */
Signature · 1 line
2.19.1
author
#3 lore raw

PATCH v4 3/5 vDPA: introduce vDPA bus

In reply to Jason Wang

vDPA device is a device that uses a datapath which complies with the virtio specifications with vendor specific control path. vDPA devices can be both physically located on the hardware or emulated by software. vDPA hardware devices are usually implemented through PCIE with the following types:

- PF (Physical Function) - A single Physical Function
- VF (Virtual Function) - Device that supports single root I/O virtualization (SR-IOV). Its Virtual Function (VF) represents a virtualized instance of the device that can be assigned to different partitions
- ADI (Assignable Device Interface) and its equivalents - With technologies such as Intel Scalable IOV, a virtual device (VDEV) composed by host OS utilizing one or more ADIs. Or its equivalent like SF (Sub function) from Mellanox.

From a driver's perspective, depends on how and where the DMA translation is done, vDPA devices are split into two types:

- Platform specific DMA translation - From the driver's perspective, the device can be used on a platform where device access to data in memory is limited and/or translated. An example is a PCIE vDPA whose DMA request was tagged via a bus (e.g PCIE) specific way. DMA translation and protection are done at PCIE bus IOMMU level.
- Device specific DMA translation - The device implements DMA isolation and protection through its own logic. An example is a vDPA device which uses on-chip IOMMU.

To hide the differences and complexity of the above types for a vDPA device/IOMMU options and in order to present a generic virtio device to the upper layer, a device agnostic framework is required.

This patch introduces a software vDPA bus which abstracts the common attributes of vDPA device, vDPA bus driver and the communication method (vdpa_config_ops) between the vDPA device abstraction and the vDPA bus driver. This allows multiple types of drivers to be used for vDPA device like the virtio_vdpa and vhost_vdpa driver to operate on the bus and allow vDPA device could be used by either kernel virtio driver or userspace vhost drivers as:

virtio drivers vhost drivers

          |             |
    [virtio bus]   [vhost uAPI]
          |             |
   virtio device   vhost device
   virtio_vdpa drv vhost_vdpa drv
             \       /
            [vDPA bus]
                 |
            vDPA device
            hardware drv
                 |
            [hardware bus]
                 |
            vDPA hardware

With the abstraction of vDPA bus and vDPA bus operations, the difference and complexity of the under layer hardware is hidden from upper layer. The vDPA bus drivers on top can use a unified vdpa_config_ops to control different types of vDPA device.

---
 MAINTAINERS                  |   1 +
 drivers/virtio/Kconfig       |   2 +
 drivers/virtio/Makefile      |   1 +
 drivers/virtio/vdpa/Kconfig  |   8 ++
 drivers/virtio/vdpa/Makefile |   2 +
 drivers/virtio/vdpa/vdpa.c   | 167 +++++++++++++++++++++++++
 include/linux/vdpa.h         | 232 +++++++++++++++++++++++++++++++++++
 7 files changed, 413 insertions(+)
 create mode 100644 drivers/virtio/vdpa/Kconfig
 create mode 100644 drivers/virtio/vdpa/Makefile
 create mode 100644 drivers/virtio/vdpa/vdpa.c
 create mode 100644 include/linux/vdpa.h
MAINTAINERS +1 -0
index 0fb645b5a7df..2b8d9fa38d9a 100644
@@ -17701,6 +17701,7 @@ F:	tools/virtio/
 F:	drivers/net/virtio_net.c
 F:	drivers/block/virtio_blk.c
 F:	include/linux/virtio*.h
+F:	include/linux/vdpa.h
 F:	include/uapi/linux/virtio_*.h
 F:	drivers/crypto/virtio/
 F:	mm/balloon_compaction.c
drivers/virtio/Kconfig +2 -0
index 078615cf2afc..9c4fdb64d9ac 100644
@@ -96,3 +96,5 @@ config VIRTIO_MMIO_CMDLINE_DEVICES
 	 If unsure, say 'N'.
 
 endif # VIRTIO_MENU
+
+source "drivers/virtio/vdpa/Kconfig"
drivers/virtio/Makefile +1 -0
index 3a2b5c5dcf46..fdf5eacd0d0a 100644
@@ -6,3 +6,4 @@ virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
 virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o
 obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o
 obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o
+obj-$(CONFIG_VDPA) += vdpa/
drivers/virtio/vdpa/Kconfig new file +8 -0
new file mode 100644
index 000000000000..9aac904a9515
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config VDPA
+	tristate
+	help
+	  Enable this module to support vDPA device that uses a
+	  datapath which complies with virtio specifications with
+	  vendor specific control path.
+
drivers/virtio/vdpa/Makefile new file +2 -0
new file mode 100644
index 000000000000..ee6a35e8a4fb
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_VDPA) += vdpa.o
drivers/virtio/vdpa/vdpa.c new file +167 -0

167 lines not on this page · raw message

include/linux/vdpa.h new file +232 -0

232 lines not on this page · raw message

Signature · 1 line
2.19.1
#4 lore raw

Re: [PATCH V4 3/5] vDPA: introduce vDPA bus

Quoted 41 lines · Jason Wang

vDPA device is a device that uses a datapath which complies with the virtio specifications with vendor specific control path. vDPA devices can be both physically located on the hardware or emulated by software. vDPA hardware devices are usually implemented through PCIE with the following types:

- PF (Physical Function) - A single Physical Function
- VF (Virtual Function) - Device that supports single root I/O virtualization (SR-IOV). Its Virtual Function (VF) represents a virtualized instance of the device that can be assigned to different partitions
- ADI (Assignable Device Interface) and its equivalents - With technologies such as Intel Scalable IOV, a virtual device (VDEV) composed by host OS utilizing one or more ADIs. Or its equivalent like SF (Sub function) from Mellanox.

From a driver's perspective, depends on how and where the DMA translation is done, vDPA devices are split into two types:

- Platform specific DMA translation - From the driver's perspective, the device can be used on a platform where device access to data in memory is limited and/or translated. An example is a PCIE vDPA whose DMA request was tagged via a bus (e.g PCIE) specific way. DMA translation and protection are done at PCIE bus IOMMU level.
- Device specific DMA translation - The device implements DMA isolation and protection through its own logic. An example is a vDPA device which uses on-chip IOMMU.

To hide the differences and complexity of the above types for a vDPA device/IOMMU options and in order to present a generic virtio device to the upper layer, a device agnostic framework is required.

This patch introduces a software vDPA bus which abstracts the common attributes of vDPA device, vDPA bus driver and the communication method (vdpa_config_ops) between the vDPA device abstraction and the vDPA bus driver. This allows multiple types of drivers to be used for vDPA device like the virtio_vdpa and vhost_vdpa driver to operate on the bus and allow vDPA device could be used by either kernel virtio driver or userspace vhost drivers as:

virtio drivers vhost drivers

Quoted 14 lines
          |             |
    [virtio bus]   [vhost uAPI]
          |             |
   virtio device   vhost device
   virtio_vdpa drv vhost_vdpa drv
             \       /
            [vDPA bus]
                 |
            vDPA device
            hardware drv
                 |
            [hardware bus]
                 |
            vDPA hardware

I still don't like this strange complexity, vhost should have been layered on top of the virtio device instead of adding an extra bus just for vdpa.

However, I don't see any technical problems with this patch now.

Thanks,
Jason

author
#5 lore raw

Re: [PATCH V4 3/5] vDPA: introduce vDPA bus

Quoted 56 lines · Jason Gunthorpe
Quoted 40 lines · Jason Wang

vDPA device is a device that uses a datapath which complies with the virtio specifications with vendor specific control path. vDPA devices can be both physically located on the hardware or emulated by software. vDPA hardware devices are usually implemented through PCIE with the following types:

- PF (Physical Function) - A single Physical Function
- VF (Virtual Function) - Device that supports single root I/O virtualization (SR-IOV). Its Virtual Function (VF) represents a virtualized instance of the device that can be assigned to different partitions
- ADI (Assignable Device Interface) and its equivalents - With technologies such as Intel Scalable IOV, a virtual device (VDEV) composed by host OS utilizing one or more ADIs. Or its equivalent like SF (Sub function) from Mellanox.

From a driver's perspective, depends on how and where the DMA
translation is done, vDPA devices are split into two types:

- Platform specific DMA translation - From the driver's perspective, the device can be used on a platform where device access to data in memory is limited and/or translated. An example is a PCIE vDPA whose DMA request was tagged via a bus (e.g PCIE) specific way. DMA translation and protection are done at PCIE bus IOMMU level.
- Device specific DMA translation - The device implements DMA isolation and protection through its own logic. An example is a vDPA device which uses on-chip IOMMU.

To hide the differences and complexity of the above types for a vDPA device/IOMMU options and in order to present a generic virtio device to the upper layer, a device agnostic framework is required.

This patch introduces a software vDPA bus which abstracts the common attributes of vDPA device, vDPA bus driver and the communication method (vdpa_config_ops) between the vDPA device abstraction and the vDPA bus driver. This allows multiple types of drivers to be used for vDPA device like the virtio_vdpa and vhost_vdpa driver to operate on the bus and allow vDPA device could be used by either kernel virtio driver or userspace vhost drivers as:

Quoted 15 lines

virtio drivers vhost drivers
| |
[virtio bus] [vhost uAPI]
| |
virtio device vhost device
virtio_vdpa drv vhost_vdpa drv
\ /
[vDPA bus]
|
vDPA device
hardware drv
|
[hardware bus]
|
vDPA hardware

Quoted 3 lines

I still don't like this strange complexity, vhost should have been layered on top of the virtio device instead of adding an extra bus just for vdpa.

We've considered such method and I think why we choose a bus is:

- vDPA device was originally named as "vhost Datapath Acceleration" which means the datapath complies virtio specification but not control path. This means the device should behave like vhost. And in order to support vhost, vDPA device requires more function than virtio. E.g the ability to query the device state (virtqueue indices, counters etc) and track dirty pages. This mean even a pure virtio hardware may not work for vhost. That's why a multi inheritance is used for a new type of vDPA device.

- As we've already discussed, virtio bus is designed for kernel driver and a brunches of devices, drivers or even buses have been implemented around that. It requires a major refactoring not only with the virtio bus but also with the drivers and devices to make it behave more like a vhost. Abstract vDPA as a kind of transport for virtio greatly simplify the work and have almost zero impact on the exist virtio core. VOP (vop_bus) use similar design.

Quoted 2 lines

However, I don't see any technical problems with this patch now.

Thanks, your review is greatly appreciated.

Quoted 4 lines

Thanks,
Jason

#6 lore raw

RE: [PATCH V4 3/5] vDPA: introduce vDPA bus

In reply to Jason Wang

Is there a plan to add an API in vDPA_config_ops for getting the notification area from the VDPA device (something similar to get_notify_area in the VDPA DPDK case)? This will make the notifications from the guest (vhost_vdpa use case) to the VDPA device more efficient - at least for virtio 1.0+ drivers in the VM.

I believe this would require enhancement to the vhost ioctl (something similar to the VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG).

Regards,
Harpreet

author
#7 lore raw

Re: [PATCH V4 3/5] vDPA: introduce vDPA bus

Quoted 3 lines · Harpreet Singh Anand

Is there a plan to add an API in vDPA_config_ops for getting the notification area from the VDPA device (something similar to get_notify_area in the VDPA DPDK case)? This will make the notifications from the guest (vhost_vdpa use case) to the VDPA device more efficient - at least for virtio 1.0+ drivers in the VM.

I believe this would require enhancement to the vhost ioctl (something similar to the VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG).

Yes, we plan to add that on top. Basically, here's what we plan to do: (sorted by urgency)

1) direct doorbell mapping as you asked here
2) direct interrupt injection (when platform support, e.g through posted interrupt)
3) control virtqueue support

Thanks

Quoted 5 lines

Regards,
Harpreet

author
#8 lore raw

PATCH v4 4/5 virtio: introduce a vDPA based transport

In reply to Jason Wang

This patch introduces a vDPA transport for virtio. This is used to use kernel virtio driver to drive the vDPA device that is capable of populating virtqueue directly.

A new virtio-vdpa driver will be registered to the vDPA bus, when a new virtio-vdpa device is probed, it will register the device with vdpa based config ops. This means it is a software transport between vDPA driver and vDPA device. The transport was implemented through bus_ops of vDPA parent.

---
 drivers/virtio/Kconfig       |  13 ++
 drivers/virtio/Makefile      |   1 +
 drivers/virtio/virtio_vdpa.c | 392 +++++++++++++++++++++++++++++++++++
 3 files changed, 406 insertions(+)
 create mode 100644 drivers/virtio/virtio_vdpa.c
drivers/virtio/Kconfig +13 -0
index 9c4fdb64d9ac..99e424570644 100644
@@ -43,6 +43,19 @@ config VIRTIO_PCI_LEGACY
 
 	  If unsure, say Y.
 
+config VIRTIO_VDPA
+	tristate "vDPA driver for virtio devices"
+	select VDPA
+	select VIRTIO
+	help
+	  This driver provides support for virtio based paravirtual
+	  device driver over vDPA bus. For this to be useful, you need
+	  an appropriate vDPA device implementation that operates on a
+	  physical device to allow the datapath of virtio to be
+	  offloaded to hardware.
+
+	  If unsure, say M.
+
 config VIRTIO_PMEM
 	tristate "Support for virtio pmem driver"
 	depends on VIRTIO
drivers/virtio/Makefile +1 -0
index fdf5eacd0d0a..3407ac03fe60 100644
@@ -6,4 +6,5 @@ virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
 virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o
 obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o
 obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o
+obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o
 obj-$(CONFIG_VDPA) += vdpa/
drivers/virtio/virtio_vdpa.c new file +392 -0

392 lines not on this page · raw message

Signature · 1 line
2.19.1
#9 lore raw

Re: [PATCH V4 4/5] virtio: introduce a vDPA based transport

Quoted 15 lines · Jason Wang
drivers/virtio/virtio_vdpa.c new file +15 -0
+static int virtio_vdpa_probe(struct vdpa_device *vdpa)
+{
+	const struct vdpa_config_ops *ops = vdpa->config;
+	struct virtio_vdpa_device *vd_dev;
+	int ret = -EINVAL;
+
+	vd_dev = kzalloc(sizeof(*vd_dev), GFP_KERNEL);
+	if (!vd_dev)
+		return -ENOMEM;
+
+	vd_dev->vdev.dev.parent = vdpa_get_dma_dev(vdpa);
+	vd_dev->vdev.dev.release = virtio_vdpa_release_dev;
+	vd_dev->vdev.config = &virtio_vdpa_config_ops;
+	vd_dev->vdpa = vdpa;
+	INIT_LIST_HEAD(&vd_dev->virtqueues);

drivers/virtio/virtio_vdpa.c line 361

Quoted 10 lines
drivers/virtio/virtio_vdpa.c new file +10 -0
+	spin_lock_init(&vd_dev->lock);
+
+	vd_dev->vdev.id.device = ops->get_device_id(vdpa);
+	if (vd_dev->vdev.id.device == 0)
+		goto err;
+
+	vd_dev->vdev.id.vendor = ops->get_vendor_id(vdpa);
+	ret = register_virtio_device(&vd_dev->vdev);
+	if (ret)
+		goto err;

This error unwind is wrong. register_virtio_device() does device_initialize() as it's first action. After that point error unwind must be done with put_device() - particularly calling kfree(vd_dev) after doing dev_set_name() leaks memory.

Looks like about half of the register_virtio_device() users did this right, the others not. Perhaps you should fix them too...

Jason

author
#10 lore raw

Re: [PATCH V4 4/5] virtio: introduce a vDPA based transport

Quoted 22 lines · Jason Gunthorpe
Quoted 21 lines · Jason Wang
drivers/virtio/virtio_vdpa.c new file +21 -0
+static int virtio_vdpa_probe(struct vdpa_device *vdpa)
+{
+	const struct vdpa_config_ops *ops = vdpa->config;
+	struct virtio_vdpa_device *vd_dev;
+	int ret = -EINVAL;
+
+	vd_dev = kzalloc(sizeof(*vd_dev), GFP_KERNEL);
+	if (!vd_dev)
+		return -ENOMEM;
+
+	vd_dev->vdev.dev.parent = vdpa_get_dma_dev(vdpa);
+	vd_dev->vdev.dev.release = virtio_vdpa_release_dev;
+	vd_dev->vdev.config = &virtio_vdpa_config_ops;
+	vd_dev->vdpa = vdpa;
+	INIT_LIST_HEAD(&vd_dev->virtqueues);
+	spin_lock_init(&vd_dev->lock);
+
+	vd_dev->vdev.id.device = ops->get_device_id(vdpa);
+	if (vd_dev->vdev.id.device == 0)
+		goto err;
+
Quoted 8 lines

drivers/virtio/virtio_vdpa.c line 361

Quoted 4 lines
drivers/virtio/virtio_vdpa.c new file +4 -0
+	vd_dev->vdev.id.vendor = ops->get_vendor_id(vdpa);
+	ret = register_virtio_device(&vd_dev->vdev);
+	if (ret)
+		goto err;

This error unwind is wrong. register_virtio_device() does device_initialize() as it's first action. After that point error unwind must be done with put_device() - particularly calling kfree(vd_dev) after doing dev_set_name() leaks memory.

Exactly.

Quoted 5 lines

Looks like about half of the register_virtio_device() users did this right, the others not. Perhaps you should fix them too...

Jason

Will do.

Thanks

Quoted 1 line
author
#11 lore raw

PATCH v4 5/5 vdpasim: vDPA device simulator

In reply to Jason Wang

This patch implements a software vDPA networking device. The datapath is implemented through vringh and workqueue. The device has an on-chip IOMMU which translates IOVA to PA. For kernel virtio drivers, vDPA simulator driver provides dma_ops. For vhost driers, set_map() methods of vdpa_config_ops is implemented to accept mappings from vhost.

Currently, vDPA device simulator will loopback TX traffic to RX. So the main use case for the device is vDPA feature testing, prototyping and development.

Note, there's no management API implemented, a vDPA device will be registered once the module is probed. We need to handle this in the future development.

---
 drivers/virtio/vdpa/Kconfig             |  18 +
 drivers/virtio/vdpa/Makefile            |   1 +
 drivers/virtio/vdpa/vdpa_sim/Makefile   |   2 +
 drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c | 660 ++++++++++++++++++++++++
 4 files changed, 681 insertions(+)
 create mode 100644 drivers/virtio/vdpa/vdpa_sim/Makefile
 create mode 100644 drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c
drivers/virtio/vdpa/Kconfig +18 -0
index 9aac904a9515..9e7dc95e0c89 100644
@@ -6,3 +6,21 @@ config VDPA
 	  datapath which complies with virtio specifications with
 	  vendor specific control path.
 
+menuconfig VDPA_MENU
+	bool "VDPA drivers"
+	default n
+
+if VDPA_MENU
+
+config VDPA_SIM
+	tristate "vDPA device simulator"
+	select VDPA
+	depends on RUNTIME_TESTING_MENU
+	default n
+	help
+	  vDPA networking device simulator which loop TX traffic back
+	  to RX. This device is used for testing, prototyping and
+	  development of vDPA.
+
+endif # VDPA_MENU
+
drivers/virtio/vdpa/Makefile +1 -0
index ee6a35e8a4fb..3814af8e097b 100644
@@ -1,2 +1,3 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_VDPA) += vdpa.o
+obj-$(CONFIG_VDPA_SIM) += vdpa_sim/
drivers/virtio/vdpa/vdpa_sim/Makefile new file +2 -0
new file mode 100644
index 000000000000..b40278f65e04
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_VDPA_SIM) += vdpa_sim.o
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +660 -0

660 lines not on this page · raw message

Signature · 1 line
2.19.1
#12 lore raw

Re: [PATCH V4 5/5] vdpasim: vDPA device simulator

Quoted 44 lines · Jason Wang
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +44 -0
+static void vdpasim_device_release(struct device *dev)
+{
+	struct vdpasim *vdpasim = dev_to_sim(dev);
+
+	cancel_work_sync(&vdpasim->work);
+	kfree(vdpasim->buffer);
+	vhost_iotlb_free(vdpasim->iommu);
+	kfree(vdpasim);
+}
+
+static struct vdpasim *vdpasim_create(void)
+{
+	struct virtio_net_config *config;
+	struct vhost_iotlb *iommu;
+	struct vdpasim *vdpasim;
+	struct device *dev;
+	void *buffer;
+	int ret = -ENOMEM;
+
+	iommu = vhost_iotlb_alloc(2048, 0);
+	if (!iommu)
+		goto err;
+
+	buffer = kmalloc(PAGE_SIZE, GFP_KERNEL);
+	if (!buffer)
+		goto err_buffer;
+
+	vdpasim = kzalloc(sizeof(*vdpasim), GFP_KERNEL);
+	if (!vdpasim)
+		goto err_alloc;
+
+	vdpasim->buffer = buffer;
+	vdpasim->iommu = iommu;
+
+	config = &vdpasim->config;
+	config->mtu = 1500;
+	config->status = VIRTIO_NET_S_LINK_UP;
+	eth_random_addr(config->mac);
+
+	INIT_WORK(&vdpasim->work, vdpasim_work);
+	spin_lock_init(&vdpasim->lock);
+
+	vringh_set_iotlb(&vdpasim->vqs[0].vring, vdpasim->iommu);
+	vringh_set_iotlb(&vdpasim->vqs[1].vring, vdpasim->iommu);

drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c line 360

Quoted 10 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +10 -0
+
+	dev = &vdpasim->dev;
+	dev->release = vdpasim_device_release;
+	dev->coherent_dma_mask = DMA_BIT_MASK(64);
+	set_dma_ops(dev, &vdpasim_dma_ops);
+	dev_set_name(dev, "%s", VDPASIM_NAME);
+
+	ret = device_register(&vdpasim->dev);
+	if (ret)
+		goto err_init;

It is a bit weird to be creating this dummy parent, couldn't this be done by just passing a NULL parent to vdpa_alloc_device, doing set_dma_ops() on the vdpasim->vdpa->dev and setting dma_device to vdpasim->vdpa->dev ?

Quoted 8 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +7 -0
+	vdpasim->vdpa = vdpa_alloc_device(dev, dev, &vdpasim_net_config_ops);
+	if (ret)
+		goto err_vdpa;
 
+	ret = vdpa_register_device(vdpasim->vdpa);
+	if (ret)
+		goto err_register;
+

drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c line 379

Quoted 10 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +10 -0
+	return vdpasim;
+
+err_register:
+	put_device(&vdpasim->vdpa->dev);
+err_vdpa:
+	device_del(&vdpasim->dev);
+	goto err;
+err_init:
+	put_device(&vdpasim->dev);
+	goto err;

If you do the vdmasim alloc first, and immediately do device_initialize() then all the failure paths can do put_device instead of having this ugly goto unwind split. Just check for vdpasim->iommu == NULL during release.

Quoted 13 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +13 -0
+static int __init vdpasim_dev_init(void)
+{
+	vdpasim_dev = vdpasim_create();
+
+	if (!IS_ERR(vdpasim_dev))
+		return 0;
+
+	return PTR_ERR(vdpasim_dev);
+}
+
+static int vdpasim_device_remove_cb(struct device *dev, void *data)
+{
+	struct vdpa_device *vdpa = dev_to_vdpa(dev);

drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c line 650

Quoted 10 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +10 -0
+
+	vdpa_unregister_device(vdpa);
+
+	return 0;
+}
+
+static void __exit vdpasim_dev_exit(void)
+{
+	device_for_each_child(&vdpasim_dev->dev, NULL,
+			      vdpasim_device_remove_cb);

Why the loop? There is only one device, and it is in the global varaible vdmasim_dev ?

Jason

author
#13 lore raw

Re: [PATCH V4 5/5] vdpasim: vDPA device simulator

Quoted 51 lines · Jason Gunthorpe
Quoted 50 lines · Jason Wang
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +50 -0
+static void vdpasim_device_release(struct device *dev)
+{
+	struct vdpasim *vdpasim = dev_to_sim(dev);
+
+	cancel_work_sync(&vdpasim->work);
+	kfree(vdpasim->buffer);
+	vhost_iotlb_free(vdpasim->iommu);
+	kfree(vdpasim);
+}
+
+static struct vdpasim *vdpasim_create(void)
+{
+	struct virtio_net_config *config;
+	struct vhost_iotlb *iommu;
+	struct vdpasim *vdpasim;
+	struct device *dev;
+	void *buffer;
+	int ret = -ENOMEM;
+
+	iommu = vhost_iotlb_alloc(2048, 0);
+	if (!iommu)
+		goto err;
+
+	buffer = kmalloc(PAGE_SIZE, GFP_KERNEL);
+	if (!buffer)
+		goto err_buffer;
+
+	vdpasim = kzalloc(sizeof(*vdpasim), GFP_KERNEL);
+	if (!vdpasim)
+		goto err_alloc;
+
+	vdpasim->buffer = buffer;
+	vdpasim->iommu = iommu;
+
+	config = &vdpasim->config;
+	config->mtu = 1500;
+	config->status = VIRTIO_NET_S_LINK_UP;
+	eth_random_addr(config->mac);
+
+	INIT_WORK(&vdpasim->work, vdpasim_work);
+	spin_lock_init(&vdpasim->lock);
+
+	vringh_set_iotlb(&vdpasim->vqs[0].vring, vdpasim->iommu);
+	vringh_set_iotlb(&vdpasim->vqs[1].vring, vdpasim->iommu);
+
+	dev = &vdpasim->dev;
+	dev->release = vdpasim_device_release;
+	dev->coherent_dma_mask = DMA_BIT_MASK(64);
+	set_dma_ops(dev, &vdpasim_dma_ops);
+	dev_set_name(dev, "%s", VDPASIM_NAME);
Quoted 8 lines

drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c line 360

Quoted 4 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +4 -0
+
+	ret = device_register(&vdpasim->dev);
+	if (ret)
+		goto err_init;

It is a bit weird to be creating this dummy parent, couldn't this be done by just passing a NULL parent to vdpa_alloc_device, doing set_dma_ops() on the vdpasim->vdpa->dev and setting dma_device to vdpasim->vdpa->dev ?

I think it works.

Quoted 13 lines
Quoted 13 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +13 -0
+	vdpasim->vdpa = vdpa_alloc_device(dev, dev, &vdpasim_net_config_ops);
+	if (ret)
+		goto err_vdpa;
+	ret = vdpa_register_device(vdpasim->vdpa);
+	if (ret)
+		goto err_register;
+
+	return vdpasim;
+
+err_register:
+	put_device(&vdpasim->vdpa->dev);
+err_vdpa:
+	device_del(&vdpasim->dev);
Quoted 8 lines

drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c line 379

Quoted 4 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +4 -0
+	goto err;
+err_init:
+	put_device(&vdpasim->dev);
+	goto err;

If you do the vdmasim alloc first, and immediately do device_initialize() then all the failure paths can do put_device instead of having this ugly goto unwind split. Just check for vdpasim->iommu == NULL during release.

Yes, that looks simpler.

Quoted 18 lines
Quoted 18 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +17 -0
+static int __init vdpasim_dev_init(void)
+{
+	vdpasim_dev = vdpasim_create();
+
+	if (!IS_ERR(vdpasim_dev))
+		return 0;
+
+	return PTR_ERR(vdpasim_dev);
+}
+
+static int vdpasim_device_remove_cb(struct device *dev, void *data)
+{
+	struct vdpa_device *vdpa = dev_to_vdpa(dev);
+
+	vdpa_unregister_device(vdpa);
+
+	return 0;
Quoted 8 lines

drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c line 650

Quoted 6 lines
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +6 -0
+}
+
+static void __exit vdpasim_dev_exit(void)
+{
+	device_for_each_child(&vdpasim_dev->dev, NULL,
+			      vdpasim_device_remove_cb);

Why the loop? There is only one device, and it is in the global varaible vdmasim_dev ?

Not necessary but doesn't harm, will remove this.

Thanks

Quoted 3 lines

Jason

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#14 lore raw

Re: [PATCH V4 5/5] vdpasim: vDPA device simulator

Quoted 63 lines · Jason Wang
Quoted 51 lines · Jason Gunthorpe
Quoted 50 lines · Jason Wang
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +50 -0
+static void vdpasim_device_release(struct device *dev)
+{
+    struct vdpasim *vdpasim = dev_to_sim(dev);
+
+    cancel_work_sync(&vdpasim->work);
+    kfree(vdpasim->buffer);
+    vhost_iotlb_free(vdpasim->iommu);
+    kfree(vdpasim);
+}
+
+static struct vdpasim *vdpasim_create(void)
+{
+    struct virtio_net_config *config;
+    struct vhost_iotlb *iommu;
+    struct vdpasim *vdpasim;
+    struct device *dev;
+    void *buffer;
+    int ret = -ENOMEM;
+
+    iommu = vhost_iotlb_alloc(2048, 0);
+    if (!iommu)
+        goto err;
+
+    buffer = kmalloc(PAGE_SIZE, GFP_KERNEL);
+    if (!buffer)
+        goto err_buffer;
+
+    vdpasim = kzalloc(sizeof(*vdpasim), GFP_KERNEL);
+    if (!vdpasim)
+        goto err_alloc;
+
+    vdpasim->buffer = buffer;
+    vdpasim->iommu = iommu;
+
+    config = &vdpasim->config;
+    config->mtu = 1500;
+    config->status = VIRTIO_NET_S_LINK_UP;
+    eth_random_addr(config->mac);
+
+    INIT_WORK(&vdpasim->work, vdpasim_work);
+    spin_lock_init(&vdpasim->lock);
+
+    vringh_set_iotlb(&vdpasim->vqs[0].vring, vdpasim->iommu);
+    vringh_set_iotlb(&vdpasim->vqs[1].vring, vdpasim->iommu);
+
+    dev = &vdpasim->dev;
+    dev->release = vdpasim_device_release;
+    dev->coherent_dma_mask = DMA_BIT_MASK(64);
+    set_dma_ops(dev, &vdpasim_dma_ops);
+    dev_set_name(dev, "%s", VDPASIM_NAME);
Quoted 8 lines
Quoted 4 lines

+
+    ret = device_register(&vdpasim->dev);
+    if (ret)
+        goto err_init;

It is a bit weird to be creating this dummy parent, couldn't this be done by just passing a NULL parent to vdpa_alloc_device, doing set_dma_ops() on the vdpasim->vdpa->dev and setting dma_device to vdpasim->vdpa->dev ?

Quoted 1 line

I think it works.

Rethink about this, since most hardware vDPA driver will have a parent and will use it to find the parent structure e.g

dev_get_drvdata(vdpa->dev->parent)

So I keep this dummy parent in V5.

Thanks

#15 lore raw

Re: [PATCH V4 5/5] vdpasim: vDPA device simulator

Quoted 6 lines · Jason Wang
Quoted 6 lines
Quoted 4 lines

It is a bit weird to be creating this dummy parent, couldn't this be done by just passing a NULL parent to vdpa_alloc_device, doing set_dma_ops() on the vdpasim->vdpa->dev and setting dma_device to vdpasim->vdpa->dev ?

Quoted 7 lines
Quoted 1 line

I think it works.

Rethink about this, since most hardware vDPA driver will have a parent and will use it to find the parent structure e.g

dev_get_drvdata(vdpa->dev->parent)

Oh that seems like a sketchy pattern

Quoted 1 line

So I keep this dummy parent in V5.

The dummy parent is visible in sysfs, I think you should remove it so the syfs for this virtual device makes sense.

Jason

#16 lore raw

RE: [PATCH V4 5/5] vdpasim: vDPA device simulator

In reply to Jason Wang

+ ret = device_register(&vdpasim->dev);
+ if (ret)
+ goto err_init;
+
+ vdpasim->vdpa = vdpa_alloc_device(dev, dev, &vdpasim_net_config_ops);
+ if (ret)
+ goto err_vdpa;

[HSA] Incorrect checking of the return value of vdpa_alloc_device.

author
#17 lore raw

Re: [PATCH V4 5/5] vdpasim: vDPA device simulator

Quoted 9 lines · Harpreet Singh Anand
drivers/virtio/vdpa/vdpa_sim/vdpa_sim.c new file +7 -0
+       ret = device_register(&vdpasim->dev);
+       if (ret)
+               goto err_init;
+
+       vdpasim->vdpa = vdpa_alloc_device(dev, dev, &vdpasim_net_config_ops);
+       if (ret)
+               goto err_vdpa;

[HSA] Incorrect checking of the return value of vdpa_alloc_device.

Yes, fixed.

Thanks