Skip to contentSkip to conversation

add generic vDPA device support

PATCH RESEND Superseded Longpeng(Mike) via 2 participants 11 messages Nov 5, 2022 – Nov 6, 2022 Arm/Amlogic Meson SoC support

v7 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v7 cover add generic vDPA device support

From: Longpeng <[email protected]>

Hi guys,

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

We can use the generic vDPA device as follow:
-device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-X Or
-M microvm -m 512m -smp 2 -kernel ... -initrd ... -device \
vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-x

Changes v6 -> v7:
(v6: https://mail.gnu.org/archive/html/qemu-devel/2022-05/msg02821.html)
- rebase. [Jason]
- add documentation . [Stefan]

Changes v5 -> v6:
Patch 2:
- Turn to the original approach in the RFC to initialize the virtio_pci_id_info array. [Michael]
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Fix logical error of exception handler around the post_init. [Stefano]
- Fix some coding style warnings. [Stefano]
Patch 4:
- Fix some coding style warnings. [Stefano]

Changes v4 -> v5:
Patch 3:
- remove vhostfd [Jason]
- support virtio-mmio [Jason]

Changes v3 -> v4:
v3: https://www.mail-archive.com/[email protected]/msg877015.html
- reorganize the series [Stefano]
- fix some typos [Stefano]
- fix logical error in vhost_vdpa_device_realize [Stefano]

Changes v2 -> v3
Patch 4 & 5:
- only call vdpa ioctls in vdpa-dev.c [Stefano, Longpeng]
- s/VQS_NUM/VQS_COUNT [Stefano]
- check both vdpa_dev_fd and vdpa_dev [Stefano]
Patch 6:
- move all steps into vhost_vdpa_device_unrealize. [Stefano]

Changes RFC -> v2
Patch 1:
- rename 'pdev_id' to 'trans_devid' [Michael]
- only use transitional device id for the devices listed in the spec [Michael]
- use macros to make the id_info table clearer [Longpeng]
- add some modern devices in the id_info table [Longpeng]
Patch 2:
- remove the GET_VECTORS_NUM command [Jason]
Patch 4:
- expose vdpa_dev_fd as a QOM preperty [Stefan]
- introduce vhost_vdpa_device_get_u32 as a common function to make the code clearer [Stefan]
- fix the misleading description of 'dc->desc' [Stefano]
Patch 5:
- check returned number of virtqueues [Stefan]
Patch 6:
- init s->num_queues [Stefano]
- free s->dev.vqs [Stefano]

Longpeng (Mike) (4):
virtio: get class_id and pci device id by the virtio id
vdpa: add vdpa-dev support
vdpa: add vdpa-dev-pci support
docs: Add generic vhost-vdpa device documentation

 docs/system/devices/vhost-vdpa-device.rst |  43 +++
 hw/virtio/Kconfig                         |   5 +
 hw/virtio/meson.build                     |   2 +
 hw/virtio/vdpa-dev-pci.c                  | 102 ++++++
 hw/virtio/vdpa-dev.c                      | 377 ++++++++++++++++++++++
 hw/virtio/virtio-pci.c                    |  88 +++++
 include/hw/virtio/vdpa-dev.h              |  43 +++
 include/hw/virtio/virtio-pci.h            |   5 +
 8 files changed, 665 insertions(+)
 create mode 100644 docs/system/devices/vhost-vdpa-device.rst
 create mode 100644 hw/virtio/vdpa-dev-pci.c
 create mode 100644 hw/virtio/vdpa-dev.c
 create mode 100644 include/hw/virtio/vdpa-dev.h
Signature · 1 line
2.23.0
#1 lore raw

PATCH RESEND v7 1/4 virtio: get class_id and pci device id by the virtio id

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the device specified by the "Virtio Device ID".

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c         | 88 ++++++++++++++++++++++++++++++++++
 include/hw/virtio/virtio-pci.h |  5 ++
 2 files changed, 93 insertions(+)
hw/virtio/virtio-pci.c +88 -0
index 34db51e241..3469b88d43 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,90 @@ static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f)
     return 0;
 }
 
+typedef struct VirtIOPCIIDInfo {
+    /* virtio id */
+    uint16_t vdev_id;
+    /* pci device id for the transitional device */
+    uint16_t trans_devid;
+    uint16_t class_id;
+} VirtIOPCIIDInfo;
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {
+        .vdev_id = VIRTIO_ID_CRYPTO,
+        .class_id = PCI_CLASS_OTHERS,
+    }, {
+        .vdev_id = VIRTIO_ID_FS,
+        .class_id = PCI_CLASS_STORAGE_OTHER,
+    }, {
+        .vdev_id = VIRTIO_ID_NET,
+        .trans_devid = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    }, {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .trans_devid = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    }, {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .trans_devid = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    }, {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .trans_devid = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI
+    }, {
+        .vdev_id = VIRTIO_ID_9P,
+        .trans_devid = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    }, {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .trans_devid = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    }, {
+        .vdev_id = VIRTIO_ID_RNG,
+        .trans_devid = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+
+static const VirtIOPCIIDInfo *virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    const VirtIOPCIIDInfo *info = NULL;
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = &virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    if (!info) {
+        /* The device id is invalid or not added to the id_info yet. */
+        error_report("Invalid virtio device(id %u)", vdev_id);
+        abort();
+    }
+
+    return info;
+}
+
+/*
+ * Get the Transitional Device ID for the specific device, return
+ * zero if the device is non-transitional.
+ */
+uint16_t virtio_pci_get_trans_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id)->trans_devid;
+}
+
+/*
+ * Get the Class ID for the specific device.
+ */
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id)->class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1683,6 +1768,9 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp)
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID, virtio_bus_get_vdev_id(bus));
+        if (proxy->trans_devid) {
+            pci_config_set_device_id(config, proxy->trans_devid);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
include/hw/virtio/virtio-pci.h +5 -0
index 2446dcd9ae..d95b1a13a5 100644
@@ -146,6 +146,8 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    /* Transitional device id */
+    uint16_t trans_devid;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -179,6 +181,9 @@ static inline void virtio_pci_disable_modern(VirtIOPCIProxy *proxy)
     proxy->disable_modern = true;
 }
 
+uint16_t virtio_pci_get_trans_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 /*
  * virtio-input-pci: This extends VirtioPCIProxy.
  */
Signature · 1 line
2.23.0
#2 lore raw

PATCH RESEND v7 2/4 vdpa: add vdpa-dev support

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Supports vdpa-dev, we can use the deivce directly:

-M microvm -m 512m -smp 2 -kernel ... -initrd ... -device \
vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-x

---
 hw/virtio/Kconfig            |   5 +
 hw/virtio/meson.build        |   1 +
 hw/virtio/vdpa-dev.c         | 377 +++++++++++++++++++++++++++++++++++
 include/hw/virtio/vdpa-dev.h |  43 ++++
 4 files changed, 426 insertions(+)
 create mode 100644 hw/virtio/vdpa-dev.c
 create mode 100644 include/hw/virtio/vdpa-dev.h
hw/virtio/Kconfig +5 -0
index cbfd8c7173..89e9e426d8 100644
@@ -85,3 +85,8 @@ config VHOST_USER_GPIO
     bool
     default y
     depends on VIRTIO && VHOST_USER
+
+config VHOST_VDPA_DEV
+    bool
+    default y
+    depends on VIRTIO && VHOST_VDPA && LINUX
hw/virtio/meson.build +1 -0
index dfed1e7af5..54d6d29af7 100644
@@ -31,6 +31,7 @@ virtio_ss.add(when: 'CONFIG_VHOST_USER_I2C', if_true: files('vhost-user-i2c.c'))
 virtio_ss.add(when: 'CONFIG_VHOST_USER_RNG', if_true: files('vhost-user-rng.c'))
 virtio_ss.add(when: 'CONFIG_VHOST_USER_GPIO', if_true: files('vhost-user-gpio.c'))
 virtio_ss.add(when: ['CONFIG_VIRTIO_PCI', 'CONFIG_VHOST_USER_GPIO'], if_true: files('vhost-user-gpio-pci.c'))
+virtio_ss.add(when: 'CONFIG_VHOST_VDPA_DEV', if_true: files('vdpa-dev.c'))
 
 virtio_pci_ss = ss.source_set()
 virtio_pci_ss.add(when: 'CONFIG_VHOST_VSOCK', if_true: files('vhost-vsock-pci.c'))
hw/virtio/vdpa-dev.c new file +377 -0

377 lines not on this page · raw message

include/hw/virtio/vdpa-dev.h new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.23.0
#3 lore raw

PATCH RESEND v7 3/4 vdpa: add vdpa-dev-pci support

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Supports vdpa-dev-pci, we can use the device as follow:

-device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-X

---
 hw/virtio/meson.build    |   1 +
 hw/virtio/vdpa-dev-pci.c | 102 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+)
 create mode 100644 hw/virtio/vdpa-dev-pci.c
hw/virtio/meson.build +1 -0
index 54d6d29af7..559b80cb28 100644
@@ -57,6 +57,7 @@ virtio_pci_ss.add(when: 'CONFIG_VIRTIO_SERIAL', if_true: files('virtio-serial-pc
 virtio_pci_ss.add(when: 'CONFIG_VIRTIO_PMEM', if_true: files('virtio-pmem-pci.c'))
 virtio_pci_ss.add(when: 'CONFIG_VIRTIO_IOMMU', if_true: files('virtio-iommu-pci.c'))
 virtio_pci_ss.add(when: 'CONFIG_VIRTIO_MEM', if_true: files('virtio-mem-pci.c'))
+virtio_pci_ss.add(when: 'CONFIG_VHOST_VDPA_DEV', if_true: files('vdpa-dev-pci.c'))
 
 virtio_ss.add_all(when: 'CONFIG_VIRTIO_PCI', if_true: virtio_pci_ss)
 
hw/virtio/vdpa-dev-pci.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line
2.23.0
#4 lore raw

PATCH RESEND v7 4/4 docs: Add generic vhost-vdpa device documentation

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

---
 docs/system/devices/vhost-vdpa-device.rst | 43 +++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 docs/system/devices/vhost-vdpa-device.rst
docs/system/devices/vhost-vdpa-device.rst new file +43 -0
new file mode 100644
index 0000000000..b758c4fce6
@@ -0,0 +1,43 @@
+
+=========================
+generic vhost-vdpa device
+=========================
+
+This document explains the usage of the generic vhost vdpa device.
+
+Description
+-----------
+
+vDPA(virtio data path acceleration) device is a device that uses a datapath
+which complies with the virtio specifications with vendor specific control
+path.
+
+QEMU provides two types of vhost-vdpa devices to enable the vDPA device, one
+is type sensitive which means QEMU needs to know the actual device type
+(e.g. net, blk, scsi) and another is called "generic vdpa device" which is
+type insensitive (likes vfio-pci).
+
+Examples
+--------
+
+Prepare the vhost-vdpa backends first:
+
+::
+  host# ls -l /dev/vhost-vdpa-*
+  crw------- 1 root root 236, 0 Nov  2 00:49 /dev/vhost-vdpa-0
+
+Start QEMU with virtio-mmio bus:
+
+::
+  host# qemu-system                                                  \
+      -M microvm -m 512 -smp 2 -kernel ... -initrd ...               \
+      -device vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-0           \
+      ...
+
+Start QEMU with virtio-pci bus:
+
+::
+  host# qemu-system                                                  \
+      -M pc -m 512 -smp 2                                            \
+      -device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-0       \
+      ...
Signature · 1 line
2.23.0
#5 lore raw
In reply to Longpeng(Mike) via
Quoted 6 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Hi guys,

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

With this kind of passthrough migration is completely MIA right? Better add a blocker...
And given this is there an advantage over VFIO?

Quoted 85 lines

We can use the generic vDPA device as follow:
-device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-X Or
-M microvm -m 512m -smp 2 -kernel ... -initrd ... -device \
vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-x

Changes v6 -> v7:
(v6: https://mail.gnu.org/archive/html/qemu-devel/2022-05/msg02821.html)
- rebase. [Jason]
- add documentation . [Stefan]

Changes v5 -> v6:
Patch 2:
- Turn to the original approach in the RFC to initialize the virtio_pci_id_info array. [Michael]
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Fix logical error of exception handler around the post_init. [Stefano]
- Fix some coding style warnings. [Stefano]
Patch 4:
- Fix some coding style warnings. [Stefano]

Changes v4 -> v5:
Patch 3:
- remove vhostfd [Jason]
- support virtio-mmio [Jason]

Changes v3 -> v4:
v3: https://www.mail-archive.com/[email protected]/msg877015.html
- reorganize the series [Stefano]
- fix some typos [Stefano]
- fix logical error in vhost_vdpa_device_realize [Stefano]

Changes v2 -> v3
Patch 4 & 5:
- only call vdpa ioctls in vdpa-dev.c [Stefano, Longpeng]
- s/VQS_NUM/VQS_COUNT [Stefano]
- check both vdpa_dev_fd and vdpa_dev [Stefano]
Patch 6:
- move all steps into vhost_vdpa_device_unrealize. [Stefano]

Changes RFC -> v2
Patch 1:
- rename 'pdev_id' to 'trans_devid' [Michael]
- only use transitional device id for the devices listed in the spec [Michael]
- use macros to make the id_info table clearer [Longpeng]
- add some modern devices in the id_info table [Longpeng]
Patch 2:
- remove the GET_VECTORS_NUM command [Jason]
Patch 4:
- expose vdpa_dev_fd as a QOM preperty [Stefan]
- introduce vhost_vdpa_device_get_u32 as a common function to make the code clearer [Stefan]
- fix the misleading description of 'dc->desc' [Stefano]
Patch 5:
- check returned number of virtqueues [Stefan]
Patch 6:
- init s->num_queues [Stefano]
- free s->dev.vqs [Stefano]

Longpeng (Mike) (4):
virtio: get class_id and pci device id by the virtio id
vdpa: add vdpa-dev support
vdpa: add vdpa-dev-pci support
docs: Add generic vhost-vdpa device documentation

 docs/system/devices/vhost-vdpa-device.rst |  43 +++
 hw/virtio/Kconfig                         |   5 +
 hw/virtio/meson.build                     |   2 +
 hw/virtio/vdpa-dev-pci.c                  | 102 ++++++
 hw/virtio/vdpa-dev.c                      | 377 ++++++++++++++++++++++
 hw/virtio/virtio-pci.c                    |  88 +++++
 include/hw/virtio/vdpa-dev.h              |  43 +++
 include/hw/virtio/virtio-pci.h            |   5 +
 8 files changed, 665 insertions(+)
 create mode 100644 docs/system/devices/vhost-vdpa-device.rst
 create mode 100644 hw/virtio/vdpa-dev-pci.c
 create mode 100644 hw/virtio/vdpa-dev.c
 create mode 100644 include/hw/virtio/vdpa-dev.h
Signature · 1 line
2.23.0
#6 lore raw
Quoted 10 lines · Michael S. Tsirkin
Quoted 6 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Hi guys,

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

With this kind of passthrough migration is completely MIA right? Better add a blocker...

Oh, I missed the "vdpa-dev: mark the device as unmigratable" since v4 and I'll add it in the next version.

We'll support passthrough migration in the next step. We have already written a demo that can migrate between some offloading cards.

Quoted 1 line

And given this is there an advantage over VFIO?

I think the answer is the same as "why we need vDPA" if we compare it with VFIO.

Quoted 88 lines
Quoted 79 lines

We can use the generic vDPA device as follow:

   -device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-X
   Or
   -M microvm -m 512m -smp 2 -kernel ... -initrd ... -device \
   vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-x

Changes v6 -> v7:
(v6: https://mail.gnu.org/archive/html/qemu-devel/2022-05/msg02821.html)
- rebase. [Jason]
- add documentation . [Stefan]

Changes v5 -> v6:
Patch 2:
- Turn to the original approach in the RFC to initialize the virtio_pci_id_info array. [Michael]
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Fix logical error of exception handler around the post_init. [Stefano]
- Fix some coding style warnings. [Stefano]
Patch 4:
- Fix some coding style warnings. [Stefano]

Changes v4 -> v5:
Patch 3:
- remove vhostfd [Jason]
- support virtio-mmio [Jason]

Changes v3 -> v4:
v3: https://www.mail-archive.com/[email protected]/msg877015.html
- reorganize the series [Stefano]
- fix some typos [Stefano]
- fix logical error in vhost_vdpa_device_realize [Stefano]

Changes v2 -> v3
Patch 4 & 5:
- only call vdpa ioctls in vdpa-dev.c [Stefano, Longpeng]
- s/VQS_NUM/VQS_COUNT [Stefano]
- check both vdpa_dev_fd and vdpa_dev [Stefano]
Patch 6:
- move all steps into vhost_vdpa_device_unrealize. [Stefano]

Changes RFC -> v2
Patch 1:
- rename 'pdev_id' to 'trans_devid' [Michael]
- only use transitional device id for the devices listed in the spec [Michael]
- use macros to make the id_info table clearer [Longpeng]
- add some modern devices in the id_info table [Longpeng]
Patch 2:
- remove the GET_VECTORS_NUM command [Jason]
Patch 4:
- expose vdpa_dev_fd as a QOM preperty [Stefan]
- introduce vhost_vdpa_device_get_u32 as a common function to make the code clearer [Stefan]
- fix the misleading description of 'dc->desc' [Stefano]
Patch 5:
- check returned number of virtqueues [Stefan]
Patch 6:
- init s->num_queues [Stefano]
- free s->dev.vqs [Stefano]

Longpeng (Mike) (4):
virtio: get class_id and pci device id by the virtio id
vdpa: add vdpa-dev support
vdpa: add vdpa-dev-pci support
docs: Add generic vhost-vdpa device documentation

  docs/system/devices/vhost-vdpa-device.rst |  43 +++
  hw/virtio/Kconfig                         |   5 +
  hw/virtio/meson.build                     |   2 +
  hw/virtio/vdpa-dev-pci.c                  | 102 ++++++
  hw/virtio/vdpa-dev.c                      | 377 ++++++++++++++++++++++
  hw/virtio/virtio-pci.c                    |  88 +++++
  include/hw/virtio/vdpa-dev.h              |  43 +++
  include/hw/virtio/virtio-pci.h            |   5 +
  8 files changed, 665 insertions(+)
Quoted 7 lines

create mode 100644 docs/system/devices/vhost-vdpa-device.rst create mode 100644 hw/virtio/vdpa-dev-pci.c
create mode 100644 hw/virtio/vdpa-dev.c
create mode 100644 include/hw/virtio/vdpa-dev.h

Signature · 1 line
2.23.0

.

#7 lore raw
Quoted 11 lines · Cloud Infrastructure Service Product Dept.)
Quoted 8 lines · Michael S. Tsirkin
Quoted 6 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Hi guys,

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

Quoted 8 lines
Quoted 2 lines

With this kind of passthrough migration is completely MIA right? Better add a blocker...

Oh, I missed the "vdpa-dev: mark the device as unmigratable" since v4 and I'll add it in the next version.

We'll support passthrough migration in the next step. We have already written a demo that can migrate between some offloading cards.

Hmm ok. Backend disconnect can't work though, can it? State is by necessity lost when backend crashes.

Quoted 4 lines
Quoted 1 line

And given this is there an advantage over VFIO?

I think the answer is the same as "why we need vDPA" if we compare it with VFIO.

The answer is mostly because you can migrate and support backend disconnect, no?

Quoted 88 lines
Quoted 88 lines
Quoted 79 lines

We can use the generic vDPA device as follow:

   -device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-X
   Or
   -M microvm -m 512m -smp 2 -kernel ... -initrd ... -device \
   vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-x

Changes v6 -> v7:
(v6: https://mail.gnu.org/archive/html/qemu-devel/2022-05/msg02821.html)
- rebase. [Jason]
- add documentation . [Stefan]

Changes v5 -> v6:
Patch 2:
- Turn to the original approach in the RFC to initialize the virtio_pci_id_info array. [Michael]
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Fix logical error of exception handler around the post_init. [Stefano]
- Fix some coding style warnings. [Stefano]
Patch 4:
- Fix some coding style warnings. [Stefano]

Changes v4 -> v5:
Patch 3:
- remove vhostfd [Jason]
- support virtio-mmio [Jason]

Changes v3 -> v4:
v3: https://www.mail-archive.com/[email protected]/msg877015.html
- reorganize the series [Stefano]
- fix some typos [Stefano]
- fix logical error in vhost_vdpa_device_realize [Stefano]

Changes v2 -> v3
Patch 4 & 5:
- only call vdpa ioctls in vdpa-dev.c [Stefano, Longpeng]
- s/VQS_NUM/VQS_COUNT [Stefano]
- check both vdpa_dev_fd and vdpa_dev [Stefano]
Patch 6:
- move all steps into vhost_vdpa_device_unrealize. [Stefano]

Changes RFC -> v2
Patch 1:
- rename 'pdev_id' to 'trans_devid' [Michael]
- only use transitional device id for the devices listed in the spec [Michael]
- use macros to make the id_info table clearer [Longpeng]
- add some modern devices in the id_info table [Longpeng]
Patch 2:
- remove the GET_VECTORS_NUM command [Jason]
Patch 4:
- expose vdpa_dev_fd as a QOM preperty [Stefan]
- introduce vhost_vdpa_device_get_u32 as a common function to make the code clearer [Stefan]
- fix the misleading description of 'dc->desc' [Stefano]
Patch 5:
- check returned number of virtqueues [Stefan]
Patch 6:
- init s->num_queues [Stefano]
- free s->dev.vqs [Stefano]

Longpeng (Mike) (4):
virtio: get class_id and pci device id by the virtio id
vdpa: add vdpa-dev support
vdpa: add vdpa-dev-pci support
docs: Add generic vhost-vdpa device documentation

  docs/system/devices/vhost-vdpa-device.rst |  43 +++
  hw/virtio/Kconfig                         |   5 +
  hw/virtio/meson.build                     |   2 +
  hw/virtio/vdpa-dev-pci.c                  | 102 ++++++
  hw/virtio/vdpa-dev.c                      | 377 ++++++++++++++++++++++
  hw/virtio/virtio-pci.c                    |  88 +++++
  include/hw/virtio/vdpa-dev.h              |  43 +++
  include/hw/virtio/virtio-pci.h            |   5 +
  8 files changed, 665 insertions(+)
Quoted 7 lines

create mode 100644 docs/system/devices/vhost-vdpa-device.rst create mode 100644 hw/virtio/vdpa-dev-pci.c
create mode 100644 hw/virtio/vdpa-dev.c
create mode 100644 include/hw/virtio/vdpa-dev.h

Signature · 1 line
2.23.0

.

#8 lore raw
Quoted 25 lines · Michael S. Tsirkin
Quoted 11 lines · Cloud Infrastructure Service Product Dept.)
Quoted 8 lines · Michael S. Tsirkin
Quoted 6 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Hi guys,

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

Quoted 8 lines
Quoted 2 lines

With this kind of passthrough migration is completely MIA right? Better add a blocker...

Oh, I missed the "vdpa-dev: mark the device as unmigratable" since v4 and I'll add it in the next version.

We'll support passthrough migration in the next step. We have already written a demo that can migrate between some offloading cards.

Hmm ok. Backend disconnect can't work though, can it? State is by necessity lost when backend crashes.
Yes, it can't.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

And given this is there an advantage over VFIO?

I think the answer is the same as "why we need vDPA" if we compare it with VFIO.

The answer is mostly because you can migrate and support backend disconnect, no?

Migrating between different hardware is the first consideration in our requirement, supporting backend disconnect is a low priority.

Quoted 91 lines
Quoted 84 lines
Quoted 84 lines
Quoted 84 lines

We can use the generic vDPA device as follow:

    -device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-X
    Or
    -M microvm -m 512m -smp 2 -kernel ... -initrd ... -device \
    vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-x

Changes v6 -> v7:
(v6: https://mail.gnu.org/archive/html/qemu-devel/2022-05/msg02821.html)
- rebase. [Jason]
- add documentation . [Stefan]

Changes v5 -> v6:
Patch 2:
- Turn to the original approach in the RFC to initialize the virtio_pci_id_info array. [Michael]
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Fix logical error of exception handler around the post_init. [Stefano]
- Fix some coding style warnings. [Stefano]
Patch 4:
- Fix some coding style warnings. [Stefano]

Changes v4 -> v5:
Patch 3:
- remove vhostfd [Jason]
- support virtio-mmio [Jason]

Changes v3 -> v4:
v3: https://www.mail-archive.com/[email protected]/msg877015.html
- reorganize the series [Stefano]
- fix some typos [Stefano]
- fix logical error in vhost_vdpa_device_realize [Stefano]

Changes v2 -> v3
Patch 4 & 5:
- only call vdpa ioctls in vdpa-dev.c [Stefano, Longpeng]
- s/VQS_NUM/VQS_COUNT [Stefano]
- check both vdpa_dev_fd and vdpa_dev [Stefano]
Patch 6:
- move all steps into vhost_vdpa_device_unrealize. [Stefano]

Changes RFC -> v2
Patch 1:
- rename 'pdev_id' to 'trans_devid' [Michael]
- only use transitional device id for the devices listed in the spec [Michael]
- use macros to make the id_info table clearer [Longpeng]
- add some modern devices in the id_info table [Longpeng]
Patch 2:
- remove the GET_VECTORS_NUM command [Jason]
Patch 4:
- expose vdpa_dev_fd as a QOM preperty [Stefan]
- introduce vhost_vdpa_device_get_u32 as a common function to make the code clearer [Stefan]
- fix the misleading description of 'dc->desc' [Stefano]
Patch 5:
- check returned number of virtqueues [Stefan]
Patch 6:
- init s->num_queues [Stefano]
- free s->dev.vqs [Stefano]

Longpeng (Mike) (4):
virtio: get class_id and pci device id by the virtio id
vdpa: add vdpa-dev support
vdpa: add vdpa-dev-pci support
docs: Add generic vhost-vdpa device documentation

   docs/system/devices/vhost-vdpa-device.rst |  43 +++
   hw/virtio/Kconfig                         |   5 +
   hw/virtio/meson.build                     |   2 +
   hw/virtio/vdpa-dev-pci.c                  | 102 ++++++
   hw/virtio/vdpa-dev.c                      | 377 ++++++++++++++++++++++
   hw/virtio/virtio-pci.c                    |  88 +++++
   include/hw/virtio/vdpa-dev.h              |  43 +++
   include/hw/virtio/virtio-pci.h            |   5 +
   8 files changed, 665 insertions(+)

create mode 100644 docs/system/devices/vhost-vdpa-device.rst create mode 100644 hw/virtio/vdpa-dev-pci.c
create mode 100644 hw/virtio/vdpa-dev.c
create mode 100644 include/hw/virtio/vdpa-dev.h

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Signature · 1 line
2.23.0

.

.

#9 lore raw
Quoted 30 lines · Cloud Infrastructure Service Product Dept.)
Quoted 27 lines · Michael S. Tsirkin
Quoted 11 lines · Cloud Infrastructure Service Product Dept.)
Quoted 8 lines · Michael S. Tsirkin
Quoted 6 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Hi guys,

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

Quoted 8 lines
Quoted 2 lines

With this kind of passthrough migration is completely MIA right? Better add a blocker...

Oh, I missed the "vdpa-dev: mark the device as unmigratable" since v4 and I'll add it in the next version.

We'll support passthrough migration in the next step. We have already written a demo that can migrate between some offloading cards.

Hmm ok. Backend disconnect can't work though, can it? State is by necessity lost when backend crashes.
Yes, it can't.

Quoted 2 lines
Quoted 1 line

And given this is there an advantage over VFIO?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I think the answer is the same as "why we need vDPA" if we compare it with VFIO.

The answer is mostly because you can migrate and support backend disconnect, no?

Migrating between different hardware is the first consideration in our requirement, supporting backend disconnect is a low priority.

I dislike non-orthogonal features though ...
And the advantage of keeping it out of process with qemu is I presume security?

Quoted 91 lines
Quoted 91 lines
Quoted 84 lines
Quoted 84 lines
Quoted 84 lines

We can use the generic vDPA device as follow:

    -device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-X
    Or
    -M microvm -m 512m -smp 2 -kernel ... -initrd ... -device \
    vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-x

Changes v6 -> v7:
(v6: https://mail.gnu.org/archive/html/qemu-devel/2022-05/msg02821.html)
- rebase. [Jason]
- add documentation . [Stefan]

Changes v5 -> v6:
Patch 2:
- Turn to the original approach in the RFC to initialize the virtio_pci_id_info array. [Michael]
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Fix logical error of exception handler around the post_init. [Stefano]
- Fix some coding style warnings. [Stefano]
Patch 4:
- Fix some coding style warnings. [Stefano]

Changes v4 -> v5:
Patch 3:
- remove vhostfd [Jason]
- support virtio-mmio [Jason]

Changes v3 -> v4:
v3: https://www.mail-archive.com/[email protected]/msg877015.html
- reorganize the series [Stefano]
- fix some typos [Stefano]
- fix logical error in vhost_vdpa_device_realize [Stefano]

Changes v2 -> v3
Patch 4 & 5:
- only call vdpa ioctls in vdpa-dev.c [Stefano, Longpeng]
- s/VQS_NUM/VQS_COUNT [Stefano]
- check both vdpa_dev_fd and vdpa_dev [Stefano]
Patch 6:
- move all steps into vhost_vdpa_device_unrealize. [Stefano]

Changes RFC -> v2
Patch 1:
- rename 'pdev_id' to 'trans_devid' [Michael]
- only use transitional device id for the devices listed in the spec [Michael]
- use macros to make the id_info table clearer [Longpeng]
- add some modern devices in the id_info table [Longpeng]
Patch 2:
- remove the GET_VECTORS_NUM command [Jason]
Patch 4:
- expose vdpa_dev_fd as a QOM preperty [Stefan]
- introduce vhost_vdpa_device_get_u32 as a common function to make the code clearer [Stefan]
- fix the misleading description of 'dc->desc' [Stefano]
Patch 5:
- check returned number of virtqueues [Stefan]
Patch 6:
- init s->num_queues [Stefano]
- free s->dev.vqs [Stefano]

Longpeng (Mike) (4):
virtio: get class_id and pci device id by the virtio id
vdpa: add vdpa-dev support
vdpa: add vdpa-dev-pci support
docs: Add generic vhost-vdpa device documentation

   docs/system/devices/vhost-vdpa-device.rst |  43 +++
   hw/virtio/Kconfig                         |   5 +
   hw/virtio/meson.build                     |   2 +
   hw/virtio/vdpa-dev-pci.c                  | 102 ++++++
   hw/virtio/vdpa-dev.c                      | 377 ++++++++++++++++++++++
   hw/virtio/virtio-pci.c                    |  88 +++++
   include/hw/virtio/vdpa-dev.h              |  43 +++
   include/hw/virtio/virtio-pci.h            |   5 +
   8 files changed, 665 insertions(+)

create mode 100644 docs/system/devices/vhost-vdpa-device.rst create mode 100644 hw/virtio/vdpa-dev-pci.c
create mode 100644 hw/virtio/vdpa-dev.c
create mode 100644 include/hw/virtio/vdpa-dev.h

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Signature · 1 line
2.23.0

.

.

#10 lore raw
Quoted 37 lines · Michael S. Tsirkin
Quoted 36 lines · Cloud Infrastructure Service Product Dept.)
Quoted 33 lines · Michael S. Tsirkin
Quoted 11 lines · Cloud Infrastructure Service Product Dept.)
Quoted 8 lines · Michael S. Tsirkin
Quoted 6 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Hi guys,

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

Quoted 8 lines
Quoted 2 lines

With this kind of passthrough migration is completely MIA right? Better add a blocker...

Oh, I missed the "vdpa-dev: mark the device as unmigratable" since v4 and I'll add it in the next version.

We'll support passthrough migration in the next step. We have already written a demo that can migrate between some offloading cards.

Hmm ok. Backend disconnect can't work though, can it? State is by necessity lost when backend crashes.
Yes, it can't.

Quoted 4 lines
Quoted 1 line

And given this is there an advantage over VFIO?

I think the answer is the same as "why we need vDPA" if we compare it with VFIO.

The answer is mostly because you can migrate and support backend disconnect, no?

Quoted 7 lines
Quoted 2 lines

Migrating between different hardware is the first consideration in our requirement, supporting backend disconnect is a low priority.

I dislike non-orthogonal features though ...
And the advantage of keeping it out of process with qemu is I presume security?

Yes, this is one of the reasons. The TCB of the generic vdpa device is smaller than the existing vdpa device (needs to use the virtio-net/blk/scsi emulation codes).

Besides, the generic vdpa device can support any virtio device, but the existing vdpa device only supports virtio-net yet.

Though the existing vdpa device is more powerful and the generic vdpa device would miss some features, it can be an alternative for some users.

Quoted 94 lines
Quoted 84 lines
Quoted 84 lines
Quoted 84 lines
Quoted 84 lines
Quoted 84 lines

We can use the generic vDPA device as follow:

     -device vhost-vdpa-device-pci,vhostdev=/dev/vhost-vdpa-X
     Or
     -M microvm -m 512m -smp 2 -kernel ... -initrd ... -device \
     vhost-vdpa-device,vhostdev=/dev/vhost-vdpa-x

Changes v6 -> v7:
(v6: https://mail.gnu.org/archive/html/qemu-devel/2022-05/msg02821.html)
- rebase. [Jason]
- add documentation . [Stefan]

Changes v5 -> v6:
Patch 2:
- Turn to the original approach in the RFC to initialize the virtio_pci_id_info array. [Michael]
https://lore.kernel.org/all/[email protected]/
Patch 3:
- Fix logical error of exception handler around the post_init. [Stefano]
- Fix some coding style warnings. [Stefano]
Patch 4:
- Fix some coding style warnings. [Stefano]

Changes v4 -> v5:
Patch 3:
- remove vhostfd [Jason]
- support virtio-mmio [Jason]

Changes v3 -> v4:
v3: https://www.mail-archive.com/[email protected]/msg877015.html
- reorganize the series [Stefano]
- fix some typos [Stefano]
- fix logical error in vhost_vdpa_device_realize [Stefano]

Changes v2 -> v3
Patch 4 & 5:
- only call vdpa ioctls in vdpa-dev.c [Stefano, Longpeng]
- s/VQS_NUM/VQS_COUNT [Stefano]
- check both vdpa_dev_fd and vdpa_dev [Stefano]
Patch 6:
- move all steps into vhost_vdpa_device_unrealize. [Stefano]

Changes RFC -> v2
Patch 1:
- rename 'pdev_id' to 'trans_devid' [Michael]
- only use transitional device id for the devices listed in the spec [Michael]
- use macros to make the id_info table clearer [Longpeng]
- add some modern devices in the id_info table [Longpeng]
Patch 2:
- remove the GET_VECTORS_NUM command [Jason]
Patch 4:
- expose vdpa_dev_fd as a QOM preperty [Stefan]
- introduce vhost_vdpa_device_get_u32 as a common function to make the code clearer [Stefan]
- fix the misleading description of 'dc->desc' [Stefano]
Patch 5:
- check returned number of virtqueues [Stefan]
Patch 6:
- init s->num_queues [Stefano]
- free s->dev.vqs [Stefano]

Longpeng (Mike) (4):
virtio: get class_id and pci device id by the virtio id
vdpa: add vdpa-dev support
vdpa: add vdpa-dev-pci support
docs: Add generic vhost-vdpa device documentation

docs/system/devices/vhost-vdpa-device.rst | 43 +++
hw/virtio/Kconfig | 5 +
hw/virtio/meson.build | 2 +
hw/virtio/vdpa-dev-pci.c | 102 ++++++
hw/virtio/vdpa-dev.c | 377 ++++++++++++++++++++++
hw/virtio/virtio-pci.c | 88 +++++
include/hw/virtio/vdpa-dev.h | 43 +++
include/hw/virtio/virtio-pci.h | 5 +
8 files changed, 665 insertions(+)
create mode 100644 docs/system/devices/vhost-vdpa-device.rst create mode 100644 hw/virtio/vdpa-dev-pci.c
create mode 100644 hw/virtio/vdpa-dev.c
create mode 100644 include/hw/virtio/vdpa-dev.h

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Signature · 1 line
2.23.0

.

.

.