Skip to contentSkip to conversation

add generic vDPA device support

RFC Superseded Longpeng(Mike) via 6 participants 52 messages Jan 5, 2022 – Jan 19, 2022 Arm/Amlogic Meson SoC support

v1 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

cover add generic vDPA device support

From: Longpeng <[email protected]>

Hi guys,

This patchset tries to support the generic vDPA device, the previous disscussion can be found here [1].

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

We can use the generic vDPA device as follow:
-device vhost-vdpa-device-pci,vdpa-dev=/dev/vhost-vdpa-X

I've done some simple tests on Huawei's offloading card (net, 0.95) and vdpa_sim_blk (1.0);

Note:
the kernel part does not send out yet, I'll send it as soon as possible.

[1] https://lore.kernel.org/all/[email protected]/

Longpeng (Mike) (10):
virtio: get class_id and pci device id by the virtio id
vhost: add 3 commands for vhost-vdpa
vdpa: add the infrastructure of vdpa-dev
vdpa-dev: implement the instance_init/class_init interface
vdpa-dev: implement the realize interface
vdpa-dev: implement the unrealize interface
vdpa-dev: implement the get_config/set_config interface
vdpa-dev: implement the get_features interface
vdpa-dev: implement the set_status interface
vdpa-dev: mark the device as unmigratable

 hw/virtio/Kconfig            |   5 +
 hw/virtio/meson.build        |   2 +
 hw/virtio/vdpa-dev-pci.c     | 127 +++++++++++++
 hw/virtio/vdpa-dev.c         | 355 +++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.c       |  93 +++++++++
 hw/virtio/virtio-pci.h       |   4 +
 include/hw/virtio/vdpa-dev.h |  26 +++
 linux-headers/linux/vhost.h  |  10 +
 8 files changed, 622 insertions(+)
 create mode 100644 hw/virtio/vdpa-dev-pci.c
 create mode 100644 hw/virtio/vdpa-dev.c
 create mode 100644 include/hw/virtio/vdpa-dev.h
Signature · 1 line
2.23.0
#11 lore raw

RFC 2/10 vhost: add 3 commands for vhost-vdpa

From: Longpeng <[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported
- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -0
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE	_IOR(VHOST_VIRTIO, 0x78, \
 					     struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM	_IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE	_IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM		_IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
Signature · 1 line
2.23.0
#1 lore raw

RFC 1/10 virtio: get class_id and pci device id by the virtio id

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +93 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f)
     return 0;
 }
 
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1674,6 +1764,9 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp)
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID, virtio_bus_get_vdev_id(bus));
+        if (proxy->pdev_id) {
+            pci_config_set_device_id(config, proxy->pdev_id);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
hw/virtio/virtio-pci.h +4 -0
index 2446dcd9ae..06aa59436e 100644
@@ -146,6 +146,7 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    uint16_t pdev_id;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -158,6 +159,9 @@ struct VirtIOPCIProxy {
     VirtioBusState bus;
 };
 
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 static inline bool virtio_pci_modern(VirtIOPCIProxy *proxy)
 {
     return !proxy->disable_modern;
Signature · 1 line
2.23.0
#37 lore raw

RFC 5/10 vdpa-dev: implement the realize interface

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Implements the .realize interface.

---
 hw/virtio/vdpa-dev.c         | 114 +++++++++++++++++++++++++++++++++++
 include/hw/virtio/vdpa-dev.h |   8 +++
 2 files changed, 122 insertions(+)
hw/virtio/vdpa-dev.c +114 -0
index 790117fb3b..2d534d837a 100644
@@ -15,9 +15,122 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 
+static void
+vhost_vdpa_device_dummy_handle_output(VirtIODevice *vdev, VirtQueue *vq)
+{
+    /* Nothing to do */
+}
+
+static int vdpa_dev_get_info_by_fd(int fd, uint64_t cmd, Error **errp)
+{
+    int val;
+
+    if (ioctl(fd, cmd, &val) < 0) {
+        error_setg(errp, "vhost-vdpa-device: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        return -1;
+    }
+
+    return val;
+}
+
+static inline int vdpa_dev_get_queue_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VRING_NUM, errp);
+}
+
+static inline int vdpa_dev_get_vqs_num(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VQS_NUM, errp);
+}
+
+static inline int vdpa_dev_get_config_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_CONFIG_SIZE, errp);
+}
+
 static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
 {
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    uint32_t device_id;
+    int max_queue_size;
+    int fd;
+    int i, ret;
+
+    fd = qemu_open(s->vdpa_dev, O_RDWR, errp);
+    if (fd == -1) {
+        return;
+    }
+    s->vdpa.device_fd = fd;
+
+    max_queue_size = vdpa_dev_get_queue_size(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
+    if (s->queue_size > max_queue_size) {
+        error_setg(errp, "vhost-vdpa-device: invalid queue_size: %d (max:%d)",
+                   s->queue_size, max_queue_size);
+        goto out;
+    } else if (!s->queue_size) {
+        s->queue_size = max_queue_size;
+    }
+
+    ret = vdpa_dev_get_vqs_num(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
+    s->dev.nvqs = ret;
+    s->dev.vqs = g_new0(struct vhost_virtqueue, s->dev.nvqs);
+    s->dev.vq_index = 0;
+    s->dev.vq_index_end = s->dev.nvqs;
+    s->dev.backend_features = 0;
+    s->started = false;
+
+    ret = vhost_dev_init(&s->dev, &s->vdpa, VHOST_BACKEND_TYPE_VDPA, 0, NULL);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device: vhost initialization failed: %s",
+                   strerror(-ret));
+        goto out;
+    }
+
+    ret = s->dev.vhost_ops->vhost_get_device_id(&s->dev, &device_id);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device: vhost get device id failed: %s",
+                   strerror(-ret));
+        goto vhost_cleanup;
+    }
+
+    s->config_size = vdpa_dev_get_config_size(fd, errp);
+    if (*errp) {
+        goto vhost_cleanup;
+    }
+
+    s->config = g_malloc0(s->config_size);
+
+    ret = vhost_dev_get_config(&s->dev, s->config, s->config_size, NULL);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device: get config failed");
+        goto config_err;
+    }
+
+    virtio_init(vdev, "vhost-vdpa", device_id, s->config_size);
+
+    s->virtqs = g_new0(VirtQueue *, s->dev.nvqs);
+    for (i = 0; i < s->dev.nvqs; i++) {
+        s->virtqs[i] = virtio_add_queue(vdev, s->queue_size,
+                                        vhost_vdpa_device_dummy_handle_output);
+    }
+
     return;
+config_err:
+    g_free(s->config);
+vhost_cleanup:
+    vhost_dev_cleanup(&s->dev);
+out:
+    close(fd);
 }
 
 static void vhost_vdpa_device_unrealize(DeviceState *dev)
@@ -51,6 +164,7 @@ static void vhost_vdpa_device_set_status(VirtIODevice *vdev, uint8_t status)
 
 static Property vhost_vdpa_device_properties[] = {
     DEFINE_PROP_STRING("vdpa-dev", VhostVdpaDevice, vdpa_dev),
+    DEFINE_PROP_UINT16("queue-size", VhostVdpaDevice, queue_size, 0),
     DEFINE_PROP_END_OF_LIST(),
 };
 
include/hw/virtio/vdpa-dev.h +8 -0
index 7a0e6bdcf8..49f8145d61 100644
@@ -13,6 +13,14 @@ struct VhostVdpaDevice {
     VirtIODevice parent_obj;
     char *vdpa_dev;
     int32_t bootindex;
+    struct vhost_dev dev;
+    struct vhost_vdpa vdpa;
+    VirtQueue **virtqs;
+    uint8_t *config;
+    int config_size;
+    uint32_t num_queues;
+    uint16_t queue_size;
+    bool started;
 };
 
 #endif
Signature · 1 line
2.23.0
#32 lore raw

RFC 4/10 vdpa-dev: implement the instance_init/class_init interface

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Implements the .instance_init and the .class_init interface.

---
 hw/virtio/vdpa-dev-pci.c     | 80 +++++++++++++++++++++++++++++++++++-
 hw/virtio/vdpa-dev.c         | 68 +++++++++++++++++++++++++++++-
 include/hw/virtio/vdpa-dev.h |  2 +
 3 files changed, 146 insertions(+), 4 deletions(-)
hw/virtio/vdpa-dev-pci.c +78 -2
index a5a7b528a9..0af54a26d4 100644
@@ -23,14 +23,90 @@ struct VhostVdpaDevicePCI {
     VhostVdpaDevice vdev;
 };
 
+static uint32_t
+vdpa_dev_pci_get_info(const char *name, uint64_t cmd, Error **errp)
+{
+    int device_fd;
+    uint32_t val;
+    int ret;
+
+    device_fd = qemu_open(name, O_RDWR, errp);
+    if (device_fd == -1) {
+        return (uint32_t)-1;
+    }
+
+    ret = ioctl(device_fd, cmd, &val);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device-pci: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        goto out;
+    }
+
+out:
+    close(device_fd);
+    return val;
+}
+
+static inline uint32_t
+vdpa_dev_pci_get_devid(VhostVdpaDevicePCI *dev, Error **errp)
+{
+    return vdpa_dev_pci_get_info(dev->vdev.vdpa_dev,
+                                 VHOST_VDPA_GET_DEVICE_ID, errp);
+}
+
+static inline uint32_t
+vdpa_dev_pci_get_vectors_num(VhostVdpaDevicePCI *dev, Error **errp)
+{
+    return vdpa_dev_pci_get_info(dev->vdev.vdpa_dev,
+                                 VHOST_VDPA_GET_VECTORS_NUM, errp);
+}
+
 static void vhost_vdpa_device_pci_instance_init(Object *obj)
 {
-    return;
+    VhostVdpaDevicePCI *dev = VHOST_VDPA_DEVICE_PCI(obj);
+
+    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
+                                TYPE_VHOST_VDPA_DEVICE);
+    object_property_add_alias(obj, "bootindex", OBJECT(&dev->vdev),
+                              "bootindex");
+}
+
+static Property vhost_vdpa_device_pci_properties[] = {
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void
+vhost_vdpa_device_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
+{
+    VhostVdpaDevicePCI *dev = VHOST_VDPA_DEVICE_PCI(vpci_dev);
+    DeviceState *vdev = DEVICE(&dev->vdev);
+    uint32_t devid;
+    uint32_t vectors;
+
+    devid = vdpa_dev_pci_get_devid(dev, errp);
+    if (*errp) {
+        return;
+    }
+
+    vectors = vdpa_dev_pci_get_vectors_num(dev, errp);
+    if (*errp) {
+        return;
+    }
+
+    vpci_dev->class_code = virtio_pci_get_class_id(devid);
+    vpci_dev->pdev_id = virtio_pci_get_pci_devid(devid);
+    vpci_dev->nvectors = vectors;
+    qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
 }
 
 static void vhost_vdpa_device_pci_class_init(ObjectClass *klass, void *data)
 {
-    return;
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
+
+    set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+    device_class_set_props(dc, vhost_vdpa_device_pci_properties);
+    k->realize = vhost_vdpa_device_pci_realize;
 }
 
 static const VirtioPCIDeviceTypeInfo vhost_vdpa_device_pci_info = {
hw/virtio/vdpa-dev.c +66 -2

82 lines not on this page · raw message

include/hw/virtio/vdpa-dev.h +2 -0
index dd94bd74a2..7a0e6bdcf8 100644
@@ -11,6 +11,8 @@ OBJECT_DECLARE_SIMPLE_TYPE(VhostVdpaDevice, VHOST_VDPA_DEVICE)
 
 struct VhostVdpaDevice {
     VirtIODevice parent_obj;
+    char *vdpa_dev;
+    int32_t bootindex;
 };
 
 #endif
Signature · 1 line
2.23.0
#47 lore raw

RFC 7/10 vdpa-dev: implement the get_config/set_config interface

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Implements the .get_config and .set_config interface.

---
 hw/virtio/vdpa-dev.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
hw/virtio/vdpa-dev.c +12 -2
index 4e4dd3d201..4f97a7521b 100644
@@ -161,13 +161,23 @@ static void vhost_vdpa_device_unrealize(DeviceState *dev)
 static void
 vhost_vdpa_device_get_config(VirtIODevice *vdev, uint8_t *config)
 {
-    return;
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+
+    memcpy(config, s->config, s->config_size);
 }
 
 static void
 vhost_vdpa_device_set_config(VirtIODevice *vdev, const uint8_t *config)
 {
-    return;
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    int ret;
+
+    ret = vhost_dev_set_config(&s->dev, s->config, 0, s->config_size,
+                               VHOST_SET_CONFIG_TYPE_MASTER);
+    if (ret) {
+        error_report("set device config space failed");
+        return;
+    }
 }
 
 static uint64_t vhost_vdpa_device_get_features(VirtIODevice *vdev,
Signature · 1 line
2.23.0
#27 lore raw

RFC 3/10 vdpa: add the infrastructure of vdpa-dev

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Add the infrastructure of vdpa-dev (the generic vDPA device), we can add a generic vDPA device as follow:
-device vhost-vdpa-device-pci,vdpa-dev=/dev/vhost-vdpa-X

---
 hw/virtio/Kconfig            |  5 ++++
 hw/virtio/meson.build        |  2 ++
 hw/virtio/vdpa-dev-pci.c     | 51 ++++++++++++++++++++++++++++++++++++
 hw/virtio/vdpa-dev.c         | 41 +++++++++++++++++++++++++++++
 include/hw/virtio/vdpa-dev.h | 16 +++++++++++
 5 files changed, 115 insertions(+)
 create mode 100644 hw/virtio/vdpa-dev-pci.c
 create mode 100644 hw/virtio/vdpa-dev.c
 create mode 100644 include/hw/virtio/vdpa-dev.h
hw/virtio/Kconfig +5 -0
index c144d42f9b..2723283382 100644
@@ -68,3 +68,8 @@ config VHOST_USER_RNG
     bool
     default y
     depends on VIRTIO && VHOST_USER
+
+config VHOST_VDPA_DEV
+    bool
+    default y if VIRTIO_PCI
+    depends on VIRTIO && VHOST_VDPA && LINUX
hw/virtio/meson.build +2 -0
index 521f7d64a8..8e8943e20b 100644
@@ -29,6 +29,7 @@ virtio_ss.add(when: 'CONFIG_VHOST_USER_I2C', if_true: files('vhost-user-i2c.c'))
 virtio_ss.add(when: ['CONFIG_VIRTIO_PCI', 'CONFIG_VHOST_USER_I2C'], if_true: files('vhost-user-i2c-pci.c'))
 virtio_ss.add(when: 'CONFIG_VHOST_USER_RNG', if_true: files('vhost-user-rng.c'))
 virtio_ss.add(when: ['CONFIG_VHOST_USER_RNG', 'CONFIG_VIRTIO_PCI'], if_true: files('vhost-user-rng-pci.c'))
+virtio_ss.add(when: 'CONFIG_VHOST_VDPA_DEV', if_true: files('vdpa-dev.c'))
 
 virtio_pci_ss = ss.source_set()
 virtio_pci_ss.add(when: 'CONFIG_VHOST_VSOCK', if_true: files('vhost-vsock-pci.c'))
@@ -49,6 +50,7 @@ virtio_pci_ss.add(when: 'CONFIG_VIRTIO_SERIAL', if_true: files('virtio-serial-pc
 virtio_pci_ss.add(when: 'CONFIG_VIRTIO_PMEM', if_true: files('virtio-pmem-pci.c'))
 virtio_pci_ss.add(when: 'CONFIG_VIRTIO_IOMMU', if_true: files('virtio-iommu-pci.c'))
 virtio_pci_ss.add(when: 'CONFIG_VIRTIO_MEM', if_true: files('virtio-mem-pci.c'))
+virtio_pci_ss.add(when: 'CONFIG_VHOST_VDPA_DEV', if_true: files('vdpa-dev-pci.c'))
 
 virtio_ss.add_all(when: 'CONFIG_VIRTIO_PCI', if_true: virtio_pci_ss)
 
hw/virtio/vdpa-dev-pci.c new file +51 -0

51 lines not on this page · raw message

hw/virtio/vdpa-dev.c new file +41 -0

41 lines not on this page · raw message

include/hw/virtio/vdpa-dev.h new file +16 -0
new file mode 100644
index 0000000000..dd94bd74a2
@@ -0,0 +1,16 @@
+#ifndef _VHOST_VDPA_DEVICE_H
+#define _VHOST_VDPA_DEVICE_H
+
+#include "hw/virtio/vhost.h"
+#include "hw/virtio/vhost-vdpa.h"
+#include "qom/object.h"
+
+
+#define TYPE_VHOST_VDPA_DEVICE "vhost-vdpa-device"
+OBJECT_DECLARE_SIMPLE_TYPE(VhostVdpaDevice, VHOST_VDPA_DEVICE)
+
+struct VhostVdpaDevice {
+    VirtIODevice parent_obj;
+};
+
+#endif
Signature · 1 line
2.23.0
#43 lore raw

RFC 6/10 vdpa-dev: implement the unrealize interface

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Implements the .unrealize interface.

---
 hw/virtio/vdpa-dev.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)
hw/virtio/vdpa-dev.c +21 -1
index 2d534d837a..4e4dd3d201 100644
@@ -133,9 +133,29 @@ out:
     close(fd);
 }
 
+static void vhost_vdpa_vdev_unrealize(VhostVdpaDevice *s)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(s);
+    int i;
+
+    for (i = 0; i < s->num_queues; i++) {
+        virtio_delete_queue(s->virtqs[i]);
+    }
+    g_free(s->virtqs);
+    virtio_cleanup(vdev);
+
+    g_free(s->config);
+}
+
 static void vhost_vdpa_device_unrealize(DeviceState *dev)
 {
-    return;
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+
+    virtio_set_status(vdev, 0);
+    vhost_dev_cleanup(&s->dev);
+    vhost_vdpa_vdev_unrealize(s);
+    close(s->vdpa.device_fd);
 }
 
 static void
Signature · 1 line
2.23.0
#50 lore raw

RFC 10/10 vdpa-dev: mark the device as unmigratable

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

The generic vDPA device doesn't support migration currently, so mark it as unmigratable temporarily.

---
 hw/virtio/vdpa-dev.c | 1 +
 1 file changed, 1 insertion(+)
hw/virtio/vdpa-dev.c +1 -0
index 64649bfb5a..0644aace22 100644
@@ -305,6 +305,7 @@ static Property vhost_vdpa_device_properties[] = {
 
 static const VMStateDescription vmstate_vhost_vdpa_device = {
     .name = "vhost-vdpa-device",
+    .unmigratable = 1,
     .minimum_version_id = 1,
     .version_id = 1,
     .fields = (VMStateField[]) {
Signature · 1 line
2.23.0
#49 lore raw

RFC 9/10 vdpa-dev: implement the set_status interface

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Implements the .set_status interface.

---
 hw/virtio/vdpa-dev.c | 100 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 99 insertions(+), 1 deletion(-)
hw/virtio/vdpa-dev.c +99 -1
index 32b3117c4b..64649bfb5a 100644
@@ -194,9 +194,107 @@ static uint64_t vhost_vdpa_device_get_features(VirtIODevice *vdev,
     return backend_features;
 }
 
+static int vhost_vdpa_device_start(VirtIODevice *vdev, Error **errp)
+{
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
+    VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
+    int i, ret;
+
+    if (!k->set_guest_notifiers) {
+        error_setg(errp, "binding does not support guest notifiers");
+        return -ENOSYS;
+    }
+
+    ret = vhost_dev_enable_notifiers(&s->dev, vdev);
+    if (ret < 0) {
+        error_setg_errno(errp, -ret, "Error enabling host notifiers");
+        return ret;
+    }
+
+    ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, true);
+    if (ret < 0) {
+        error_setg_errno(errp, -ret, "Error binding guest notifier");
+        goto err_host_notifiers;
+    }
+
+    s->dev.acked_features = vdev->guest_features;
+
+    ret = vhost_dev_start(&s->dev, vdev);
+    if (ret < 0) {
+        error_setg_errno(errp, -ret, "Error starting vhost");
+        goto err_guest_notifiers;
+    }
+    s->started = true;
+
+    /*
+     * guest_notifier_mask/pending not used yet, so just unmask
+     * everything here. virtio-pci will do the right thing by
+     * enabling/disabling irqfd.
+     */
+    for (i = 0; i < s->dev.nvqs; i++) {
+        vhost_virtqueue_mask(&s->dev, vdev, i, false);
+    }
+
+    return ret;
+
+err_guest_notifiers:
+    k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false);
+err_host_notifiers:
+    vhost_dev_disable_notifiers(&s->dev, vdev);
+    return ret;
+}
+
+static void vhost_vdpa_device_stop(VirtIODevice *vdev)
+{
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
+    VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
+    int ret;
+
+    if (!s->started) {
+        return;
+    }
+    s->started = false;
+
+    if (!k->set_guest_notifiers) {
+        return;
+    }
+
+    vhost_dev_stop(&s->dev, vdev);
+
+    ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false);
+    if (ret < 0) {
+        error_report("vhost guest notifier cleanup failed: %d", ret);
+        return;
+    }
+
+    vhost_dev_disable_notifiers(&s->dev, vdev);
+}
+
 static void vhost_vdpa_device_set_status(VirtIODevice *vdev, uint8_t status)
 {
-    return;
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    bool should_start = virtio_device_started(vdev, status);
+    Error *local_err = NULL;
+    int ret;
+
+    if (!vdev->vm_running) {
+        should_start = false;
+    }
+
+    if (s->started == should_start) {
+        return;
+    }
+
+    if (should_start) {
+        ret = vhost_vdpa_device_start(vdev, &local_err);
+        if (ret < 0) {
+            error_reportf_err(local_err, "vhost-vdpa-device: start failed: ");
+        }
+    } else {
+        vhost_vdpa_device_stop(vdev);
+    }
 }
 
 static Property vhost_vdpa_device_properties[] = {
Signature · 1 line
2.23.0
#48 lore raw

RFC 8/10 vdpa-dev: implement the get_features interface

In reply to Longpeng(Mike) via

From: Longpeng <[email protected]>

Implements the .get_features interface.

---
 hw/virtio/vdpa-dev.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
hw/virtio/vdpa-dev.c +8 -1
index 4f97a7521b..32b3117c4b 100644
@@ -184,7 +184,14 @@ static uint64_t vhost_vdpa_device_get_features(VirtIODevice *vdev,
                                                uint64_t features,
                                                Error **errp)
 {
-    return (uint64_t)-1;
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    uint64_t backend_features = s->dev.features;
+
+    if (!virtio_has_feature(features, VIRTIO_F_IOMMU_PLATFORM)) {
+        virtio_clear_feature(&backend_features, VIRTIO_F_IOMMU_PLATFORM);
+    }
+
+    return backend_features;
 }
 
 static void vhost_vdpa_device_set_status(VirtIODevice *vdev, uint8_t status)
Signature · 1 line
2.23.0
#12 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Longpeng(Mike) via
Quoted 5 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent. And it reveals device implementation details which block (cross vendor) migration.

Thanks

Quoted 30 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#2 lore raw

Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

In reply to Longpeng(Mike) via
Quoted 28 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f)
     return 0;

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 224

Quoted 10 lines
hw/virtio/virtio-pci.c +8 -0
 }
 
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {

Any way to get rid of this array? E.g using the algorithm that is used by the kernel virtio driver.

Thanks

Quoted 119 lines
hw/virtio/virtio-pci.c +84 -0
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1674,6 +1764,9 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp)
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID, virtio_bus_get_vdev_id(bus));
+        if (proxy->pdev_id) {
+            pci_config_set_device_id(config, proxy->pdev_id);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
hw/virtio/virtio-pci.h +4 -1
index 2446dcd9ae..06aa59436e 100644
@@ -146,6 +146,7 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    uint16_t pdev_id;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -158,6 +159,9 @@ struct VirtIOPCIProxy {
     VirtioBusState bus;
 };
 
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 static inline bool virtio_pci_modern(VirtIOPCIProxy *proxy)
 {
     return !proxy->disable_modern;
--

2.23.0

#3 lore raw

RE: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

Quoted 47 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 12:38 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; mst <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n,

Quoted 27 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n,

QEMUFile *f)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
     return 0;
 }
 
Quoted 12 lines

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 224

Quoted 8 lines
hw/virtio/virtio-pci.c +8 -0
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {

Any way to get rid of this array? E.g using the algorithm that is used by the kernel virtio driver.

For device id, we can use the algorithm if we no need to support Transitional id. But how to get the class id ?

Quoted 123 lines

Thanks

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d,

Quoted 85 lines
hw/virtio/virtio-pci.c +81 -0
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1674,6 +1764,9 @@ static void virtio_pci_device_plugged(DeviceState *d,

Error **errp)

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d, Error **errp)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID,

virtio_bus_get_vdev_id(bus));

Quoted 31 lines
hw/virtio/virtio-pci.c +3 -0
+        if (proxy->pdev_id) {
+            pci_config_set_device_id(config, proxy->pdev_id);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
hw/virtio/virtio-pci.h +4 -1
index 2446dcd9ae..06aa59436e 100644
@@ -146,6 +146,7 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    uint16_t pdev_id;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -158,6 +159,9 @@ struct VirtIOPCIProxy {
     VirtioBusState bus;
 };
 
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 static inline bool virtio_pci_modern(VirtIOPCIProxy *proxy)
 {
     return !proxy->disable_modern;
--

2.23.0

#4 lore raw

Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

Quoted 61 lines · Cloud Infrastructure Service Product Dept.)
Quoted 48 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 12:38 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; mst <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n,

Quoted 27 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n,

QEMUFile *f)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
     return 0;
 }
 
Quoted 12 lines

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 224

Quoted 8 lines
hw/virtio/virtio-pci.c +8 -0
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {

Any way to get rid of this array? E.g using the algorithm that is used by the kernel virtio driver.

Quoted 2 lines

For device id, we can use the algorithm if we no need to support Transitional id. But how to get the class id ?

Right, I miss this. So the current code should be fine.

Thanks

Quoted 125 lines
Quoted 124 lines

Thanks

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d,

Quoted 85 lines
hw/virtio/virtio-pci.c +81 -0
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1674,6 +1764,9 @@ static void virtio_pci_device_plugged(DeviceState *d,

Error **errp)

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d, Error **errp)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID,

virtio_bus_get_vdev_id(bus));

Quoted 31 lines
hw/virtio/virtio-pci.c +3 -0
+        if (proxy->pdev_id) {
+            pci_config_set_device_id(config, proxy->pdev_id);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
hw/virtio/virtio-pci.h +4 -1
index 2446dcd9ae..06aa59436e 100644
@@ -146,6 +146,7 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    uint16_t pdev_id;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -158,6 +159,9 @@ struct VirtIOPCIProxy {
     VirtioBusState bus;
 };
 
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 static inline bool virtio_pci_modern(VirtIOPCIProxy *proxy)
 {
     return !proxy->disable_modern;
--

2.23.0

#13 lore raw

RE: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Jason Wang
Quoted 16 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 12:36 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; mst <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 1 line · Longpeng(Mike)
Quoted 9 lines
Quoted 4 lines

From: Longpeng <[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent. And it reveals device implementation details which block (cross vendor) migration.

Can we set the VirtIOPCIProxy.nvectors to "the count of virtqueues + 1 (config)" ?

Quoted 32 lines

Thanks

Quoted 30 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#14 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 16 lines · Cloud Infrastructure Service Product Dept.)
Quoted 16 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 12:36 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; mst <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 2 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent. And it reveals device implementation details which block (cross vendor) migration.

Can we set the VirtIOPCIProxy.nvectors to "the count of virtqueues + 1 (config)" ?

That should work.

Thanks

Quoted 33 lines
Quoted 33 lines

Thanks

Quoted 30 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
  linux-headers/linux/vhost.h | 10 ++++++++++
  1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
  /* Get the valid iova range */
  #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                              struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
  #endif
--

2.23.0

#15 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Jason Wang
Quoted 9 lines · Jason Wang
Quoted 5 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you? Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Not necessarily, userspace can hide this from guest if it wants to, just validate.

Quoted 30 lines
Quoted 30 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#16 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 6 lines · Michael S. Tsirkin
Quoted 4 lines · Jason Wang
Quoted 3 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 9 lines

Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#17 lore raw

RE: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 38 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 3:54 PM
To: Michael S. Tsirkin <[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 6 lines · Michael S. Tsirkin
Quoted 4 lines · Jason Wang
Quoted 3 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 5 lines

Otherwise guests will at best be suboptimal.

Quoted 3 lines

And it reveals device implementation
details which block (cross vendor) migration.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Thanks

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Not only MSI vectors, but also queue-size, #vqs, etc.

Maybe the vhost level could expose the hardware's real capabilities and let the userspace (QEMU) do the hiding? The userspace know how to process them.

Quoted 35 lines

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#18 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 41 lines · Cloud Infrastructure Service Product Dept.)
Quoted 41 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 3:54 PM
To: Michael S. Tsirkin <[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 6 lines · Michael S. Tsirkin
Quoted 4 lines · Jason Wang
Quoted 3 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 7 lines

Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Not only MSI vectors, but also queue-size, #vqs, etc.

MSI is PCI specific, we have non PCI vDPA parent e.g VDUSE/simulator/mlx5

And it's something that is not guaranteed to be not changed. E.g some drivers may choose to allocate MSI during set_status() which can fail for various reasons.

Quoted 4 lines

Maybe the vhost level could expose the hardware's real capabilities and let the userspace (QEMU) do the hiding? The userspace know how to process them.

#MSI vectors is much more easier to be mediated than queue-size and #vqs.

For interrupts, we've already had VHOST_SET_X_KICK, we can keep allocating eventfd based on #MSI vectors to make it work with any number of MSI vectors that the virtual device had.

For queue-size, it's Ok to have a new uAPI but it's not a must, Qemu can simply fail if SET_VRING_NUM fail.

For #vqs, it's OK to have a new uAPI since the emulated virtio-pci device requires knowledge the #vqs in the config space. (still not a must, we can enumerate #vqs per device type)

For the config size, it's OK but not a must, technically we can simply relay what guest write to vhost-vdpa. It's just because current Qemu require to have it during virtio device initialization.

Thanks

Quoted 37 lines
Quoted 36 lines

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#25 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Jason Wang
Quoted 10 lines · Jason Wang
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 5 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Some hardware may be more limited. E.g. there could be a bunch of vqs which don't need a dedicated interrupt. Or device could support a single interrupt shared between VQs.

Quoted 47 lines
Quoted 9 lines

Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#26 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 18 lines · Michael S. Tsirkin
Quoted 16 lines · Jason Wang
Quoted 14 lines · Michael S. Tsirkin
Quoted 9 lines · Jason Wang
Quoted 5 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

Quoted 6 lines
Quoted 2 lines

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Some hardware may be more limited. E.g. there could be a bunch of vqs which don't need a dedicated interrupt. Or device could support a single interrupt shared between VQs.

Right, but in the worst case, we may just waste some eventfds?

Or if we want to be optimized in the case you mentioned above, we need to know the association among vectors and vqs which requires more extensions.

1) API to know which vectors does a dedicated VQ belong
2) the kick eventfd is set based on the vectors but not vq

And such mappings are not static, e.g IFCVF requesting MSI-X vectors during DRIVER_OK.

Thanks

Quoted 50 lines
Quoted 49 lines
Quoted 11 lines

Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#28 lore raw

Re: [RFC 03/10] vdpa: add the infrastructure of vdpa-dev

In reply to Longpeng(Mike) via

hw/virtio/vdpa-dev-pci.c line 40

Quoted 5 lines · Longpeng(Mike)
hw/virtio/vdpa-dev-pci.c new file +5 -0
+static const VirtioPCIDeviceTypeInfo vhost_vdpa_device_pci_info = {
+    .base_name               = TYPE_VHOST_VDPA_DEVICE_PCI,
+    .generic_name            = "vhost-vdpa-device-pci",
+    .transitional_name       = "vhost-vdpa-device-pci-transitional",
+    .non_transitional_name   = "vhost-vdpa-device-pci-non-transitional",

Does vDPA support Transitional VIRTIO devices?

I expected this device to support Modern devices only.

Stefan

#33 lore raw

Re: [RFC 04/10] vdpa-dev: implement the instance_init/class_init interface

In reply to Longpeng(Mike) via
Quoted 16 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .instance_init and the .class_init interface.

---
 hw/virtio/vdpa-dev-pci.c     | 80 +++++++++++++++++++++++++++++++++++-
 hw/virtio/vdpa-dev.c         | 68 +++++++++++++++++++++++++++++-
 include/hw/virtio/vdpa-dev.h |  2 +
 3 files changed, 146 insertions(+), 4 deletions(-)
hw/virtio/vdpa-dev-pci.c +0 -0
index a5a7b528a9..0af54a26d4 100644

hw/virtio/vdpa-dev-pci.c struct VhostVdpaDevicePCI { · line 27

Quoted 5 lines
hw/virtio/vdpa-dev-pci.c +2 -0
@@ -23,14 +23,90 @@ struct VhostVdpaDevicePCI {
     VhostVdpaDevice vdev;
 };
 
+static uint32_t
+vdpa_dev_pci_get_info(const char *name, uint64_t cmd, Error **errp)

vdpa_dev_pci_get_u32() might be a clearer name.

Quoted 9 lines
hw/virtio/vdpa-dev-pci.c +9 -0
+{
+    int device_fd;
+    uint32_t val;
+    int ret;
+
+    device_fd = qemu_open(name, O_RDWR, errp);
+    if (device_fd == -1) {
+        return (uint32_t)-1;
+    }

hw/virtio/vdpa-dev-pci.c struct VhostVdpaDevicePCI { · line 46

Quoted 10 lines
hw/virtio/vdpa-dev-pci.c +10 -0
+
+    ret = ioctl(device_fd, cmd, &val);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device-pci: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        goto out;
+    }
+
+out:
+    close(device_fd);

Race conditions are possible if the device node is replaced between calls. Why not open the file once and reuse the fd across ioctl calls?

Both VhostVdpaDevicePCI and VhostVdpaDevice need the fd but VhostVdpaDevicePCI needs it first. I suggest passing ownership of the fd to the VhostVdpaDevice. One way of doing this is using QOM properties so that VhostVdpaDevice can use the given fd instead of reopening the file. (If fd is -1 then VhostVdpaDevice can try to open the file. That is necessary when VhostVdpaDevice is used directly with virtio-mmio because there is no proxy object.)

#38 lore raw

Re: [RFC 05/10] vdpa-dev: implement the realize interface

In reply to Longpeng(Mike) via
Quoted 15 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .realize interface.

---
 hw/virtio/vdpa-dev.c         | 114 +++++++++++++++++++++++++++++++++++
 include/hw/virtio/vdpa-dev.h |   8 +++
 2 files changed, 122 insertions(+)
hw/virtio/vdpa-dev.c +0 -0
index 790117fb3b..2d534d837a 100644

hw/virtio/vdpa-dev.c line 24

Quoted 10 lines
hw/virtio/vdpa-dev.c +7 -0
@@ -15,9 +15,122 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 
+static void
+vhost_vdpa_device_dummy_handle_output(VirtIODevice *vdev, VirtQueue *vq)
+{
+    /* Nothing to do */
+}
+
+static int vdpa_dev_get_info_by_fd(int fd, uint64_t cmd, Error **errp)

This looks similar to the helper function in a previous patch but this time the return value type is int instead of uint32_t. Please make the types consistent.

Quoted 31 lines
hw/virtio/vdpa-dev.c +29 -0
+{
+    int val;
+
+    if (ioctl(fd, cmd, &val) < 0) {
+        error_setg(errp, "vhost-vdpa-device: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        return -1;
+    }
+
+    return val;
+}
+
+static inline int vdpa_dev_get_queue_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VRING_NUM, errp);
+}
+
+static inline int vdpa_dev_get_vqs_num(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VQS_NUM, errp);
+}
+
+static inline int vdpa_dev_get_config_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_CONFIG_SIZE, errp);
+}
+
 static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
 {
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);

hw/virtio/vdpa-dev.c line 65

Quoted 10 lines
hw/virtio/vdpa-dev.c +10 -0
+    uint32_t device_id;
+    int max_queue_size;
+    int fd;
+    int i, ret;
+
+    fd = qemu_open(s->vdpa_dev, O_RDWR, errp);
+    if (fd == -1) {
+        return;
+    }
+    s->vdpa.device_fd = fd;

This is the field I suggest exposing as a QOM property so it can be set from the proxy object (e.g. when the PCI proxy opens the vdpa device before our .realize() function is called).

Quoted 10 lines
hw/virtio/vdpa-dev.c +10 -0
+
+    max_queue_size = vdpa_dev_get_queue_size(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
+    if (s->queue_size > max_queue_size) {
+        error_setg(errp, "vhost-vdpa-device: invalid queue_size: %d (max:%d)",
+                   s->queue_size, max_queue_size);
+        goto out;

hw/virtio/vdpa-dev.c line 85

Quoted 10 lines
hw/virtio/vdpa-dev.c +10 -0
+    } else if (!s->queue_size) {
+        s->queue_size = max_queue_size;
+    }
+
+    ret = vdpa_dev_get_vqs_num(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
+    s->dev.nvqs = ret;

There is no input validation because we trust the kernel vDPA return values. That seems okay for now but if there is a vhost-user version of this in the future then input validation will be necessary to achieve isolation between QEMU and the vhost-user processes. I suggest including input validation code right away because it's harder to audit the code and fix missing input validation later on.

Quoted 4 lines
hw/virtio/vdpa-dev.c +4 -0
+    s->dev.vqs = g_new0(struct vhost_virtqueue, s->dev.nvqs);
+    s->dev.vq_index = 0;
+    s->dev.vq_index_end = s->dev.nvqs;
+    s->dev.backend_features = 0;

hw/virtio/vdpa-dev.c line 99

Quoted 10 lines
hw/virtio/vdpa-dev.c +10 -0
+    s->started = false;
+
+    ret = vhost_dev_init(&s->dev, &s->vdpa, VHOST_BACKEND_TYPE_VDPA, 0, NULL);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device: vhost initialization failed: %s",
+                   strerror(-ret));
+        goto out;
+    }
+
+    ret = s->dev.vhost_ops->vhost_get_device_id(&s->dev, &device_id);

The vhost_*() API abstracts the ioctl calls but this source file and the PCI proxy have ioctl calls. I wonder if it's possible to move the ioctls calls into the vhost_*() API? That would be cleaner and also make it easier to add vhost-user vDPA support in the future.

#51 lore raw
In reply to Longpeng(Mike) via
Quoted 21 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Hi guys,

This patchset tries to support the generic vDPA device, the previous disscussion can be found here [1].

With the generic vDPA device, QEMU won't need to touch the device types any more, such like vfio.

We can use the generic vDPA device as follow:
-device vhost-vdpa-device-pci,vdpa-dev=/dev/vhost-vdpa-X

I've done some simple tests on Huawei's offloading card (net, 0.95) and vdpa_sim_blk (1.0);

Note:
the kernel part does not send out yet, I'll send it as soon as possible.

[1] https://lore.kernel.org/all/[email protected]/

Quoted 11 lines

Longpeng (Mike) (10):
virtio: get class_id and pci device id by the virtio id
vhost: add 3 commands for vhost-vdpa
vdpa: add the infrastructure of vdpa-dev
vdpa-dev: implement the instance_init/class_init interface
vdpa-dev: implement the realize interface
vdpa-dev: implement the unrealize interface
vdpa-dev: implement the get_config/set_config interface
vdpa-dev: implement the get_features interface
vdpa-dev: implement the set_status interface
vdpa-dev: mark the device as unmigratable

Nice and small.

Stefan

#6 lore raw

Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

In reply to Longpeng(Mike) via
Quoted 107 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +79 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f)
     return 0;
 }
 
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 304

Quoted 10 lines
hw/virtio/virtio-pci.c +10 -0
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}

What happens if these functions are called for a device_id that is not in the array, e.g. if we forgot to add a new id to the array?

Can the array be generated in some way?

#44 lore raw

Re: [RFC 06/10] vdpa-dev: implement the unrealize interface

In reply to Longpeng(Mike) via
Quoted 14 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .unrealize interface.

---
hw/virtio/vdpa-dev.c | 22 +++++++++++++++++++++-

1 file changed, 21 insertions(+), 1 deletion(-)
hw/virtio/vdpa-dev.c +0 -0
index 2d534d837a..4e4dd3d201 100644

hw/virtio/vdpa-dev.c out: · line 141

Quoted 9 lines
hw/virtio/vdpa-dev.c +6 -0
@@ -133,9 +133,29 @@ out:
    close(fd);
 }
 
+static void vhost_vdpa_vdev_unrealize(VhostVdpaDevice *s)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(s);
+    int i;
+
+    for (i = 0; i < s->num_queues; i++) {
                      ^

s->num_queues seems uninitialized to me, maybe we could just remove the num_queues field from VhostVdpaDevice, and use s->dev.nvqs as in vhost_vdpa_device_realize().

Quoted 6 lines
hw/virtio/vdpa-dev.c +6 -0
+        virtio_delete_queue(s->virtqs[i]);
+    }
+    g_free(s->virtqs);
+    virtio_cleanup(vdev);
+
+    g_free(s->config);

hw/virtio/vdpa-dev.c out: · line 156

Quoted 10 lines
hw/virtio/vdpa-dev.c +7 -1
+}
+
 static void vhost_vdpa_device_unrealize(DeviceState *dev)
 {
-    return;
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+
+    virtio_set_status(vdev, 0);
+    vhost_dev_cleanup(&s->dev);

If we will use s->dev.nvqs in vhost_vdpa_vdev_unrealize(), we should call vhost_dev_cleanup() after it, just before close() as we already do in the error path of vhost_vdpa_device_realize().

Quoted 8 lines
hw/virtio/vdpa-dev.c +2 -0
+    vhost_vdpa_vdev_unrealize(s);
+    close(s->vdpa.device_fd);
 }
 

static void

Signature · 1 line
2.23.0
#35 lore raw

Re: [RFC 04/10] vdpa-dev: implement the instance_init/class_init interface

In reply to Longpeng(Mike) via
Quoted 164 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .instance_init and the .class_init interface.

---

hw/virtio/vdpa-dev-pci.c     | 80 +++++++++++++++++++++++++++++++++++-
hw/virtio/vdpa-dev.c         | 68 +++++++++++++++++++++++++++++-
include/hw/virtio/vdpa-dev.h |  2 +
3 files changed, 146 insertions(+), 4 deletions(-)
hw/virtio/vdpa-dev-pci.c +78 -2
index a5a7b528a9..0af54a26d4 100644
@@ -23,14 +23,90 @@ struct VhostVdpaDevicePCI {
    VhostVdpaDevice vdev;
 };
 
+static uint32_t
+vdpa_dev_pci_get_info(const char *name, uint64_t cmd, Error **errp)
+{
+    int device_fd;
+    uint32_t val;
+    int ret;
+
+    device_fd = qemu_open(name, O_RDWR, errp);
+    if (device_fd == -1) {
+        return (uint32_t)-1;
+    }
+
+    ret = ioctl(device_fd, cmd, &val);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device-pci: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        goto out;
+    }
+
+out:
+    close(device_fd);
+    return val;
+}
+
+static inline uint32_t
+vdpa_dev_pci_get_devid(VhostVdpaDevicePCI *dev, Error **errp)
+{
+    return vdpa_dev_pci_get_info(dev->vdev.vdpa_dev,
+                                 VHOST_VDPA_GET_DEVICE_ID, errp);
+}
+
+static inline uint32_t
+vdpa_dev_pci_get_vectors_num(VhostVdpaDevicePCI *dev, Error **errp)
+{
+    return vdpa_dev_pci_get_info(dev->vdev.vdpa_dev,
+                                 VHOST_VDPA_GET_VECTORS_NUM, errp);
+}
+
 static void vhost_vdpa_device_pci_instance_init(Object *obj)
 {
-    return;
+    VhostVdpaDevicePCI *dev = VHOST_VDPA_DEVICE_PCI(obj);
+
+    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
+                                TYPE_VHOST_VDPA_DEVICE);
+    object_property_add_alias(obj, "bootindex", OBJECT(&dev->vdev),
+                              "bootindex");
+}
+
+static Property vhost_vdpa_device_pci_properties[] = {
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void
+vhost_vdpa_device_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
+{
+    VhostVdpaDevicePCI *dev = VHOST_VDPA_DEVICE_PCI(vpci_dev);
+    DeviceState *vdev = DEVICE(&dev->vdev);
+    uint32_t devid;
+    uint32_t vectors;
+
+    devid = vdpa_dev_pci_get_devid(dev, errp);
+    if (*errp) {
+        return;
+    }
+
+    vectors = vdpa_dev_pci_get_vectors_num(dev, errp);
+    if (*errp) {
+        return;
+    }
+
+    vpci_dev->class_code = virtio_pci_get_class_id(devid);
+    vpci_dev->pdev_id = virtio_pci_get_pci_devid(devid);
+    vpci_dev->nvectors = vectors;
+    qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
 }
 
 static void vhost_vdpa_device_pci_class_init(ObjectClass *klass, void *data)
 {
-    return;
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
+
+    set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+    device_class_set_props(dc, vhost_vdpa_device_pci_properties);
+    k->realize = vhost_vdpa_device_pci_realize;
 }
 
 static const VirtioPCIDeviceTypeInfo vhost_vdpa_device_pci_info = {
hw/virtio/vdpa-dev.c +38 -2
index f4f92b90b0..790117fb3b 100644
@@ -15,16 +15,80 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 
-static void vhost_vdpa_device_class_init(ObjectClass *klass, void *data)
+static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
 {
    return;
 }
 
-static void vhost_vdpa_device_instance_init(Object *obj)
+static void vhost_vdpa_device_unrealize(DeviceState *dev)
+{
+    return;
+}
+
+static void
+vhost_vdpa_device_get_config(VirtIODevice *vdev, uint8_t *config)
+{
+    return;
+}
+
+static void
+vhost_vdpa_device_set_config(VirtIODevice *vdev, const uint8_t *config)
 {
    return;
 }
 
+static uint64_t vhost_vdpa_device_get_features(VirtIODevice *vdev,
+                                               uint64_t features,
+                                               Error **errp)
+{
+    return (uint64_t)-1;
+}
+
+static void vhost_vdpa_device_set_status(VirtIODevice *vdev, uint8_t status)
+{
+    return;
+}
+
+static Property vhost_vdpa_device_properties[] = {
+    DEFINE_PROP_STRING("vdpa-dev", VhostVdpaDevice, vdpa_dev),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static const VMStateDescription vmstate_vhost_vdpa_device = {
+    .name = "vhost-vdpa-device",
+    .minimum_version_id = 1,
+    .version_id = 1,
+    .fields = (VMStateField[]) {
+        VMSTATE_VIRTIO_DEVICE,
+        VMSTATE_END_OF_LIST()

hw/virtio/vdpa-dev.c line 73

Quoted 10 lines
hw/virtio/vdpa-dev.c +10 -0
+    },
+};
+
+static void vhost_vdpa_device_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
+
+    device_class_set_props(dc, vhost_vdpa_device_properties);
+    dc->desc = "VDPA-based generic PCI device assignment";

IIUC, this should be the description of the generic vhost vDPA device, not the PCI implementation, right?

Thanks,
Stefano

#19 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Jason Wang
Quoted 62 lines · Jason Wang
Quoted 41 lines · Cloud Infrastructure Service Product Dept.)
Quoted 41 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 3:54 PM
To: Michael S. Tsirkin <[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 6 lines · Michael S. Tsirkin
Quoted 4 lines · Jason Wang
Quoted 3 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 7 lines

Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Not only MSI vectors, but also queue-size, #vqs, etc.

MSI is PCI specific, we have non PCI vDPA parent e.g VDUSE/simulator/mlx5

And it's something that is not guaranteed to be not changed. E.g some drivers may choose to allocate MSI during set_status() which can fail for various reasons.

Quoted 4 lines

Maybe the vhost level could expose the hardware's real capabilities and let the userspace (QEMU) do the hiding? The userspace know how to process them.

Quoted 5 lines

#MSI vectors is much more easier to be mediated than queue-size and #vqs.

For interrupts, we've already had VHOST_SET_X_KICK, we can keep allocating eventfd based on #MSI vectors to make it work with any number of MSI vectors that the virtual device had.

Right but if hardware does not support so many then what? Just fail? Having a query API would make things somewhat cleaner imho.

Quoted 7 lines

For queue-size, it's Ok to have a new uAPI but it's not a must, Qemu can simply fail if SET_VRING_NUM fail.

For #vqs, it's OK to have a new uAPI since the emulated virtio-pci device requires knowledge the #vqs in the config space. (still not a must, we can enumerate #vqs per device type)

Quoted 5 lines

For the config size, it's OK but not a must, technically we can simply relay what guest write to vhost-vdpa. It's just because current Qemu require to have it during virtio device initialization.

Thanks

I agree but these ok things make for a cleaner API I think.

Quoted 37 lines
Quoted 37 lines
Quoted 36 lines

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#29 lore raw

RE: [RFC 03/10] vdpa: add the infrastructure of vdpa-dev

In reply to Stefan Hajnoczi
Quoted 20 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 5:49 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 03/10] vdpa: add the infrastructure of vdpa-dev

hw/virtio/vdpa-dev-pci.c line 40

Quoted 5 lines · Longpeng(Mike)
hw/virtio/vdpa-dev-pci.c new file +5 -0
+static const VirtioPCIDeviceTypeInfo vhost_vdpa_device_pci_info = {
+    .base_name               = TYPE_VHOST_VDPA_DEVICE_PCI,
+    .generic_name            = "vhost-vdpa-device-pci",
+    .transitional_name       = "vhost-vdpa-device-pci-transitional",
+    .non_transitional_name   = "vhost-vdpa-device-pci-non-transitional",
Quoted 4 lines

Does vDPA support Transitional VIRTIO devices?

I expected this device to support Modern devices only.

There's already a 0.95 vdpa driver (Alibaba ENI) in the kernel source and supporting 0.95 devices is necessary for some older GuestOS.

I'm OK if other guys also approve of supporting 1.0+ devices only :)

Quoted 1 line

Stefan

#7 lore raw

RE: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

In reply to Cornelia Huck
Quoted 130 lines

-----Original Message-----
From: Cornelia Huck [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 6:46 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]; Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Subject: Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n,

Quoted 26 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n,

QEMUFile *f)

Quoted 86 lines
hw/virtio/virtio-pci.c +83 -0
     return 0;
 }
 
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
Quoted 9 lines

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 304

Quoted 5 lines
hw/virtio/virtio-pci.c +5 -0
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}

What happens if these functions are called for a device_id that is not in the array, e.g. if we forgot to add a new id to the array?

It would return pdev_id=0 or class_id=0 as a result, the virtio device with pdev_id=0 is undefined and class_id=0 is also treated as undefined (PCI_CLASS_NOT_DEFINED), so the caller should check the returned value.

Quoted 1 line

Can the array be generated in some way?

For PCI Device ID:
- If we need to support Transitional VIRTIO devices, there's no algorithm can map a VIRTIO ID to a PCI Device ID.
- If we only need to support 1.0+ VIRTIO devices, then we can calculate the PCI Device ID based on the VIRTIO ID.

For Class ID, seems no way :(

#20 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Michael S. Tsirkin
Quoted 64 lines · Michael S. Tsirkin
Quoted 62 lines · Jason Wang
Quoted 41 lines · Cloud Infrastructure Service Product Dept.)
Quoted 41 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 3:54 PM
To: Michael S. Tsirkin <[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 6 lines · Michael S. Tsirkin
Quoted 4 lines · Jason Wang
Quoted 3 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 7 lines

Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Not only MSI vectors, but also queue-size, #vqs, etc.

MSI is PCI specific, we have non PCI vDPA parent e.g VDUSE/simulator/mlx5

And it's something that is not guaranteed to be not changed. E.g some drivers may choose to allocate MSI during set_status() which can fail for various reasons.

Quoted 4 lines

Maybe the vhost level could expose the hardware's real capabilities and let the userspace (QEMU) do the hiding? The userspace know how to process them.

Quoted 8 lines
Quoted 5 lines

#MSI vectors is much more easier to be mediated than queue-size and #vqs.

For interrupts, we've already had VHOST_SET_X_KICK, we can keep allocating eventfd based on #MSI vectors to make it work with any number of MSI vectors that the virtual device had.

Right but if hardware does not support so many then what? Just fail?

Or just trigger the callback of vqs that shares the vector.

Quoted 1 line

Having a query API would make things somewhat cleaner imho.

I may miss something, even if we know #vectors, we still don't know the associated virtqueues for a dedicated vector?

Quoted 8 lines
Quoted 8 lines

For queue-size, it's Ok to have a new uAPI but it's not a must, Qemu can simply fail if SET_VRING_NUM fail.

For #vqs, it's OK to have a new uAPI since the emulated virtio-pci device requires knowledge the #vqs in the config space. (still not a must, we can enumerate #vqs per device type)

Quoted 8 lines
Quoted 5 lines

For the config size, it's OK but not a must, technically we can simply relay what guest write to vhost-vdpa. It's just because current Qemu require to have it during virtio device initialization.

Thanks

I agree but these ok things make for a cleaner API I think.

Right.

Thanks

Quoted 39 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#34 lore raw

RE: [RFC 04/10] vdpa-dev: implement the instance_init/class_init interface

In reply to Stefan Hajnoczi
Quoted 34 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 6:01 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 04/10] vdpa-dev: implement the instance_init/class_init interface

Quoted 19 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .instance_init and the .class_init interface.

---
 hw/virtio/vdpa-dev-pci.c     | 80 +++++++++++++++++++++++++++++++++++-
 hw/virtio/vdpa-dev.c         | 68 +++++++++++++++++++++++++++++-
 include/hw/virtio/vdpa-dev.h |  2 +
 3 files changed, 146 insertions(+), 4 deletions(-)
hw/virtio/vdpa-dev-pci.c +0 -0
index a5a7b528a9..0af54a26d4 100644
@@ -23,14 +23,90 @@ struct VhostVdpaDevicePCI {
     VhostVdpaDevice vdev;
 };
 
Quoted 5 lines

hw/virtio/vdpa-dev-pci.c struct VhostVdpaDevicePCI { · line 27

Quoted 2 lines
hw/virtio/vdpa-dev-pci.c +2 -0
+static uint32_t
+vdpa_dev_pci_get_info(const char *name, uint64_t cmd, Error **errp)

vdpa_dev_pci_get_u32() might be a clearer name.

OK.

Quoted 23 lines
Quoted 9 lines
hw/virtio/vdpa-dev-pci.c +9 -0
+{
+    int device_fd;
+    uint32_t val;
+    int ret;
+
+    device_fd = qemu_open(name, O_RDWR, errp);
+    if (device_fd == -1) {
+        return (uint32_t)-1;
+    }

hw/virtio/vdpa-dev-pci.c struct VhostVdpaDevicePCI { · line 46

Quoted 10 lines
hw/virtio/vdpa-dev-pci.c +10 -0
+
+    ret = ioctl(device_fd, cmd, &val);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device-pci: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        goto out;
+    }
+
+out:
+    close(device_fd);

Race conditions are possible if the device node is replaced between calls. Why not open the file once and reuse the fd across ioctl calls?

Quoted 7 lines

Both VhostVdpaDevicePCI and VhostVdpaDevice need the fd but VhostVdpaDevicePCI needs it first. I suggest passing ownership of the fd to the VhostVdpaDevice. One way of doing this is using QOM properties so that VhostVdpaDevice can use the given fd instead of reopening the file. (If fd is -1 then VhostVdpaDevice can try to open the file. That is necessary when VhostVdpaDevice is used directly with virtio-mmio because there is no proxy object.)

Adding the fd field into the VhostVdpaDevice looks fine! I'll do it in V2. Thanks.

#36 lore raw

RE: [RFC 04/10] vdpa-dev: implement the instance_init/class_init interface

In reply to Stefano Garzarella
Quoted 184 lines

-----Original Message-----
From: Stefano Garzarella [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 7:29 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 04/10] vdpa-dev: implement the instance_init/class_init interface

Quoted 169 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .instance_init and the .class_init interface.

---

hw/virtio/vdpa-dev-pci.c     | 80 +++++++++++++++++++++++++++++++++++-
hw/virtio/vdpa-dev.c         | 68 +++++++++++++++++++++++++++++-
include/hw/virtio/vdpa-dev.h |  2 +
3 files changed, 146 insertions(+), 4 deletions(-)
hw/virtio/vdpa-dev-pci.c +78 -2
index a5a7b528a9..0af54a26d4 100644
@@ -23,14 +23,90 @@ struct VhostVdpaDevicePCI {
    VhostVdpaDevice vdev;
 };
 
+static uint32_t
+vdpa_dev_pci_get_info(const char *name, uint64_t cmd, Error **errp)
+{
+    int device_fd;
+    uint32_t val;
+    int ret;
+
+    device_fd = qemu_open(name, O_RDWR, errp);
+    if (device_fd == -1) {
+        return (uint32_t)-1;
+    }
+
+    ret = ioctl(device_fd, cmd, &val);
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device-pci: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        goto out;
+    }
+
+out:
+    close(device_fd);
+    return val;
+}
+
+static inline uint32_t
+vdpa_dev_pci_get_devid(VhostVdpaDevicePCI *dev, Error **errp)
+{
+    return vdpa_dev_pci_get_info(dev->vdev.vdpa_dev,
+                                 VHOST_VDPA_GET_DEVICE_ID, errp);
+}
+
+static inline uint32_t
+vdpa_dev_pci_get_vectors_num(VhostVdpaDevicePCI *dev, Error **errp)
+{
+    return vdpa_dev_pci_get_info(dev->vdev.vdpa_dev,
+                                 VHOST_VDPA_GET_VECTORS_NUM, errp);
+}
+
 static void vhost_vdpa_device_pci_instance_init(Object *obj)
 {
-    return;
+    VhostVdpaDevicePCI *dev = VHOST_VDPA_DEVICE_PCI(obj);
+
+    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
+                                TYPE_VHOST_VDPA_DEVICE);
+    object_property_add_alias(obj, "bootindex", OBJECT(&dev->vdev),
+                              "bootindex");
+}
+
+static Property vhost_vdpa_device_pci_properties[] = {
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void
+vhost_vdpa_device_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
+{
+    VhostVdpaDevicePCI *dev = VHOST_VDPA_DEVICE_PCI(vpci_dev);
+    DeviceState *vdev = DEVICE(&dev->vdev);
+    uint32_t devid;
+    uint32_t vectors;
+
+    devid = vdpa_dev_pci_get_devid(dev, errp);
+    if (*errp) {
+        return;
+    }
+
+    vectors = vdpa_dev_pci_get_vectors_num(dev, errp);
+    if (*errp) {
+        return;
+    }
+
+    vpci_dev->class_code = virtio_pci_get_class_id(devid);
+    vpci_dev->pdev_id = virtio_pci_get_pci_devid(devid);
+    vpci_dev->nvectors = vectors;
+    qdev_realize(vdev, BUS(&vpci_dev->bus), errp);
 }
 
 static void vhost_vdpa_device_pci_class_init(ObjectClass *klass, void *data)
 {
-    return;
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
+
+    set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+    device_class_set_props(dc, vhost_vdpa_device_pci_properties);
+    k->realize = vhost_vdpa_device_pci_realize;
 }
 
 static const VirtioPCIDeviceTypeInfo vhost_vdpa_device_pci_info = {
hw/virtio/vdpa-dev.c +43 -2
index f4f92b90b0..790117fb3b 100644
@@ -15,16 +15,80 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 
-static void vhost_vdpa_device_class_init(ObjectClass *klass, void *data)
+static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
 {
    return;
 }
 
-static void vhost_vdpa_device_instance_init(Object *obj)
+static void vhost_vdpa_device_unrealize(DeviceState *dev)
+{
+    return;
+}
+
+static void
+vhost_vdpa_device_get_config(VirtIODevice *vdev, uint8_t *config)
+{
+    return;
+}
+
+static void
+vhost_vdpa_device_set_config(VirtIODevice *vdev, const uint8_t *config)
 {
    return;
 }
 
+static uint64_t vhost_vdpa_device_get_features(VirtIODevice *vdev,
+                                               uint64_t features,
+                                               Error **errp)
+{
+    return (uint64_t)-1;
+}
+
+static void vhost_vdpa_device_set_status(VirtIODevice *vdev, uint8_t status)
+{
+    return;
+}
+
+static Property vhost_vdpa_device_properties[] = {
+    DEFINE_PROP_STRING("vdpa-dev", VhostVdpaDevice, vdpa_dev),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static const VMStateDescription vmstate_vhost_vdpa_device = {
+    .name = "vhost-vdpa-device",
+    .minimum_version_id = 1,
+    .version_id = 1,
+    .fields = (VMStateField[]) {
+        VMSTATE_VIRTIO_DEVICE,
+        VMSTATE_END_OF_LIST()
+    },
+};
+
+static void vhost_vdpa_device_class_init(ObjectClass *klass, void *data)
+{
Quoted 9 lines

hw/virtio/vdpa-dev.c line 73

Quoted 5 lines
hw/virtio/vdpa-dev.c +5 -0
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
+
+    device_class_set_props(dc, vhost_vdpa_device_properties);
+    dc->desc = "VDPA-based generic PCI device assignment";

IIUC, this should be the description of the generic vhost vDPA device, not the PCI implementation, right?

Good catch, thanks.

Quoted 2 lines

Thanks,
Stefano

#39 lore raw

RE: [RFC 05/10] vdpa-dev: implement the realize interface

In reply to Stefan Hajnoczi
Quoted 32 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 6:18 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 05/10] vdpa-dev: implement the realize interface

Quoted 18 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .realize interface.

---
 hw/virtio/vdpa-dev.c         | 114 +++++++++++++++++++++++++++++++++++
 include/hw/virtio/vdpa-dev.h |   8 +++
 2 files changed, 122 insertions(+)
hw/virtio/vdpa-dev.c +0 -0
index 790117fb3b..2d534d837a 100644
@@ -15,9 +15,122 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 
Quoted 12 lines

hw/virtio/vdpa-dev.c line 24

Quoted 7 lines
hw/virtio/vdpa-dev.c +7 -0
+static void
+vhost_vdpa_device_dummy_handle_output(VirtIODevice *vdev, VirtQueue *vq)
+{
+    /* Nothing to do */
+}
+
+static int vdpa_dev_get_info_by_fd(int fd, uint64_t cmd, Error **errp)

This looks similar to the helper function in a previous patch but this time the return value type is int instead of uint32_t. Please make the types consistent.

OK.

Quoted 37 lines
Quoted 37 lines
hw/virtio/vdpa-dev.c +35 -0
+{
+    int val;
+
+    if (ioctl(fd, cmd, &val) < 0) {
+        error_setg(errp, "vhost-vdpa-device: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        return -1;
+    }
+
+    return val;
+}
+
+static inline int vdpa_dev_get_queue_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VRING_NUM, errp);
+}
+
+static inline int vdpa_dev_get_vqs_num(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VQS_NUM, errp);
+}
+
+static inline int vdpa_dev_get_config_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_CONFIG_SIZE, errp);
+}
+
 static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
 {
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    uint32_t device_id;
+    int max_queue_size;
+    int fd;
+    int i, ret;
+
+    fd = qemu_open(s->vdpa_dev, O_RDWR, errp);
Quoted 9 lines

hw/virtio/vdpa-dev.c line 65

Quoted 4 lines
hw/virtio/vdpa-dev.c +4 -0
+    if (fd == -1) {
+        return;
+    }
+    s->vdpa.device_fd = fd;

This is the field I suggest exposing as a QOM property so it can be set from the proxy object (e.g. when the PCI proxy opens the vdpa device before our .realize() function is called).

OK.

Quoted 20 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +8 -0
+
+    max_queue_size = vdpa_dev_get_queue_size(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
+    if (s->queue_size > max_queue_size) {
+        error_setg(errp, "vhost-vdpa-device: invalid queue_size: %d

(max:%d)",

Quoted 11 lines
hw/virtio/vdpa-dev.c +11 -0
+                   s->queue_size, max_queue_size);
+        goto out;
+    } else if (!s->queue_size) {
+        s->queue_size = max_queue_size;
+    }
+
+    ret = vdpa_dev_get_vqs_num(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
Quoted 9 lines

hw/virtio/vdpa-dev.c line 85

Quoted 1 line
hw/virtio/vdpa-dev.c +1 -0
+    s->dev.nvqs = ret;

There is no input validation because we trust the kernel vDPA return values. That seems okay for now but if there is a vhost-user version of this in the future then input validation will be necessary to achieve isolation between QEMU and the vhost-user processes. I suggest including input validation code right away because it's harder to audit the code and fix missing input validation later on.

Make sense!

Should we only need to validate the upper boundary (e.g. <VIRTIO_QUEUE_MAX)?

Quoted 17 lines

hw/virtio/vdpa-dev.c

Quoted 7 lines
hw/virtio/vdpa-dev.c +7 -0
+    s->dev.vqs = g_new0(struct vhost_virtqueue, s->dev.nvqs);
+    s->dev.vq_index = 0;
+    s->dev.vq_index_end = s->dev.nvqs;
+    s->dev.backend_features = 0;
+    s->started = false;
+
+    ret = vhost_dev_init(&s->dev, &s->vdpa, VHOST_BACKEND_TYPE_VDPA, 0,

NULL);

hw/virtio/vdpa-dev.c line 94

Quoted 2 lines
hw/virtio/vdpa-dev.c +2 -0
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device: vhost initialization

failed: %s",

hw/virtio/vdpa-dev.c line 99

Quoted 5 lines
hw/virtio/vdpa-dev.c +5 -0
+                   strerror(-ret));
+        goto out;
+    }
+
+    ret = s->dev.vhost_ops->vhost_get_device_id(&s->dev, &device_id);
Quoted 4 lines

The vhost_*() API abstracts the ioctl calls but this source file and the PCI proxy have ioctl calls. I wonder if it's possible to move the ioctls calls into the vhost_*() API? That would be cleaner and also make it easier to add vhost-user vDPA support in the future.

We need these ioctls calls because we need invoke them before the vhost-dev object is initialized.

#45 lore raw

RE: [RFC 06/10] vdpa-dev: implement the unrealize interface

In reply to Stefano Garzarella
Quoted 38 lines

-----Original Message-----
From: Stefano Garzarella [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 7:16 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 06/10] vdpa-dev: implement the unrealize interface

Quoted 14 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .unrealize interface.

---
hw/virtio/vdpa-dev.c | 22 +++++++++++++++++++++-

1 file changed, 21 insertions(+), 1 deletion(-)
hw/virtio/vdpa-dev.c +0 -0
index 2d534d837a..4e4dd3d201 100644

hw/virtio/vdpa-dev.c out: · line 141

Quoted 9 lines
hw/virtio/vdpa-dev.c +6 -0
@@ -133,9 +133,29 @@ out:
    close(fd);
 }
 
+static void vhost_vdpa_vdev_unrealize(VhostVdpaDevice *s)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(s);
+    int i;
+
+    for (i = 0; i < s->num_queues; i++) {
                       ^
Quoted 4 lines

s->num_queues seems uninitialized to me, maybe we could just remove the num_queues field from VhostVdpaDevice, and use s->dev.nvqs as in vhost_vdpa_device_realize().

Good catch, I'll fix the bug.

But I think we should keep the num_queues field, we need it if we support migration in the next step, right?

Quoted 12 lines
Quoted 12 lines
hw/virtio/vdpa-dev.c +9 -1
+        virtio_delete_queue(s->virtqs[i]);
+    }
+    g_free(s->virtqs);
+    virtio_cleanup(vdev);
+
+    g_free(s->config);
+}
+
 static void vhost_vdpa_device_unrealize(DeviceState *dev)
 {
-    return;
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
Quoted 9 lines

hw/virtio/vdpa-dev.c out: · line 156

Quoted 4 lines
hw/virtio/vdpa-dev.c +4 -0
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+
+    virtio_set_status(vdev, 0);
+    vhost_dev_cleanup(&s->dev);

If we will use s->dev.nvqs in vhost_vdpa_vdev_unrealize(), we should call vhost_dev_cleanup() after it, just before close() as we already do in the error path of vhost_vdpa_device_realize().

I'll try to fix the above bug first if you agree that we should keep the num_queues field.

I just realize that I forgot to free s->dev.vqs here...

Quoted 8 lines
Quoted 8 lines
hw/virtio/vdpa-dev.c +2 -0
+    vhost_vdpa_vdev_unrealize(s);
+    close(s->vdpa.device_fd);
 }
 

static void
--
2.23.0

#21 lore raw

RE: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Jason Wang
Quoted 84 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Thursday, January 6, 2022 10:34 AM
To: Michael S. Tsirkin <[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 7 lines · Michael S. Tsirkin
Quoted 5 lines · Jason Wang
Quoted 3 lines · Cloud Infrastructure Service Product Dept.)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 3:54 PM
To: Michael S. Tsirkin <[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>;

pbonzini

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

<[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>;

qemu-devel

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

<[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

On Wed, Jan 5, 2022 at 8:59 AM Longpeng(Mike) <[email protected]>

Quoted 43 lines · wrote:
Quoted 43 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

From: Longpeng <[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 7 lines

Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Not only MSI vectors, but also queue-size, #vqs, etc.

MSI is PCI specific, we have non PCI vDPA parent e.g VDUSE/simulator/mlx5

And it's something that is not guaranteed to be not changed. E.g some drivers may choose to allocate MSI during set_status() which can fail for various reasons.

Quoted 4 lines

Maybe the vhost level could expose the hardware's real capabilities and let the userspace (QEMU) do the hiding? The userspace know how to process them.

#MSI vectors is much more easier to be mediated than queue-size and #vqs.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

For interrupts, we've already had VHOST_SET_X_KICK, we can keep allocating eventfd based on #MSI vectors to make it work with any number of MSI vectors that the virtual device had.

Right but if hardware does not support so many then what? Just fail?

Or just trigger the callback of vqs that shares the vector.

Then we should disable PI if we need to share a vector in this case?

Quoted 71 lines
Quoted 1 line

Having a query API would make things somewhat cleaner imho.

I may miss something, even if we know #vectors, we still don't know the associated virtqueues for a dedicated vector?

Quoted 8 lines
Quoted 8 lines

For queue-size, it's Ok to have a new uAPI but it's not a must, Qemu can simply fail if SET_VRING_NUM fail.

For #vqs, it's OK to have a new uAPI since the emulated virtio-pci device requires knowledge the #vqs in the config space. (still not a must, we can enumerate #vqs per device type)

Quoted 8 lines
Quoted 5 lines

For the config size, it's OK but not a must, technically we can simply relay what guest write to vhost-vdpa. It's just because current Qemu require to have it during virtio device initialization.

Thanks

I agree but these ok things make for a cleaner API I think.

Right.

Thanks

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Thanks

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Review

Quoted 6 lines
---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
(quoted diff) +0 -0

b/linux-headers/linux/vhost.h

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

linux-headers/linux/vhost.h line 151

Quoted 6 lines

index c998860d7b..c5edd75d15 100644

linux-headers/linux/vhost.h +0 -0
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78,

\

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

linux-headers/linux/vhost.h line 152

Quoted 1 line
linux-headers/linux/vhost.h +0 -0
                                             struct

vhost_vdpa_iova_range)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

linux-headers/linux/vhost.h line 155

Quoted 3 lines
linux-headers/linux/vhost.h +3 -0
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79,

int)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

linux-headers/linux/vhost.h line 158

Quoted 3 lines
linux-headers/linux/vhost.h +3 -0
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80,

int)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

linux-headers/linux/vhost.h line 161

Quoted 3 lines
linux-headers/linux/vhost.h +3 -0
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81,

int)

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
linux-headers/linux/vhost.h +1 -1
+
 #endif
--

2.23.0

#30 lore raw

Re: [RFC 03/10] vdpa: add the infrastructure of vdpa-dev

In reply to longpeng2--- via
Quoted 22 lines · Cloud Infrastructure Service Product Dept.)
Quoted 22 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 5:49 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 03/10] vdpa: add the infrastructure of vdpa-dev

hw/virtio/vdpa-dev-pci.c line 40

Quoted 5 lines · Longpeng(Mike)
hw/virtio/vdpa-dev-pci.c new file +5 -0
+static const VirtioPCIDeviceTypeInfo vhost_vdpa_device_pci_info = {
+    .base_name               = TYPE_VHOST_VDPA_DEVICE_PCI,
+    .generic_name            = "vhost-vdpa-device-pci",
+    .transitional_name       = "vhost-vdpa-device-pci-transitional",
+    .non_transitional_name   = "vhost-vdpa-device-pci-non-transitional",

Does vDPA support Transitional VIRTIO devices?

Quoted 7 lines
Quoted 2 lines

I expected this device to support Modern devices only.

There's already a 0.95 vdpa driver (Alibaba ENI) in the kernel source and supporting 0.95 devices is necessary for some older GuestOS.

I'm OK if other guys also approve of supporting 1.0+ devices only :)

If vDPA supports Transitional VIRTIO devices then it's fine to keep this code unchanged in this patch series.

Stefan

#40 lore raw

Re: [RFC 05/10] vdpa-dev: implement the realize interface

In reply to longpeng2--- via
Quoted 126 lines · Cloud Infrastructure Service Product Dept.)
Quoted 32 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 6:18 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 05/10] vdpa-dev: implement the realize interface

Quoted 18 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .realize interface.

---
 hw/virtio/vdpa-dev.c         | 114 +++++++++++++++++++++++++++++++++++
 include/hw/virtio/vdpa-dev.h |   8 +++
 2 files changed, 122 insertions(+)
hw/virtio/vdpa-dev.c +0 -0
index 790117fb3b..2d534d837a 100644
@@ -15,9 +15,122 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 
Quoted 12 lines

hw/virtio/vdpa-dev.c line 24

Quoted 7 lines
hw/virtio/vdpa-dev.c +7 -0
+static void
+vhost_vdpa_device_dummy_handle_output(VirtIODevice *vdev, VirtQueue *vq)
+{
+    /* Nothing to do */
+}
+
+static int vdpa_dev_get_info_by_fd(int fd, uint64_t cmd, Error **errp)

This looks similar to the helper function in a previous patch but this time the return value type is int instead of uint32_t. Please make the types consistent.

OK.

Quoted 37 lines
Quoted 37 lines
hw/virtio/vdpa-dev.c +35 -0
+{
+    int val;
+
+    if (ioctl(fd, cmd, &val) < 0) {
+        error_setg(errp, "vhost-vdpa-device: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        return -1;
+    }
+
+    return val;
+}
+
+static inline int vdpa_dev_get_queue_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VRING_NUM, errp);
+}
+
+static inline int vdpa_dev_get_vqs_num(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VQS_NUM, errp);
+}
+
+static inline int vdpa_dev_get_config_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_CONFIG_SIZE, errp);
+}
+
 static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
 {
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    uint32_t device_id;
+    int max_queue_size;
+    int fd;
+    int i, ret;
+
+    fd = qemu_open(s->vdpa_dev, O_RDWR, errp);
Quoted 9 lines

hw/virtio/vdpa-dev.c line 65

Quoted 4 lines
hw/virtio/vdpa-dev.c +4 -0
+    if (fd == -1) {
+        return;
+    }
+    s->vdpa.device_fd = fd;

This is the field I suggest exposing as a QOM property so it can be set from the proxy object (e.g. when the PCI proxy opens the vdpa device before our .realize() function is called).

OK.

Quoted 20 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +8 -0
+
+    max_queue_size = vdpa_dev_get_queue_size(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
+    if (s->queue_size > max_queue_size) {
+        error_setg(errp, "vhost-vdpa-device: invalid queue_size: %d

(max:%d)",

Quoted 11 lines
hw/virtio/vdpa-dev.c +11 -0
+                   s->queue_size, max_queue_size);
+        goto out;
+    } else if (!s->queue_size) {
+        s->queue_size = max_queue_size;
+    }
+
+    ret = vdpa_dev_get_vqs_num(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
Quoted 9 lines

hw/virtio/vdpa-dev.c line 85

Quoted 1 line
hw/virtio/vdpa-dev.c +1 -0
+    s->dev.nvqs = ret;

There is no input validation because we trust the kernel vDPA return values. That seems okay for now but if there is a vhost-user version of this in the future then input validation will be necessary to achieve isolation between QEMU and the vhost-user processes. I suggest including input validation code right away because it's harder to audit the code and fix missing input validation later on.

Quoted 3 lines

Make sense!

Should we only need to validate the upper boundary (e.g. <VIRTIO_QUEUE_MAX)?

Careful, ret is currently an int so negative values would bypass the < VIRTIO_QUEUE_MAX check.

Quoted 18 lines
Quoted 18 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +7 -0
+    s->dev.vqs = g_new0(struct vhost_virtqueue, s->dev.nvqs);
+    s->dev.vq_index = 0;
+    s->dev.vq_index_end = s->dev.nvqs;
+    s->dev.backend_features = 0;
+    s->started = false;
+
+    ret = vhost_dev_init(&s->dev, &s->vdpa, VHOST_BACKEND_TYPE_VDPA, 0,

NULL);

hw/virtio/vdpa-dev.c line 94

Quoted 2 lines
hw/virtio/vdpa-dev.c +2 -0
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device: vhost initialization

failed: %s",

Quoted 5 lines
hw/virtio/vdpa-dev.c +5 -0
+                   strerror(-ret));
+        goto out;
+    }
+
+    ret = s->dev.vhost_ops->vhost_get_device_id(&s->dev, &device_id);
Quoted 7 lines
Quoted 4 lines

The vhost_*() API abstracts the ioctl calls but this source file and the PCI proxy have ioctl calls. I wonder if it's possible to move the ioctls calls into the vhost_*() API? That would be cleaner and also make it easier to add vhost-user vDPA support in the future.

We need these ioctls calls because we need invoke them before the vhost-dev object is initialized.

It may be possible to clean this up by changing how vhost_dev_init() works but I haven't investigated. The issue is that the vhost_dev_init() API requires information from the caller that has to be fetched from the vDPA device. This forces the caller to communicate directly with the vDPA device before calling vhost_dev_init(). It may be possible to move this setup code inside vhost_dev_init() (and vhost_ops callbacks).

Stefan

#23 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Jason Wang
Quoted 67 lines · Jason Wang
Quoted 66 lines · Michael S. Tsirkin
Quoted 64 lines · Jason Wang
Quoted 41 lines · Cloud Infrastructure Service Product Dept.)
Quoted 41 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 3:54 PM
To: Michael S. Tsirkin <[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 6 lines · Michael S. Tsirkin
Quoted 4 lines · Jason Wang
Quoted 3 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 7 lines

Otherwise guests will at best be suboptimal.

Quoted 4 lines

And it reveals device implementation
details which block (cross vendor) migration.

Thanks

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Not only MSI vectors, but also queue-size, #vqs, etc.

MSI is PCI specific, we have non PCI vDPA parent e.g VDUSE/simulator/mlx5

And it's something that is not guaranteed to be not changed. E.g some drivers may choose to allocate MSI during set_status() which can fail for various reasons.

Quoted 4 lines

Maybe the vhost level could expose the hardware's real capabilities and let the userspace (QEMU) do the hiding? The userspace know how to process them.

#MSI vectors is much more easier to be mediated than queue-size and #vqs.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

For interrupts, we've already had VHOST_SET_X_KICK, we can keep allocating eventfd based on #MSI vectors to make it work with any number of MSI vectors that the virtual device had.

Right but if hardware does not support so many then what? Just fail?

Or just trigger the callback of vqs that shares the vector.

Right but we want userspace to be able to report this to guest accurately if it wants to. Guest can then configure itself correctly.

Quoted 4 lines
Quoted 1 line

Having a query API would make things somewhat cleaner imho.

I may miss something, even if we know #vectors, we still don't know the associated virtqueues for a dedicated vector?

This is up to the guest.

Quoted 60 lines
Quoted 8 lines
Quoted 8 lines

For queue-size, it's Ok to have a new uAPI but it's not a must, Qemu can simply fail if SET_VRING_NUM fail.

For #vqs, it's OK to have a new uAPI since the emulated virtio-pci device requires knowledge the #vqs in the config space. (still not a must, we can enumerate #vqs per device type)

Quoted 8 lines
Quoted 5 lines

For the config size, it's OK but not a must, technically we can simply relay what guest write to vhost-vdpa. It's just because current Qemu require to have it during virtio device initialization.

Thanks

I agree but these ok things make for a cleaner API I think.

Right.

Thanks

Quoted 39 lines
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines

Thanks

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
 linux-headers/linux/vhost.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
 /* Get the valid iova range */
 #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                             struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
 #endif
--

2.23.0

#31 lore raw

Re: [RFC 03/10] vdpa: add the infrastructure of vdpa-dev

In reply to Stefan Hajnoczi
Quoted 27 lines · Stefan Hajnoczi
Quoted 25 lines · Cloud Infrastructure Service Product Dept.)
Quoted 24 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 5:49 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 03/10] vdpa: add the infrastructure of vdpa-dev

hw/virtio/vdpa-dev-pci.c line 40

Quoted 5 lines · Longpeng(Mike)
hw/virtio/vdpa-dev-pci.c new file +5 -0
+static const VirtioPCIDeviceTypeInfo vhost_vdpa_device_pci_info = {
+    .base_name               = TYPE_VHOST_VDPA_DEVICE_PCI,
+    .generic_name            = "vhost-vdpa-device-pci",
+    .transitional_name       = "vhost-vdpa-device-pci-transitional",
+    .non_transitional_name   = "vhost-vdpa-device-pci-non-transitional",

Does vDPA support Transitional VIRTIO devices?

I expected this device to support Modern devices only.

Quoted 7 lines
Quoted 4 lines

There's already a 0.95 vdpa driver (Alibaba ENI) in the kernel source and supporting 0.95 devices is necessary for some older GuestOS.

I'm OK if other guys also approve of supporting 1.0+ devices only :)

If vDPA supports Transitional VIRTIO devices then it's fine to keep this code unchanged in this patch series.

Right, and I think that's the plan.

Thanks

Quoted 2 lines

Stefan

#22 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to longpeng2--- via
Quoted 68 lines · Cloud Infrastructure Service Product Dept.)
Quoted 68 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Thursday, January 6, 2022 10:34 AM
To: Michael S. Tsirkin<[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi<[email protected]>; Stefano Garzarella<[email protected]>; Cornelia Huck<[email protected]>; pbonzini <[email protected]>; Gonglei (Arei)<[email protected]>; Yechuan <[email protected]>; Huangzhichao<[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jason Wang
Quoted 8 lines · Cloud Infrastructure Service Product Dept.)
Quoted 8 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 3:54 PM
To: Michael S. Tsirkin<[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi<[email protected]>; Stefano Garzarella<[email protected]>; Cornelia Huck<[email protected]>;

pbonzini

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

<[email protected]>; Gonglei (Arei)<[email protected]>; Yechuan <[email protected]>; Huangzhichao<[email protected]>;

qemu-devel

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

<[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 2 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

On Wed, Jan 5, 2022 at 8:59 AM Longpeng(Mike)<[email protected]>

Quoted 33 lines · wrote:
Quoted 33 lines
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

From: Longpeng<[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well*guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 5 lines

Otherwise guests will at best be suboptimal.

Quoted 3 lines

And it reveals device implementation
details which block (cross vendor) migration.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Thanks

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Not only MSI vectors, but also queue-size, #vqs, etc.

MSI is PCI specific, we have non PCI vDPA parent e.g VDUSE/simulator/mlx5

And it's something that is not guaranteed to be not changed. E.g some drivers may choose to allocate MSI during set_status() which can fail for various reasons.

Quoted 3 lines

Maybe the vhost level could expose the hardware's real capabilities and let the userspace (QEMU) do the hiding? The userspace know how to process them.

#MSI vectors is much more easier to be mediated than queue-size and #vqs.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

For interrupts, we've already had VHOST_SET_X_KICK, we can keep allocating eventfd based on #MSI vectors to make it work with any number of MSI vectors that the virtual device had.

Right but if hardware does not support so many then what? Just fail?

Or just trigger the callback of vqs that shares the vector.

Then we should disable PI if we need to share a vector in this case?

I may miss something, but I don't see any reason for doing this. I think the irqbypass manager and the arch specific PI codes should deal with this case.

Ling Shan (cced) told me it works in the past.

Thanks

Quoted 1 line
#24 lore raw

Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

In reply to Michael S. Tsirkin
Quoted 62 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 51 lines · Michael S. Tsirkin
Quoted 50 lines · Jason Wang
Quoted 33 lines · Cloud Infrastructure Service Product Dept.)
Quoted 33 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 3:54 PM
To: Michael S. Tsirkin <[email protected]>
Cc: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>; Stefan Hajnoczi <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 02/10] vhost: add 3 commands for vhost-vdpa

Quoted 10 lines · Michael S. Tsirkin
Quoted 7 lines · Jason Wang
Quoted 4 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

To support generic vdpa deivce, we need add the following ioctls:
- GET_VECTORS_NUM: the count of vectors that supported

Does this mean MSI vectors? If yes, it looks like a layer violation:
vhost is transport independent.

Well *guest* needs to know how many vectors device supports. I don't think there's a way around that. Do you?

We have VHOST_SET_VRING/CONFIG_CALL which is per vq. I think we can simply assume #vqs + 1?

Quoted 5 lines

Otherwise guests will at best be suboptimal.

Quoted 3 lines

And it reveals device implementation
details which block (cross vendor) migration.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Thanks

Not necessarily, userspace can hide this from guest if it wants to, just validate.

If we can hide it at vhost/uAPI level, it would be even better?

Not only MSI vectors, but also queue-size, #vqs, etc.

MSI is PCI specific, we have non PCI vDPA parent e.g VDUSE/simulator/mlx5

And it's something that is not guaranteed to be not changed. E.g some drivers may choose to allocate MSI during set_status() which can fail for various reasons.

Quoted 3 lines

Maybe the vhost level could expose the hardware's real capabilities and let the userspace (QEMU) do the hiding? The userspace know how to process them.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

#MSI vectors is much more easier to be mediated than queue-size and #vqs.

For interrupts, we've already had VHOST_SET_X_KICK, we can keep allocating eventfd based on #MSI vectors to make it work with any number of MSI vectors that the virtual device had.

Right but if hardware does not support so many then what? Just fail?

Or just trigger the callback of vqs that shares the vector.

Quoted 8 lines

Right but we want userspace to be able to report this to guest accurately if it wants to. Guest can then configure itself correctly.

Quoted 3 lines
Quoted 1 line

Having a query API would make things somewhat cleaner imho.

I may miss something, even if we know #vectors, we still don't know the associated virtqueues for a dedicated vector?

This is up to the guest.

Just to clarify the possible issue, this only works if vDPA parent is using the same irq binding policy as what viritio-pci did in the guest.

Consider vDPA has 3 vectors allocated:

host vector 0: tx/rx
host vector 1: cvq
host vector 2: config

So we return 3 for get_vectors. So the virtual device will have 3 vectors in this case.

But a guest driver may do:

guest vector 0: tx (eventfd0)
guest vector 1: rx (eventfd1)
guest vector 2: cvq/config (eventfd2)

The irq handler of host vector0 will notify both eventfd0(guest vector0) and eventfd1(guest vector1) in this case.

And using such "vectors passthrough" may block migration between the vDPA device where the #vectors is the only difference.

Thanks

Quoted 52 lines
Quoted 52 lines
Quoted 8 lines
Quoted 8 lines

For queue-size, it's Ok to have a new uAPI but it's not a must, Qemu can simply fail if SET_VRING_NUM fail.

For #vqs, it's OK to have a new uAPI since the emulated virtio-pci device requires knowledge the #vqs in the config space. (still not a must, we can enumerate #vqs per device type)

Quoted 7 lines
Quoted 5 lines

For the config size, it's OK but not a must, technically we can simply relay what guest write to vhost-vdpa. It's just because current Qemu require to have it during virtio device initialization.

Thanks

I agree but these ok things make for a cleaner API I think.

Right.

Thanks

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Thanks

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines

- GET_CONFIG_SIZE: the size of the virtio config space
- GET_VQS_NUM: the count of virtqueues that exported

---
  linux-headers/linux/vhost.h | 10 ++++++++++
  1 file changed, 10 insertions(+)
linux-headers/linux/vhost.h +10 -1
index c998860d7b..c5edd75d15 100644
@@ -150,4 +150,14 @@
  /* Get the valid iova range */
  #define VHOST_VDPA_GET_IOVA_RANGE      _IOR(VHOST_VIRTIO, 0x78, \
                                              struct vhost_vdpa_iova_range)
+
+/* Get the number of vectors */
+#define VHOST_VDPA_GET_VECTORS_NUM     _IOR(VHOST_VIRTIO, 0x79, int)
+
+/* Get the virtio config size */
+#define VHOST_VDPA_GET_CONFIG_SIZE     _IOR(VHOST_VIRTIO, 0x80, int)
+
+/* Get the number of virtqueues */
+#define VHOST_VDPA_GET_VQS_NUM         _IOR(VHOST_VIRTIO, 0x81, int)
+
  #endif
--

2.23.0

#5 lore raw

RE: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

In reply to Jason Wang
Quoted 80 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 2:15 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; mst <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>; pbonzini <[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

Quoted 10 lines · Cloud Infrastructure Service Product Dept.)
Quoted 10 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 12:38 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; mst <[email protected]>; Stefano Garzarella <[email protected]>; Cornelia Huck <[email protected]>;

pbonzini

Quoted 27 lines
Quoted 27 lines

<[email protected]>; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; qemu-devel <[email protected]>
Subject: Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

Quoted 20 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +0 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
Quoted 7 lines
Quoted 7 lines

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int

Quoted 7 lines
hw/virtio/virtio-pci.c +1 -0
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int

n,

Quoted 17 lines
Quoted 16 lines

QEMUFile *f)

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 224

Quoted 11 lines
hw/virtio/virtio-pci.c +8 -0
     return 0;
 }
 
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {

Any way to get rid of this array? E.g using the algorithm that is used by the kernel virtio driver.

Quoted 5 lines
Quoted 2 lines

For device id, we can use the algorithm if we no need to support Transitional id. But how to get the class id ?

Right, I miss this. So the current code should be fine.

Maybe the following way would be better? It can save about 40 lines.

#define VIRTIO_PCI_ID_INFO(name, class)   \
    {VIRTIO_ID_##name, PCI_DEVICE_ID_VIRTIO_##name, class}

static const VirtIOPCIIDInfo virtio_pci_id_info[] = {

    VIRTIO_PCI_ID_INFO(NET,     PCI_CLASS_NETWORK_ETHERNET),
    VIRTIO_PCI_ID_INFO(BLOCK,   PCI_CLASS_STORAGE_SCSI),
    VIRTIO_PCI_ID_INFO(SCSI,    PCI_CLASS_STORAGE_SCSI),

VIRTIO_PCI_ID_INFO(CONSOLE, PCI_CLASS_COMMUNICATION_OTHER),
VIRTIO_PCI_ID_INFO(VSOCK, PCI_CLASS_COMMUNICATION_OTHER),
VIRTIO_PCI_ID_INFO(IOMMU, PCI_CLASS_OTHERS),
VIRTIO_PCI_ID_INFO(MEM, PCI_CLASS_OTHERS),
VIRTIO_PCI_ID_INFO(PMEM, PCI_CLASS_OTHERS),
VIRTIO_PCI_ID_INFO(RNG, PCI_CLASS_OTHERS),
VIRTIO_PCI_ID_INFO(BALLOON, PCI_CLASS_OTHERS),
VIRTIO_PCI_ID_INFO(9P, PCI_BASE_CLASS_NETWORK),
};

Quoted 128 lines

Thanks

Quoted 80 lines
Quoted 80 lines

Thanks

Quoted 77 lines
hw/virtio/virtio-pci.c +77 -0
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
Quoted 8 lines
Quoted 8 lines

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState

Quoted 8 lines
hw/virtio/virtio-pci.c +4 -0
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1674,6 +1764,9 @@ static void virtio_pci_device_plugged(DeviceState

*d,

Quoted 37 lines
Quoted 36 lines

Error **errp)

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d, Error **errp)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID,

virtio_bus_get_vdev_id(bus));

Quoted 31 lines
hw/virtio/virtio-pci.c +3 -0
+        if (proxy->pdev_id) {
+            pci_config_set_device_id(config, proxy->pdev_id);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
hw/virtio/virtio-pci.h +4 -1
index 2446dcd9ae..06aa59436e 100644
@@ -146,6 +146,7 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    uint16_t pdev_id;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -158,6 +159,9 @@ struct VirtIOPCIProxy {
     VirtioBusState bus;
 };
 
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 static inline bool virtio_pci_modern(VirtIOPCIProxy *proxy)
 {
     return !proxy->disable_modern;
--

2.23.0

#8 lore raw

Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

In reply to Longpeng(Mike) via
Quoted 4 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

ton of typos here.

Quoted 21 lines

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 221

Quoted 8 lines
hw/virtio/virtio-pci.c +5 -0
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f)
     return 0;
 }
 
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;

if this is transitional as comment says make it explicit in the names and comments.

Quoted 49 lines
hw/virtio/virtio-pci.c +49 -0
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 280

Quoted 10 lines
hw/virtio/virtio-pci.c +10 -0
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+

this is the list from the spec:

So this is the list from the spec:

0x1000 network card
0x1001 block device
0x1002 memory ballooning (traditional)
0x1003 console
0x1004 SCSI host
0x1005 entropy source
0x1009 9P transport

I'd drop all the rest, use the algorithm for non transitional. And when class is other I'd just not include it in the array, make this the default.

Quoted 62 lines
hw/virtio/virtio-pci.c +28 -0
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1674,6 +1764,9 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp)
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID, virtio_bus_get_vdev_id(bus));
+        if (proxy->pdev_id) {
+            pci_config_set_device_id(config, proxy->pdev_id);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
hw/virtio/virtio-pci.h +4 -0
index 2446dcd9ae..06aa59436e 100644
@@ -146,6 +146,7 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    uint16_t pdev_id;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -158,6 +159,9 @@ struct VirtIOPCIProxy {
     VirtioBusState bus;
 };
 
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 static inline bool virtio_pci_modern(VirtIOPCIProxy *proxy)
 {
     return !proxy->disable_modern;
Signature · 1 line
2.23.0
#9 lore raw

RE: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

Quoted 14 lines

-----Original Message-----
From: Michael S. Tsirkin [mailto:[email protected]]
Sent: Monday, January 10, 2022 1:43 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

Quoted 8 lines
Quoted 4 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

ton of typos here.

Will fix all in the V2.

Quoted 25 lines

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n,

Quoted 21 lines

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n,

QEMUFile *f)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
     return 0;
 }
 
Quoted 10 lines

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 221

Quoted 5 lines
hw/virtio/virtio-pci.c +5 -0
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;

if this is transitional as comment says make it explicit in the names and comments.

OK.

Quoted 66 lines
Quoted 49 lines
hw/virtio/virtio-pci.c +49 -0
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 280

Quoted 10 lines
hw/virtio/virtio-pci.c +10 -0
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+

this is the list from the spec:

So this is the list from the spec:

Quoted 8 lines

0x1000 network card
0x1001 block device
0x1002 memory ballooning (traditional)
0x1003 console
0x1004 SCSI host
0x1005 entropy source
0x1009 9P transport

Why the following device IDs are introduced? They are non transitional devices.

#define PCI_DEVICE_ID_VIRTIO_VSOCK       0x1012
#define PCI_DEVICE_ID_VIRTIO_PMEM        0x1013
#define PCI_DEVICE_ID_VIRTIO_IOMMU       0x1014
#define PCI_DEVICE_ID_VIRTIO_MEM         0x1015
Quoted 71 lines

I'd drop all the rest, use the algorithm for non transitional. And when class is other I'd just not include it in the array, make this the default.

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d,

Quoted 29 lines
hw/virtio/virtio-pci.c +25 -0
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1674,6 +1764,9 @@ static void virtio_pci_device_plugged(DeviceState *d,

Error **errp)

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d, Error **errp)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID,

virtio_bus_get_vdev_id(bus));

Quoted 30 lines
hw/virtio/virtio-pci.c +3 -0
+        if (proxy->pdev_id) {
+            pci_config_set_device_id(config, proxy->pdev_id);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
hw/virtio/virtio-pci.h +4 -1
index 2446dcd9ae..06aa59436e 100644
@@ -146,6 +146,7 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    uint16_t pdev_id;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -158,6 +159,9 @@ struct VirtIOPCIProxy {
     VirtioBusState bus;
 };
 
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 static inline bool virtio_pci_modern(VirtIOPCIProxy *proxy)
 {
     return !proxy->disable_modern;
--

2.23.0

#10 lore raw

Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

Quoted 138 lines · Cloud Infrastructure Service Product Dept.)
Quoted 14 lines

-----Original Message-----
From: Michael S. Tsirkin [mailto:[email protected]]
Sent: Monday, January 10, 2022 1:43 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 01/10] virtio: get class_id and pci device id by the virtio id

Quoted 8 lines
Quoted 4 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Add helpers to get the "Transitional PCI Device ID" and "class_id" of the deivce which is specificed by the "Virtio Device ID".

ton of typos here.

Will fix all in the V2.

Quoted 25 lines

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n,

Quoted 21 lines

These helpers will be used to build the generic vDPA device later.

---
 hw/virtio/virtio-pci.c | 93 ++++++++++++++++++++++++++++++++++++++++++
 hw/virtio/virtio-pci.h |  4 ++
 2 files changed, 97 insertions(+)
hw/virtio/virtio-pci.c +1 -0
index 750aa47ec1..843085c4ea 100644
@@ -19,6 +19,7 @@
 
 #include "exec/memop.h"
 #include "standard-headers/linux/virtio_pci.h"
+#include "standard-headers/linux/virtio_ids.h"
 #include "hw/boards.h"
 #include "hw/virtio/virtio.h"
 #include "migration/qemu-file-types.h"
@@ -213,6 +214,95 @@ static int virtio_pci_load_queue(DeviceState *d, int n,

QEMUFile *f)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
     return 0;
 }
 
Quoted 10 lines

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 221

Quoted 5 lines
hw/virtio/virtio-pci.c +5 -0
+typedef struct VirtIOPCIIDInfo {
+    uint16_t vdev_id; /* virtio id */
+    uint16_t pdev_id; /* pci device id */
+    uint16_t class_id;
+} VirtIOPCIIDInfo;

if this is transitional as comment says make it explicit in the names and comments.

OK.

Quoted 66 lines
Quoted 49 lines
hw/virtio/virtio-pci.c +49 -0
+
+static const VirtIOPCIIDInfo virtio_pci_id_info[] = {
+    {
+        .vdev_id = VIRTIO_ID_NET,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_NET,
+        .class_id = PCI_CLASS_NETWORK_ETHERNET,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BLOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BLOCK,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_CONSOLE,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_CONSOLE,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_SCSI,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_SCSI,
+        .class_id = PCI_CLASS_STORAGE_SCSI,
+    },
+    {
+        .vdev_id = VIRTIO_ID_9P,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_9P,
+        .class_id = PCI_BASE_CLASS_NETWORK,
+    },
+    {
+        .vdev_id = VIRTIO_ID_VSOCK,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_VSOCK,
+        .class_id = PCI_CLASS_COMMUNICATION_OTHER,
+    },
+    {
+        .vdev_id = VIRTIO_ID_IOMMU,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_IOMMU,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_MEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_MEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_PMEM,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_PMEM,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_RNG,

hw/virtio/virtio-pci.c static int virtio_pci_load_queue(DeviceState *d, int n, QEMUFile *f) · line 280

Quoted 10 lines
hw/virtio/virtio-pci.c +10 -0
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_RNG,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+    {
+        .vdev_id = VIRTIO_ID_BALLOON,
+        .pdev_id = PCI_DEVICE_ID_VIRTIO_BALLOON,
+        .class_id = PCI_CLASS_OTHERS,
+    },
+};
+

this is the list from the spec:

So this is the list from the spec:

Quoted 8 lines

0x1000 network card
0x1001 block device
0x1002 memory ballooning (traditional)
0x1003 console
0x1004 SCSI host
0x1005 entropy source
0x1009 9P transport

Quoted 7 lines

Why the following device IDs are introduced? They are non transitional devices.

#define PCI_DEVICE_ID_VIRTIO_VSOCK       0x1012
#define PCI_DEVICE_ID_VIRTIO_PMEM        0x1013
#define PCI_DEVICE_ID_VIRTIO_IOMMU       0x1014
#define PCI_DEVICE_ID_VIRTIO_MEM         0x1015

Just a single place to put these things.
E.g. vsock id is used in more than 1 place.

Quoted 71 lines
Quoted 71 lines

I'd drop all the rest, use the algorithm for non transitional. And when class is other I'd just not include it in the array, make this the default.

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d,

Quoted 29 lines
hw/virtio/virtio-pci.c +25 -0
+static VirtIOPCIIDInfo virtio_pci_get_id_info(uint16_t vdev_id)
+{
+    VirtIOPCIIDInfo info = {};
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(virtio_pci_id_info); i++) {
+        if (virtio_pci_id_info[i].vdev_id == vdev_id) {
+            info = virtio_pci_id_info[i];
+            break;
+        }
+    }
+
+    return info;
+}
+
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).pdev_id;
+}
+
+uint16_t virtio_pci_get_class_id(uint16_t device_id)
+{
+    return virtio_pci_get_id_info(device_id).class_id;
+}
+
 static bool virtio_pci_ioeventfd_enabled(DeviceState *d)
 {
     VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
@@ -1674,6 +1764,9 @@ static void virtio_pci_device_plugged(DeviceState *d,

Error **errp)

hw/virtio/virtio-pci.c static void virtio_pci_device_plugged(DeviceState *d, Error **errp)

Quoted 3 lines
hw/virtio/virtio-pci.c +0 -0
          * is set to PCI_SUBVENDOR_ID_REDHAT_QUMRANET by default.
          */
         pci_set_word(config + PCI_SUBSYSTEM_ID,

virtio_bus_get_vdev_id(bus));

Quoted 30 lines
hw/virtio/virtio-pci.c +3 -0
+        if (proxy->pdev_id) {
+            pci_config_set_device_id(config, proxy->pdev_id);
+        }
     } else {
         /* pure virtio-1.0 */
         pci_set_word(config + PCI_VENDOR_ID,
hw/virtio/virtio-pci.h +4 -1
index 2446dcd9ae..06aa59436e 100644
@@ -146,6 +146,7 @@ struct VirtIOPCIProxy {
     bool disable_modern;
     bool ignore_backend_features;
     OnOffAuto disable_legacy;
+    uint16_t pdev_id;
     uint32_t class_code;
     uint32_t nvectors;
     uint32_t dfselect;
@@ -158,6 +159,9 @@ struct VirtIOPCIProxy {
     VirtioBusState bus;
 };
 
+uint16_t virtio_pci_get_pci_devid(uint16_t device_id);
+uint16_t virtio_pci_get_class_id(uint16_t device_id);
+
 static inline bool virtio_pci_modern(VirtIOPCIProxy *proxy)
 {
     return !proxy->disable_modern;
--

2.23.0

#46 lore raw

Re: [RFC 06/10] vdpa-dev: implement the unrealize interface

In reply to longpeng2--- via
Quoted 70 lines · Cloud Infrastructure Service Product Dept.)
Quoted 38 lines

-----Original Message-----
From: Stefano Garzarella [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 7:16 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 06/10] vdpa-dev: implement the unrealize interface

Quoted 14 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .unrealize interface.

---
hw/virtio/vdpa-dev.c | 22 +++++++++++++++++++++-

1 file changed, 21 insertions(+), 1 deletion(-)
hw/virtio/vdpa-dev.c +0 -0
index 2d534d837a..4e4dd3d201 100644

hw/virtio/vdpa-dev.c out: · line 141

Quoted 9 lines
hw/virtio/vdpa-dev.c +6 -0
@@ -133,9 +133,29 @@ out:
    close(fd);
 }
 
+static void vhost_vdpa_vdev_unrealize(VhostVdpaDevice *s)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(s);
+    int i;
+
+    for (i = 0; i < s->num_queues; i++) {
                       ^
Quoted 4 lines

s->num_queues seems uninitialized to me, maybe we could just remove the num_queues field from VhostVdpaDevice, and use s->dev.nvqs as in vhost_vdpa_device_realize().

Good catch, I'll fix the bug.

But I think we should keep the num_queues field, we need it if we support migration in the next step, right?

Quoted 12 lines
Quoted 12 lines
hw/virtio/vdpa-dev.c +9 -1
+        virtio_delete_queue(s->virtqs[i]);
+    }
+    g_free(s->virtqs);
+    virtio_cleanup(vdev);
+
+    g_free(s->config);
+}
+
 static void vhost_vdpa_device_unrealize(DeviceState *dev)
 {
-    return;
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
Quoted 9 lines

hw/virtio/vdpa-dev.c out: · line 156

Quoted 4 lines
hw/virtio/vdpa-dev.c +4 -0
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+
+    virtio_set_status(vdev, 0);
+    vhost_dev_cleanup(&s->dev);

If we will use s->dev.nvqs in vhost_vdpa_vdev_unrealize(), we should call vhost_dev_cleanup() after it, just before close() as we already do in the error path of vhost_vdpa_device_realize().

Quoted 2 lines

I'll try to fix the above bug first if you agree that we should keep the num_queues field.

Yep, if it could be useful, we can keep it.

Quoted 2 lines

I just realize that I forgot to free s->dev.vqs here...

Oh right, I miss it too.
We should free it also in the error path of vhost_vdpa_device_realize().

Thanks,
Stefano

#41 lore raw

RE: [RFC 05/10] vdpa-dev: implement the realize interface

In reply to Stefan Hajnoczi
Quoted 176 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Thursday, January 6, 2022 7:34 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 05/10] vdpa-dev: implement the realize interface

Quoted 32 lines · Cloud Infrastructure Service Product Dept.)
Quoted 32 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 6:18 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 05/10] vdpa-dev: implement the realize interface

Quoted 18 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .realize interface.

---
 hw/virtio/vdpa-dev.c         | 114 +++++++++++++++++++++++++++++++++++
 include/hw/virtio/vdpa-dev.h |   8 +++
 2 files changed, 122 insertions(+)
hw/virtio/vdpa-dev.c +0 -0
index 790117fb3b..2d534d837a 100644
@@ -15,9 +15,122 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 
Quoted 2 lines
Quoted 2 lines

hw/virtio/vdpa-dev.c line 19

Quoted 2 lines
hw/virtio/vdpa-dev.c +2 -0
+static void
+vhost_vdpa_device_dummy_handle_output(VirtIODevice *vdev, VirtQueue

*vq)

Quoted 30 lines
Quoted 10 lines

hw/virtio/vdpa-dev.c line 24

Quoted 5 lines
hw/virtio/vdpa-dev.c +5 -0
+{
+    /* Nothing to do */
+}
+
+static int vdpa_dev_get_info_by_fd(int fd, uint64_t cmd, Error **errp)

This looks similar to the helper function in a previous patch but this time the return value type is int instead of uint32_t. Please make the types consistent.

OK.

Quoted 17 lines
Quoted 17 lines
hw/virtio/vdpa-dev.c +17 -0
+{
+    int val;
+
+    if (ioctl(fd, cmd, &val) < 0) {
+        error_setg(errp, "vhost-vdpa-device: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        return -1;
+    }
+
+    return val;
+}
+
+static inline int vdpa_dev_get_queue_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VRING_NUM, errp);
+}
+
Quoted 8 lines
Quoted 8 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +8 -0
+static inline int vdpa_dev_get_vqs_num(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VQS_NUM, errp);
+}
+
+static inline int vdpa_dev_get_config_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_CONFIG_SIZE,

errp);

Quoted 54 lines
Quoted 12 lines
Quoted 12 lines
hw/virtio/vdpa-dev.c +10 -0
+}
+
 static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
 {
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    uint32_t device_id;
+    int max_queue_size;
+    int fd;
+    int i, ret;
+
+    fd = qemu_open(s->vdpa_dev, O_RDWR, errp);
Quoted 9 lines

hw/virtio/vdpa-dev.c line 65

Quoted 4 lines
hw/virtio/vdpa-dev.c +4 -0
+    if (fd == -1) {
+        return;
+    }
+    s->vdpa.device_fd = fd;

This is the field I suggest exposing as a QOM property so it can be set from the proxy object (e.g. when the PCI proxy opens the vdpa device before our .realize() function is called).

OK.

Quoted 20 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +8 -0
+
+    max_queue_size = vdpa_dev_get_queue_size(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
+    if (s->queue_size > max_queue_size) {
+        error_setg(errp, "vhost-vdpa-device: invalid queue_size: %d

(max:%d)",

Quoted 11 lines
hw/virtio/vdpa-dev.c +11 -0
+                   s->queue_size, max_queue_size);
+        goto out;
+    } else if (!s->queue_size) {
+        s->queue_size = max_queue_size;
+    }
+
+    ret = vdpa_dev_get_vqs_num(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
Quoted 9 lines

hw/virtio/vdpa-dev.c line 85

Quoted 1 line
hw/virtio/vdpa-dev.c +1 -0
+    s->dev.nvqs = ret;

There is no input validation because we trust the kernel vDPA return values. That seems okay for now but if there is a vhost-user version of this in the future then input validation will be necessary to achieve isolation between QEMU and the vhost-user processes. I suggest including input validation code right away because it's harder to audit the code and fix missing input validation later on.

Quoted 3 lines

Make sense!

Should we only need to validate the upper boundary (e.g. <VIRTIO_QUEUE_MAX)?

Careful, ret is currently an int so negative values would bypass the < VIRTIO_QUEUE_MAX check.

Quoted 18 lines
Quoted 18 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +7 -0
+    s->dev.vqs = g_new0(struct vhost_virtqueue, s->dev.nvqs);
+    s->dev.vq_index = 0;
+    s->dev.vq_index_end = s->dev.nvqs;
+    s->dev.backend_features = 0;
+    s->started = false;
+
+    ret = vhost_dev_init(&s->dev, &s->vdpa, VHOST_BACKEND_TYPE_VDPA, 0,

NULL);

hw/virtio/vdpa-dev.c line 94

Quoted 2 lines
hw/virtio/vdpa-dev.c +2 -0
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device: vhost initialization

failed: %s",

Quoted 5 lines
hw/virtio/vdpa-dev.c +5 -0
+                   strerror(-ret));
+        goto out;
+    }
+
+    ret = s->dev.vhost_ops->vhost_get_device_id(&s->dev, &device_id);
Quoted 7 lines
Quoted 4 lines

The vhost_*() API abstracts the ioctl calls but this source file and the PCI proxy have ioctl calls. I wonder if it's possible to move the ioctls calls into the vhost_*() API? That would be cleaner and also make it easier to add vhost-user vDPA support in the future.

We need these ioctls calls because we need invoke them before the vhost-dev object is initialized.

Quoted 7 lines

It may be possible to clean this up by changing how vhost_dev_init() works but I haven't investigated. The issue is that the vhost_dev_init() API requires information from the caller that has to be fetched from the vDPA device. This forces the caller to communicate directly with the vDPA device before calling vhost_dev_init(). It may be possible to move this setup code inside vhost_dev_init() (and vhost_ops callbacks).

Hmm, this is still not clear to me, so let's continue to discuss this in v2 if you think it's necessary.

Quoted 1 line

Stefan

#42 lore raw

Re: [RFC 05/10] vdpa-dev: implement the realize interface

Quoted 184 lines · Cloud Infrastructure Service Product Dept.)
Quoted 176 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Thursday, January 6, 2022 7:34 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 05/10] vdpa-dev: implement the realize interface

Quoted 32 lines · Cloud Infrastructure Service Product Dept.)
Quoted 32 lines

-----Original Message-----
From: Stefan Hajnoczi [mailto:[email protected]]
Sent: Wednesday, January 5, 2022 6:18 PM
To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei) <[email protected]>; Yechuan <[email protected]>; Huangzhichao <[email protected]>; [email protected]
Subject: Re: [RFC 05/10] vdpa-dev: implement the realize interface

Quoted 18 lines · Longpeng(Mike)

From: Longpeng <[email protected]>

Implements the .realize interface.

---
 hw/virtio/vdpa-dev.c         | 114 +++++++++++++++++++++++++++++++++++
 include/hw/virtio/vdpa-dev.h |   8 +++
 2 files changed, 122 insertions(+)
hw/virtio/vdpa-dev.c +0 -0
index 790117fb3b..2d534d837a 100644
@@ -15,9 +15,122 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/runstate.h"
 
Quoted 2 lines
Quoted 2 lines

hw/virtio/vdpa-dev.c line 19

Quoted 2 lines
hw/virtio/vdpa-dev.c +2 -0
+static void
+vhost_vdpa_device_dummy_handle_output(VirtIODevice *vdev, VirtQueue

*vq)

Quoted 30 lines
Quoted 10 lines

hw/virtio/vdpa-dev.c line 24

Quoted 5 lines
hw/virtio/vdpa-dev.c +5 -0
+{
+    /* Nothing to do */
+}
+
+static int vdpa_dev_get_info_by_fd(int fd, uint64_t cmd, Error **errp)

This looks similar to the helper function in a previous patch but this time the return value type is int instead of uint32_t. Please make the types consistent.

OK.

Quoted 17 lines
Quoted 17 lines
hw/virtio/vdpa-dev.c +17 -0
+{
+    int val;
+
+    if (ioctl(fd, cmd, &val) < 0) {
+        error_setg(errp, "vhost-vdpa-device: cmd 0x%lx failed: %s",
+                   cmd, strerror(errno));
+        return -1;
+    }
+
+    return val;
+}
+
+static inline int vdpa_dev_get_queue_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VRING_NUM, errp);
+}
+
Quoted 8 lines
Quoted 8 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +8 -0
+static inline int vdpa_dev_get_vqs_num(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_VQS_NUM, errp);
+}
+
+static inline int vdpa_dev_get_config_size(int fd, Error **errp)
+{
+    return vdpa_dev_get_info_by_fd(fd, VHOST_VDPA_GET_CONFIG_SIZE,

errp);

Quoted 54 lines
Quoted 12 lines
Quoted 12 lines
hw/virtio/vdpa-dev.c +10 -0
+}
+
 static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
 {
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VhostVdpaDevice *s = VHOST_VDPA_DEVICE(vdev);
+    uint32_t device_id;
+    int max_queue_size;
+    int fd;
+    int i, ret;
+
+    fd = qemu_open(s->vdpa_dev, O_RDWR, errp);
Quoted 9 lines

hw/virtio/vdpa-dev.c line 65

Quoted 4 lines
hw/virtio/vdpa-dev.c +4 -0
+    if (fd == -1) {
+        return;
+    }
+    s->vdpa.device_fd = fd;

This is the field I suggest exposing as a QOM property so it can be set from the proxy object (e.g. when the PCI proxy opens the vdpa device before our .realize() function is called).

OK.

Quoted 20 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +8 -0
+
+    max_queue_size = vdpa_dev_get_queue_size(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
+    if (s->queue_size > max_queue_size) {
+        error_setg(errp, "vhost-vdpa-device: invalid queue_size: %d

(max:%d)",

Quoted 11 lines
hw/virtio/vdpa-dev.c +11 -0
+                   s->queue_size, max_queue_size);
+        goto out;
+    } else if (!s->queue_size) {
+        s->queue_size = max_queue_size;
+    }
+
+    ret = vdpa_dev_get_vqs_num(fd, errp);
+    if (*errp) {
+        goto out;
+    }
+
Quoted 9 lines

hw/virtio/vdpa-dev.c line 85

Quoted 1 line
hw/virtio/vdpa-dev.c +1 -0
+    s->dev.nvqs = ret;

There is no input validation because we trust the kernel vDPA return values. That seems okay for now but if there is a vhost-user version of this in the future then input validation will be necessary to achieve isolation between QEMU and the vhost-user processes. I suggest including input validation code right away because it's harder to audit the code and fix missing input validation later on.

Quoted 3 lines

Make sense!

Should we only need to validate the upper boundary (e.g. <VIRTIO_QUEUE_MAX)?

Careful, ret is currently an int so negative values would bypass the < VIRTIO_QUEUE_MAX check.

Quoted 18 lines
Quoted 18 lines

hw/virtio/vdpa-dev.c

Quoted 8 lines
hw/virtio/vdpa-dev.c +7 -0
+    s->dev.vqs = g_new0(struct vhost_virtqueue, s->dev.nvqs);
+    s->dev.vq_index = 0;
+    s->dev.vq_index_end = s->dev.nvqs;
+    s->dev.backend_features = 0;
+    s->started = false;
+
+    ret = vhost_dev_init(&s->dev, &s->vdpa, VHOST_BACKEND_TYPE_VDPA, 0,

NULL);

hw/virtio/vdpa-dev.c line 94

Quoted 2 lines
hw/virtio/vdpa-dev.c +2 -0
+    if (ret < 0) {
+        error_setg(errp, "vhost-vdpa-device: vhost initialization

failed: %s",

Quoted 5 lines
hw/virtio/vdpa-dev.c +5 -0
+                   strerror(-ret));
+        goto out;
+    }
+
+    ret = s->dev.vhost_ops->vhost_get_device_id(&s->dev, &device_id);
Quoted 7 lines
Quoted 4 lines

The vhost_*() API abstracts the ioctl calls but this source file and the PCI proxy have ioctl calls. I wonder if it's possible to move the ioctls calls into the vhost_*() API? That would be cleaner and also make it easier to add vhost-user vDPA support in the future.

We need these ioctls calls because we need invoke them before the vhost-dev object is initialized.

Quoted 7 lines

It may be possible to clean this up by changing how vhost_dev_init() works but I haven't investigated. The issue is that the vhost_dev_init() API requires information from the caller that has to be fetched from the vDPA device. This forces the caller to communicate directly with the vDPA device before calling vhost_dev_init(). It may be possible to move this setup code inside vhost_dev_init() (and vhost_ops callbacks).

Quoted 2 lines

Hmm, this is still not clear to me, so let's continue to discuss this in v2 if you think it's necessary.

Okay.

Stefan