Skip to contentSkip to conversation

Shadow VirtQueue event index support

PATCH In review Eugenio Pérez 3 participants 19 messages Oct 20, 2022 – Nov 1, 2022

lore
Share & tools
lore
#0 lore raw

cover Shadow VirtQueue event index support

Event idx helps to reduce the number of notifications between the device and the driver. It allows them to specify an index on the circular descriptors rings where to issue the notification, instead of a single binary indicator.

Adding support for SVQ.

These patches are sent on top of [1] series, so trivial conflicts could arise if it is applied directly on master. Future versions can be not based on it is more convenient.

[1] https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03280.html

Eugenio Pérez (4):
vhost: allocate event_idx fields on vring
vhost: toggle device callbacks using used event idx
vhost: use avail event idx on vhost_svq_kick
vhost: Accept event idx flag

 hw/virtio/vhost-shadow-virtqueue.c | 39 ++++++++++++++++++++++++------
 1 file changed, 31 insertions(+), 8 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH 1/4 vhost: allocate event_idx fields on vring

There was not enough room to accomodate them.

---
 hw/virtio/vhost-shadow-virtqueue.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +4 -4
index 596d4434d2..a518f84772 100644
@@ -570,16 +570,16 @@ void vhost_svq_get_vring_addr(const VhostShadowVirtqueue *svq,
 size_t vhost_svq_driver_area_size(const VhostShadowVirtqueue *svq)
 {
     size_t desc_size = sizeof(vring_desc_t) * svq->vring.num;
-    size_t avail_size = offsetof(vring_avail_t, ring) +
-                                             sizeof(uint16_t) * svq->vring.num;
+    size_t avail_size = offsetof(vring_avail_t, ring[svq->vring.num]) +
+                                                              sizeof(uint16_t);
 
     return ROUND_UP(desc_size + avail_size, qemu_real_host_page_size());
 }
 
 size_t vhost_svq_device_area_size(const VhostShadowVirtqueue *svq)
 {
-    size_t used_size = offsetof(vring_used_t, ring) +
-                                    sizeof(vring_used_elem_t) * svq->vring.num;
+    size_t used_size = offsetof(vring_used_t, ring[svq->vring.num]) +
+                                                              sizeof(uint16_t);
     return ROUND_UP(used_size, qemu_real_host_page_size());
 }
 
Signature · 1 line
2.31.1
#2 lore raw

PATCH 2/4 vhost: toggle device callbacks using used event idx

In reply to Eugenio Pérez

Actually use the new field of the used ring and tell the device if SVQ wants to be notified.

The code is not reachable at the moment.

---
 hw/virtio/vhost-shadow-virtqueue.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +15 -3
index a518f84772..f5c0fad3fc 100644
@@ -369,15 +369,27 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)
  */
 static bool vhost_svq_enable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
-    /* Make sure the flag is written before the read of used_idx */
+    if (virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        uint16_t *used_event = (uint16_t *)&svq->vring.avail->ring[svq->vring.num];
+        *used_event = svq->shadow_used_idx;
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
Signature · 1 line
2.31.1
#3 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 16 lines · Eugenio Pérez

Actually use the new field of the used ring and tell the device if SVQ wants to be notified.

The code is not reachable at the moment.

---
 hw/virtio/vhost-shadow-virtqueue.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +0 -0
index a518f84772..f5c0fad3fc 100644

hw/virtio/vhost-shadow-virtqueue.c static bool vhost_svq_more_used(VhostShadowVirtqueue *svq) · line 374

Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +3 -2
@@ -369,15 +369,27 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)
  */
 static bool vhost_svq_enable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
-    /* Make sure the flag is written before the read of used_idx */
+    if (virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        uint16_t *used_event = (uint16_t *)&svq->vring.avail->ring[svq->vring.num];
+        *used_event = svq->shadow_used_idx;

Do we need to care about the endian here?

E.g vduse has:

    *((uint16_t *)&vq->vring.used->ring[vq->vring.num]) = htole16(val);

Thanks

Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#4 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 29 lines · Jason Wang
Quoted 16 lines · Eugenio Pérez

Actually use the new field of the used ring and tell the device if SVQ wants to be notified.

The code is not reachable at the moment.

---
 hw/virtio/vhost-shadow-virtqueue.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +0 -0
index a518f84772..f5c0fad3fc 100644

hw/virtio/vhost-shadow-virtqueue.c static bool vhost_svq_more_used(VhostShadowVirtqueue *svq) · line 374

Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +3 -2
@@ -369,15 +369,27 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)
  */
 static bool vhost_svq_enable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
-    /* Make sure the flag is written before the read of used_idx */
+    if (virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        uint16_t *used_event = (uint16_t *)&svq->vring.avail->ring[svq->vring.num];
+        *used_event = svq->shadow_used_idx;

Do we need to care about the endian here?

Quoted 4 lines

E.g vduse has:

    *((uint16_t *)&vq->vring.used->ring[vq->vring.num]) = htole16(val);

Good catch, I forgot about endianness.

I'll review the series, thanks!

Quoted 28 lines

Thanks

Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#5 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 30 lines · Eugenio Perez Martin
Quoted 29 lines · Jason Wang
Quoted 16 lines · Eugenio Pérez

Actually use the new field of the used ring and tell the device if SVQ wants to be notified.

The code is not reachable at the moment.

---
 hw/virtio/vhost-shadow-virtqueue.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +0 -0
index a518f84772..f5c0fad3fc 100644

hw/virtio/vhost-shadow-virtqueue.c static bool vhost_svq_more_used(VhostShadowVirtqueue *svq) · line 374

Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +3 -2
@@ -369,15 +369,27 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)
  */
 static bool vhost_svq_enable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
-    /* Make sure the flag is written before the read of used_idx */
+    if (virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        uint16_t *used_event = (uint16_t *)&svq->vring.avail->ring[svq->vring.num];
+        *used_event = svq->shadow_used_idx;

Do we need to care about the endian here?

Quoted 8 lines
Quoted 4 lines

E.g vduse has:

    *((uint16_t *)&vq->vring.used->ring[vq->vring.num]) = htole16(val);

Good catch, I forgot about endianness.

I'll review the series, thanks!

It's generally a waste that we don't use endian-ness annotations the way linux does.

Quoted 28 lines
Quoted 28 lines

Thanks

Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#6 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 37 lines · Michael S. Tsirkin
Quoted 35 lines · Eugenio Perez Martin
Quoted 33 lines · Jason Wang
Quoted 16 lines · Eugenio Pérez

Actually use the new field of the used ring and tell the device if SVQ wants to be notified.

The code is not reachable at the moment.

---
 hw/virtio/vhost-shadow-virtqueue.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +0 -0
index a518f84772..f5c0fad3fc 100644

hw/virtio/vhost-shadow-virtqueue.c static bool vhost_svq_more_used(VhostShadowVirtqueue *svq) · line 374

Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +3 -2
@@ -369,15 +369,27 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)
  */
 static bool vhost_svq_enable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
-    /* Make sure the flag is written before the read of used_idx */
+    if (virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        uint16_t *used_event = (uint16_t *)&svq->vring.avail->ring[svq->vring.num];
+        *used_event = svq->shadow_used_idx;

Do we need to care about the endian here?

E.g vduse has:

    *((uint16_t *)&vq->vring.used->ring[vq->vring.num]) = htole16(val);
Quoted 6 lines
Quoted 3 lines

Good catch, I forgot about endianness.

I'll review the series, thanks!

It's generally a waste that we don't use endian-ness annotations the way linux does.

Yes, it's worth doing something similar sometime.

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

Thanks

Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#7 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 39 lines · Jason Wang
Quoted 37 lines · Michael S. Tsirkin
Quoted 35 lines · Eugenio Perez Martin
Quoted 33 lines · Jason Wang
Quoted 16 lines · Eugenio Pérez

Actually use the new field of the used ring and tell the device if SVQ wants to be notified.

The code is not reachable at the moment.

---
 hw/virtio/vhost-shadow-virtqueue.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +0 -0
index a518f84772..f5c0fad3fc 100644

hw/virtio/vhost-shadow-virtqueue.c static bool vhost_svq_more_used(VhostShadowVirtqueue *svq) · line 374

Quoted 8 lines
hw/virtio/vhost-shadow-virtqueue.c +3 -2
@@ -369,15 +369,27 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *svq)
  */
 static bool vhost_svq_enable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
-    /* Make sure the flag is written before the read of used_idx */
+    if (virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        uint16_t *used_event = (uint16_t *)&svq->vring.avail->ring[svq->vring.num];
+        *used_event = svq->shadow_used_idx;

Do we need to care about the endian here?

E.g vduse has:

    *((uint16_t *)&vq->vring.used->ring[vq->vring.num]) = htole16(val);
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Good catch, I forgot about endianness.

I'll review the series, thanks!

It's generally a waste that we don't use endian-ness annotations the way linux does.

Yes, it's worth doing something similar sometime.

Maybe we could wrap them in some struct like virtio_le16 or virtio_16, avoiding at least integer direct assignment? Wrappers like cpu_to_virtio16 could return these structs and I think all compilers should emit the same code as direct assignment.

Thanks!

Quoted 34 lines

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

Thanks

Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#8 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 6 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 2 lines

It's generally a waste that we don't use endian-ness annotations the way linux does.

Yes, it's worth doing something similar sometime.

Quoted 7 lines

Maybe we could wrap them in some struct like virtio_le16 or virtio_16, avoiding at least integer direct assignment? Wrappers like cpu_to_virtio16 could return these structs and I think all compilers should emit the same code as direct assignment.

Thanks!

This will break bitwise operations such as | and &. Generally Linux has solved the problem and I don't think we should go look for another solution.

Quoted 36 lines
Quoted 36 lines

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

Thanks

Quoted 25 lines
hw/virtio/vhost-shadow-virtqueue.c +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#9 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 2 lines

It's generally a waste that we don't use endian-ness annotations the way linux does.

Yes, it's worth doing something similar sometime.

Maybe we could wrap them in some struct like virtio_le16 or virtio_16, avoiding at least integer direct assignment? Wrappers like cpu_to_virtio16 could return these structs and I think all compilers should emit the same code as direct assignment.

Quoted 6 lines
Quoted 2 lines

Thanks!

This will break bitwise operations such as | and &. Generally Linux has solved the problem and I don't think we should go look for another solution.

Yes, but it should not block this series (we can do that in the future if we had bandwidth).

Thanks

Quoted 39 lines
Quoted 38 lines
Quoted 38 lines

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

Thanks

Quoted 25 lines
(quoted diff) +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#10 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 17 lines · Jason Wang
Quoted 16 lines · Michael S. Tsirkin
Quoted 13 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 2 lines

It's generally a waste that we don't use endian-ness annotations the way linux does.

Yes, it's worth doing something similar sometime.

Maybe we could wrap them in some struct like virtio_le16 or virtio_16, avoiding at least integer direct assignment? Wrappers like cpu_to_virtio16 could return these structs and I think all compilers should emit the same code as direct assignment.

Thanks!

Quoted 8 lines
Quoted 3 lines

This will break bitwise operations such as | and &. Generally Linux has solved the problem and I don't think we should go look for another solution.

Yes, but it should not block this series (we can do that in the future if we had bandwidth).

Thanks

Sorry I don't get what you are saying. Which series?
Making LE tags structs is not going to fly, this is why sparse implements the __bitwise__ tag and in fact this is where the name comes from - bitwise operations need to work.

Quoted 39 lines
Quoted 39 lines
Quoted 38 lines
Quoted 38 lines

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

Thanks

Quoted 25 lines
(quoted diff) +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#11 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Jason Wang
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 2 lines

It's generally a waste that we don't use endian-ness annotations the way linux does.

Yes, it's worth doing something similar sometime.

Quoted 7 lines

Maybe we could wrap them in some struct like virtio_le16 or virtio_16, avoiding at least integer direct assignment? Wrappers like cpu_to_virtio16 could return these structs and I think all compilers should emit the same code as direct assignment.

Thanks!

This will break bitwise operations such as | and &. Generally Linux has solved the problem and I don't think we should go look for another solution.

Quoted 6 lines
Quoted 4 lines

Yes, but it should not block this series (we can do that in the future if we had bandwidth).

Thanks

Sorry I don't get what you are saying. Which series?

I meant this series.

Quoted 3 lines

Making LE tags structs is not going to fly, this is why sparse implements the __bitwise__ tag and in fact this is where the name comes from - bitwise operations need to work.

Yes but do we want to add sparse checks in this series only for shadow virtqueue code?

Thanks

Quoted 43 lines
Quoted 42 lines
Quoted 42 lines
Quoted 41 lines
Quoted 41 lines

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

Thanks

Quoted 25 lines
(quoted diff) +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#12 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

Quoted 33 lines · Jason Wang
Quoted 23 lines · Michael S. Tsirkin
Quoted 21 lines · Jason Wang
Quoted 19 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 2 lines

It's generally a waste that we don't use endian-ness annotations the way linux does.

Yes, it's worth doing something similar sometime.

Quoted 7 lines

Maybe we could wrap them in some struct like virtio_le16 or virtio_16, avoiding at least integer direct assignment? Wrappers like cpu_to_virtio16 could return these structs and I think all compilers should emit the same code as direct assignment.

Thanks!

This will break bitwise operations such as | and &. Generally Linux has solved the problem and I don't think we should go look for another solution.

Quoted 6 lines
Quoted 4 lines

Yes, but it should not block this series (we can do that in the future if we had bandwidth).

Thanks

Sorry I don't get what you are saying. Which series?

I meant this series.

Quoted 8 lines
Quoted 3 lines

Making LE tags structs is not going to fly, this is why sparse implements the __bitwise__ tag and in fact this is where the name comes from - bitwise operations need to work.

Yes but do we want to add sparse checks in this series only for shadow virtqueue code?

Thanks

No, I think a concerted effort to add endian-ness tagging is more appropriate. Has nothing to do with shadow vq.

Quoted 43 lines
Quoted 43 lines
Quoted 42 lines
Quoted 42 lines
Quoted 41 lines
Quoted 41 lines

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

Thanks

Quoted 25 lines
(quoted diff) +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#13 lore raw

Re: [PATCH 2/4] vhost: toggle device callbacks using used event idx

In reply to Michael S. Tsirkin
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · Eugenio Perez Martin
Quoted 5 lines
Quoted 2 lines

It's generally a waste that we don't use endian-ness annotations the way linux does.

Yes, it's worth doing something similar sometime.

Maybe we could wrap them in some struct like virtio_le16 or virtio_16, avoiding at least integer direct assignment? Wrappers like cpu_to_virtio16 could return these structs and I think all compilers should emit the same code as direct assignment.

Quoted 7 lines
Quoted 2 lines

Thanks!

This will break bitwise operations such as | and &. Generally Linux has solved the problem and I don't think we should go look for another solution.

That's right, we would need to do it with functions like we do with the Int128 type. The idea is the same, do not mix operations with bitwise integers and cpu type ones.

But I totally agree a sparse tag or similar is way more clean and convenient.

Quoted 38 lines
Quoted 37 lines
Quoted 37 lines

Thanks

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines

Thanks

Quoted 25 lines
(quoted diff) +12 -2
+    } else {
+        svq->vring.avail->flags &= ~cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
+
+    /* Make sure the event is enabled before the read of used_idx */
     smp_mb();
     return !vhost_svq_more_used(svq);
 }
 
 static void vhost_svq_disable_notification(VhostShadowVirtqueue *svq)
 {
-    svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    /*
+     * No need to disable notification in the event idx case, since used event
+     * index is already an index too far away.
+     */
+    if (!virtio_vdev_has_feature(svq->vdev, VIRTIO_RING_F_EVENT_IDX)) {
+        svq->vring.avail->flags |= cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT);
+    }
 }
 
 static uint16_t vhost_svq_last_desc_of_chain(const VhostShadowVirtqueue *svq,
--

2.31.1

#14 lore raw

PATCH 4/4 vhost: Accept event idx flag

In reply to Eugenio Pérez

Enabling all the code path created before.

---
 hw/virtio/vhost-shadow-virtqueue.c | 1 +
 1 file changed, 1 insertion(+)
hw/virtio/vhost-shadow-virtqueue.c +1 -0
index f306ebef72..5bd14cad96 100644
@@ -33,6 +33,7 @@ bool vhost_svq_valid_features(uint64_t features, Error **errp)
          ++b) {
         switch (b) {
         case VIRTIO_F_ANY_LAYOUT:
+        case VIRTIO_RING_F_EVENT_IDX:
             continue;
 
         case VIRTIO_F_ACCESS_PLATFORM:
Signature · 1 line
2.31.1
Reviewed-by on the v1 series
#15 lore raw
In reply to Eugenio Pérez
Quoted 6 lines · Eugenio Pérez

Event idx helps to reduce the number of notifications between the device and the driver. It allows them to specify an index on the circular descriptors rings where to issue the notification, instead of a single binary indicator.

Adding support for SVQ.

Jason seems to be taking this through net

Quoted 18 lines

These patches are sent on top of [1] series, so trivial conflicts could arise if it is applied directly on master. Future versions can be not based on it is more convenient.

[1] https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03280.html

Eugenio Pérez (4):
vhost: allocate event_idx fields on vring
vhost: toggle device callbacks using used event idx
vhost: use avail event idx on vhost_svq_kick
vhost: Accept event idx flag

 hw/virtio/vhost-shadow-virtqueue.c | 39 ++++++++++++++++++++++++------
 1 file changed, 31 insertions(+), 8 deletions(-)
Signature · 1 line
2.31.1
#16 lore raw
Quoted 11 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez

Event idx helps to reduce the number of notifications between the device and the driver. It allows them to specify an index on the circular descriptors rings where to issue the notification, instead of a single binary indicator.

Adding support for SVQ.

Jason seems to be taking this through net

Ok, I've queued this.

Eugenio, please post the fix for endian on top.

Thanks

Quoted 19 lines
Quoted 19 lines

These patches are sent on top of [1] series, so trivial conflicts could arise if it is applied directly on master. Future versions can be not based on it is more convenient.

[1] https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03280.html

Eugenio Pérez (4):
vhost: allocate event_idx fields on vring
vhost: toggle device callbacks using used event idx
vhost: use avail event idx on vhost_svq_kick
vhost: Accept event idx flag

  hw/virtio/vhost-shadow-virtqueue.c | 39 ++++++++++++++++++++++++------
  1 file changed, 31 insertions(+), 8 deletions(-)
Signature · 1 line
2.31.1
#17 lore raw
Quoted 11 lines · Jason Wang
Quoted 8 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez

Event idx helps to reduce the number of notifications between the device and the driver. It allows them to specify an index on the circular descriptors rings where to issue the notification, instead of a single binary indicator.

Adding support for SVQ.

Quoted 9 lines
Quoted 3 lines

Jason seems to be taking this through net

Ok, I've queued this.

Eugenio, please post the fix for endian on top.

I've got a v2 ready to send, would it be possible to send it right now and send a v2 pull? That would save a few commits where the vdpa devices do not work with big endian architectures.

Thanks!

#18 lore raw
Quoted 18 lines · Eugenio Perez Martin
Quoted 16 lines · Jason Wang
Quoted 11 lines · Michael S. Tsirkin
Quoted 6 lines · Eugenio Pérez

Event idx helps to reduce the number of notifications between the device and the driver. It allows them to specify an index on the circular descriptors rings where to issue the notification, instead of a single binary indicator.

Adding support for SVQ.

Jason seems to be taking this through net

Quoted 8 lines
Quoted 4 lines

Ok, I've queued this.

Eugenio, please post the fix for endian on top.

I've got a v2 ready to send, would it be possible to send it right now and send a v2 pull? That would save a few commits where the vdpa devices do not work with big endian architectures.

Probably too late, but I queue the fixes on top to rc1.

Thanks

Quoted 3 lines

Thanks!