Skip to contentSkip to conversation

virtio-pci: fix up config interrupt handling

PATCH In review Michael S. Tsirkin 3 participants 4 messages Jan 9, 2022 – Jan 10, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 virtio-pci: fix up config interrupt handling

Fixes a couple of issues with irqfd use by config interrupt:
- Rearrange initialization so cleanup happens in the reverse order
- Don't use irqfd for config when not in use for data path I am not sure this is a complete fix though: I think we are better off limiting the effect to vdpa devices
with config interrupt support. Or even bypass irqfd
for config completely and inject into KVM using ioctl? The advantage would be less FDs used.
This would mean mostly reverting the patchset though.

Fixes d5d24d859c ("virtio-pci: add support for configure interrupt")
Cc "Cindy Lu" <[email protected]>
---
 hw/virtio/virtio-pci.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
hw/virtio/virtio-pci.c +7 -5
index 98fb5493ae..b77cd69f97 100644
@@ -1130,15 +1130,15 @@ static int virtio_pci_set_guest_notifiers(DeviceState *d, int nvqs, bool assign)
             proxy->vector_irqfd =
                 g_malloc0(sizeof(*proxy->vector_irqfd) *
                           msix_nr_vectors_allocated(&proxy->pci_dev));
+            r = kvm_virtio_pci_vector_config_use(proxy);
+            if (r < 0) {
+                goto config_error;
+            }
             r = kvm_virtio_pci_vector_use(proxy, nvqs);
             if (r < 0) {
                 goto config_assign_error;
             }
         }
-        r = kvm_virtio_pci_vector_config_use(proxy);
-        if (r < 0) {
-            goto config_error;
-        }
         r = msix_set_vector_notifiers(&proxy->pci_dev, virtio_pci_vector_unmask,
                                       virtio_pci_vector_mask,
                                       virtio_pci_vector_poll);
@@ -1155,7 +1155,9 @@ notifiers_error:
         kvm_virtio_pci_vector_release(proxy, nvqs);
     }
 config_error:
-    kvm_virtio_pci_vector_config_release(proxy);
+    if (with_irqfd) {
+        kvm_virtio_pci_vector_config_release(proxy);
+    }
 config_assign_error:
     virtio_pci_set_guest_notifier(d, VIRTIO_CONFIG_IRQ_IDX, !assign,
                                   with_irqfd);
Signature · 1 line
MST
#1 lore raw
Quoted 10 lines · Michael S. Tsirkin

Fixes a couple of issues with irqfd use by config interrupt:
- Rearrange initialization so cleanup happens in the reverse order
- Don't use irqfd for config when not in use for data path I am not sure this is a complete fix though: I think we are better off limiting the effect to vdpa devices
with config interrupt support. Or even bypass irqfd
for config completely and inject into KVM using ioctl? The advantage would be less FDs used.
This would mean mostly reverting the patchset though.

Quoted 3 lines
Fixes d5d24d859c ("virtio-pci: add support for configure interrupt")
Cc "Cindy Lu" <[email protected]>

On a KVM guest with vhost, I am still seeing at reboot an issue :/

../hw/pci/msix.c:622: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier'

C.

Quoted 40 lines
---
  hw/virtio/virtio-pci.c | 12 +++++++-----
  1 file changed, 7 insertions(+), 5 deletions(-)
hw/virtio/virtio-pci.c +7 -5
index 98fb5493ae..b77cd69f97 100644
@@ -1130,15 +1130,15 @@ static int virtio_pci_set_guest_notifiers(DeviceState *d, int nvqs, bool assign)
              proxy->vector_irqfd =
                  g_malloc0(sizeof(*proxy->vector_irqfd) *
                            msix_nr_vectors_allocated(&proxy->pci_dev));
+            r = kvm_virtio_pci_vector_config_use(proxy);
+            if (r < 0) {
+                goto config_error;
+            }
              r = kvm_virtio_pci_vector_use(proxy, nvqs);
              if (r < 0) {
                  goto config_assign_error;
              }
          }
-        r = kvm_virtio_pci_vector_config_use(proxy);
-        if (r < 0) {
-            goto config_error;
-        }
          r = msix_set_vector_notifiers(&proxy->pci_dev, virtio_pci_vector_unmask,
                                        virtio_pci_vector_mask,
                                        virtio_pci_vector_poll);
@@ -1155,7 +1155,9 @@ notifiers_error:
          kvm_virtio_pci_vector_release(proxy, nvqs);
      }
  config_error:
-    kvm_virtio_pci_vector_config_release(proxy);
+    if (with_irqfd) {
+        kvm_virtio_pci_vector_config_release(proxy);
+    }
  config_assign_error:
      virtio_pci_set_guest_notifier(d, VIRTIO_CONFIG_IRQ_IDX, !assign,
                                    with_irqfd);
#2 lore raw
In reply to Michael S. Tsirkin

Hi Michael,

Quoted 11 lines · Michael S. Tsirkin

Fixes a couple of issues with irqfd use by config interrupt:
- Rearrange initialization so cleanup happens in the reverse order
- Don't use irqfd for config when not in use for data path I am not sure this is a complete fix though: I think we are better off limiting the effect to vdpa devices
with config interrupt support. Or even bypass irqfd
for config completely and inject into KVM using ioctl? The advantage would be less FDs used.
This would mean mostly reverting the patchset though.

Quoted 3 lines
Fixes d5d24d859c ("virtio-pci: add support for configure interrupt")
Cc "Cindy Lu" <[email protected]>

This doesn't seem to fix the problems I'm seeing here with a KVM/arm64 guest (the issue exists with and without this patch)

On initial boot:

<quote>
Loading Linux 5.7.0-1-arm64 ...
Loading initial ramdisk ...
qemu-system-aarch64: virtio-blk failed to set guest notifier (-16), ensure -accel kvm is set.
qemu-system-aarch64: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-system-aarch64: virtio-scsi: Failed to set guest notifiers (-16), ensure -accel kvm is set.
qemu-system-aarch64: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower). </quote>

The guest is functional though. However, on reboot:

<quote>
Loading Linux 5.7.0-1-arm64 ...
Loading initial ramdisk ...
qemu-system-aarch64: ../hw/pci/msix.c:622: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier' failed. </quote>

Reverting d5d24d859c fixes the issue. For the record, my qemu command line:

/home/maz/qemu/build/qemu-system-aarch64 -m 1G -smp 8 -cpu host,aarch64=on -machine virt,accel=kvm,gic-version=host -nographic -drive if=pflash,format=raw,readonly=on,file=/usr/share/AAVMF/AAVMF_CODE.fd -drive if=pflash,format=raw,file=bullseye/bBwcgtDY2UwXklV6.fd -netdev user,id=hostnet0 -device virtio-net-pci,netdev=hostnet0 -drive if=none,format=raw,cache=none,aio=native,file=bullseye/bBwcgtDY2UwXklV6.img,id=disk0 -device virtio-blk-pci,drive=disk0 -drive file=debian-testing-arm64-netinst-preseed.iso,id=cdrom,if=none,media=cdrom -device virtio-scsi-pci -device scsi-cd,drive=cdrom

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#3 lore raw
Quoted 41 lines · Marc Zyngier

Hi Michael,

Quoted 11 lines · Michael S. Tsirkin

Fixes a couple of issues with irqfd use by config interrupt:
- Rearrange initialization so cleanup happens in the reverse order
- Don't use irqfd for config when not in use for data path I am not sure this is a complete fix though: I think we are better off limiting the effect to vdpa devices
with config interrupt support. Or even bypass irqfd
for config completely and inject into KVM using ioctl? The advantage would be less FDs used.
This would mean mostly reverting the patchset though.

Quoted 3 lines
Fixes d5d24d859c ("virtio-pci: add support for configure interrupt")
Cc "Cindy Lu" <[email protected]>

This doesn't seem to fix the problems I'm seeing here with a KVM/arm64 guest (the issue exists with and without this patch)

On initial boot:

<quote>
Loading Linux 5.7.0-1-arm64 ...
Loading initial ramdisk ...
qemu-system-aarch64: virtio-blk failed to set guest notifier (-16), ensure -accel kvm is set.
qemu-system-aarch64: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
qemu-system-aarch64: virtio-scsi: Failed to set guest notifiers (-16), ensure -accel kvm is set.
qemu-system-aarch64: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower). </quote>

The guest is functional though. However, on reboot:

<quote>
Loading Linux 5.7.0-1-arm64 ...
Loading initial ramdisk ...
qemu-system-aarch64: ../hw/pci/msix.c:622: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier' failed. </quote>

Quoted 8 lines

Reverting d5d24d859c fixes the issue. For the record, my qemu command line:

/home/maz/qemu/build/qemu-system-aarch64 -m 1G -smp 8 -cpu host,aarch64=on -machine virt,accel=kvm,gic-version=host -nographic -drive if=pflash,format=raw,readonly=on,file=/usr/share/AAVMF/AAVMF_CODE.fd -drive if=pflash,format=raw,file=bullseye/bBwcgtDY2UwXklV6.fd -netdev user,id=hostnet0 -device virtio-net-pci,netdev=hostnet0 -drive if=none,format=raw,cache=none,aio=native,file=bullseye/bBwcgtDY2UwXklV6.img,id=disk0 -device virtio-blk-pci,drive=disk0 -drive file=debian-testing-arm64-netinst-preseed.iso,id=cdrom,if=none,media=cdrom -device virtio-scsi-pci -device scsi-cd,drive=cdrom

Thanks,

	M.

Thanks, I'm reverting these changes for now.
Cindy, once you have your patches ready pinging people who encountered problems with the previous versions is a very good idea.

Quoted 2 lines
Signature · 1 line
Without deviation from the norm, progress is not possible.