vhost: force vhost off for non-MSI guests

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

---

I need to report some error from virtio-pci
that would be handled specially (disable but don't
report an error) so I wanted one that's never likely to be used by a userspace ioctl. I selected ERANGE but it'd
be easy to switch to something else. Comments?

  hw/vhost.c      |    4 +++-
  hw/virtio-net.c |    6 ++++--
  hw/virtio-pci.c |    3 +++
  hw/virtio.h     |    2 ++
  4 files changed, 12 insertions(+), 3 deletions(-)

hw/vhost.c +3 -1

index 1d09ed0..c79765a 100644
@@ -649,7 +649,9 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
 
      r = vdev->binding->set_guest_notifiers(vdev->binding_opaque, true);
      if (r<  0) {
-        fprintf(stderr, "Error binding guest notifier: %d\n", -r);
+	if (r != -EVIRTIO_DISABLED) {
+		fprintf(stderr, "Error binding guest notifier: %d\n", -r);
+	}
          goto fail_notifiers;
      }
 

hw/virtio-net.c +4 -2

index ccb3e63..5de3fee 100644
@@ -121,8 +121,10 @@ static void virtio_net_vhost_status(VirtIONet *n, uint8_t status)
      if (!n->vhost_started) {
          int r = vhost_net_start(tap_get_vhost_net(n->nic->nc.peer),&n->vdev);
          if (r<  0) {
-            error_report("unable to start vhost net: %d: "
-                         "falling back on userspace virtio", -r);
+            if (r != -EVIRTIO_DISABLED) {
+                error_report("unable to start vhost net: %d: "
+                             "falling back on userspace virtio", -r);
+            }
          } else {
              n->vhost_started = 1;
          }

hw/virtio-pci.c +3 -0

index dd8887a..dbf4be0 100644
@@ -628,6 +628,9 @@ static int virtio_pci_set_guest_notifier(void *opaque, int n, bool assign)
      EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
 
      if (assign) {
+        if (!msix_enabled(&proxy->pci_dev)) {
+            return -EVIRTIO_DISABLED;
+        }
          int r = event_notifier_init(notifier, 0);
          if (r<  0) {
              return r;

hw/virtio.h +2 -0

index d8546d5..53bbdba 100644
@@ -98,6 +98,8 @@ typedef struct {
      void (*vmstate_change)(void * opaque, bool running);
  } VirtIOBindings;
 
+#define EVIRTIO_DISABLED ERANGE
+
  #define VIRTIO_PCI_QUEUE_MAX 64
 
  #define VIRTIO_NO_VECTOR 0xffff

Quoted 9 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

Quoted 15 lines · Anthony Liguori

Quoted 10 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Quoted 8 lines

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

The issue is that user has no control of the guest, and can not know whether the guest enables MSI. So what you ask for will just make some guests fail, and others fail sometimes.
The user also has no way to know that version X of kvm does not expose a way to inject level interrupts with irqfd.

We could have *another* flag that says "use vhost where it helps" but then I think this is what everyone wants to do, anyway, and libvirt already sets vhost=on so I prefer redefining the meaning of an existing flag.

Maybe this is best handled by a documentation update?

We always said:
" use vhost=on to enable experimental in kernel accelerator\n"

note 'enable' not 'require'. This is similar to how we specify nvectors : you can not make guest use the feature.

How about this:

qemu-options.hx +1 -0

index 898561d..3c937c1 100644
@@ -1061,6 +1061,7 @@ DEF("net", HAS_ARG, QEMU_OPTION_net,
      "                use vnet_hdr=off to avoid enabling the IFF_VNET_HDR tap flag\n"
      "                use vnet_hdr=on to make the lack of IFF_VNET_HDR support an error condition\n"
      "                use vhost=on to enable experimental in kernel accelerator\n"
+    "                (note: vhost=on has no effect unless guest uses MSI-X)\n"
      "                use 'vhostfd=h' to connect to an already opened vhost net device\n"
  #endif
      "-net socket[,vlan=n][,name=str][,fd=h][,listen=[host]:port][,connect=host:port]\n"

Quoted 31 lines · Michael S. Tsirkin

Quoted 15 lines · Anthony Liguori

Quoted 10 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Quoted 8 lines

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

The issue is that user has no control of the guest, and can not know whether the guest enables MSI. So what you ask for will just make some guests fail, and others fail sometimes.
The user also has no way to know that version X of kvm does not expose a way to inject level interrupts with irqfd.

Quoted 5 lines

We could have *another* flag that says "use vhost where it helps" but then I think this is what everyone wants to do, anyway, and libvirt already sets vhost=on so I prefer redefining the meaning of an existing flag.

In the very least, there needs to be a vhost=force.

Having some sort of friendly default policy is fine but we need to provide a mechanism for a user to have the final say. If you want to redefine vhost=on to really mean, use the friendly default, that's fine by me, but only if the vhost=force option exists.

I actually would think libvirt would want to use vhost=force. Debugging with vhost=on is going to be a royal pain in the ass if a user reports bad performance. Given the libvirt XML, you can't actually tell from the guest and the XML whether or not vhost was actually in use or not.

Quoted 45 lines · Anthony Liguori

Quoted 31 lines · Michael S. Tsirkin

Quoted 15 lines · Anthony Liguori

Quoted 10 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Quoted 8 lines

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

The issue is that user has no control of the guest, and can not know whether the guest enables MSI. So what you ask for will just make some guests fail, and others fail sometimes.
The user also has no way to know that version X of kvm does not expose a way to inject level interrupts with irqfd.

Quoted 5 lines

We could have *another* flag that says "use vhost where it helps" but then I think this is what everyone wants to do, anyway, and libvirt already sets vhost=on so I prefer redefining the meaning of an existing flag.

In the very least, there needs to be a vhost=force.

Having some sort of friendly default policy is fine but we need to provide a mechanism for a user to have the final say. If you want to redefine vhost=on to really mean, use the friendly default, that's fine by me, but only if the vhost=force option exists.

Quoted 4 lines

I actually would think libvirt would want to use vhost=force. Debugging with vhost=on is going to be a royal pain in the ass if a user reports bad performance. Given the libvirt XML, you can't actually tell from the guest and the XML whether or not vhost was actually in use or not.

If we add a force option, let's please distinguish hotplug from VM creation time. The latter can abort. Hotplug should print an error and fail the initfn.

Thanks,

Alex

Quoted 51 lines · Alex Williamson

Quoted 49 lines · Anthony Liguori

Quoted 31 lines · Michael S. Tsirkin

Quoted 15 lines · Anthony Liguori

Quoted 10 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Quoted 8 lines

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

The issue is that user has no control of the guest, and can not know whether the guest enables MSI. So what you ask for will just make some guests fail, and others fail sometimes.
The user also has no way to know that version X of kvm does not expose a way to inject level interrupts with irqfd.

Quoted 5 lines

We could have *another* flag that says "use vhost where it helps" but then I think this is what everyone wants to do, anyway, and libvirt already sets vhost=on so I prefer redefining the meaning of an existing flag.

In the very least, there needs to be a vhost=force.

Having some sort of friendly default policy is fine but we need to provide a mechanism for a user to have the final say. If you want to redefine vhost=on to really mean, use the friendly default, that's fine by me, but only if the vhost=force option exists.

I actually would think libvirt would want to use vhost=force. Debugging with vhost=on is going to be a royal pain in the ass if a user reports bad performance. Given the libvirt XML, you can't actually tell from the guest and the XML whether or not vhost was actually in use or not.

Quoted 3 lines

If we add a force option, let's please distinguish hotplug from VM creation time. The latter can abort. Hotplug should print an error and fail the initfn.

It can't abort at init - MSI is disabled at init, it needs to be enabled by the guest later. And aborting the guest in the middle of the run is a very bad idea.

Quoted 52 lines · Michael S. Tsirkin

Quoted 51 lines · Alex Williamson

Quoted 49 lines · Anthony Liguori

Quoted 31 lines · Michael S. Tsirkin

Quoted 15 lines · Anthony Liguori

Quoted 10 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Quoted 8 lines

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

The issue is that user has no control of the guest, and can not know whether the guest enables MSI. So what you ask for will just make some guests fail, and others fail sometimes.
The user also has no way to know that version X of kvm does not expose a way to inject level interrupts with irqfd.

Quoted 5 lines

We could have *another* flag that says "use vhost where it helps" but then I think this is what everyone wants to do, anyway, and libvirt already sets vhost=on so I prefer redefining the meaning of an existing flag.

In the very least, there needs to be a vhost=force.

Having some sort of friendly default policy is fine but we need to provide a mechanism for a user to have the final say. If you want to redefine vhost=on to really mean, use the friendly default, that's fine by me, but only if the vhost=force option exists.

I actually would think libvirt would want to use vhost=force. Debugging with vhost=on is going to be a royal pain in the ass if a user reports bad performance. Given the libvirt XML, you can't actually tell from the guest and the XML whether or not vhost was actually in use or not.

Quoted 7 lines

Quoted 3 lines

If we add a force option, let's please distinguish hotplug from VM creation time. The latter can abort. Hotplug should print an error and fail the initfn.

It can't abort at init - MSI is disabled at init, it needs to be enabled by the guest later. And aborting the guest in the middle of the run is a very bad idea.

Yeah, I was thinking about the ordering of device being added vs guest enabling MSI this morning. Waiting until the guest decides to try to start using the device to NAK it with an abort is very undesirable. What if when we have vhost=on,force, the device doesn't advertise an INTx (PCI_INTERRUPT_PIN = 0)?

Alex

Quoted 52 lines · Alex Williamson

Quoted 50 lines · Anthony Liguori

Quoted 35 lines · Michael S. Tsirkin

Quoted 17 lines · Anthony Liguori

Quoted 11 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Quoted 9 lines

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

The issue is that user has no control of the guest, and can not know whether the guest enables MSI. So what you ask for will just make some guests fail, and others fail sometimes.
The user also has no way to know that version X of kvm does not expose a way to inject level interrupts with irqfd.

Quoted 6 lines

We could have *another* flag that says "use vhost where it helps" but then I think this is what everyone wants to do, anyway, and libvirt already sets vhost=on so I prefer redefining the meaning of an existing flag.

In the very least, there needs to be a vhost=force.

Having some sort of friendly default policy is fine but we need to provide a mechanism for a user to have the final say. If you want to redefine vhost=on to really mean, use the friendly default, that's fine by me, but only if the vhost=force option exists.

Quoted 9 lines

Quoted 5 lines

I actually would think libvirt would want to use vhost=force. Debugging with vhost=on is going to be a royal pain in the ass if a user reports bad performance. Given the libvirt XML, you can't actually tell from the guest and the XML whether or not vhost was actually in use or not.

If we add a force option, let's please distinguish hotplug from VM creation time. The latter can abort. Hotplug should print an error and fail the initfn.

It can't abort at init - MSI is disabled at init, it needs to be enabled by the guest later. And aborting the guest in the middle of the run is a very bad idea.

What vhostforce=true will do is force vhost backend to be used even if it is slower.

Quoted 5 lines

Thanks,

Alex

Quoted 27 lines · Michael S. Tsirkin

Quoted 13 lines · Anthony Liguori

Quoted 9 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Quoted 7 lines

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

The issue is that user has no control of the guest, and can not know whether the guest enables MSI. So what you ask for will just make some guests fail, and others fail sometimes.
The user also has no way to know that version X of kvm does not expose a way to inject level interrupts with irqfd.

Quoted 4 lines

We could have *another* flag that says "use vhost where it helps" but then I think this is what everyone wants to do, anyway, and libvirt already sets vhost=on so I prefer redefining the meaning of an existing flag.

In the very least, there needs to be a vhost=force. Having some sort of friendly default policy is fine but we need to provide a mechanism for a user to have the final say. If you want to redefine vhost=on to really mean, use the friendly default, that's fine by me, but only if the vhost=force option exists.

I actually would think libvirt would want to use vhost=force. Debugging with vhost=on is going to be a royal pain in the ass if a user reports bad performance. Given the libvirt XML, you can't actually tell from the guest and the XML whether or not vhost was actually in use or not.

Regards,

Anthony Liguori

Quoted 24 lines

Maybe this is best handled by a documentation update?

We always said:
" use vhost=on to enable experimental in kernel accelerator\n"

note 'enable' not 'require'. This is similar to how we specify nvectors : you can not make guest use the feature.

How about this:

qemu-options.hx +1 -0

index 898561d..3c937c1 100644
@@ -1061,6 +1061,7 @@ DEF("net", HAS_ARG, QEMU_OPTION_net,
     "                use vnet_hdr=off to avoid enabling the IFF_VNET_HDR tap flag\n"
     "                use vnet_hdr=on to make the lack of IFF_VNET_HDR support an error condition\n"
     "                use vhost=on to enable experimental in kernel accelerator\n"
+    "                (note: vhost=on has no effect unless guest uses MSI-X)\n"
     "                use 'vhostfd=h' to connect to an already opened vhost net device\n"
 #endif
     "-net socket[,vlan=n][,name=str][,fd=h][,listen=[host]:port][,connect=host:port]\n"

Quoted 38 lines · Anthony Liguori

Quoted 31 lines · Michael S. Tsirkin

Quoted 15 lines · Anthony Liguori

Quoted 10 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin<[email protected]>

I actually think this should be a terminal error. The user asks for vhost-net, if we cannot enable it, we should exit.

Quoted 8 lines

Or we should warn the user that they should expect bad performance. Silently doing something that the user has explicitly asked us not to do is not a good behavior.

Regards,

Anthony Liguori

The issue is that user has no control of the guest, and can not know whether the guest enables MSI. So what you ask for will just make some guests fail, and others fail sometimes.
The user also has no way to know that version X of kvm does not expose a way to inject level interrupts with irqfd.

Quoted 5 lines

We could have *another* flag that says "use vhost where it helps" but then I think this is what everyone wants to do, anyway, and libvirt already sets vhost=on so I prefer redefining the meaning of an existing flag.

Quoted 6 lines

In the very least, there needs to be a vhost=force. Having some sort of friendly default policy is fine but we need to provide a mechanism for a user to have the final say. If you want to redefine vhost=on to really mean, use the friendly default, that's fine by me, but only if the vhost=force option exists.

OK, I will add that, probably as a separate flag as vhost is a boolean. This will get worse performance but it will be what the user asked for.

Quoted 6 lines

I actually would think libvirt would want to use vhost=force. Debugging with vhost=on is going to be a royal pain in the ass if a user reports bad performance. Given the libvirt XML, you can't actually tell from the guest and the XML whether or not vhost was actually in use or not.

Yes you can: check MSI enabled in the guest, if it is - check vhost enabled in the XML. Not that bad at all, is it?

Quoted 5 lines

Regards,

Anthony Liguori

We get worse performance without MSI anyway, how is this different?

Quoted 25 lines

Quoted 25 lines

Maybe this is best handled by a documentation update?

We always said:
" use vhost=on to enable experimental in kernel accelerator\n"

note 'enable' not 'require'. This is similar to how we specify nvectors : you can not make guest use the feature.

How about this:

qemu-options.hx +1 -0

index 898561d..3c937c1 100644
@@ -1061,6 +1061,7 @@ DEF("net", HAS_ARG, QEMU_OPTION_net,
      "                use vnet_hdr=off to avoid enabling the IFF_VNET_HDR tap flag\n"
      "                use vnet_hdr=on to make the lack of IFF_VNET_HDR support an error condition\n"
      "                use vhost=on to enable experimental in kernel accelerator\n"
+    "                (note: vhost=on has no effect unless guest uses MSI-X)\n"
      "                use 'vhostfd=h' to connect to an already opened vhost net device\n"
  #endif
      "-net socket[,vlan=n][,name=str][,fd=h][,listen=[host]:port][,connect=host:port]\n"

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin <[email protected]>

---

I need to report some error from virtio-pci
that would be handled specially (disable but don't
report an error) so I wanted one that's never likely to be used by a userspace ioctl. I selected ERANGE but it'd
be easy to switch to something else. Comments?

 hw/vhost.c      |    4 +++-
 hw/virtio-net.c |    6 ++++--
 hw/virtio-pci.c |    3 +++
 hw/virtio.h     |    2 ++
 4 files changed, 12 insertions(+), 3 deletions(-)

hw/vhost.c +3 -1

index 1d09ed0..c79765a 100644
@@ -649,7 +649,9 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
 
     r = vdev->binding->set_guest_notifiers(vdev->binding_opaque, true);
     if (r < 0) {
-        fprintf(stderr, "Error binding guest notifier: %d\n", -r);
+	if (r != -EVIRTIO_DISABLED) {
+		fprintf(stderr, "Error binding guest notifier: %d\n", -r);
+	}
         goto fail_notifiers;
     }
 

hw/virtio-net.c +4 -2

index ccb3e63..5de3fee 100644
@@ -121,8 +121,10 @@ static void virtio_net_vhost_status(VirtIONet *n, uint8_t status)
     if (!n->vhost_started) {
         int r = vhost_net_start(tap_get_vhost_net(n->nic->nc.peer), &n->vdev);
         if (r < 0) {
-            error_report("unable to start vhost net: %d: "
-                         "falling back on userspace virtio", -r);
+            if (r != -EVIRTIO_DISABLED) {
+                error_report("unable to start vhost net: %d: "
+                             "falling back on userspace virtio", -r);
+            }
         } else {
             n->vhost_started = 1;
         }

hw/virtio-pci.c +3 -0

index dd8887a..dbf4be0 100644
@@ -628,6 +628,9 @@ static int virtio_pci_set_guest_notifier(void *opaque, int n, bool assign)
     EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
 
     if (assign) {
+        if (!msix_enabled(&proxy->pci_dev)) {
+            return -EVIRTIO_DISABLED;
+        }
         int r = event_notifier_init(notifier, 0);
         if (r < 0) {
             return r;

hw/virtio.h +2 -0

index d8546d5..53bbdba 100644
@@ -98,6 +98,8 @@ typedef struct {
     void (*vmstate_change)(void * opaque, bool running);
 } VirtIOBindings;
 
+#define EVIRTIO_DISABLED ERANGE
+
 #define VIRTIO_PCI_QUEUE_MAX 64
 
 #define VIRTIO_NO_VECTOR 0xffff

Quoted 8 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Quoted 8 lines

Signed-off-by Michael S. Tsirkin <[email protected]>

---

I need to report some error from virtio-pci
that would be handled specially (disable but don't
report an error) so I wanted one that's never likely to be used by a userspace ioctl. I selected ERANGE but it'd
be easy to switch to something else. Comments?

Should this error be EVHOST_DISABLED rather than EVIRTIO_DISABLED?

-Sridhar

Quoted 66 lines

 hw/vhost.c      |    4 +++-
 hw/virtio-net.c |    6 ++++--
 hw/virtio-pci.c |    3 +++
 hw/virtio.h     |    2 ++
 4 files changed, 12 insertions(+), 3 deletions(-)

hw/vhost.c +3 -1

index 1d09ed0..c79765a 100644
@@ -649,7 +649,9 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
 
     r = vdev->binding->set_guest_notifiers(vdev->binding_opaque, true);
     if (r < 0) {
-        fprintf(stderr, "Error binding guest notifier: %d\n", -r);
+	if (r != -EVIRTIO_DISABLED) {
+		fprintf(stderr, "Error binding guest notifier: %d\n", -r);
+	}
         goto fail_notifiers;
     }
 

hw/virtio-net.c +4 -2

index ccb3e63..5de3fee 100644
@@ -121,8 +121,10 @@ static void virtio_net_vhost_status(VirtIONet *n, uint8_t status)
     if (!n->vhost_started) {
         int r = vhost_net_start(tap_get_vhost_net(n->nic->nc.peer), &n->vdev);
         if (r < 0) {
-            error_report("unable to start vhost net: %d: "
-                         "falling back on userspace virtio", -r);
+            if (r != -EVIRTIO_DISABLED) {
+                error_report("unable to start vhost net: %d: "
+                             "falling back on userspace virtio", -r);
+            }
         } else {
             n->vhost_started = 1;
         }

hw/virtio-pci.c +3 -0

index dd8887a..dbf4be0 100644
@@ -628,6 +628,9 @@ static int virtio_pci_set_guest_notifier(void *opaque, int n, bool assign)
     EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
 
     if (assign) {
+        if (!msix_enabled(&proxy->pci_dev)) {
+            return -EVIRTIO_DISABLED;
+        }
         int r = event_notifier_init(notifier, 0);
         if (r < 0) {
             return r;

hw/virtio.h +2 -0

index d8546d5..53bbdba 100644
@@ -98,6 +98,8 @@ typedef struct {
     void (*vmstate_change)(void * opaque, bool running);
 } VirtIOBindings;
 
+#define EVIRTIO_DISABLED ERANGE
+
 #define VIRTIO_PCI_QUEUE_MAX 64
 
 #define VIRTIO_NO_VECTOR 0xffff

Quoted 18 lines · Sridhar Samudrala

Quoted 16 lines · Michael S. Tsirkin

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin <[email protected]>

---

I need to report some error from virtio-pci
that would be handled specially (disable but don't
report an error) so I wanted one that's never likely to be used by a userspace ioctl. I selected ERANGE but it'd
be easy to switch to something else. Comments?

Quoted 3 lines

Should this error be EVHOST_DISABLED rather than EVIRTIO_DISABLED?

-Sridhar

The error is reported by virtio-pci which does not know about vhost. I started with EVIRTIO_MSIX_DISABLED and made is shorter. Would EVIRTIO_MSIX_DISABLED be better?

Quoted 67 lines

Quoted 67 lines

 hw/vhost.c      |    4 +++-
 hw/virtio-net.c |    6 ++++--
 hw/virtio-pci.c |    3 +++
 hw/virtio.h     |    2 ++
 4 files changed, 12 insertions(+), 3 deletions(-)

hw/vhost.c +3 -1

index 1d09ed0..c79765a 100644
@@ -649,7 +649,9 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
 
     r = vdev->binding->set_guest_notifiers(vdev->binding_opaque, true);
     if (r < 0) {
-        fprintf(stderr, "Error binding guest notifier: %d\n", -r);
+	if (r != -EVIRTIO_DISABLED) {
+		fprintf(stderr, "Error binding guest notifier: %d\n", -r);
+	}
         goto fail_notifiers;
     }
 

hw/virtio-net.c +4 -2

index ccb3e63..5de3fee 100644
@@ -121,8 +121,10 @@ static void virtio_net_vhost_status(VirtIONet *n, uint8_t status)
     if (!n->vhost_started) {
         int r = vhost_net_start(tap_get_vhost_net(n->nic->nc.peer), &n->vdev);
         if (r < 0) {
-            error_report("unable to start vhost net: %d: "
-                         "falling back on userspace virtio", -r);
+            if (r != -EVIRTIO_DISABLED) {
+                error_report("unable to start vhost net: %d: "
+                             "falling back on userspace virtio", -r);
+            }
         } else {
             n->vhost_started = 1;
         }

hw/virtio-pci.c +3 -0

index dd8887a..dbf4be0 100644
@@ -628,6 +628,9 @@ static int virtio_pci_set_guest_notifier(void *opaque, int n, bool assign)
     EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
 
     if (assign) {
+        if (!msix_enabled(&proxy->pci_dev)) {
+            return -EVIRTIO_DISABLED;
+        }
         int r = event_notifier_init(notifier, 0);
         if (r < 0) {
             return r;

hw/virtio.h +2 -0

index d8546d5..53bbdba 100644
@@ -98,6 +98,8 @@ typedef struct {
     void (*vmstate_change)(void * opaque, bool running);
 } VirtIOBindings;
 
+#define EVIRTIO_DISABLED ERANGE
+
 #define VIRTIO_PCI_QUEUE_MAX 64
 
 #define VIRTIO_NO_VECTOR 0xffff

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

Signed-off-by Michael S. Tsirkin <[email protected]>

---

I need to report some error from virtio-pci
that would be handled specially (disable but don't
report an error) so I wanted one that's never likely to be used by a userspace ioctl. I selected ERANGE but it'd
be easy to switch to something else. Comments?

 hw/vhost.c      |    4 +++-
 hw/virtio-net.c |    6 ++++--
 hw/virtio-pci.c |    3 +++
 hw/virtio.h     |    2 ++
 4 files changed, 12 insertions(+), 3 deletions(-)

hw/vhost.c +0 -0

index 1d09ed0..c79765a 100644

hw/vhost.c +0 -0

         goto fail_notifiers;
     }
 

hw/virtio-net.c +4 -2

index ccb3e63..5de3fee 100644
@@ -121,8 +121,10 @@ static void virtio_net_vhost_status(VirtIONet *n, uint8_t status)
     if (!n->vhost_started) {
         int r = vhost_net_start(tap_get_vhost_net(n->nic->nc.peer), &n->vdev);
         if (r < 0) {
-            error_report("unable to start vhost net: %d: "
-                         "falling back on userspace virtio", -r);
+            if (r != -EVIRTIO_DISABLED) {
+                error_report("unable to start vhost net: %d: "
+                             "falling back on userspace virtio", -r);
+            }
         } else {
             n->vhost_started = 1;
         }

hw/virtio-pci.c +3 -0

index dd8887a..dbf4be0 100644
@@ -628,6 +628,9 @@ static int virtio_pci_set_guest_notifier(void *opaque, int n, bool assign)
     EventNotifier *notifier = virtio_queue_get_guest_notifier(vq);
 
     if (assign) {
+        if (!msix_enabled(&proxy->pci_dev)) {
+            return -EVIRTIO_DISABLED;
+        }
         int r = event_notifier_init(notifier, 0);
         if (r < 0) {
             return r;

hw/virtio.h +0 -0

index d8546d5..53bbdba 100644

When MSI is off, each interrupt needs to be bounced through the io thread when it's set/cleared, so vhost-net causes more context switches and higher CPU utilization than userspace virtio which handles networking in the same thread.

We'll need to fix this by adding level irq support in kvm irqfd, for now disable vhost-net in these configurations.

This maybe a novice question - Would appreciate it if you can you provide a

Quoted 2 lines

This maybe a novice question - Would appreciate it if you can you provide

pointer to documentation or relevant code that explains what is the *limitation in supporting level irq support in kvm irqfd.*

Seeking clarification to the original question I posted:

Quoted 8 lines

Quoted 2 lines

This maybe a novice question - Would appreciate it if you can you provide a pointer to documentation or relevant code that explains what is the limitation in supporting level irq support in kvm irqfd.

After browsing the KVM kernel code, it does look like direct assignment of PCI devices allows support for level-triggered interrupts to be injected to the guest from the kernel.  (as opposed to not supporting it for vhost irqfd mechanism)
This occurs when the guest device supports INTX.
Reference:  kvm_assigned_dev_interrupt_work_handler() in assigned-dev.c calls kvm_set_irq()
with the guest_irq.
This function in turn invokes the assigned set function  (either kvm_set_pic_irq or kvm_set_ioapic_irq) which was setup at kvm_irq_chip creation time when kvm_setup_default_irq_routing () called for handling ioctl KVM_CREATE_IRQCHIP.

So, it isn't clear why level-triggered interrupt isn't supported for irqfd mechanism.
Would greatly appreciate clarification here

Thanks
-Rukhsana

Quoted 22 lines · rukhsana ansari

Seeking clarification to the original question I posted:

Quoted 8 lines

Quoted 2 lines

This maybe a novice question - Would appreciate it if you can you provide a pointer to documentation or relevant code that explains what is the limitation in supporting level irq support in kvm irqfd.

After browsing the KVM kernel code, it does look like direct assignment of PCI devices allows support for level-triggered interrupts to be injected to the guest from the kernel. (as opposed to not supporting it for vhost irqfd mechanism)
This occurs when the guest device supports INTX.
Reference: kvm_assigned_dev_interrupt_work_handler() in assigned-dev.c calls kvm_set_irq()
with the guest_irq.
This function in turn invokes the assigned set function (either kvm_set_pic_irq or kvm_set_ioapic_irq) which was setup at kvm_irq_chip creation time when kvm_setup_default_irq_routing () called for handling ioctl KVM_CREATE_IRQCHIP.

Quoted 7 lines

So, it isn't clear why level-triggered interrupt isn't supported for irqfd mechanism.
Would greatly appreciate clarification here

Thanks
-Rukhsana

Mostly, no one came up with an implementation so far.

If the point is to use irqfd with vhost-net, there's also a question of adding interfaces to
1. pass IO read transactions directly to another kernel module
2. add an interface to clear the irq level

Maybe the right thing is to combine the two somehow:
irqfd might get an oiption to set a bit in memory, ioeventfd might get an option to read and clear from memory and clear irqfd line at the same time.

wrote:

Quoted 5 lines · Michael S. Tsirkin

Quoted 4 lines · rukhsana ansari

Seeking clarification to the original question I posted:

Quoted 3 lines

Quoted 2 lines

This maybe a novice question - Would appreciate it if you can you

provide a

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

pointer to documentation or relevant code that explains what is the limitation in supporting level irq support in kvm irqfd.

After browsing the KVM kernel code, it does look like direct assignment

of PCI

Quoted 1 line

Quoted 1 line

devices allows support for level-triggered interrupts to be injected to

the

Quoted 1 line

Quoted 1 line

guest from the kernel. (as opposed to not supporting it for vhost

irqfd

Quoted 3 lines

Quoted 3 lines

mechanism)
This occurs when the guest device supports INTX.
Reference: kvm_assigned_dev_interrupt_work_handler() in assigned-dev.c

calls

Quoted 4 lines

Quoted 4 lines

kvm_set_irq()
with the guest_irq.
This function in turn invokes the assigned set function (either kvm_set_pic_irq or kvm_set_ioapic_irq) which was setup at kvm_irq_chip

creation

Quoted 4 lines

Quoted 4 lines

time when kvm_setup_default_irq_routing () called for handling ioctl KVM_CREATE_IRQCHIP.

So, it isn't clear why level-triggered interrupt isn't supported for

irqfd

Quoted 14 lines

Quoted 6 lines

mechanism.
Would greatly appreciate clarification here

Thanks
-Rukhsana

Mostly, no one came up with an implementation so far.

If the point is to use irqfd with vhost-net, there's also a question of adding interfaces to
1. pass IO read transactions directly to another kernel module
2. add an interface to clear the irq level

Quoted 4 lines

Maybe the right thing is to combine the two somehow:
irqfd might get an oiption to set a bit in memory, ioeventfd might get an option to read and clear from memory and clear irqfd line at the same time.

I had wanted this for VFIO too and it gets pretty complicated. The first problem with level triggered interrupts is that you need to know which GSI your device triggers. This means translating PCI INTA through bridge swizzles and chipset mapping to an IOAPIC. Current device assignment does this through a complete hack in qemu. Then you can set the IRQ, but being level triggered, we need to know when the guest has serviced the IRQ so we can de-assert it. This requires a hook into the in-kernel APIC to sent the EOI back out to userspace.

I posted RFC patches for doing all this a while back, but they didn't go anywhere. I think the feeling was that it was too intrusive for "slow" interrupts. The current thinking for VFIO based device assignment is to use qemu for level interrupts until we find something that actually needs low latency in this path. We generally consider INTx to be like supporting i/o port space or non-4k BARs, ie. necessary for compatibility, but not necessarily a performance path. High performance devices should always be using some kind of MSI because it bypasses all of the APIC complications and slowness. Thanks,

Alex