Extend vhost-user to support VFIO based accelerators

Signed-off-by Tiwei Bie <[email protected]>

---
 docs/interop/vhost-user.txt    |  57 ++++++
 hw/vfio/common.c               |   2 +-
 hw/virtio/vhost-user.c         | 381 ++++++++++++++++++++++++++++++++++++++++-
 hw/virtio/vhost.c              |   3 +-
 hw/virtio/virtio-pci.c         |   8 -
 hw/virtio/virtio-pci.h         |   8 +
 include/hw/vfio/vfio.h         |   2 +
 include/hw/virtio/vhost-user.h |  26 +++
 8 files changed, 476 insertions(+), 11 deletions(-)

hw/vfio/common.c +0 -0

index 7b2924c0ef..53d8700581 100644

Quoted 12 lines · On Fri, 22 Dec 2017 14:41:51 +0800 Tiwei Bie <[email protected]> wrote:

Signed-off-by Tiwei Bie <[email protected]>

---
 docs/interop/vhost-user.txt    |  57 ++++++
 hw/vfio/common.c               |   2 +-
 hw/virtio/vhost-user.c         | 381 ++++++++++++++++++++++++++++++++++++++++-
 hw/virtio/vhost.c              |   3 +-
 hw/virtio/virtio-pci.c         |   8 -
 hw/virtio/virtio-pci.h         |   8 +
 include/hw/vfio/vfio.h         |   2 +
 include/hw/virtio/vhost-user.h |  26 +++
 8 files changed, 476 insertions(+), 11 deletions(-)

hw/vfio/common.c struct vfio_as_head vfio_address_spaces = · line 53

Quoted 11 lines

hw/vfio/common.c +1 -1

index 7b2924c0ef..53d8700581 100644
@@ -49,7 +49,7 @@ struct vfio_as_head vfio_address_spaces =
  * initialized, this file descriptor is only released on QEMU exit and
  * we'll re-use it should another vfio device be attached before then.
  */
-static int vfio_kvm_device_fd = -1;
+int vfio_kvm_device_fd = -1;
 #endif

It seems troublesome for vhost to maintain it's own list of groups and register them with the vfio-kvm device. These should likely be made into services provided by vfio/common.c such that we can have a single group list and interfaces for adding and deleting them. Thanks,

This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU.

When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU.

The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to):

- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;

But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support:

1) guest driver notifies the device directly;
2) device interrupts the guest directly;

So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through.

A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch.

There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated!

-----Original Message-----
From: Alexey Kardashevskiy [mailto:[email protected]]
Sent: Tuesday, January 2, 2018 10:42 AM
To: Bie, Tiwei <[email protected]>; [email protected]; qemu- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: Tan, Jianfeng <[email protected]>; Liang, Cunming <[email protected]>; Wang, Xiao W <[email protected]>; Wang, Zhihong <[email protected]>; Daly, Dan <[email protected]>
Subject: Re: [Qemu-devel] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

Quoted 43 lines · Tiwei Bie

This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU.

When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU.

The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to):

- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;

But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support:

1) guest driver notifies the device directly;
2) device interrupts the guest directly;

So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through.

A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch.

Quoted 3 lines

There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated!

I am missing a lot of context here. Out of curiosity - how is this all supposed to work? QEMU command line example would be useful, what will the guest see? A virtio device (i.e. Redhat vendor ID) or an actual PCI device (since VFIO is mentioned)? Thanks.

--
Alexey

Quoted 58 lines

-----Original Message-----
From: Alexey Kardashevskiy [mailto:[email protected]]
Sent: Tuesday, January 2, 2018 10:42 AM
To: Bie, Tiwei <[email protected]>; [email protected]; qemu- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: Tan, Jianfeng <[email protected]>; Liang, Cunming <[email protected]>; Wang, Xiao W <[email protected]>; Wang, Zhihong <[email protected]>; Daly, Dan <[email protected]>
Subject: Re: [Qemu-devel] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

Quoted 43 lines · Tiwei Bie

This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU.

When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU.

The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to):

- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;

But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support:

1) guest driver notifies the device directly;
2) device interrupts the guest directly;

So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through.

A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch.

Quoted 8 lines

Quoted 3 lines

There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated!

I am missing a lot of context here. Out of curiosity - how is this all supposed to work? QEMU command line example would be useful, what will the guest see? A virtio device (i.e. Redhat vendor ID) or an actual PCI device (since VFIO is mentioned)? Thanks.

It's a normal virtio PCIe devices in the guest. Extensions on the host are transparent to the guest.

In terms of the usage, there's a sample may help.
http://dpdk.org/ml/archives/dev/2017-December/085044.html
The sample takes virtio-net device in VM as data path accelerator of virtio-net in nested VM.

When taking physical device on bare metal, it accelerates virtio-net in VM equivalently. There's no additional params of QEMU command line needed for vhost-user.

One more context, including vDPA enabling in DPDK vhost-user library.
http://dpdk.org/ml/archives/dev/2017-December/084792.html

-----Original Message-----
From: Alexey Kardashevskiy [mailto:[email protected]]
Sent: Tuesday, January 2, 2018 2:01 PM
To: Liang, Cunming <[email protected]>; Bie, Tiwei <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: Tan, Jianfeng <[email protected]>; Wang, Xiao W <[email protected]>; Wang, Zhihong <[email protected]>; Daly, Dan <[email protected]>
Subject: Re: [Qemu-devel] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

Quoted 64 lines · Cunming

Quoted 64 lines

-----Original Message-----
From: Alexey Kardashevskiy [mailto:[email protected]]
Sent: Tuesday, January 2, 2018 10:42 AM
To: Bie, Tiwei <[email protected]>; [email protected]; qemu- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: Tan, Jianfeng <[email protected]>; Liang, Cunming <[email protected]>; Wang, Xiao W <[email protected]>; Wang, Zhihong <[email protected]>; Daly, Dan <[email protected]>
Subject: Re: [Qemu-devel] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

Quoted 46 lines · Tiwei Bie

This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU.

When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU.

The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to):

- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;

But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support:

1) guest driver notifies the device directly;
2) device interrupts the guest directly;

So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through.

A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch.

There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated!

Quoted 6 lines

Quoted 4 lines

I am missing a lot of context here. Out of curiosity - how is this all supposed to work? QEMU command line example would be useful, what will the guest see? A virtio device (i.e. Redhat vendor ID) or an actual PCI device (since VFIO is mentioned)? Thanks.

It's a normal virtio PCIe devices in the guest. Extensions on the host are

transparent to the guest.

Quoted 1 line

Quoted 3 lines

In terms of the usage, there's a sample may help.
http://dpdk.org/ml/archives/dev/2017-December/085044.html
The sample takes virtio-net device in VM as data path accelerator of virtio-net

in nested VM.

Aaah, this is for nested VMs, the original description was not clear about this. I get it now, thanks.

Quoted 3 lines

When taking physical device on bare metal, it accelerates virtio-net in VM

equivalently.

Quoted 4 lines

There's no additional params of QEMU command line needed for vhost-user.

One more context, including vDPA enabling in DPDK vhost-user library.
http://dpdk.org/ml/archives/dev/2017-December/084792.html

--
Alexey

This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU.

When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU.

The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to):

- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;

But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support:

1) guest driver notifies the device directly;
2) device interrupts the guest directly;

So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through.

A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch.

There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated!

Tiwei Bie (3):
vhost-user: support receiving file descriptors in slave_read
vhost-user: introduce shared vhost-user state
vhost-user: add VFIO based accelerators support

  docs/interop/vhost-user.txt     |  57 ++++++
  hw/scsi/vhost-user-scsi.c       |   6 +-
  hw/vfio/common.c                |   2 +-
  hw/virtio/vhost-user.c          | 430 +++++++++++++++++++++++++++++++++++++++-
  hw/virtio/vhost.c               |   3 +-
  hw/virtio/virtio-pci.c          |   8 -
  hw/virtio/virtio-pci.h          |   8 +
  include/hw/vfio/vfio.h          |   2 +
  include/hw/virtio/vhost-user.h  |  43 ++++
  include/hw/virtio/virtio-scsi.h |   6 +-
  net/vhost-user.c                |  30 +--
  11 files changed, 561 insertions(+), 34 deletions(-)

create mode 100644 include/hw/virtio/vhost-user.h

Quoted 64 lines · Tiwei Bie

This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU.

When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU.

The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to):

- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;

But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support:

1) guest driver notifies the device directly;
2) device interrupts the guest directly;

So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through.

A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch.

There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated!

Tiwei Bie (3):
vhost-user: support receiving file descriptors in slave_read
vhost-user: introduce shared vhost-user state
vhost-user: add VFIO based accelerators support

  docs/interop/vhost-user.txt     |  57 ++++++
  hw/scsi/vhost-user-scsi.c       |   6 +-
  hw/vfio/common.c                |   2 +-
  hw/virtio/vhost-user.c          | 430 +++++++++++++++++++++++++++++++++++++++-
  hw/virtio/vhost.c               |   3 +-
  hw/virtio/virtio-pci.c          |   8 -
  hw/virtio/virtio-pci.h          |   8 +
  include/hw/vfio/vfio.h          |   2 +
  include/hw/virtio/vhost-user.h  |  43 ++++
  include/hw/virtio/virtio-scsi.h |   6 +-
  net/vhost-user.c                |  30 +--
  11 files changed, 561 insertions(+), 34 deletions(-)

Quoted 2 lines

create mode 100644 include/hw/virtio/vhost-user.h

I may miss something, but may I ask why you must implement them through vhost-use/dpdk. It looks to me you could put all of them in qemu which could simplify a lots of things (just like userspace NVME driver wrote by Fam).

Quoted 67 lines · Jason Wang

Quoted 64 lines · Tiwei Bie

This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU.

When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU.

The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to):

- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;

But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support:

1) guest driver notifies the device directly;
2) device interrupts the guest directly;

So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through.

A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch.

There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really appreciated!

Tiwei Bie (3):
vhost-user: support receiving file descriptors in slave_read
vhost-user: introduce shared vhost-user state
vhost-user: add VFIO based accelerators support

   docs/interop/vhost-user.txt     |  57 ++++++
   hw/scsi/vhost-user-scsi.c       |   6 +-
   hw/vfio/common.c                |   2 +-
   hw/virtio/vhost-user.c          | 430 +++++++++++++++++++++++++++++++++++++++-
   hw/virtio/vhost.c               |   3 +-
   hw/virtio/virtio-pci.c          |   8 -
   hw/virtio/virtio-pci.h          |   8 +
   include/hw/vfio/vfio.h          |   2 +
   include/hw/virtio/vhost-user.h  |  43 ++++
   include/hw/virtio/virtio-scsi.h |   6 +-
   net/vhost-user.c                |  30 +--
   11 files changed, 561 insertions(+), 34 deletions(-)

Quoted 2 lines

create mode 100644 include/hw/virtio/vhost-user.h

Quoted 4 lines

I may miss something, but may I ask why you must implement them through vhost-use/dpdk. It looks to me you could put all of them in qemu which could simplify a lots of things (just like userspace NVME driver wrote by Fam).

Thanks for your comments! :-)

Yeah, you're right. We can also implement everything in QEMU like the userspace NVME driver by Fam. It was also described by Cunming on the KVM Forum 2017. Below is the link to the slides:

https://events.static.linuxfound.org/sites/events/files/slides/KVM17%27-vDPA.pdf

We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details).

And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands to QEMU:

- the switching among different accelerators and software version can be done at runtime in vhost process;
- use different accelerators to accelerate different queue pairs or just accelerate some (instead of all) queue pairs;

Best regards,
Tiwei Bie

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Thursday, January 4, 2018 3:22 PM
To: Bie, Tiwei <[email protected]>
Cc: Tan, Jianfeng <[email protected]>; [email protected]; [email protected]; Liang, Cunming <[email protected]>; qemu- [email protected]; [email protected]; Wang, Xiao W <[email protected]>; [email protected]; Wang, Zhihong <[email protected]>; [email protected]; Daly, Dan <[email protected]>
Subject: Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

Quoted 45 lines · Tiwei Bie

Quoted 44 lines · Jason Wang

Quoted 43 lines · Tiwei Bie

This RFC patch set does some small extensions to vhost-user protocol to support VFIO based accelerators, and makes it possible to get the similar performance of VFIO passthru while keeping the virtio device emulation in QEMU.

When we have virtio ring compatible devices, it's possible to setup the device (DMA mapping, PCI config, etc) based on the existing info (memory-table, features, vring info, etc) which is available on the vhost-backend (e.g. DPDK vhost library). Then, we will be able to use such devices to accelerate the emulated device for the VM. And we call it vDPA: vhost DataPath Acceleration. The key difference between VFIO passthru and vDPA is that, in vDPA only the data path (e.g. ring, notify and queue interrupt) is pass-throughed, the device control path (e.g. PCI configuration space and MMIO regions) is still defined and emulated by QEMU.

The benefits of keeping virtio device emulation in QEMU compared with virtio device VFIO passthru include (but not limit to):

- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;

But the critical issue in vDPA is that the data path performance is relatively low and some host threads are needed for the data path, because some necessary mechanisms are missing to support:

1) guest driver notifies the device directly;
2) device interrupts the guest directly;

So this patch set does some small extensions to vhost-user protocol to make both of them possible. It leverages the same mechanisms (e.g. EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to achieve the data path pass through.

A new protocol feature bit is added to negotiate the accelerator feature support. Two new slave message types are added to enable the notify and interrupt passthru for each queue. From the view of vhost-user protocol design, it's very flexible. The passthru can be enabled/disabled for each queue individually, and it's possible to accelerate each queue by different devices. More design and implementation details can be found from the last patch.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

There are some rough edges in this patch set (so this is a RFC patch set for now), but it's never too early to hear the thoughts from the community! So any comments and suggestions would be really

appreciated!

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Tiwei Bie (3):
vhost-user: support receiving file descriptors in slave_read
vhost-user: introduce shared vhost-user state
vhost-user: add VFIO based accelerators support

   docs/interop/vhost-user.txt     |  57 ++++++
   hw/scsi/vhost-user-scsi.c       |   6 +-
   hw/vfio/common.c                |   2 +-
   hw/virtio/vhost-user.c          | 430

+++++++++++++++++++++++++++++++++++++++-

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

   hw/virtio/vhost.c               |   3 +-
   hw/virtio/virtio-pci.c          |   8 -
   hw/virtio/virtio-pci.h          |   8 +
   include/hw/vfio/vfio.h          |   2 +
   include/hw/virtio/vhost-user.h  |  43 ++++
   include/hw/virtio/virtio-scsi.h |   6 +-
   net/vhost-user.c                |  30 +--
   11 files changed, 561 insertions(+), 34 deletions(-)

create mode 100644 include/hw/virtio/vhost-user.h

Quoted 3 lines

Quoted 3 lines

I may miss something, but may I ask why you must implement them through vhost-use/dpdk. It looks to me you could put all of them in qemu which could simplify a lots of things (just like userspace NVME

driver wrote by Fam).

Quoted 10 lines

Quoted 1 line

Thanks for your comments! :-)

Yeah, you're right. We can also implement everything in QEMU like the userspace NVME driver by Fam. It was also described by Cunming on the KVM Forum 2017. Below is the link to the
slides:

https://events.static.linuxfound.org/sites/events/files/slides/KVM17%2
7-vDPA.pdf

Thanks for the pointer. Looks rather interesting.

Quoted 1 line

Quoted 3 lines

We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details).

This is exactly what I mean. Form my point of view, there's no need for any extension for vhost protocol, we just need to reuse qemu iothread to implement a userspace vhost dataplane and do the mdev inside that thread.

Quoted 7 lines

And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands to QEMU:

Maybe I was wrong but I don't think we care about the complexity of command line or monitor command in this case.

Quoted 1 line

Quoted 4 lines

- the switching among different accelerators and software version can be done at runtime in vhost process;
- use different accelerators to accelerate different queue pairs or just accelerate some (instead of all) queue pairs;

Well, technically, if we want, these could be implemented in qemu too.

And here's some more advantages if you implement it in qemu:

1) Avoid extra dependency like dpdk
2) More flexible, mdev could even choose to not use VFIO or not depend on vDPA
3) More efficient guest IOMMU integration especially for dynamic mappings (device IOTLB transactions could be done by function calls instead of slow UDP messages)
4) Zerocopy (for non intel vDPA) is more easier to be implemented
5) Compare to vhost-user, tightly coupled with device emulation can simplify lots of things (an example is programmable flow director/RSS implementation). And any future enhancement to virtio does not need to introduce new type of vhost-user messages.

I don't object vhost-user/dpdk method but I second for implementing all the stuffs in qemu.

Thanks

Quoted 4 lines

Best regards,
Tiwei Bie

Quoted 8 lines

Thanks for the pointer. Looks rather interesting.

Quoted 3 lines

We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details).

This is exactly what I mean. Form my point of view, there's no need for any extension for vhost protocol, we just need to reuse qemu iothread to implement a userspace vhost dataplane and do the mdev inside that thread.

On functional perspective, it makes sense to have qemu native support of those certain usage. However, qemu doesn't have to take responsibility for dataplane. There're already huge amounts of codes for different devices emulation, leveraging external dataplane library is an effective way to introduce more.

The beauty of vhost_user is to open a door for variable userland workloads(e.g. vswitch). The dataplane connected with VM usually need to be close integrated with those userland workloads, a control place interface(vhost-user) is better than a datapath interface(e.g. provided by dataplace in qemu iothread).

On workloads point of view, it's not excited to be part of qemu process.

That comes up with the idea of vhost-user extension. Userland workloads decides to enable accelerators or not, qemu provides the common control plane infrastructure.

Quoted 8 lines

Quoted 5 lines

And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands to QEMU:

Maybe I was wrong but I don't think we care about the complexity of command line or monitor command in this case.

Quoted 5 lines

Quoted 4 lines

- the switching among different accelerators and software version

    can be done at runtime in vhost process;
- use different accelerators to accelerate different queue pairs
    or just accelerate some (instead of all) queue pairs;

Well, technically, if we want, these could be implemented in qemu too.

You're right if just considering I/O. The ways to consume those I/O is another perspective. Simply 1:1 associating guest virtio-net and accelerator w/ SW datapath fallback is not the whole picture.

It's variable usages on workload side to abstract the device (e.g. port re-presenter for vswitch) and etc. I don't think qemu is interested for all bunch of things there.

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Friday, January 5, 2018 4:39 PM
To: Liang, Cunming <[email protected]>; Bie, Tiwei <[email protected]>
Cc: Tan, Jianfeng <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Wang, Xiao W <[email protected]>; [email protected]; Wang, Zhihong <[email protected]>; [email protected]; Daly, Dan <[email protected]>
Subject: Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

Quoted 8 lines · Cunming

Quoted 8 lines

Thanks for the pointer. Looks rather interesting.

Quoted 3 lines

We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details).

This is exactly what I mean. Form my point of view, there's no need for any extension for vhost protocol, we just need to reuse qemu iothread to implement a userspace vhost dataplane and do the mdev

inside that thread.

Quoted 1 line

On functional perspective, it makes sense to have qemu native support of

those certain usage. However, qemu doesn't have to take responsibility for dataplane. There're already huge amounts of codes for different devices emulation, leveraging external dataplane library is an effective way to introduce more.

This does not mean to drop external dataplane library. Actually, you can link dpdk to qemu directly.

Quoted 2 lines

The beauty of vhost_user is to open a door for variable userland

workloads(e.g. vswitch). The dataplane connected with VM usually need to be close integrated with those userland workloads, a control place interface(vhost-user) is better than a datapath interface(e.g. provided by dataplace in qemu iothread).

Do we really need vswitch for vDPA?

Quoted 2 lines

On workloads point of view, it's not excited to be part of qemu process.

Don't see why, qemu have dataplane for virtio-blk/scsi.

Quoted 2 lines

That comes up with the idea of vhost-user extension. Userland workloads

decides to enable accelerators or not, qemu provides the common control plane infrastructure.

It brings extra complexity: endless new types of messages and a huge brunch of bugs. And what's more important, the split model tends to be less efficient in some cases, e.g guest IOMMU integration. I'm pretty sure we will meet more in the future.

Quoted 9 lines

Quoted 9 lines

Quoted 6 lines

And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands to QEMU:

Maybe I was wrong but I don't think we care about the complexity of command line or monitor command in this case.

Quoted 6 lines

Quoted 5 lines

Quoted 4 lines

- the switching among different accelerators and software version

    can be done at runtime in vhost process;
- use different accelerators to accelerate different queue pairs
    or just accelerate some (instead of all) queue pairs;

Well, technically, if we want, these could be implemented in qemu too.

You're right if just considering I/O. The ways to consume those I/O is

another perspective.

Quoted 1 line

Simply 1:1 associating guest virtio-net and accelerator w/ SW datapath

fallback is not the whole picture.

Pay attention:

1) What I mean is not a fallback here. You can still do a lot of tricks e.g offloading datapath to hardware or doorbell map.
2) Qemu supports (very old and inefficient) a split model of device emulation and network backend. This means we can switch between backends (though not implemented).

Quoted 2 lines

It's variable usages on workload side to abstract the device (e.g. port re-

presenter for vswitch) and etc. I don't think qemu is interested for all bunch of things there.

Quoted 1 line

Again, you can link any dataplane to qemu directly instead of using vhost- user if vhost-user tends to be less useful in some cases (vDPA is one of the case I think).

Thanks

Quoted 27 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Friday, January 5, 2018 4:39 PM
To: Liang, Cunming <[email protected]>; Bie, Tiwei <[email protected]>
Cc: Tan, Jianfeng <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Wang, Xiao W <[email protected]>; [email protected]; Wang, Zhihong <[email protected]>; [email protected]; Daly, Dan <[email protected]>
Subject: Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

Quoted 8 lines · Cunming

Quoted 8 lines

Thanks for the pointer. Looks rather interesting.

Quoted 3 lines

We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details).

This is exactly what I mean. Form my point of view, there's no need for any extension for vhost protocol, we just need to reuse qemu iothread to implement a userspace vhost dataplane and do the mdev

inside that thread.

Quoted 1 line

On functional perspective, it makes sense to have qemu native support of

Quoted 7 lines

those certain usage. However, qemu doesn't have to take responsibility for dataplane. There're already huge amounts of codes for different devices emulation, leveraging external dataplane library is an effective way to introduce more.

This does not mean to drop external dataplane library. Actually, you can link dpdk to qemu directly.

It's not a bad idea, then the interface comes to be new API/ABI definition of external dataplane library instead of existing vhost protocol.

dpdk as a library is not a big deal to link with, customized application is.
In addition, it will ask for qemu to provide flexible process model then. Lots of application level features (e.g. hot upgrade/fix) becomes burden.

I'm open to that option, keep eyes on any proposal there.

Quoted 7 lines

Quoted 1 line

The beauty of vhost_user is to open a door for variable userland

workloads(e.g. vswitch). The dataplane connected with VM usually need to be close integrated with those userland workloads, a control place interface(vhost-user) is better than a datapath interface(e.g. provided by dataplace in qemu iothread).

Do we really need vswitch for vDPA?

Accelerators come into the picture of vswitch, which usually provides in-chip EMC for early classification. It gives a fast path for those throughput sensitive(SLA) VNF to bypass the further table lookup. It co-exists other VNF whose SLA level is best effort but requires more functions(e.g. stateful conntrack, security check, even higher layer WAF support) support, DPDK based datapath still boost the throughput there. It's not used to be a single choice of dedicated or shared datapath, usually they're co-exist.

Quoted 3 lines

Quoted 2 lines

On workloads point of view, it's not excited to be part of qemu process.

Don't see why, qemu have dataplane for virtio-blk/scsi.

Qemu has vhost-user for scsi too. I'm not saying which one is bad, just point out sometime it's very workloads driven. Network is different with blk/scsi/crypto.

Quoted 8 lines

Quoted 1 line

That comes up with the idea of vhost-user extension. Userland workloads

decides to enable accelerators or not, qemu provides the common control plane infrastructure.

It brings extra complexity: endless new types of messages and a huge brunch of bugs. And what's more important, the split model tends to be less efficient in some cases, e.g guest IOMMU integration. I'm pretty sure we will meet more in the future.

vIOMMU relevant message has been supported by vhost protocol. It's independent effort there.

I don't see this patch introduce endless new types.

My taking of your fundamental concern is about continues adding new features on vhost-user. Feel free to correct me if I misunderstood your point.

Quoted 19 lines

Quoted 9 lines

Quoted 9 lines

Quoted 6 lines

And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands to QEMU:

Maybe I was wrong but I don't think we care about the complexity of command line or monitor command in this case.

Quoted 6 lines

Quoted 5 lines

Quoted 4 lines

- the switching among different accelerators and software version

     can be done at runtime in vhost process;
- use different accelerators to accelerate different queue pairs
     or just accelerate some (instead of all) queue pairs;

Well, technically, if we want, these could be implemented in qemu too.

You're right if just considering I/O. The ways to consume those I/O is

another perspective.

Quoted 1 line

Simply 1:1 associating guest virtio-net and accelerator w/ SW datapath

fallback is not the whole picture.

Quoted 7 lines

Pay attention:

1) What I mean is not a fallback here. You can still do a lot of tricks e.g offloading datapath to hardware or doorbell map.
2) Qemu supports (very old and inefficient) a split model of device emulation and network backend. This means we can switch between backends (though not implemented).

Accelerator won't be defined in the same device layout, it means there're different kinds of drivers.

Qemu definitely won't like to have HW relevant driver there,

that's end up with another vhost-vfio in my slides.

A mediated device can help to unify the device layout definition, and leave the driver part in its own place.

This approach is quite good when application doesn't need to put userland SW datapath and accelerator datapath in the same picture as which I mentioned(vswitch cases).

Quoted 6 lines

Quoted 1 line

It's variable usages on workload side to abstract the device (e.g. port re-

presenter for vswitch) and etc. I don't think qemu is interested for all bunch of things there.
Again, you can link any dataplane to qemu directly instead of using vhost- user if vhost-user tends to be less useful in some cases (vDPA is one of the case I think).

See my previous words.

Quoted 1 line

Thanks

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jason Wang
Sent: Monday, January 8, 2018 11:24 AM
To: Liang, Cunming <[email protected]>; Bie, Tiwei <[email protected]>
Cc: Tan, Jianfeng <[email protected]>; [email protected]; [email protected]; Daly, Dan <[email protected]>; qemu- [email protected]; [email protected]; Wang, Xiao W <[email protected]>; Maxime Coquelin <[email protected]>; [email protected]; Wang, Zhihong <[email protected]>; [email protected]
Subject: [virtio-dev] Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost- user to support VFIO based accelerators

Quoted 29 lines · Cunming

Quoted 29 lines

-----Original Message-----
From: Jason Wang [mailto:[email protected]]
Sent: Friday, January 5, 2018 4:39 PM
To: Liang, Cunming <[email protected]>; Bie, Tiwei <[email protected]>
Cc: Tan, Jianfeng <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Wang, Xiao W <[email protected]>; [email protected]; Wang, Zhihong <[email protected]>; [email protected]; Daly, Dan <[email protected]>
Subject: Re: [Qemu-devel] [virtio-dev] [RFC 0/3] Extend vhost-user to support VFIO based accelerators

Quoted 8 lines · Cunming

Quoted 8 lines

Thanks for the pointer. Looks rather interesting.

Quoted 3 lines

We're also working on it (including defining a standard device for vhost data path acceleration based on mdev to hide vendor specific details).

This is exactly what I mean. Form my point of view, there's no need for any extension for vhost protocol, we just need to reuse qemu iothread to implement a userspace vhost dataplane and do the mdev

inside that thread.

Quoted 2 lines

On functional perspective, it makes sense to have qemu native support of

Quoted 8 lines

Quoted 7 lines

those certain usage. However, qemu doesn't have to take responsibility for dataplane. There're already huge amounts of codes for different devices emulation, leveraging external dataplane library is an effective way to introduce more.

This does not mean to drop external dataplane library. Actually, you can link dpdk to qemu directly.

It's not a bad idea, then the interface comes to be new API/ABI definition of

external dataplane library instead of existing vhost protocol.

These API/ABI should be qemu internal which should be much flexible than vhost-user.

Quoted 2 lines

dpdk as a library is not a big deal to link with, customized application is.
In addition, it will ask for qemu to provide flexible process model then. Lots

of application level features (e.g. hot upgrade/fix) becomes burden.

Don't quite get this, I think we can solve this by migration. Even if a dpdk userspace backend can do this, it can only do upgrade and fix for network datapath. This is not a complete solution obviously.

It's nice to discuss this but it was a little bit out of the topic.

Quoted 10 lines

I'm open to that option, keep eyes on any proposal there.

Quoted 7 lines

Quoted 1 line

The beauty of vhost_user is to open a door for variable userland

workloads(e.g. vswitch). The dataplane connected with VM usually need to be close integrated with those userland workloads, a control place interface(vhost-user) is better than a datapath interface(e.g. provided by dataplace in qemu iothread).

Do we really need vswitch for vDPA?

Accelerators come into the picture of vswitch, which usually provides in-

chip EMC for early classification. It gives a fast path for those throughput sensitive(SLA) VNF to bypass the further table lookup. It co-exists other VNF whose SLA level is best effort but requires more functions(e.g. stateful conntrack, security check, even higher layer WAF support) support, DPDK based datapath still boost the throughput there. It's not used to be a single choice of dedicated or shared datapath, usually they're co-exist.

So if I understand this correctly, the "vswtich" here is a hardware function (something like smart NICs or OVS offloaded). So the question still, is vhost- user a must in this case?

Quoted 5 lines

Quoted 4 lines

Quoted 3 lines

On workloads point of view, it's not excited to be part of qemu process.

Don't see why, qemu have dataplane for virtio-blk/scsi.

Qemu has vhost-user for scsi too. I'm not saying which one is bad, just

point out sometime it's very workloads driven. Network is different with blk/scsi/crypto.

What's the main difference from your point of view which makes vhost-user a must in this case?

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

That comes up with the idea of vhost-user extension. Userland

workloads

Quoted 4 lines

Quoted 4 lines

decides to enable accelerators or not, qemu provides the common control plane infrastructure.

It brings extra complexity: endless new types of messages and a huge

brunch

Quoted 1 line

Quoted 1 line

of bugs. And what's more important, the split model tends to be less

efficient

Quoted 3 lines

Quoted 2 lines

in some cases, e.g guest IOMMU integration. I'm pretty sure we will meet more in the future.

vIOMMU relevant message has been supported by vhost protocol. It's

independent effort there.

The point is vIOMMU integration is very inefficient in vhost-user for some cases. If you have lots of dynamic mappings, it can have only 5%-10% performance compared to vIOMMU disabled. A huge amount of translation request will be generated in this case. The main issue here is you can not offload datapath completely to vhost-user backends completely, IOMMU translations were still done in qemu. This is one of the defect of vhost-user when datapath need to access the device state.

Quoted 2 lines

I don't see this patch introduce endless new types.

Not this patch but we can imagine vhost-user protocol will become complex in the future.

Quoted 1 line

My taking of your fundamental concern is about continues adding new

features on vhost-user.

Quoted 1 line

Feel free to correct me if I misunderstood your point.

Unfortunately not, endless itself is not a problem but we'd better only try to extend it only when it was really needed. The main questions are:

1) whether or not we need to split things like what you suggested here?
2) if needed, is vhost-user the best method?

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

And IMO it's also not a bad idea to extend vhost-user protocol to support the accelerators if possible. And it could be more flexible because it could support (for example) below things easily without introducing any complex command line options or monitor commands

to

Quoted 11 lines

Quoted 11 lines

Quoted 10 lines

Quoted 9 lines

Quoted 1 line

QEMU:

Maybe I was wrong but I don't think we care about the complexity of command line or monitor command in this case.

Quoted 4 lines

- the switching among different accelerators and software version

     can be done at runtime in vhost process;
- use different accelerators to accelerate different queue pairs
     or just accelerate some (instead of all) queue pairs;

Well, technically, if we want, these could be implemented in qemu too.

You're right if just considering I/O. The ways to consume those I/O is

another perspective.

Quoted 8 lines

Quoted 8 lines

Quoted 1 line

Simply 1:1 associating guest virtio-net and accelerator w/ SW datapath

fallback is not the whole picture.

Pay attention:

1) What I mean is not a fallback here. You can still do a lot of tricks e.g offloading datapath to hardware or doorbell map.
2) Qemu supports (very old and inefficient) a split model of device

emulation

Quoted 1 line

Quoted 1 line

and network backend. This means we can switch between backends

(though

Quoted 2 lines

Quoted 1 line

not implemented).

Accelerator won't be defined in the same device layout, it means there're

different kinds of drivers.

Well, you can still use different drivers if you link dpdk or whatever other dataplane library to qemu.

Quoted 1 line

Qemu definitely won't like to have HW relevant driver there,

Why not? We've already had userspace NVME driver.

Quoted 2 lines

that's end up with another vhost-vfio in my slides.

I don't get why we can't implement it purely through a userspace driver inside qemu.

Quoted 2 lines

A mediated device can help to unify the device layout definition, and leave

the driver part in its own place.

The point is not about mediated device but why you must use vhost-user to do it.

Thanks

Quoted 1 line

This approach is quite good when application doesn't need to put userland

SW datapath and accelerator datapath in the same picture as which I mentioned(vswitch cases).

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

It's variable usages on workload side to abstract the device (e.g. port

re-

Quoted 1 line

Quoted 1 line

presenter for vswitch) and etc. I don't think qemu is interested for all

bunch

Quoted 7 lines

Quoted 4 lines

of things there.
Again, you can link any dataplane to qemu directly instead of using vhost- user if vhost-user tends to be less useful in some cases (vDPA is one of the case I think).

See my previous words.

Quoted 1 line

Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Quoted 7 lines

chip EMC for early classification. It gives a fast path for those throughput sensitive(SLA) VNF to bypass the further table lookup. It co-exists other VNF whose SLA level is best effort but requires more functions(e.g. stateful conntrack, security check, even higher layer WAF support) support, DPDK based datapath still boost the throughput there. It's not used to be a single choice of dedicated or shared datapath, usually they're co-exist.

Quoted 3 lines

So if I understand this correctly, the "vswtich" here is a hardware function (something like smart NICs or OVS offloaded). So the question still, is vhost- user a must in this case?

"vswitch" point to SW vswitch(e.g. OVS-DPDK). Accelerators stands for different offloading IPs on the device(e.g. smart NIC) which can be used from a userland driver.
EMC IP used to offload OVS fastpath, so as move traffic to VM directly. Either SRIOV device assignment or vDPA helps to build datapath pass-thru context which represented by a virtual interface on management perspective. For entire "vswitch", there still co-exist none pass-thru interface(SW backend) which uses vhost-user for virtual interface. Both of them shall be able to replace each other.

There's no other user space choice yet recently for network except vhost-user. The patch of vhost-user extension has lower impact for qemu. If you read this patch, it's really about to reduce the doorbell and interrupt overhead.

Basic vDPA works even without any qemu change. As vhost-user is well-recognized as the vhost interface for userland backend, it's reasonable to well-support the usage of userland backend w/ I/O accelerator.

Before moving forward, it's necessary to get some alignment on two basic things.
- Do you agree that providing userland backend via vhost-user is the right way to do with vswitch workload. Otherwise, we probably shall go back to revisit vhost-user itself rather than talking anything new happening on vhost-user.

- Do you agree vhost-user is a right way for qemu to allow multi-process? Please refer to https://www.linux-kvm.org/images/f/fc/KVM_FORUM_multi-process.pdf

Quoted 9 lines

Quoted 4 lines

Quoted 3 lines

Quoted 2 lines

On workloads point of view, it's not excited to be part of qemu process.

Don't see why, qemu have dataplane for virtio-blk/scsi.

Qemu has vhost-user for scsi too. I'm not saying which one is bad, just

point out sometime it's very workloads driven. Network is different with blk/scsi/crypto.

What's the main difference from your point of view which makes vhost-user a must in this case?

Network devices, a NIC or a Smart NIC usually has vendor specific driver. DPDK takes devices by its user space drivers to run OVS. Virtual interface is all vhost-user based talking with qemu. For some virtual interface, it now tries to bypass the traffic. It's looking forward a consistent vhost-user interface there.

Linking OVS-DPDK with qemu, TBH, it's far away from today's usage.

Quoted 14 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

That comes up with the idea of vhost-user extension. Userland

workloads

Quoted 4 lines

Quoted 4 lines

decides to enable accelerators or not, qemu provides the common control plane infrastructure.

It brings extra complexity: endless new types of messages and a huge

brunch

Quoted 1 line

Quoted 1 line

of bugs. And what's more important, the split model tends to be less

efficient

Quoted 3 lines

Quoted 2 lines

in some cases, e.g guest IOMMU integration. I'm pretty sure we will meet more in the future.

vIOMMU relevant message has been supported by vhost protocol. It's

independent effort there.

Quoted 7 lines

The point is vIOMMU integration is very inefficient in vhost-user for some cases. If you have lots of dynamic mappings, it can have only 5%-10% performance compared to vIOMMU disabled. A huge amount of translation request will be generated in this case. The main issue here is you can not offload datapath completely to vhost-user backends completely, IOMMU translations were still done in qemu. This is one of the defect of vhost-user when datapath need to access the device state.

It's vIOMMU's challenge of dynamic mapping, besides vhost-user, kernel vhost shall face the same situation. Static mapping w/ DPDK looks much better. It's not fair to blame vhost-user by vIOMMU overhead.

Quoted 6 lines

Quoted 2 lines

I don't see this patch introduce endless new types.

Not this patch but we can imagine vhost-user protocol will become complex in the future.

Quoted 1 line

My taking of your fundamental concern is about continues adding new

Quoted 7 lines

features on vhost-user.

Quoted 1 line

Feel free to correct me if I misunderstood your point.

Unfortunately not, endless itself is not a problem but we'd better only try to extend it only when it was really needed. The main questions are:

1) whether or not we need to split things like what you suggested here?
2) if needed, is vhost-user the best method?

Sounds good. BTW, this patch(vhost-user extention) is a performance improvement patch for DPDK vDPA usage(Refer DPDK patches). Another RFC patch stay tuned for kernel space usage which will propose a qemu native vhost adaptor for in-kernel mediated device driver.

Quoted 1 line

Why not? We've already had userspace NVME driver.

There's huge amount of vendor specific driver for network. NVMe is much generalized than NIC.
The idea of linking an external dataplane sounds interesting, but it's not used in real world. Looking forward the progress.

Quoted 3 lines

Quoted 1 line

that's end up with another vhost-vfio in my slides.

I don't get why we can't implement it purely through a userspace driver inside qemu.

TBH, we think about this before. There're a few reasons stopping us.
- qemu hasn't an abstraction layout of network device(HW NIC) for userspace drivers

- qemu launch process, linking dpdk w/ qemu is not problem. Gap is on ovs integration, effort/impact is not small

- for qemu native virtio SW backend, it lacks of efficient ways to talk with external process. The change efforts/impact is not small.

- qemu native userspace driver only used for qemu, userspace driver in DPDK can be used for others