VFIO based vGPU(was Re: [Announcement] 2015-Q3 release of XenGT - a Mediated ...)

Hi Alex, let's continue with a new thread :)

Basically we agree with you: exposing vGPU via VFIO can make QEMU share as much code as possible with pcidev(PF or VF) assignment. And yes, different vGPU vendors can share quite a lot of the QEMU part, which will do good for upper layers such as libvirt.

To achieve this, there are quite a lot to do, I'll summarize it below. I dived into VFIO for a while but still may have things misunderstood, so please correct me :)

First, let me illustrate my understanding of current VFIO framework used to pass through a pcidev to guest:

                 +----------------------------------+
                 |            vfio qemu             |
                 +-----+------------------------+---+
                       |DMA                  ^  |CFG
QEMU                   |map               IRQ|  |
-----------------------|---------------------|--|-----------
KERNEL    +------------|---------------------|--|----------+
          | VFIO       |                     |  |          |
          |            v                     |  v          |
          |  +-------------------+     +-----+-----------+ |
IOMMU     |  | vfio iommu driver |     | vfio bus driver | |
API  <-------+                   |     |                 | |
Layer     |  | e.g. type1        |     | e.g. vfio_pci   | |
          |  +-------------------+     +-----------------+ |
          +------------------------------------------------+

Here when a particular pcidev is passed-through to a KVM guest, it is attached to vfio_pci driver in host, and guest memory is mapped into IOMMU via the type1 iommu driver.

Then, the draft infrastructure of future VFIO-based vgpu:

                 +-------------------------------------+
                 |              vfio qemu              |
                 +----+-------------------------+------+
                      |DMA                   ^  |CFG
QEMU                  |map                IRQ|  |
----------------------|----------------------|--|-----------
KERNEL                |                      |  |
         +------------|----------------------|--|----------+
         |VFIO        |                      |  |          |
         |            v                      |  v          |
         | +--------------------+      +-----+-----------+ |
DMA      | | vfio iommu driver  |      | vfio bus driver | |
API <------+                    |      |                 | |
Layer    | |  e.g. vfio_type2   |      |  e.g. vfio_vgpu | |
         | +--------------------+      +-----------------+ |
         |         |  ^                      |  ^          |
         +---------|--|----------------------|--|----------+
                   |  |                      |  |
                   |  |                      v  |
         +---------|--|----------+   +---------------------+
         | +-------v-----------+ |   |                     |
         | |                   | |   |                     |
         | |      KVMGT        | |   |                     |
         | |                   | |   |   host gfx driver   |
         | +-------------------+ |   |                     |
         |                       |   |                     |
         |    KVM hypervisor     |   |                     |
         +-----------------------+   +---------------------+

        NOTE    vfio_type2 and vfio_vgpu are only *logically* parts of VFIO, they may be implemented in KVM hypervisor or host gfx driver.

Here we need to implement a new vfio IOMMU driver instead of type1, let's call it vfio_type2 temporarily. The main difference from pcidev assignment is, vGPU doesn't have its own DMA requester id, so it has to share mappings with host and other vGPUs.

        - type1 iommu driver maps gpa to hpa for passing through; whereas type2 maps iova to hpa;

        - hardware iommu is always needed by type1, whereas for type2, hardware iommu is optional;

        - type1 will invoke low-level IOMMU API (iommu_map et al) to setup IOMMU page table directly, whereas type2 dosen't (only need to invoke higher level DMA API like dma_map_page);

We also need to implement a new 'bus' driver instead of vfio_pci, let's call it vfio_vgpu temporarily:

        - vfio_pci is a real pci driver, it has a probe method called during dev attaching; whereas the vfio_vgpu is a pseudo driver, it won't attach any devivce - the GPU is always owned by host gfx driver. It has to do 'probing' elsewhere, but still in host gfx driver attached to the device;

        - pcidev(PF or VF) attached to vfio_pci has a natural path in sysfs; whereas vgpu is purely a software concept:
          vfio_vgpu needs to create create/destory vgpu instances, maintain their paths in sysfs (e.g. "/sys/class/vgpu/intel/vgpu0") etc. There should be something added in a higher layer to do this (VFIO or DRM).

        - vfio_pci in most case will allow QEMU to access pcidev hardware; whereas vfio_vgpu is to access virtual resource emulated by another device model;

        - vfio_pci will inject an IRQ to guest only when physical IRQ generated; whereas vfio_vgpu may inject an IRQ for emulation purpose. Anyway they can share the same injection interface;

Questions:

        [1] For VFIO No-IOMMU mode (!iommu_present), I saw it was reverted in upstream ae5515d66362(Revert: "vfio: Include No-IOMMU mode"). In my opinion, vfio_type2 doesn't rely on it to support No-IOMMU case, instead it needs a new implementation which fits both w/ and w/o IOMMU. Is this correct?

For things not mentioned above, we might have them discussed in other threads, or temporarily maintained in a TODO list (we might get back to them after the big picture get agreed):

        - How to expose guest framebuffer via VFIO for SPICE;

        - How to avoid double translation with two-stage: GTT + IOMMU, whether identity map is possible, and if yes, how to make it more effectively;

        - Application acceleration
          You mentioned that with VFIO, a vGPU may be used by applications to get GPU acceleration. It's a potential opportunity to use vGPU for container usage, worthy of further investigation.

Hi Jike,

Quoted 87 lines · Jike Song

Hi Alex, let's continue with a new thread :)

Basically we agree with you: exposing vGPU via VFIO can make QEMU share as much code as possible with pcidev(PF or VF) assignment. And yes, different vGPU vendors can share quite a lot of the QEMU part, which will do good for upper layers such as libvirt.

To achieve this, there are quite a lot to do, I'll summarize it below. I dived into VFIO for a while but still may have things misunderstood, so please correct me :)

First, let me illustrate my understanding of current VFIO framework used to pass through a pcidev to guest:

                 +----------------------------------+
                 |            vfio qemu             |
                 +-----+------------------------+---+
                       |DMA                  ^  |CFG
QEMU                   |map               IRQ|  |
-----------------------|---------------------|--|-----------
KERNEL    +------------|---------------------|--|----------+
          | VFIO       |                     |  |          |
          |            v                     |  v          |
          |  +-------------------+     +-----+-----------+ |
IOMMU     |  | vfio iommu driver |     | vfio bus driver | |
API  <-------+                   |     |                 | |
Layer     |  | e.g. type1        |     | e.g. vfio_pci   | |
          |  +-------------------+     +-----------------+ |
          +------------------------------------------------+

Here when a particular pcidev is passed-through to a KVM guest, it is attached to vfio_pci driver in host, and guest memory is mapped into IOMMU via the type1 iommu driver.

Then, the draft infrastructure of future VFIO-based vgpu:

                 +-------------------------------------+
                 |              vfio qemu              |
                 +----+-------------------------+------+
                      |DMA                   ^  |CFG
QEMU                  |map                IRQ|  |
----------------------|----------------------|--|-----------
KERNEL                |                      |  |
         +------------|----------------------|--|----------+
         |VFIO        |                      |  |          |
         |            v                      |  v          |
         | +--------------------+      +-----+-----------+ |
DMA      | | vfio iommu driver  |      | vfio bus driver | |
API <------+                    |      |                 | |
Layer    | |  e.g. vfio_type2   |      |  e.g. vfio_vgpu | |
         | +--------------------+      +-----------------+ |
         |         |  ^                      |  ^          |
         +---------|--|----------------------|--|----------+
                   |  |                      |  |
                   |  |                      v  |
         +---------|--|----------+   +---------------------+
         | +-------v-----------+ |   |                     |
         | |                   | |   |                     |
         | |      KVMGT        | |   |                     |
         | |                   | |   |   host gfx driver   |
         | +-------------------+ |   |                     |
         |                       |   |                     |
         |    KVM hypervisor     |   |                     |
         +-----------------------+   +---------------------+

NOTE vfio_type2 and vfio_vgpu are only *logically* parts
of VFIO, they may be implemented in KVM hypervisor or host gfx driver.

Here we need to implement a new vfio IOMMU driver instead of type1, let's call it vfio_type2 temporarily. The main difference from pcidev assignment is, vGPU doesn't have its own DMA requester id, so it has to share mappings with host and other vGPUs.

- type1 iommu driver maps gpa to hpa for passing through; whereas type2 maps iova to hpa;

Quoted 6 lines

- hardware iommu is always needed by type1, whereas for type2, hardware iommu is optional;

- type1 will invoke low-level IOMMU API (iommu_map et al) to setup IOMMU page table directly, whereas type2 dosen't (only need to invoke higher level DMA API like dma_map_page);

Yes, the current type1 implementation is not compatible with vgpu since there are not separate requester IDs on the bus and you probably don't want or need to pin all of guest memory like we do for direct assignment. However, let's separate the type1 user API from the current implementation. It's quite easy within the vfio code to consider "type1" to be an API specification that may have multiple implementations. A minor code change would allow us to continue looking for compatible iommu backends if the group we're trying to attach is rejected.

The benefit here is that QEMU could work
unmodified, using the type1 vfio-iommu API regardless of whether a device is directly assigned or virtual.

Let's look at the type1 interface; we have simple map and unmap interfaces which map and unmap process virtual address space (vaddr) to the device address space (iova). The host physical address is obtained by pinning the vaddr. In the current implementation, a map operation pins pages and populates the hardware iommu. A vgpu compatible implementation might simply register the translation into a kernel- based database to be called upon later. When the host graphics driver needs to enable dma for the vgpu, it doesn't need to go to QEMU for the translation, it already possesses the iova to vaddr mapping, which becomes iova to hpa after a pinning operation.

So, I would encourage you to look at creating a vgpu vfio iommu backened that makes use of the type1 api since it will reduce the changes necessary for userspace.

Quoted 16 lines

We also need to implement a new 'bus' driver instead of vfio_pci, let's call it vfio_vgpu temporarily:

- vfio_pci is a real pci driver, it has a probe method called during dev attaching; whereas the vfio_vgpu is a pseudo driver, it won't attach any devivce - the GPU is always owned by host gfx driver. It has to do 'probing' elsewhere, but still in host gfx driver attached to the device;

- pcidev(PF or VF) attached to vfio_pci has a natural path in sysfs; whereas vgpu is purely a software concept:
vfio_vgpu needs to create create/destory vgpu instances, maintain their paths in sysfs (e.g. "/sys/class/vgpu/intel/vgpu0") etc. There should be something added in a higher layer to do this (VFIO or DRM).

Quoted 7 lines

- vfio_pci in most case will allow QEMU to access pcidev hardware; whereas vfio_vgpu is to access virtual resource emulated by another device model;

- vfio_pci will inject an IRQ to guest only when physical IRQ generated; whereas vfio_vgpu may inject an IRQ for emulation purpose. Anyway they can share the same injection interface;

Here too, I think you're making assumptions based on an implementation path. Personally, I think each vgpu should be a struct device and that an iommu group should be created for each. I think this is a valid abstraction; dma isolation is provided through something other than a system-level iommu, but it's still provided. Without this, the entire vfio core would need to be aware of vgpu, since the core operates on devices and groups. I believe creating a struct device also gives you basic probe and release support for a driver.

There will be a need for some sort of lifecycle management of a vgpu.
How is it created? Destroyed? Can it be given more or less resources
than other vgpus, etc. This could be implemented in sysfs for each physical gpu with vgpu support, sort of like how we support sr-iov now, the PF exports controls for creating VFs. The more commonality we can get for lifecycle and device access for userspace, the better.

As for virtual vs physical resources and interrupts, part of the purpose of vfio is to abstract a device into basic components. It's up to the bus driver how accesses to each space map to the physical device. Take for instance PCI config space, the existing vfio-pci driver emulates some portions of config space for the user.

Quoted 8 lines

Questions:

[1] For VFIO No-IOMMU mode (!iommu_present), I saw it was reverted in upstream ae5515d66362(Revert: "vfio: Include No-IOMMU mode"). In my opinion, vfio_type2 doesn't rely on it to support No-IOMMU case, instead it needs a new implementation which fits both w/ and w/o IOMMU. Is this correct?

vfio no-iommu has also been re-added for v4.5 (03a76b60f8ba), this was simply a case that the kernel development outpaced the intended user and I didn't want to commit to the user api changes until it had been completely vetted. In any case, vgpu should have no dependency whatsoever on no-iommu. As above, I think vgpu should create virtual devices and add them to an iommu group, similar to how no-iommu does, but without the kernel tainting because you are actually providing isolation through other means than a system iommu.

Quoted 6 lines

For things not mentioned above, we might have them discussed in other threads, or temporarily maintained in a TODO list (we might get back to them after the big picture get agreed):

- How to expose guest framebuffer via VFIO for SPICE;

Potentially through a new, device specific region, which I think can be done within the existing vfio API. The API can already expose an arbitrary number of regions to the user, it's just a matter of how we tell the user the purpose of a region index beyond the fixed set we map to PCI resources.

Quoted 4 lines

- How to avoid double translation with two-stage: GTT + IOMMU, whether identity map is possible, and if yes, how to make it more effectively;

Quoted 5 lines

- Application acceleration
You mentioned that with VFIO, a vGPU may be used by applications to get GPU acceleration. It's a potential opportunity to use vGPU for container usage, worthy of further investigation.

Yes, interesting topics. Thanks,

Alex

Quoted 97 lines · Alex Williamson

Hi Jike,

Quoted 87 lines · Jike Song

Hi Alex, let's continue with a new thread :)

Basically we agree with you: exposing vGPU via VFIO can make QEMU share as much code as possible with pcidev(PF or VF) assignment. And yes, different vGPU vendors can share quite a lot of the QEMU part, which will do good for upper layers such as libvirt.

To achieve this, there are quite a lot to do, I'll summarize it below. I dived into VFIO for a while but still may have things misunderstood, so please correct me :)

First, let me illustrate my understanding of current VFIO framework used to pass through a pcidev to guest:

                 +----------------------------------+
                 |            vfio qemu             |
                 +-----+------------------------+---+
                       |DMA                  ^  |CFG
QEMU                   |map               IRQ|  |
-----------------------|---------------------|--|-----------
KERNEL    +------------|---------------------|--|----------+
          | VFIO       |                     |  |          |
          |            v                     |  v          |
          |  +-------------------+     +-----+-----------+ |
IOMMU     |  | vfio iommu driver |     | vfio bus driver | |
API  <-------+                   |     |                 | |
Layer     |  | e.g. type1        |     | e.g. vfio_pci   | |
          |  +-------------------+     +-----------------+ |
          +------------------------------------------------+

Here when a particular pcidev is passed-through to a KVM guest, it is attached to vfio_pci driver in host, and guest memory is mapped into IOMMU via the type1 iommu driver.

Then, the draft infrastructure of future VFIO-based vgpu:

                 +-------------------------------------+
                 |              vfio qemu              |
                 +----+-------------------------+------+
                      |DMA                   ^  |CFG
QEMU                  |map                IRQ|  |
----------------------|----------------------|--|-----------
KERNEL                |                      |  |
         +------------|----------------------|--|----------+
         |VFIO        |                      |  |          |
         |            v                      |  v          |
         | +--------------------+      +-----+-----------+ |
DMA      | | vfio iommu driver  |      | vfio bus driver | |
API <------+                    |      |                 | |
Layer    | |  e.g. vfio_type2   |      |  e.g. vfio_vgpu | |
         | +--------------------+      +-----------------+ |
         |         |  ^                      |  ^          |
         +---------|--|----------------------|--|----------+
                   |  |                      |  |
                   |  |                      v  |
         +---------|--|----------+   +---------------------+
         | +-------v-----------+ |   |                     |
         | |                   | |   |                     |
         | |      KVMGT        | |   |                     |
         | |                   | |   |   host gfx driver   |
         | +-------------------+ |   |                     |
         |                       |   |                     |
         |    KVM hypervisor     |   |                     |
         +-----------------------+   +---------------------+

        NOTE    vfio_type2 and vfio_vgpu are only *logically* parts of VFIO, they may be implemented in KVM hypervisor or host gfx driver.

Here we need to implement a new vfio IOMMU driver instead of type1, let's call it vfio_type2 temporarily. The main difference from pcidev assignment is, vGPU doesn't have its own DMA requester id, so it has to share mappings with host and other vGPUs.

        - type1 iommu driver maps gpa to hpa for passing through; whereas type2 maps iova to hpa;

Quoted 6 lines

        - hardware iommu is always needed by type1, whereas for type2, hardware iommu is optional;

        - type1 will invoke low-level IOMMU API (iommu_map et al) to setup IOMMU page table directly, whereas type2 dosen't (only need to invoke higher level DMA API like dma_map_page);

Quoted 9 lines

Yes, the current type1 implementation is not compatible with vgpu since there are not separate requester IDs on the bus and you probably don't want or need to pin all of guest memory like we do for direct assignment.  However, let's separate the type1 user API from the current implementation.  It's quite easy within the vfio code to consider "type1" to be an API specification that may have multiple implementations.  A minor code change would allow us to continue looking for compatible iommu backends if the group we're trying to attach is rejected.

Would you elaborate a bit about 'iommu backends' here? Previously I thought that entire type1 will be duplicated. If not, what is supposed to add, a new vfio_dma_do_map?

Quoted 15 lines

The benefit here is that QEMU could work
unmodified, using the type1 vfio-iommu API regardless of whether a device is directly assigned or virtual.

Let's look at the type1 interface; we have simple map and unmap interfaces which map and unmap process virtual address space (vaddr) to the device address space (iova).  The host physical address is obtained by pinning the vaddr.  In the current implementation, a map operation pins pages and populates the hardware iommu.  A vgpu compatible implementation might simply register the translation into a kernel- based database to be called upon later.  When the host graphics driver needs to enable dma for the vgpu, it doesn't need to go to QEMU for the translation, it already possesses the iova to vaddr mapping, which becomes iova to hpa after a pinning operation.

Quoted 4 lines

So, I would encourage you to look at creating a vgpu vfio iommu backened that makes use of the type1 api since it will reduce the changes necessary for userspace.

Yes, keeping type1 API sounds a great idea.

Quoted 24 lines

Quoted 16 lines

We also need to implement a new 'bus' driver instead of vfio_pci, let's call it vfio_vgpu temporarily:

        - vfio_pci is a real pci driver, it has a probe method called during dev attaching; whereas the vfio_vgpu is a pseudo driver, it won't attach any devivce - the GPU is always owned by host gfx driver. It has to do 'probing' elsewhere, but still in host gfx driver attached to the device;

        - pcidev(PF or VF) attached to vfio_pci has a natural path in sysfs; whereas vgpu is purely a software concept:
          vfio_vgpu needs to create create/destory vgpu instances, maintain their paths in sysfs (e.g. "/sys/class/vgpu/intel/vgpu0") etc. There should be something added in a higher layer to do this (VFIO or DRM).

Quoted 7 lines

        - vfio_pci in most case will allow QEMU to access pcidev hardware; whereas vfio_vgpu is to access virtual resource emulated by another device model;

        - vfio_pci will inject an IRQ to guest only when physical IRQ generated; whereas vfio_vgpu may inject an IRQ for emulation purpose. Anyway they can share the same injection interface;

Quoted 9 lines

Here too, I think you're making assumptions based on an implementation path.  Personally, I think each vgpu should be a struct device and that an iommu group should be created for each.  I think this is a valid abstraction; dma isolation is provided through something other than a system-level iommu, but it's still provided.  Without this, the entire vfio core would need to be aware of vgpu, since the core operates on devices and groups.  I believe creating a struct device also gives you basic probe and release support for a driver.

Indeed.
BTW, that should be done in the 'bus' driver, right?

Quoted 7 lines

There will be a need for some sort of lifecycle management of a vgpu. How is it created?  Destroyed?  Can it be given more or less resources than other vgpus, etc.  This could be implemented in sysfs for each physical gpu with vgpu support, sort of like how we support sr-iov now, the PF exports controls for creating VFs.  The more commonality we can get for lifecycle and device access for userspace, the better.

Will have a look at the VF managements, thanks for the info.

Quoted 15 lines

As for virtual vs physical resources and interrupts, part of the purpose of vfio is to abstract a device into basic components.  It's up to the bus driver how accesses to each space map to the physical device.  Take for instance PCI config space, the existing vfio-pci driver emulates some portions of config space for the user.

Quoted 8 lines

Questions:

        [1] For VFIO No-IOMMU mode (!iommu_present), I saw it was reverted in upstream ae5515d66362(Revert: "vfio: Include No-IOMMU mode"). In my opinion, vfio_type2 doesn't rely on it to support No-IOMMU case, instead it needs a new implementation which fits both w/ and w/o IOMMU. Is this correct?

Quoted 9 lines

vfio no-iommu has also been re-added for v4.5 (03a76b60f8ba), this was simply a case that the kernel development outpaced the intended user and I didn't want to commit to the user api changes until it had been completely vetted.  In any case, vgpu should have no dependency whatsoever on no-iommu.  As above, I think vgpu should create virtual devices and add them to an iommu group, similar to how no-iommu does, but without the kernel tainting because you are actually providing isolation through other means than a system iommu.

Thanks for confirmation.

Quoted 23 lines

Quoted 6 lines

For things not mentioned above, we might have them discussed in other threads, or temporarily maintained in a TODO list (we might get back to them after the big picture get agreed):

        - How to expose guest framebuffer via VFIO for SPICE;

Potentially through a new, device specific region, which I think can be done within the existing vfio API.  The API can already expose an arbitrary number of regions to the user, it's just a matter of how we tell the user the purpose of a region index beyond the fixed set we map to PCI resources.

Quoted 9 lines

        - How to avoid double translation with two-stage: GTT + IOMMU, whether identity map is possible, and if yes, how to make it more effectively;

        - Application acceleration
          You mentioned that with VFIO, a vGPU may be used by applications to get GPU acceleration. It's a potential opportunity to use vGPU for container usage, worthy of further investigation.

Quoted 2 lines

Yes, interesting topics.  Thanks,

Looks that things get more clear overall, with small exceptions. Thanks for the advice:)

Quoted 4 lines · Jike Song

Would you elaborate a bit about 'iommu backends' here? Previously I thought that entire type1 will be duplicated. If not, what is supposed to add, a new vfio_dma_do_map?

I don't know that you necessarily want to re-use any of the vfio_iommu_type1.c code as-is, it's just the API that we'll want to keep consistent so QEMU doesn't need to learn about a new iommu backend. Opportunities for sharing certainly may arise, you may want to use a similar red-black tree for storing current mappings, the pinning code may be similar, etc. We can evaluate on a case by case basis whether it makes sense to pull out common code for each of those.

As for an iommu backend in general, if you look at the code flow example in Documentation/vfio.txt, the user opens a container (/dev/vfio/vfio) and a group (/dev/vfio/$GROUPNUM). The group is set to associate with a container instance via VFIO_GROUP_SET_CONTAINER and then an iommu model is set for the container with VFIO_SET_IOMMU.
Looking at drivers/vfio/vfio.c:vfio_ioctl_set_iommu(), we look for an
iommu backend that supports the requested extension (VFIO_TYPE1_IOMMU), call the open() callback on it and then attempt to attach the group via the attach_group() callback. At this latter callback, the iommu backend can compare the device to those that it actually supports. For instance the existing vfio_iommu_type1 will attempt to use the IOMMU API and should fail if the device cannot be supported with that. The current loop in vfio_ioctl_set_iommu() will exit in this case, but as you can see in the code, it's easy to make it continue and look for another iommu backend that supports the requested extension.

Quoted 15 lines

Quoted 15 lines

The benefit here is that QEMU could work
unmodified, using the type1 vfio-iommu API regardless of whether a device is directly assigned or virtual.

Let's look at the type1 interface; we have simple map and unmap interfaces which map and unmap process virtual address space (vaddr) to the device address space (iova). The host physical address is obtained by pinning the vaddr. In the current implementation, a map operation pins pages and populates the hardware iommu. A vgpu compatible implementation might simply register the translation into a kernel- based database to be called upon later. When the host graphics driver needs to enable dma for the vgpu, it doesn't need to go to QEMU for the translation, it already possesses the iova to vaddr mapping, which becomes iova to hpa after a pinning operation.

Quoted 6 lines

Quoted 4 lines

So, I would encourage you to look at creating a vgpu vfio iommu backened that makes use of the type1 api since it will reduce the changes necessary for userspace.

BTW, that should be done in the 'bus' driver, right?

I think you have some flexibility between the graphics driver and the vfio-vgpu driver in where this is done. If we want vfio-vgpu to be more generic, then vgpu device creation and management should probably be done in the graphics driver and vfio-vgpu should be able to probe that device and call back into the graphics driver to handle requests. If it turns out there's not much for vfio-vgpu to share, ie. it's just a passthrough for device specific emulation, then maybe we want a vfio- intel-vgpu instead.

Quoted 3 lines

Looks that things get more clear overall, with small exceptions. Thanks for the advice:)

Yes, please let me know how I can help. Thanks,

Alex

From: Song, Jike
Sent: Wednesday, January 20, 2016 5:00 PM

Quoted 11 lines

Quoted 1 line

BTW, that should be done in the 'bus' driver, right?

I think you have some flexibility between the graphics driver and the vfio-vgpu driver in where this is done. If we want vfio-vgpu to be more generic, then vgpu device creation and management should probably be done in the graphics driver and vfio-vgpu should be able to probe that device and call back into the graphics driver to handle requests. If it turns out there's not much for vfio-vgpu to share, ie. it's just a passthrough for device specific emulation, then maybe we want a vfio- intel-vgpu instead.

Good to know that.

Quoted 5 lines

Quoted 4 lines

Looks that things get more clear overall, with small exceptions. Thanks for the advice:)

Quoted 4 lines

Yes, please let me know how I can help. Thanks,

Alex

I will start the coding soon, will do :)

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

Bus Driver

{ in i915/vgt/xxx.c }

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

	IOMMU

		{ in a new vfio_xxx.c }

		- allocate: struct device & IOMMU group

- map/unmap functions for vgpu
- rb-tree to maintain iova/hpa mappings

- interacts with kvmgt.c

vgpu instance management

{ in i915 }

- path, create/destroy

From: Alex Williamson [mailto:[email protected]]
Sent: Tuesday, January 26, 2016 5:30 AM

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

Quoted 6 lines

	IOMMU

		{ in a new vfio_xxx.c }

		- allocate: struct device & IOMMU group

It seems like the vgpu instance management would do this.

Quoted 2 lines

- map/unmap functions for vgpu
- rb-tree to maintain iova/hpa mappings

Yep, pretty much what type1 does now, but without mapping through the IOMMU API.  Essentially just a database of the current userspace mappings that can be accessed for page pinning and IOVA->HPA translation.

Quoted 10 lines

- interacts with kvmgt.c

vgpu instance management

{ in i915 }

- path, create/destroy

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Nvidia has also been looking at this and has some ideas how we might standardize on some of the interfaces and create a vgpu framework to help share code between vendors and hopefully make a more consistent userspace interface for libvirt as well.  I'll let Neo provide some details.  Thanks,

From: Tian, Kevin
Sent: Tuesday, January 26, 2016 5:45 AM

Quoted 11 lines

Quoted 10 lines

- interacts with kvmgt.c

vgpu instance management

{ in i915 }

- path, create/destroy

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

It's invoked here, but expecting the function exposed by vfio_vgpu.c. It's not good to touch vfio internal structures from another module (such as i915.ko)

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset? Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right? That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

Quoted 5 lines

	IOMMU

		{ in a new vfio_xxx.c }

		- allocate: struct device & IOMMU group

It seems like the vgpu instance management would do this.

Quoted 2 lines

- map/unmap functions for vgpu
- rb-tree to maintain iova/hpa mappings

Yep, pretty much what type1 does now, but without mapping through the IOMMU API. Essentially just a database of the current userspace mappings that can be accessed for page pinning and IOVA->HPA translation.

Quoted 9 lines

- interacts with kvmgt.c

vgpu instance management

{ in i915 }

- path, create/destroy

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group. The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Nvidia has also been looking at this and has some ideas how we might standardize on some of the interfaces and create a vgpu framework to help share code between vendors and hopefully make a more consistent userspace interface for libvirt as well. I'll let Neo provide some details. Thanks,

Alex

Quoted 22 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Tuesday, January 26, 2016 5:30 AM

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 8 lines

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

That is the next level detail Jike will figure out and discuss soon.

yes, basic region info/access should be necessary. For interrupt, could you elaborate a bit what current interface is doing? If just about creating an eventfd for virtual interrupt injection, it applies to vgpu too.

Quoted 9 lines

Quoted 6 lines

	IOMMU

		{ in a new vfio_xxx.c }

		- allocate: struct device & IOMMU group

It seems like the vgpu instance management would do this.

Quoted 7 lines

Quoted 2 lines

- map/unmap functions for vgpu
- rb-tree to maintain iova/hpa mappings

Yep, pretty much what type1 does now, but without mapping through the IOMMU API.  Essentially just a database of the current userspace mappings that can be accessed for page pinning and IOVA->HPA translation.

The thought is to reuse iommu_type1.c, by abstracting several underlying operations and then put vgpu specific implementation in a vfio_vgpu.c (e.g. for map/unmap instead of using IOMMU API, an iova/hpa mapping is updated accordingly), etc.

This file will also connect between VFIO and vendor specific vgpu driver, e.g. exposing interfaces to allow the latter querying iova<->hpa and also creating necessary VFIO structures like aforementioned device/IOMMUas...

Quoted 11 lines

Quoted 10 lines

- interacts with kvmgt.c

vgpu instance management

{ in i915 }

- path, create/destroy

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

It's invoked here, but expecting the function exposed by vfio_vgpu.c. It's not good to touch vfio internal structures from another module (such as i915.ko)

Quoted 1 line

Quoted 6 lines

Nvidia has also been looking at this and has some ideas how we might standardize on some of the interfaces and create a vgpu framework to help share code between vendors and hopefully make a more consistent userspace interface for libvirt as well.  I'll let Neo provide some details.  Thanks,

Nice to know that. Neo, please share your thought here.

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

- vfio_pci will inject an IRQ to guest only when physical IRQ generated; whereas vfio_vgpu may inject an IRQ for emulation purpose. Anyway they can share the same injection interface;

cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types

Jike will provide next level API definitions based on KVMGT requirement. We can further refine it to match requirements of multi-vendors.

Thanks
Kevin
From dc8ca387f7b06c6dfc85fb4bd79a760dca76e831 Mon Sep 17 00:00:00 2001

From 380156ade7053664bdb318af0659708357f40050 Mon Sep 17 00:00:00 2001

Quoted 22 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Tuesday, January 26, 2016 5:30 AM

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 8 lines

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

That is the next level detail Jike will figure out and discuss soon.

yes, basic region info/access should be necessary. For interrupt, could you elaborate a bit what current interface is doing? If just about creating an eventfd for virtual interrupt injection, it applies to vgpu too.

Quoted 9 lines

Quoted 6 lines

	IOMMU

		{ in a new vfio_xxx.c }

		- allocate: struct device & IOMMU group

It seems like the vgpu instance management would do this.

Quoted 7 lines

Quoted 2 lines

- map/unmap functions for vgpu
- rb-tree to maintain iova/hpa mappings

Yep, pretty much what type1 does now, but without mapping through the IOMMU API.  Essentially just a database of the current userspace mappings that can be accessed for page pinning and IOVA->HPA translation.

The thought is to reuse iommu_type1.c, by abstracting several underlying operations and then put vgpu specific implementation in a vfio_vgpu.c (e.g. for map/unmap instead of using IOMMU API, an iova/hpa mapping is updated accordingly), etc.

This file will also connect between VFIO and vendor specific vgpu driver, e.g. exposing interfaces to allow the latter querying iova<->hpa and also creating necessary VFIO structures like aforementioned device/IOMMUas...

Quoted 11 lines

Quoted 10 lines

- interacts with kvmgt.c

vgpu instance management

{ in i915 }

- path, create/destroy

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

It's invoked here, but expecting the function exposed by vfio_vgpu.c. It's not good to touch vfio internal structures from another module (such as i915.ko)

Quoted 1 line

Quoted 6 lines

Nvidia has also been looking at this and has some ideas how we might standardize on some of the interfaces and create a vgpu framework to help share code between vendors and hopefully make a more consistent userspace interface for libvirt as well.  I'll let Neo provide some details.  Thanks,

Nice to know that. Neo, please share your thought here.

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

- vfio_pci will inject an IRQ to guest only when physical IRQ generated; whereas vfio_vgpu may inject an IRQ for emulation purpose. Anyway they can share the same injection interface;

cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types

From dc8ca387f7b06c6dfc85fb4bd79a760dca76e831 Mon Sep 17 00:00:00 2001

From 380156ade7053664bdb318af0659708357f40050 Mon Sep 17 00:00:00 2001

Jike will provide next level API definitions based on KVMGT requirement. We can further refine it to match requirements of multi-vendors.

Thanks
Kevin

Quoted 19 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 9 lines

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset? Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right? That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:) For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

- VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
- VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

- VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

- VFIO_PCI_ROM_REGION_INDEX
- VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

- VFIO_DEVICE_GET_INFO
- VFIO_DEVICE_GET_REGION_INFO
- VFIO_DEVICE_GET_IRQ_INFO
- VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

b) adding other information. For example, for the OpRegion, QEMU need to do more than mmap a region, it has to:

- allocate a region
- copy contents from somewhere in host to that region
- mmap it to guest

I remember you already have a prototype for this?

Quoted 8 lines

Quoted 5 lines

	IOMMU

		{ in a new vfio_xxx.c }

		- allocate: struct device & IOMMU group

It seems like the vgpu instance management would do this.

Yes, it can be removed from here.

Quoted 8 lines

Quoted 2 lines

- map/unmap functions for vgpu
- rb-tree to maintain iova/hpa mappings

Yep, pretty much what type1 does now, but without mapping through the IOMMU API. Essentially just a database of the current userspace mappings that can be accessed for page pinning and IOVA->HPA translation.

Yes.

Quoted 10 lines

Quoted 9 lines

- interacts with kvmgt.c

vgpu instance management

{ in i915 }

- path, create/destroy

Quoted 7 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group. The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Yes, just like the SRIOV does.

Quoted 5 lines

Nvidia has also been looking at this and has some ideas how we might standardize on some of the interfaces and create a vgpu framework to help share code between vendors and hopefully make a more consistent userspace interface for libvirt as well. I'll let Neo provide some details. Thanks,

Good to know that, so we can possibly cooperate on some common part, e.g. the instance management :)

Quoted 3 lines

Alex

--
Thanks,
Jike
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 19 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 9 lines

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:)  For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

- VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
- VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

- VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

- VFIO_PCI_ROM_REGION_INDEX
- VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

- VFIO_DEVICE_GET_INFO
- VFIO_DEVICE_GET_REGION_INFO
- VFIO_DEVICE_GET_IRQ_INFO
- VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

b) adding other information. For example, for the OpRegion, QEMU need to do more than mmap a region, it has to:

- allocate a region
- copy contents from somewhere in host to that region
- mmap it to guest

I remember you already have a prototype for this?

Quoted 63 lines · Jike Song

Quoted 19 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 9 lines

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:)  For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

- VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
- VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

- VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

- VFIO_PCI_ROM_REGION_INDEX
- VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

- VFIO_DEVICE_GET_INFO
- VFIO_DEVICE_GET_REGION_INFO
- VFIO_DEVICE_GET_IRQ_INFO
- VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

Quoted 2 lines

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

From: Neo Jia [mailto:[email protected]]
Sent: Tuesday, January 26, 2016 6:21 PM

0. High level overview
=====================================================
=============================

user space:

                                +-----------+  VFIO IOMMU IOCTLs
                      +---------| QEMU VFIO |-------------------------+
        VFIO IOCTLs   |         +-----------+                         |
                      |                                               |
 ---------------------|-----------------------------------------------|---------
                      |                                               |
  kernel space:       |  +--->----------->---+  (callback)            V
                      |  |                   v                 +------V-----+
  +----------+   +----V--^--+          +--+--+-----+           | VGPU       |
  |          |   |          |     +----| nvidia.ko +----->-----> TYPE1 IOMMU|
  | VFIO Bus <===| VGPU.ko  |<----|    +-----------+     |     +---++-------+
  |          |   |          |     | (register)           ^         ||
  +----------+   +-------+--+     |    +-----------+     |         ||
                         V        +----| i915.ko   +-----+     +---VV-------+
                         |             +-----^-----+           | TYPE1      |
                         |  (callback)       |                 | IOMMU      |
                         +-->------------>---+                 +------------+

access flow:

Guest MMIO / PCI config access
|
-------------------------------------------------
|

  +-----> KVM VM_EXITs  (kernel)
          |
  -------------------------------------------------
          |
          +-----> QEMU VFIO driver (user)
                  |
  -------------------------------------------------
                  |
                  +---->  VGPU kernel driver (kernel)
                          |
                          |
                          +----> vendor driver callback

Quoted 32 lines

From: Neo Jia [mailto:[email protected]]
Sent: Tuesday, January 26, 2016 6:21 PM

0. High level overview
=====================================================
=============================

user space:

                                +-----------+  VFIO IOMMU IOCTLs
                      +---------| QEMU VFIO |-------------------------+
        VFIO IOCTLs   |         +-----------+                         |
                      |                                               |
 ---------------------|-----------------------------------------------|---------
                      |                                               |
  kernel space:       |  +--->----------->---+  (callback)            V
                      |  |                   v                 +------V-----+
  +----------+   +----V--^--+          +--+--+-----+           | VGPU       |
  |          |   |          |     +----| nvidia.ko +----->-----> TYPE1 IOMMU|
  | VFIO Bus <===| VGPU.ko  |<----|    +-----------+     |     +---++-------+
  |          |   |          |     | (register)           ^         ||
  +----------+   +-------+--+     |    +-----------+     |         ||
                         V        +----| i915.ko   +-----+     +---VV-------+
                         |             +-----^-----+           | TYPE1      |
                         |  (callback)       |                 | IOMMU      |
                         +-->------------>---+                 +------------+

access flow:

Guest MMIO / PCI config access
|
-------------------------------------------------
|

Quoted 14 lines

  +-----> KVM VM_EXITs  (kernel)
          |
  -------------------------------------------------
          |
          +-----> QEMU VFIO driver (user)
                  |
  -------------------------------------------------
                  |
                  +---->  VGPU kernel driver (kernel)
                          |
                          |
                          +----> vendor driver callback

There is one difference between nvidia and intel implementations. We have vgpu device model in kernel, as part of i915.ko. So I/O emulation requests are forwarded directly in kernel side.

Thanks
Kevin

Quoted 1 line · Kevin

Quoted 1 line

From: Alex Williamson [mailto:[email protected]]

Hi Alex, Kevin and Jike,

(Seems I shouldn't use attachment, resend it again to the list, patches are inline at the end)

Thanks for adding me to this technical discussion, a great opportunity for us to design together which can bring both Intel and NVIDIA vGPU solution to KVM platform.

Instead of directly jumping to the proposal that we have been working on recently for NVIDIA vGPU on KVM, I think it is better for me to put out couple quick comments / thoughts regarding the existing discussions on this thread as fundamentally I think we are solving the same problem, DMA, interrupt and MMIO.
Then we can look at what we have, hopefully we can reach some consensus soon.

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Infact to keep vfio-vgpu to be more generic, vgpu device creation and management can be centralized and done in vfio-vgpu. That also include adding to IOMMU group and VFIO group.

Graphics driver can register with vfio-vgpu to get management and emulation call backs to graphics driver.

We already have struct vgpu_device in our proposal that keeps pointer to physical device.

Quoted 3 lines

- vfio_pci will inject an IRQ to guest only when physical IRQ generated; whereas vfio_vgpu may inject an IRQ for emulation purpose. Anyway they can share the same injection interface;

eventfd to inject the interrupt is known to vfio-vgpu, that fd should be available to graphics driver so that graphics driver can inject interrupts directly when physical device triggers interrupt.

Here is the proposal we have, please review.

Please note the patches we have put out here is mainly for POC purpose to verify our understanding also can serve the purpose to reduce confusions and speed up our design, although we are very happy to refine that to something eventually can be used for both parties and upstreamed.

Linux vGPU kernel design
==================================================================================

Here we are proposing a generic Linux kernel module based on VFIO framework which allows different GPU vendors to plugin and provide their GPU virtualization solution on KVM, the benefits of having such generic kernel module are:

1) Reuse QEMU VFIO driver, supporting VFIO UAPI

2) GPU HW agnostic management API for upper layer software such as libvirt
3) No duplicated VFIO kernel logic reimplemented by different GPU driver vendor
0. High level overview
==================================================================================


  user space:

                                +-----------+  VFIO IOMMU IOCTLs
                      +---------| QEMU VFIO |-------------------------+
        VFIO IOCTLs   |         +-----------+                         |
                      |                                               | 
 ---------------------|-----------------------------------------------|---------
                      |                                               |
  kernel space:       |  +--->----------->---+  (callback)            V
                      |  |                   v                 +------V-----+
  +----------+   +----V--^--+          +--+--+-----+           | VGPU       |
  |          |   |          |     +----| nvidia.ko +----->-----> TYPE1 IOMMU|
  | VFIO Bus <===| VGPU.ko  |<----|    +-----------+     |     +---++-------+ 
  |          |   |          |     | (register)           ^         ||
  +----------+   +-------+--+     |    +-----------+     |         ||
                         V        +----| i915.ko   +-----+     +---VV-------+ 
                         |             +-----^-----+           | TYPE1      |
                         |  (callback)       |                 | IOMMU      |
                         +-->------------>---+                 +------------+

 access flow:

  Guest MMIO / PCI config access
  |
  -------------------------------------------------
  |

  +-----> KVM VM_EXITs  (kernel)
          |
  -------------------------------------------------
          |
          +-----> QEMU VFIO driver (user)
                  | 
  -------------------------------------------------
                  |
                  +---->  VGPU kernel driver (kernel)
                          |  
                          | 
                          +----> vendor driver callback

1. VGPU management interface
==================================================================================

This is the interface allows upper layer software (mostly libvirt) to query and configure virtual GPU device in a HW agnostic fashion. Also, this management interface has provided flexibility to underlying GPU vendor to support virtual device hotplug, multiple virtual devices per VM, multiple virtual devices from different physical devices, etc.

1.1 Under per-physical device sysfs:
----------------------------------------------------------------------------------

vgpu_supported_types - RO, list the current supported virtual GPU types and its

VGPU_ID. VGPU_ID - a vGPU type identifier returned from reads of "vgpu_supported_types".

vgpu_create - WO, input syntax <VM_UUID:idx:VGPU_ID>, create a virtual gpu device on a target physical GPU. idx: virtual device index inside a VM
vgpu_destroy - WO, input syntax <VM_UUID:idx>, destroy a virtual gpu device on a target physical GPU

1.3 Under vgpu class sysfs:
----------------------------------------------------------------------------------

vgpu_start - WO, input syntax <VM_UUID>, this will trigger the registration interface to notify the GPU vendor driver to commit virtual GPU resource for this target VM.

Also, the vgpu_start function is a synchronized call, the successful return of this call will indicate all the requested vGPU resource has been fully committed, the VMM should continue.

vgpu_shutdown - WO, input syntax <VM_UUID>, this will trigger the registration interface to notify the GPU vendor driver to release virtual GPU resource of this target VM.

1.4 Virtual device Hotplug
----------------------------------------------------------------------------------

To support virtual device hotplug, <vgpu_create> and <vgpu_destroy> can be accessed during VM runtime, and the corresponding registration callback will be invoked to allow GPU vendor support hotplug.

To support hotplug, vendor driver would take necessary action to handle the situation when a vgpu_create is done on a VM_UUID after vgpu_start, and that implies both create and start for that vgpu device.

Same, vgpu_destroy implies a vgpu_shudown on a running VM only if vendor driver supports vgpu hotplug.

If hotplug is not supported and VM is still running, vendor driver can return error code to indicate not supported.

Separate create from start gives flixibility to have:

- multiple vgpu instances for single VM and
- hotplug feature.

2. GPU driver vendor registration interface
==================================================================================

2.1 Registration interface definition (include/linux/vgpu.h)
----------------------------------------------------------------------------------

extern int vgpu_register_device(struct pci_dev *dev,
                                const struct gpu_device_ops *ops);

extern void vgpu_unregister_device(struct pci_dev *dev);

/**
 * struct gpu_device_ops - Structure to be registered for each physical GPU to
 * register the device to vgpu module.
 *
 * @owner:                      The module owner.
 * @vgpu_supported_config:      Called to get information about supported vgpu
 * types.
 *                              @dev : pci device structure of physical GPU. *                              @config: should return string listing supported *                              config
 *                              Returns integer: success (0) or error (< 0)
 * @vgpu_create:                Called to allocate basic resouces in graphics *                              driver for a particular vgpu.
 *                              @dev: physical pci device structure on which *                              vgpu
 *                                    should be created
 *                              @vm_uuid: VM's uuid for which VM it is intended *                              to
 *                              @instance: vgpu instance in that VM *                              @vgpu_id: This represents the type of vgpu to be *                                        created
 *                              Returns integer: success (0) or error (< 0)
 * @vgpu_destroy:               Called to free resources in graphics driver for *                              a vgpu instance of that VM.
 *                              @dev: physical pci device structure to which *                              this vgpu points to.
 *                              @vm_uuid: VM's uuid for which the vgpu belongs *                              to.
 *                              @instance: vgpu instance in that VM *                              Returns integer: success (0) or error (< 0) *                              If VM is running and vgpu_destroy is called that *                              means the vGPU is being hotunpluged. Return *                              error
 *                              if VM is running and graphics driver doesn't *                              support vgpu hotplug.
 * @vgpu_start:                 Called to do initiate vGPU initialization *                              process in graphics driver when VM boots before *                              qemu starts.
 *                              @vm_uuid: VM's UUID which is booting. *                              Returns integer: success (0) or error (< 0)
 * @vgpu_shutdown:              Called to teardown vGPU related resources for *                              the VM
 *                              @vm_uuid: VM's UUID which is shutting down . *                              Returns integer: success (0) or error (< 0)
 * @read:                       Read emulation callback
 *                              @vdev: vgpu device structure
 *                              @buf: read buffer
 *                              @count: number bytes to read
 *                              @address_space: specifies for which address *                              space
 *                              the request is: pci_config_space, IO register *                              space or MMIO space.
 *                              Retuns number on bytes read on success or error.
 * @write:                      Write emulation callback
 *                              @vdev: vgpu device structure
 *                              @buf: write buffer
 *                              @count: number bytes to be written *                              @address_space: specifies for which address *                              space
 *                              the request is: pci_config_space, IO register *                              space or MMIO space.
 *                              Retuns number on bytes written on success or *                              error.
 * @vgpu_set_irqs:              Called to send about interrupts configuration *                              information that qemu set.
 *                              @vdev: vgpu device structure
 *                              @flags, index, start, count and *data : same as *                              that of struct vfio_irq_set of
 *                              VFIO_DEVICE_SET_IRQS API.
 *
 * Physical GPU that support vGPU should be register with vgpu module with
 * gpu_device_ops structure.
 */

struct gpu_device_ops {
        struct module   *owner;
        int     (*vgpu_supported_config)(struct pci_dev *dev, char *config); int     (*vgpu_create)(struct pci_dev *dev, uuid_le vm_uuid, uint32_t instance, uint32_t vgpu_id); int     (*vgpu_destroy)(struct pci_dev *dev, uuid_le vm_uuid, uint32_t instance);
        int     (*vgpu_start)(uuid_le vm_uuid);
        int     (*vgpu_shutdown)(uuid_le vm_uuid);
        ssize_t (*read) (struct vgpu_device *vdev, char *buf, size_t count,

                         uint32_t address_space, loff_t pos); ssize_t (*write)(struct vgpu_device *vdev, char *buf, size_t count, uint32_t address_space,loff_t pos);
        int     (*vgpu_set_irqs)(struct vgpu_device *vdev, uint32_t flags, unsigned index, unsigned start, unsigned count, void *data);

};

2.2 Details for callbacks we haven't mentioned above.
---------------------------------------------------------------------------------

vgpu_supported_config: allows the vendor driver to specify the supported vGPU type/configuration

vgpu_create          : create a virtual GPU device, can be used for device hotplug.
vgpu_destroy         : destroy a virtual GPU device, can be used for device hotplug.
vgpu_start           : callback function to notify vendor driver vgpu device come to live for a given virtual machine.

vgpu_shutdown        : callback function to notify vendor driver

read                 : callback to vendor driver to handle virtual device config space or MMIO read access

write                : callback to vendor driver to handle virtual device config space or MMIO write access

vgpu_set_irqs        : callback to vendor driver to pass along the interrupt information for the target virtual device, then vendor driver can inject interrupt into virtual machine for this device.

2.3 Potential additional virtual device configuration registration interface:
---------------------------------------------------------------------------------

callback function to describe the MMAP behavior of the virtual GPU

callback function to allow GPU vendor driver to provide PCI config space backing memory.

3. VGPU TYPE1 IOMMU
==================================================================================

Here we are providing a TYPE1 IOMMU for vGPU which will basically keep track the <iova, hva, size, flag> and save the QEMU mm for later reference.

You can find the quick/ugly implementation in the attached patch file, which is actually just a simple version Alex's type1 IOMMU without actual real mapping when IOMMU_MAP_DMA / IOMMU_UNMAP_DMA is called.

We have thought about providing another vendor driver registration interface so such tracking information will be sent to vendor driver and he will use the QEMU mm to do the get_user_pages / remap_pfn_range when it is required. After doing a quick implementation within our driver, I noticed following issues:

1) OS/VFIO logic into vendor driver which will be a maintenance issue.
2) Every driver vendor has to implement their own RB tree, instead of reusing the common existing VFIO code (vfio_find/link/unlink_dma)

3) IOMMU_UNMAP_DMA is expecting to get "unmapped bytes" back to the caller/QEMU, better not have anything inside a vendor driver that the VFIO caller immediately depends on.

Based on the above consideration, we decide to implement the DMA tracking logic within VGPU TYPE1 IOMMU code (ideally, this should be merged into current TYPE1 IOMMU code) and expose two symbols to outside for MMIO mapping and page translation and pinning.

Also, with a mmap MMIO interface between virtual and physical, this allows para-virtualized guest driver can access his virtual MMIO without taking a MMAP fault hit, also we can support different MMIO size between virtual and physical device.

int vgpu_map_virtual_bar

(
    uint64_t virt_bar_addr,
    uint64_t phys_bar_addr,
    uint32_t len,
    uint32_t flags
)

EXPORT_SYMBOL(vgpu_map_virtual_bar);

int vgpu_dma_do_translate(dma_addr_t *gfn_buffer, uint32_t count)

EXPORT_SYMBOL(vgpu_dma_do_translate);

Still a lot to be added and modified, such as supporting multiple VMs and multiple virtual devices, tracking the mapped / pinned region within VGPU IOMMU kernel driver, error handling, roll-back and locked memory size per user, etc.

4. Modules
==================================================================================

Two new modules are introduced: vfio_iommu_type1_vgpu.ko and vgpu.ko
vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU TYPE1 v1 and v2 interface.

vgpu.ko                  - provide registration interface and virtual device
                           VFIO access.

5. QEMU note
==================================================================================

To allow us focus on the VGPU kernel driver prototyping, we have introduced a new VFIO class - vgpu inside QEMU, so we don't have to change the existing vfio/pci.c file and use it as a reference for our implementation. It is basically just a quick c & p from vfio/pci.c to quickly meet our needs.

Once this proposal is finalized, we will move to vfio/pci.c instead of a new class, and probably the only thing required is to have a new way to discover the device.

6. Examples
==================================================================================

On this server, we have two NVIDIA M60 GPUs.

[root@cjia-vgx-kvm ~]# lspci -d 10de:13f2
86:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1)
87:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1)

After nvidia.ko gets initialized, we can query the supported vGPU type by accessing the "vgpu_supported_types" like following:

[root@cjia-vgx-kvm ~]# cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types 11:GRID M60-0B
12:GRID M60-0Q
13:GRID M60-1B
14:GRID M60-1Q
15:GRID M60-2B
16:GRID M60-2Q
17:GRID M60-4Q
18:GRID M60-8Q

For example the VM_UUID is c0b26072-dd1b-4340-84fe-bf338c510818, and we would
like to create "GRID M60-4Q" VM on it.

echo "c0b26072-dd1b-4340-84fe-bf338c510818:0:17" > /sys/bus/pci/devices/0000\:86\:00.0/vgpu_create
Note: the number 0 here is for vGPU device index. So far the change is not tested for multiple vgpu devices yet, but we will support it.

At this moment, if you query the "vgpu_supported_types" it will still show all supported virtual GPU types as no virtual GPU resource is committed yet.

Starting VM:

echo "c0b26072-dd1b-4340-84fe-bf338c510818" > /sys/class/vgpu/vgpu_start

then, the supported vGPU type query will return:

[root@cjia-vgx-kvm /home/cjia]$

Quoted 1 line

cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types

17:GRID M60-4Q

So vgpu_supported_config needs to be called whenever a new virtual device gets created as the underlying HW might limit the supported types if there are any existing VM runnings.

Then, VM gets shutdown, writes to /sys/class/vgpu/vgpu_shutdown will info the GPU driver vendor to clean up resource.

Eventually, those virtual GPUs can be removed by writing to vgpu_destroy under device sysfs.

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 12:37 AM

Quoted 64 lines · Yang Zhang

Quoted 63 lines · Jike Song

Quoted 19 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 9 lines

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:)  For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

- VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
- VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

- VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

- VFIO_PCI_ROM_REGION_INDEX
- VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

- VFIO_DEVICE_GET_INFO
- VFIO_DEVICE_GET_REGION_INFO
- VFIO_DEVICE_GET_IRQ_INFO
- VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

Quoted 7 lines

Quoted 2 lines

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

VFIO supports the following region attributes:

#define VFIO_REGION_INFO_FLAG_READ      (1 << 0) /* Region supports read */ #define VFIO_REGION_INFO_FLAG_WRITE     (1 << 1) /* Region supports write */ #define VFIO_REGION_INFO_FLAG_MMAP      (1 << 2) /* Region supports mmap */

If MMAP is not set, then the QEMU driver will do pread and/or pwrite to the specified offsets of the device file descriptor, depending on what accesses are supported.  This is all reported through the REGION_INFO ioctl for a given index.  If mmap is supported, the VM will have direct access to the area, without faulting to KVM other than to populate the mapping.  Without mmap support, a VM MMIO access traps into KVM, which returns out to QEMU to service the request, which then finds the MemoryRegion serviced through vfio, which will then perform a pread/pwrite through to the kernel vfio bus driver to handle the access.  Thanks,

Quoted 82 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 12:37 AM

Quoted 64 lines · Yang Zhang

Quoted 63 lines · Jike Song

Quoted 19 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 9 lines

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:)  For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

- VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
- VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

- VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

- VFIO_PCI_ROM_REGION_INDEX
- VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

- VFIO_DEVICE_GET_INFO
- VFIO_DEVICE_GET_REGION_INFO
- VFIO_DEVICE_GET_IRQ_INFO
- VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

Quoted 7 lines

Quoted 2 lines

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

VFIO supports the following region attributes:

#define VFIO_REGION_INFO_FLAG_READ      (1 << 0) /* Region supports read */ #define VFIO_REGION_INFO_FLAG_WRITE     (1 << 1) /* Region supports write */ #define VFIO_REGION_INFO_FLAG_MMAP      (1 << 2) /* Region supports mmap */

Quoted 11 lines

If MMAP is not set, then the QEMU driver will do pread and/or pwrite to the specified offsets of the device file descriptor, depending on what accesses are supported.  This is all reported through the REGION_INFO ioctl for a given index.  If mmap is supported, the VM will have direct access to the area, without faulting to KVM other than to populate the mapping.  Without mmap support, a VM MMIO access traps into KVM, which returns out to QEMU to service the request, which then finds the MemoryRegion serviced through vfio, which will then perform a pread/pwrite through to the kernel vfio bus driver to handle the access.  Thanks,

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Thanks
Kevin

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 4:06 AM

Quoted 26 lines · Neo Jia

Quoted 1 line · Kevin

Quoted 1 line

From: Alex Williamson [mailto:[email protected]]

Hi Alex, Kevin and Jike,

(Seems I shouldn't use attachment, resend it again to the list, patches are inline at the end)

Thanks for adding me to this technical discussion, a great opportunity for us to design together which can bring both Intel and NVIDIA vGPU solution to KVM platform.

Instead of directly jumping to the proposal that we have been working on recently for NVIDIA vGPU on KVM, I think it is better for me to put out couple quick comments / thoughts regarding the existing discussions on this thread as fundamentally I think we are solving the same problem, DMA, interrupt and MMIO.

Then we can look at what we have, hopefully we can reach some consensus soon.

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Quoted 3 lines

Infact to keep vfio-vgpu to be more generic, vgpu device creation and management can be centralized and done in vfio-vgpu. That also include adding to IOMMU group and VFIO group.

Is this really a good idea?  The concept of a vgpu is not unique to vfio, we want vfio to be a driver for a vgpu, not an integral part of the lifecycle of a vgpu.  That certainly doesn't exclude adding infrastructure to make lifecycle management of a vgpu more consistent between drivers, but it should be done independently of vfio.  I'll go back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfio does not create the VF, that's done in coordination with the PF making use of some PCI infrastructure for consistency between drivers.

It seems like we need to take more advantage of the class and driver core support to perhaps setup a vgpu bus and class with vfio-vgpu just being a driver for those devices.

Quoted 3 lines

6. Examples

=====================================================
=============================

Quoted 11 lines

On this server, we have two NVIDIA M60 GPUs.

[root@cjia-vgx-kvm ~]# lspci -d 10de:13f2
86:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1) 87:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1)

After nvidia.ko gets initialized, we can query the supported vGPU type by accessing the "vgpu_supported_types" like following:

[root@cjia-vgx-kvm ~]# cat

/sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types

Quoted 9 lines

11:GRID M60-0B
12:GRID M60-0Q
13:GRID M60-1B
14:GRID M60-1Q
15:GRID M60-2B
16:GRID M60-2Q
17:GRID M60-4Q
18:GRID M60-8Q

Quoted 4 lines

For example the VM_UUID is c0b26072-dd1b-4340-84fe-bf338c510818, and we would like to create "GRID M60-4Q" VM on it.

echo "c0b26072-dd1b-4340-84fe-bf338c510818:0:17" >

/sys/bus/pci/devices/0000\:86\:00.0/vgpu_create

Quoted 24 lines

Note: the number 0 here is for vGPU device index. So far the change is not tested for multiple vgpu devices yet, but we will support it.

At this moment, if you query the "vgpu_supported_types" it will still show all supported virtual GPU types as no virtual GPU resource is committed yet.

Starting VM:

echo "c0b26072-dd1b-4340-84fe-bf338c510818" > /sys/class/vgpu/vgpu_start

then, the supported vGPU type query will return:

[root@cjia-vgx-kvm /home/cjia]$

Quoted 1 line

cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types

17:GRID M60-4Q

So vgpu_supported_config needs to be called whenever a new virtual device gets created as the underlying HW might limit the supported types if there are any existing VM runnings.

Then, VM gets shutdown, writes to /sys/class/vgpu/vgpu_shutdown will info the GPU driver vendor to clean up resource.

Quoted 2 lines

Eventually, those virtual GPUs can be removed by writing to vgpu_destroy under device sysfs.

I'd like to hear Intel's thoughts on this interface.  Are there different vgpu capacities or priority classes that would necessitate different types of vcpus on Intel?

Does Intel have a need for start and shutdown interfaces?

From: Neo Jia [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 5:31 AM

Quoted 94 lines · Kevin

Quoted 82 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 12:37 AM

Quoted 64 lines · Yang Zhang

Quoted 63 lines · Jike Song

Quoted 19 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

	Bus Driver

		{ in i915/vgt/xxx.c }

Quoted 9 lines

Quoted 3 lines

- define a subset of vfio_pci interfaces
- selective pass-through (say aperture)
- trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:)  For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

- VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
- VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

- VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

- VFIO_PCI_ROM_REGION_INDEX
- VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

- VFIO_DEVICE_GET_INFO
- VFIO_DEVICE_GET_REGION_INFO
- VFIO_DEVICE_GET_IRQ_INFO
- VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

Quoted 7 lines

Quoted 2 lines

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

VFIO supports the following region attributes:

#define VFIO_REGION_INFO_FLAG_READ      (1 << 0) /* Region supports read */ #define VFIO_REGION_INFO_FLAG_WRITE     (1 << 1) /* Region supports write */ #define VFIO_REGION_INFO_FLAG_MMAP      (1 << 2) /* Region supports mmap */

Quoted 11 lines

If MMAP is not set, then the QEMU driver will do pread and/or pwrite to the specified offsets of the device file descriptor, depending on what accesses are supported.  This is all reported through the REGION_INFO ioctl for a given index.  If mmap is supported, the VM will have direct access to the area, without faulting to KVM other than to populate the mapping.  Without mmap support, a VM MMIO access traps into KVM, which returns out to QEMU to service the request, which then finds the MemoryRegion serviced through vfio, which will then perform a pread/pwrite through to the kernel vfio bus driver to handle the access.  Thanks,

Quoted 8 lines

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Hi Kevin,

Maybe there is some confusion about the VFIO interface that we are going to use here. I thought we were going to adopt VFIO so nobody would need to directly plug into kvm module.

Quoted 85 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 12:37 AM

Quoted 66 lines · Yang Zhang

Quoted 57 lines · Jike Song

Quoted 22 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 4 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Quoted 13 lines

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

  Bus Driver

  { in i915/vgt/xxx.c }

  - define a subset of vfio_pci interfaces
  - selective pass-through (say aperture)
  - trap MMIO: interface w/ QEMU

Quoted 6 lines

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:)  For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

  - VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
  - VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

  - VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

  - VFIO_PCI_ROM_REGION_INDEX
  - VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

  - VFIO_DEVICE_GET_INFO
  - VFIO_DEVICE_GET_REGION_INFO
  - VFIO_DEVICE_GET_IRQ_INFO
  - VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

Quoted 8 lines

We will need to extend:

  - VFIO_DEVICE_GET_REGION_INFO


a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

Quoted 5 lines

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

VFIO supports the following region attributes:

#define VFIO_REGION_INFO_FLAG_READ      (1 << 0) /* Region supports read */ #define VFIO_REGION_INFO_FLAG_WRITE     (1 << 1) /* Region supports write */ #define VFIO_REGION_INFO_FLAG_MMAP      (1 << 2) /* Region supports mmap */
If MMAP is not set, then the QEMU driver will do pread and/or pwrite to the specified offsets of the device file descriptor, depending on what accesses are supported.  This is all reported through the REGION_INFO

Quoted 8 lines

ioctl for a given index.  If mmap is supported, the VM will have direct access to the area, without faulting to KVM other than to populate the mapping.  Without mmap support, a VM MMIO access traps into KVM, which returns out to QEMU to service the request, which then finds the MemoryRegion serviced through vfio, which will then perform a pread/pwrite through to the kernel vfio bus driver to handle the access.  Thanks,

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 5:44 AM

Quoted 94 lines · Kevin

Quoted 82 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 12:37 AM

Quoted 64 lines · Yang Zhang

Quoted 63 lines · Jike Song

Quoted 19 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 14 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

  Bus Driver

  { in i915/vgt/xxx.c }

Quoted 9 lines

Quoted 3 lines

  - define a subset of vfio_pci interfaces
  - selective pass-through (say aperture)
  - trap MMIO: interface w/ QEMU

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:)  For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

  - VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
  - VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

  - VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

  - VFIO_PCI_ROM_REGION_INDEX
  - VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

  - VFIO_DEVICE_GET_INFO
  - VFIO_DEVICE_GET_REGION_INFO
  - VFIO_DEVICE_GET_IRQ_INFO
  - VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

We will need to extend:

  - VFIO_DEVICE_GET_REGION_INFO

Quoted 7 lines

Quoted 2 lines

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

VFIO supports the following region attributes:

#define VFIO_REGION_INFO_FLAG_READ      (1 << 0) /* Region supports read */ #define VFIO_REGION_INFO_FLAG_WRITE     (1 << 1) /* Region supports write */ #define VFIO_REGION_INFO_FLAG_MMAP      (1 << 2) /* Region supports mmap */

Quoted 11 lines

If MMAP is not set, then the QEMU driver will do pread and/or pwrite to the specified offsets of the device file descriptor, depending on what accesses are supported.  This is all reported through the REGION_INFO ioctl for a given index.  If mmap is supported, the VM will have direct access to the area, without faulting to KVM other than to populate the mapping.  Without mmap support, a VM MMIO access traps into KVM, which returns out to QEMU to service the request, which then finds the MemoryRegion serviced through vfio, which will then perform a pread/pwrite through to the kernel vfio bus driver to handle the access.  Thanks,

Quoted 7 lines

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

From: Alex Williamson
Sent: Wednesday, January 27, 2016 12:13 AM

Quoted 8 lines

b) adding other information. For example, for the OpRegion, QEMU need to do more than mmap a region, it has to:

- allocate a region
- copy contents from somewhere in host to that region
- mmap it to guest

Quoted 1 line

I remember you already have a prototype for this?

Yes, I'm working on this currently, it will by a device specific region and QEMU can either copy the contents to a new buffer in guest memory or provided trapped access to the host opregion.  I thought vgpus weren't going to need opregions though, I figured it was more for GVT-d support.  Thanks,

Quoted 105 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 5:44 AM

Quoted 93 lines · Kevin

Quoted 85 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 12:37 AM

Quoted 66 lines · Yang Zhang

Quoted 57 lines · Jike Song

Quoted 22 lines · Alex Williamson

[cc +Neo @Nvidia]

Hi Jike,

Quoted 4 lines · Jike Song

Quoted 3 lines · Kevin

I would expect we can spell out next level tasks toward above direction, upon which Alex can easily judge whether there are some common VFIO framework changes that he can help :-)

Quoted 13 lines

Hi Alex,

Here is a draft task list after a short discussion w/ Kevin, would you please have a look?

  Bus Driver

  { in i915/vgt/xxx.c }

  - define a subset of vfio_pci interfaces
  - selective pass-through (say aperture)
  - trap MMIO: interface w/ QEMU

Quoted 6 lines

What's included in the subset?  Certainly the bus reset ioctls really don't apply, but you'll need to support the full device interface, right?  That includes the region info ioctl and access through the vfio device file descriptor as well as the interrupt info and setup ioctls.

[All interfaces I thought are via ioctl:)  For other stuff like file descriptor we'll definitely keep it.]

The list of ioctl commands provided by vfio_pci:

  - VFIO_DEVICE_GET_PCI_HOT_RESET_INFO
  - VFIO_DEVICE_PCI_HOT_RESET

As you said, above 2 don't apply. But for this:

  - VFIO_DEVICE_RESET

In my opinion it should be kept, no matter what will be provided in the bus driver.

  - VFIO_PCI_ROM_REGION_INDEX
  - VFIO_PCI_VGA_REGION_INDEX

I suppose above 2 don't apply neither? For a vgpu we don't provide a ROM BAR or VGA region.

  - VFIO_DEVICE_GET_INFO
  - VFIO_DEVICE_GET_REGION_INFO
  - VFIO_DEVICE_GET_IRQ_INFO
  - VFIO_DEVICE_SET_IRQS

Above 4 are needed of course.

Quoted 8 lines

We will need to extend:

  - VFIO_DEVICE_GET_REGION_INFO


a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

Quoted 5 lines

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

VFIO supports the following region attributes:

#define VFIO_REGION_INFO_FLAG_READ      (1 << 0) /* Region supports read */ #define VFIO_REGION_INFO_FLAG_WRITE     (1 << 1) /* Region supports write */ #define VFIO_REGION_INFO_FLAG_MMAP      (1 << 2) /* Region supports mmap */
If MMAP is not set, then the QEMU driver will do pread and/or pwrite to the specified offsets of the device file descriptor, depending on what accesses are supported.  This is all reported through the REGION_INFO

Quoted 8 lines

ioctl for a given index.  If mmap is supported, the VM will have direct access to the area, without faulting to KVM other than to populate the mapping.  Without mmap support, a VM MMIO access traps into KVM, which returns out to QEMU to service the request, which then finds the MemoryRegion serviced through vfio, which will then perform a pread/pwrite through to the kernel vfio bus driver to handle the access.  Thanks,

Quoted 8 lines

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 9 lines

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 19 lines

Quoted 10 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 8 lines

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

Quoted 5 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Are you jumping to the conclusion that it cannot be done with proper layering in place?  Performance is important, but it's not an excuse to abandon designing interfaces between independent components.  Thanks,

Quoted 27 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 18 lines

Quoted 9 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 9 lines

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

Quoted 6 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 5 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place?  Performance is important, but it's not an excuse to abandon designing interfaces between independent components.  Thanks,

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
  - KVMGT will expose a private control variable whether in-kernel delivery is required;

  - when the variable is true, KVMGT will register in-kernel MMIO emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
  - when the variable is false, KVMGT will not register anything. VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

Quoted 25 lines · Neo Jia

Quoted 1 line · Kevin

Quoted 1 line

From: Alex Williamson [mailto:[email protected]]

Hi Alex, Kevin and Jike,

(Seems I shouldn't use attachment, resend it again to the list, patches are inline at the end)

Thanks for adding me to this technical discussion, a great opportunity for us to design together which can bring both Intel and NVIDIA vGPU solution to KVM platform.

Instead of directly jumping to the proposal that we have been working on recently for NVIDIA vGPU on KVM, I think it is better for me to put out couple quick comments / thoughts regarding the existing discussions on this thread as fundamentally I think we are solving the same problem, DMA, interrupt and MMIO.
Then we can look at what we have, hopefully we can reach some consensus soon.

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Quoted 4 lines

Infact to keep vfio-vgpu to be more generic, vgpu device creation and management can be centralized and done in vfio-vgpu. That also include adding to IOMMU group and VFIO group.

Is this really a good idea?  The concept of a vgpu is not unique to vfio, we want vfio to be a driver for a vgpu, not an integral part of the lifecycle of a vgpu.  That certainly doesn't exclude adding infrastructure to make lifecycle management of a vgpu more consistent between drivers, but it should be done independently of vfio.  I'll go back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfio does not create the VF, that's done in coordination with the PF making use of some PCI infrastructure for consistency between drivers.

It seems like we need to take more advantage of the class and driver core support to perhaps setup a vgpu bus and class with vfio-vgpu just being a driver for those devices.

Quoted 89 lines

Graphics driver can register with vfio-vgpu to get management and emulation call backs to graphics driver.

We already have struct vgpu_device in our proposal that keeps pointer to physical device.

Quoted 3 lines

- vfio_pci will inject an IRQ to guest only when physical IRQ generated; whereas vfio_vgpu may inject an IRQ for emulation purpose. Anyway they can share the same injection interface;

eventfd to inject the interrupt is known to vfio-vgpu, that fd should be available to graphics driver so that graphics driver can inject interrupts directly when physical device triggers interrupt.

Here is the proposal we have, please review.

Please note the patches we have put out here is mainly for POC purpose to verify our understanding also can serve the purpose to reduce confusions and speed up our design, although we are very happy to refine that to something eventually can be used for both parties and upstreamed.

Linux vGPU kernel design
==================================================================================

Here we are proposing a generic Linux kernel module based on VFIO framework which allows different GPU vendors to plugin and provide their GPU virtualization solution on KVM, the benefits of having such generic kernel module are:

1) Reuse QEMU VFIO driver, supporting VFIO UAPI

2) GPU HW agnostic management API for upper layer software such as libvirt
3) No duplicated VFIO kernel logic reimplemented by different GPU driver vendor
0. High level overview
==================================================================================


  user space:

                                +-----------+  VFIO IOMMU IOCTLs
                      +---------| QEMU VFIO |-------------------------+
        VFIO IOCTLs   |         +-----------+                         |
                      |                                               | 
 ---------------------|-----------------------------------------------|---------
                      |                                               |
  kernel space:       |  +--->----------->---+  (callback)            V
                      |  |                   v                 +------V-----+
  +----------+   +----V--^--+          +--+--+-----+           | VGPU       |
  |          |   |          |     +----| nvidia.ko +----->-----> TYPE1 IOMMU|
  | VFIO Bus <===| VGPU.ko  |<----|    +-----------+     |     +---++-------+ 
  |          |   |          |     | (register)           ^         ||
  +----------+   +-------+--+     |    +-----------+     |         ||
                         V        +----| i915.ko   +-----+     +---VV-------+ 
                         |             +-----^-----+           | TYPE1      |
                         |  (callback)       |                 | IOMMU      |
                         +-->------------>---+                 +------------+

 access flow:

  Guest MMIO / PCI config access
  |
  -------------------------------------------------
  |

  +-----> KVM VM_EXITs  (kernel)
          |
  -------------------------------------------------
          |
          +-----> QEMU VFIO driver (user)
                  | 
  -------------------------------------------------
                  |
                  +---->  VGPU kernel driver (kernel)
                          |  
                          | 
                          +----> vendor driver callback

1. VGPU management interface
==================================================================================

This is the interface allows upper layer software (mostly libvirt) to query and configure virtual GPU device in a HW agnostic fashion. Also, this management interface has provided flexibility to underlying GPU vendor to support virtual device hotplug, multiple virtual devices per VM, multiple virtual devices from different physical devices, etc.

1.1 Under per-physical device sysfs:
----------------------------------------------------------------------------------

vgpu_supported_types - RO, list the current supported virtual GPU types and its

Quoted 8 lines

VGPU_ID. VGPU_ID - a vGPU type identifier returned from reads of "vgpu_supported_types".

vgpu_create - WO, input syntax <VM_UUID:idx:VGPU_ID>, create a virtual gpu device on a target physical GPU. idx: virtual device index inside a VM
vgpu_destroy - WO, input syntax <VM_UUID:idx>, destroy a virtual gpu device on a target physical GPU

I've noted in previous discussions that we need to separate user policy from kernel policy here, the kernel policy should not require a "VM UUID".  A UUID simply represents a set of one or more devices and an index picks the device within the set.  Whether that UUID matches a VM or is independently used is up to the user policy when creating the device.

Personally I'd also prefer to get rid of the concept of indexes within a UUID set of devices and instead have each device be independent.  This seems to be an imposition on the nvidia implementation into the kernel interface design.

Quoted 133 lines

1.3 Under vgpu class sysfs:
----------------------------------------------------------------------------------

vgpu_start - WO, input syntax <VM_UUID>, this will trigger the registration interface to notify the GPU vendor driver to commit virtual GPU resource for this target VM.

Also, the vgpu_start function is a synchronized call, the successful return of this call will indicate all the requested vGPU resource has been fully committed, the VMM should continue.

vgpu_shutdown - WO, input syntax <VM_UUID>, this will trigger the registration interface to notify the GPU vendor driver to release virtual GPU resource of this target VM.

1.4 Virtual device Hotplug
----------------------------------------------------------------------------------

To support virtual device hotplug, <vgpu_create> and <vgpu_destroy> can be accessed during VM runtime, and the corresponding registration callback will be invoked to allow GPU vendor support hotplug.

To support hotplug, vendor driver would take necessary action to handle the situation when a vgpu_create is done on a VM_UUID after vgpu_start, and that implies both create and start for that vgpu device.

Same, vgpu_destroy implies a vgpu_shudown on a running VM only if vendor driver supports vgpu hotplug.

If hotplug is not supported and VM is still running, vendor driver can return error code to indicate not supported.

Separate create from start gives flixibility to have:

- multiple vgpu instances for single VM and
- hotplug feature.

2. GPU driver vendor registration interface
==================================================================================

2.1 Registration interface definition (include/linux/vgpu.h)
----------------------------------------------------------------------------------

extern int vgpu_register_device(struct pci_dev *dev,
                                const struct gpu_device_ops *ops);

extern void vgpu_unregister_device(struct pci_dev *dev);

/**
 * struct gpu_device_ops - Structure to be registered for each physical GPU to
 * register the device to vgpu module.
 *
 * @owner:                      The module owner.
 * @vgpu_supported_config:      Called to get information about supported vgpu
 * types.
 *                              @dev : pci device structure of physical GPU. *                              @config: should return string listing supported *                              config
 *                              Returns integer: success (0) or error (< 0)
 * @vgpu_create:                Called to allocate basic resouces in graphics *                              driver for a particular vgpu.
 *                              @dev: physical pci device structure on which *                              vgpu
 *                                    should be created
 *                              @vm_uuid: VM's uuid for which VM it is intended *                              to
 *                              @instance: vgpu instance in that VM *                              @vgpu_id: This represents the type of vgpu to be *                                        created
 *                              Returns integer: success (0) or error (< 0)
 * @vgpu_destroy:               Called to free resources in graphics driver for *                              a vgpu instance of that VM.
 *                              @dev: physical pci device structure to which *                              this vgpu points to.
 *                              @vm_uuid: VM's uuid for which the vgpu belongs *                              to.
 *                              @instance: vgpu instance in that VM *                              Returns integer: success (0) or error (< 0) *                              If VM is running and vgpu_destroy is called that *                              means the vGPU is being hotunpluged. Return *                              error
 *                              if VM is running and graphics driver doesn't *                              support vgpu hotplug.
 * @vgpu_start:                 Called to do initiate vGPU initialization *                              process in graphics driver when VM boots before *                              qemu starts.
 *                              @vm_uuid: VM's UUID which is booting. *                              Returns integer: success (0) or error (< 0)
 * @vgpu_shutdown:              Called to teardown vGPU related resources for *                              the VM
 *                              @vm_uuid: VM's UUID which is shutting down . *                              Returns integer: success (0) or error (< 0)
 * @read:                       Read emulation callback
 *                              @vdev: vgpu device structure
 *                              @buf: read buffer
 *                              @count: number bytes to read
 *                              @address_space: specifies for which address *                              space
 *                              the request is: pci_config_space, IO register *                              space or MMIO space.
 *                              Retuns number on bytes read on success or error.
 * @write:                      Write emulation callback
 *                              @vdev: vgpu device structure
 *                              @buf: write buffer
 *                              @count: number bytes to be written *                              @address_space: specifies for which address *                              space
 *                              the request is: pci_config_space, IO register *                              space or MMIO space.
 *                              Retuns number on bytes written on success or *                              error.
 * @vgpu_set_irqs:              Called to send about interrupts configuration *                              information that qemu set.
 *                              @vdev: vgpu device structure
 *                              @flags, index, start, count and *data : same as *                              that of struct vfio_irq_set of
 *                              VFIO_DEVICE_SET_IRQS API.
 *
 * Physical GPU that support vGPU should be register with vgpu module with
 * gpu_device_ops structure.
 */

struct gpu_device_ops {
        struct module   *owner;
        int     (*vgpu_supported_config)(struct pci_dev *dev, char *config); int     (*vgpu_create)(struct pci_dev *dev, uuid_le vm_uuid, uint32_t instance, uint32_t vgpu_id); int     (*vgpu_destroy)(struct pci_dev *dev, uuid_le vm_uuid, uint32_t instance);
        int     (*vgpu_start)(uuid_le vm_uuid);
        int     (*vgpu_shutdown)(uuid_le vm_uuid);
        ssize_t (*read) (struct vgpu_device *vdev, char *buf, size_t count,

Quoted 8 lines

                         uint32_t address_space, loff_t pos); ssize_t (*write)(struct vgpu_device *vdev, char *buf, size_t count, uint32_t address_space,loff_t pos);
        int     (*vgpu_set_irqs)(struct vgpu_device *vdev, uint32_t flags, unsigned index, unsigned start, unsigned count, void *data);

};

I wonder if it shouldn't be vfio-vgpu sub-drivers (ie, Intel and Nvidia) that register these ops with the main vfio-vgpu driver and they should also include a probe() function which allows us to associate a given vgpu device with a set of vendor ops.

Quoted 70 lines

2.2 Details for callbacks we haven't mentioned above.
---------------------------------------------------------------------------------

vgpu_supported_config: allows the vendor driver to specify the supported vGPU type/configuration

vgpu_create          : create a virtual GPU device, can be used for device hotplug.
vgpu_destroy         : destroy a virtual GPU device, can be used for device hotplug.
vgpu_start           : callback function to notify vendor driver vgpu device come to live for a given virtual machine.

vgpu_shutdown        : callback function to notify vendor driver

read                 : callback to vendor driver to handle virtual device config space or MMIO read access

write                : callback to vendor driver to handle virtual device config space or MMIO write access

vgpu_set_irqs        : callback to vendor driver to pass along the interrupt information for the target virtual device, then vendor driver can inject interrupt into virtual machine for this device.

2.3 Potential additional virtual device configuration registration interface:
---------------------------------------------------------------------------------

callback function to describe the MMAP behavior of the virtual GPU

callback function to allow GPU vendor driver to provide PCI config space backing memory.

3. VGPU TYPE1 IOMMU
==================================================================================

Here we are providing a TYPE1 IOMMU for vGPU which will basically keep track the <iova, hva, size, flag> and save the QEMU mm for later reference.

You can find the quick/ugly implementation in the attached patch file, which is actually just a simple version Alex's type1 IOMMU without actual real mapping when IOMMU_MAP_DMA / IOMMU_UNMAP_DMA is called.

We have thought about providing another vendor driver registration interface so such tracking information will be sent to vendor driver and he will use the QEMU mm to do the get_user_pages / remap_pfn_range when it is required. After doing a quick implementation within our driver, I noticed following issues:

1) OS/VFIO logic into vendor driver which will be a maintenance issue.
2) Every driver vendor has to implement their own RB tree, instead of reusing the common existing VFIO code (vfio_find/link/unlink_dma)

3) IOMMU_UNMAP_DMA is expecting to get "unmapped bytes" back to the caller/QEMU, better not have anything inside a vendor driver that the VFIO caller immediately depends on.

Based on the above consideration, we decide to implement the DMA tracking logic within VGPU TYPE1 IOMMU code (ideally, this should be merged into current TYPE1 IOMMU code) and expose two symbols to outside for MMIO mapping and page translation and pinning.

Also, with a mmap MMIO interface between virtual and physical, this allows para-virtualized guest driver can access his virtual MMIO without taking a MMAP fault hit, also we can support different MMIO size between virtual and physical device.

int vgpu_map_virtual_bar

Quoted 8 lines

(
    uint64_t virt_bar_addr,
    uint64_t phys_bar_addr,
    uint32_t len,
    uint32_t flags
)

EXPORT_SYMBOL(vgpu_map_virtual_bar);

Per the implementation provided, this needs to be implemented in the vfio device driver, not in the iommu interface.  Finding the DMA mapping of the device and replacing it is wrong.  It should be remapped at the vfio device file interface using vm_ops.

Quoted 8 lines

int vgpu_dma_do_translate(dma_addr_t *gfn_buffer, uint32_t count)

EXPORT_SYMBOL(vgpu_dma_do_translate);

Still a lot to be added and modified, such as supporting multiple VMs and multiple virtual devices, tracking the mapped / pinned region within VGPU IOMMU kernel driver, error handling, roll-back and locked memory size per user, etc.

Particularly, handling of mapping changes is completely missing.  This cannot be a point in time translation, the user is free to remap addresses whenever they wish and device translations need to be updated accordingly.

Quoted 8 lines

4. Modules
==================================================================================

Two new modules are introduced: vfio_iommu_type1_vgpu.ko and vgpu.ko
vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU TYPE1 v1 and v2 interface.

Depending on how intrusive it is, this can possibly by done within the existing type1 driver.  Either that or we can split out common code for use by a separate module.

Quoted 56 lines

vgpu.ko                  - provide registration interface and virtual device
                           VFIO access.

5. QEMU note
==================================================================================

To allow us focus on the VGPU kernel driver prototyping, we have introduced a new VFIO class - vgpu inside QEMU, so we don't have to change the existing vfio/pci.c file and use it as a reference for our implementation. It is basically just a quick c & p from vfio/pci.c to quickly meet our needs.

Once this proposal is finalized, we will move to vfio/pci.c instead of a new class, and probably the only thing required is to have a new way to discover the device.

6. Examples
==================================================================================

On this server, we have two NVIDIA M60 GPUs.

[root@cjia-vgx-kvm ~]# lspci -d 10de:13f2
86:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1)
87:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1)

After nvidia.ko gets initialized, we can query the supported vGPU type by accessing the "vgpu_supported_types" like following:

[root@cjia-vgx-kvm ~]# cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types 11:GRID M60-0B
12:GRID M60-0Q
13:GRID M60-1B
14:GRID M60-1Q
15:GRID M60-2B
16:GRID M60-2Q
17:GRID M60-4Q
18:GRID M60-8Q

For example the VM_UUID is c0b26072-dd1b-4340-84fe-bf338c510818, and we would
like to create "GRID M60-4Q" VM on it.

echo "c0b26072-dd1b-4340-84fe-bf338c510818:0:17" > /sys/bus/pci/devices/0000\:86\:00.0/vgpu_create
Note: the number 0 here is for vGPU device index. So far the change is not tested for multiple vgpu devices yet, but we will support it.

At this moment, if you query the "vgpu_supported_types" it will still show all supported virtual GPU types as no virtual GPU resource is committed yet.

Starting VM:

echo "c0b26072-dd1b-4340-84fe-bf338c510818" > /sys/class/vgpu/vgpu_start

then, the supported vGPU type query will return:

[root@cjia-vgx-kvm /home/cjia]$

Quoted 1 line

cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types

Quoted 11 lines

17:GRID M60-4Q

So vgpu_supported_config needs to be called whenever a new virtual device gets created as the underlying HW might limit the supported types if there are any existing VM runnings.

Then, VM gets shutdown, writes to /sys/class/vgpu/vgpu_shutdown will info the GPU driver vendor to clean up resource.

Eventually, those virtual GPUs can be removed by writing to vgpu_destroy under device sysfs.

I'd like to hear Intel's thoughts on this interface.  Are there different vgpu capacities or priority classes that would necessitate different types of vcpus on Intel?

I think there are some gaps in translating from named vgpu types to indexes here, along with my previous mention of the UUID/set oddity.

Does Intel have a need for start and shutdown interfaces?

Neo, wasn't there at some point information about how many of each type could be supported through these interfaces?  How does a user know their capacity limits?

Thanks,
Alex

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:27 AM

Quoted 33 lines · Kevin

Quoted 28 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 19 lines

Quoted 10 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 8 lines

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

Quoted 5 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 4 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place?  Performance is important, but it's not an excuse to abandon designing interfaces between independent components.  Thanks,

Quoted 5 lines

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
  - KVMGT will expose a private control variable whether in-kernel delivery is required;

But in-kernel delivery is never *required*.  Wouldn't userspace want to deliver in-kernel any time it possibly could?

Quoted 6 lines

  - when the variable is true, KVMGT will register in-kernel MMIO emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
  - when the variable is false, KVMGT will not register anything. VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

No, that means the interface is entirely dependent on a backdoor through KVM.  Why can't userspace (QEMU) do something like register an MMIO region with KVM handled via a provided file descriptor and offset, couldn't KVM then call the file ops without a kernel exit?  Thanks,

Quoted 52 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:27 AM

Quoted 32 lines · Kevin

Quoted 27 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 18 lines

Quoted 9 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 9 lines

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

Quoted 6 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 5 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place?  Performance is important, but it's not an excuse to abandon designing interfaces between independent components.  Thanks,

Quoted 6 lines

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
  - KVMGT will expose a private control variable whether in-kernel delivery is required;

But in-kernel delivery is never *required*.  Wouldn't userspace want to deliver in-kernel any time it possibly could?

Quoted 6 lines

  - when the variable is true, KVMGT will register in-kernel MMIO emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
  - when the variable is false, KVMGT will not register anything. VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

Quoted 6 lines

No, that means the interface is entirely dependent on a backdoor through KVM.  Why can't userspace (QEMU) do something like register an MMIO region with KVM handled via a provided file descriptor and offset, couldn't KVM then call the file ops without a kernel exit?  Thanks,

Could you elaborate this thought? If it can achieve the purpose w/o a kernel exit definitely we can adapt to it. :-)

Quoted 12 lines · Alex Williamson

Quoted 4 lines

1.1 Under per-physical device sysfs:
----------------------------------------------------------------------------------

vgpu_supported_types - RO, list the current supported virtual GPU types and its

Quoted 8 lines

VGPU_ID. VGPU_ID - a vGPU type identifier returned from reads of "vgpu_supported_types".

vgpu_create - WO, input syntax <VM_UUID:idx:VGPU_ID>, create a virtual gpu device on a target physical GPU. idx: virtual device index inside a VM
vgpu_destroy - WO, input syntax <VM_UUID:idx>, destroy a virtual gpu device on a target physical GPU

Quoted 14 lines

I've noted in previous discussions that we need to separate user policy from kernel policy here, the kernel policy should not require a "VM UUID".  A UUID simply represents a set of one or more devices and an index picks the device within the set.  Whether that UUID matches a VM or is independently used is up to the user policy when creating the device.

Personally I'd also prefer to get rid of the concept of indexes within a UUID set of devices and instead have each device be independent.  This seems to be an imposition on the nvidia implementation into the kernel interface design.

Hi Alex,

I agree with you that we should not put UUID concept into a kernel API. At this point (without any prototyping), I am thinking of using a list of virtual devices instead of UUID.

Quoted 10 lines

Quoted 10 lines

int vgpu_map_virtual_bar
(
    uint64_t virt_bar_addr,
    uint64_t phys_bar_addr,
    uint32_t len,
    uint32_t flags
)

EXPORT_SYMBOL(vgpu_map_virtual_bar);

Quoted 7 lines

Per the implementation provided, this needs to be implemented in the vfio device driver, not in the iommu interface.  Finding the DMA mapping of the device and replacing it is wrong.  It should be remapped at the vfio device file interface using vm_ops.

So you are basically suggesting that we are going to take a mmap fault and within that fault handler, we will go into vendor driver to look up the "pre-registered" mapping and remap there.

Is my understanding correct?

Quoted 8 lines

Quoted 7 lines

int vgpu_dma_do_translate(dma_addr_t *gfn_buffer, uint32_t count)

EXPORT_SYMBOL(vgpu_dma_do_translate);

Still a lot to be added and modified, such as supporting multiple VMs and multiple virtual devices, tracking the mapped / pinned region within VGPU IOMMU kernel driver, error handling, roll-back and locked memory size per user, etc.

Quoted 6 lines

Particularly, handling of mapping changes is completely missing.  This cannot be a point in time translation, the user is free to remap addresses whenever they wish and device translations need to be updated accordingly.

When you say "user", do you mean the QEMU?

Here, whenever the DMA that
the guest driver is going to launch will be first pinned within VM, and then registered to QEMU, therefore the IOMMU memory listener, eventually the pages will be pinned by the GPU or DMA engine.

Since we are keeping the upper level code same, thinking about passthru case, where the GPU has already put the real IOVA into his PTEs, I don't know how QEMU can change that mapping without causing an IOMMU fault on a active DMA device.

Quoted 80 lines

Quoted 7 lines

4. Modules
==================================================================================

Two new modules are introduced: vfio_iommu_type1_vgpu.ko and vgpu.ko
vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU TYPE1 v1 and v2 interface.

Depending on how intrusive it is, this can possibly by done within the existing type1 driver.  Either that or we can split out common code for use by a separate module.

Quoted 57 lines

vgpu.ko                  - provide registration interface and virtual device VFIO access.

5. QEMU note
==================================================================================

To allow us focus on the VGPU kernel driver prototyping, we have introduced a new VFIO class - vgpu inside QEMU, so we don't have to change the existing vfio/pci.c file and use it as a reference for our implementation. It is basically just a quick c & p from vfio/pci.c to quickly meet our needs.

Once this proposal is finalized, we will move to vfio/pci.c instead of a new class, and probably the only thing required is to have a new way to discover the device.

6. Examples
==================================================================================

On this server, we have two NVIDIA M60 GPUs.

[root@cjia-vgx-kvm ~]# lspci -d 10de:13f2
86:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1) 87:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1)
After nvidia.ko gets initialized, we can query the supported vGPU type by accessing the "vgpu_supported_types" like following:

[root@cjia-vgx-kvm ~]# cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types 11:GRID M60-0B
12:GRID M60-0Q
13:GRID M60-1B
14:GRID M60-1Q
15:GRID M60-2B
16:GRID M60-2Q
17:GRID M60-4Q
18:GRID M60-8Q

For example the VM_UUID is c0b26072-dd1b-4340-84fe-bf338c510818, and we would like to create "GRID M60-4Q" VM on it.

echo "c0b26072-dd1b-4340-84fe-bf338c510818:0:17" > /sys/bus/pci/devices/0000\:86\:00.0/vgpu_create

Note: the number 0 here is for vGPU device index. So far the change is not tested for multiple vgpu devices yet, but we will support it.

At this moment, if you query the "vgpu_supported_types" it will still show all supported virtual GPU types as no virtual GPU resource is committed yet.
Starting VM:

echo "c0b26072-dd1b-4340-84fe-bf338c510818" > /sys/class/vgpu/vgpu_start
then, the supported vGPU type query will return:

[root@cjia-vgx-kvm /home/cjia]$

Quoted 1 line

cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types

Quoted 11 lines

17:GRID M60-4Q

So vgpu_supported_config needs to be called whenever a new virtual device gets created as the underlying HW might limit the supported types if there are any existing VM runnings.

Then, VM gets shutdown, writes to /sys/class/vgpu/vgpu_shutdown will info the GPU driver vendor to clean up resource.

Eventually, those virtual GPUs can be removed by writing to vgpu_destroy under device sysfs.

Quoted 15 lines

I'd like to hear Intel's thoughts on this interface.  Are there different vgpu capacities or priority classes that would necessitate different types of vcpus on Intel?

I think there are some gaps in translating from named vgpu types to indexes here, along with my previous mention of the UUID/set oddity.
Does Intel have a need for start and shutdown interfaces?

Neo, wasn't there at some point information about how many of each type could be supported through these interfaces?  How does a user know their capacity limits?

Thanks for reminding me that, I think we probably forget to put that *important* information as the output of "vgpu_supported_types".

Regarding the capacity, we can provide the frame buffer size as part of the "vgpu_supported_types" output as well, I would imagine those will be eventually show up on the openstack management interface or virt-mgr.

Basically, yes there would be a separate col to show the number of instance you can create for each type of VGPU on a specific physical GPU.

Quoted 5 lines · Jike Song

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

Quoted 2 lines

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

Quoted 7 lines · Yang Zhang

Quoted 5 lines · Jike Song

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

Quoted 8 lines

Quoted 2 lines

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

Hi Yang,

MMIO accesses are supposed to be handled in kernel, without vm-exiting to QEMU, similar to in-kernel irqchip :)

Quoted 53 lines · Kevin

Quoted 53 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:27 AM

Quoted 33 lines · Kevin

Quoted 28 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 19 lines

Quoted 10 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 8 lines

That sounds like a future optimization TBH. There's very strict layering between vfio and kvm. Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do. Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device. We can't simply take shortcuts to provide such direct access. Thanks,

Quoted 5 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 4 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place? Performance is important, but it's not an excuse to abandon designing interfaces between independent components. Thanks,

Quoted 5 lines

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
- KVMGT will expose a private control variable whether in-kernel
delivery is required;

But in-kernel delivery is never *required*. Wouldn't userspace want to deliver in-kernel any time it possibly could?

Quoted 6 lines

- when the variable is true, KVMGT will register in-kernel MMIO
emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
- when the variable is false, KVMGT will not register anything.
VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

Quoted 8 lines

Quoted 5 lines

No, that means the interface is entirely dependent on a backdoor through KVM. Why can't userspace (QEMU) do something like register an MMIO region with KVM handled via a provided file descriptor and offset, couldn't KVM then call the file ops without a kernel exit? Thanks,

Could you elaborate this thought? If it can achieve the purpose w/o a kernel exit definitely we can adapt to it. :-)

I only thought of it when replying to the last email and have been doing some research, but we already do quite a bit of synchronization through file descriptors. The kvm-vfio pseudo device uses a group file descriptor to ensure a user has access to a group, allowing some degree of interaction between modules. Eventfds and irqfds already make use of f_ops on file descriptors to poke data. So, if KVM had information that an MMIO region was backed by a file descriptor for which it already has a reference via fdget() (and verified access rights and whatnot), then it ought to be a simple matter to get to f_ops->read/write knowing the base offset of that MMIO region. Perhaps it could even simply use __vfs_read/write(). Then we've got a proper reference to the file descriptor for ownership purposes and we've transparently jumped across modules without any implicit knowledge of the other end. Could it work?

Thanks,

Alex

Quoted 17 lines · Jike Song

Quoted 15 lines · Yang Zhang

Quoted 7 lines · Jike Song

We will need to extend:

- VFIO_DEVICE_GET_REGION_INFO

a) adding a flag: DONT_MAP. For example, the MMIO of vgpu should be trapped instead of being mmap-ed.

I may not in the context, but i am curious how to handle the DONT_MAP in vfio driver? Since there are no real MMIO maps into the region and i suppose the access to the region should be handled by vgpu in i915 driver, but currently most of the mmio accesses are handled by Qemu.

Quoted 4 lines

Hi Yang,

MMIO accesses are supposed to be handled in kernel, without vm-exiting to QEMU, similar to in-kernel irqchip :)

The question is current vfio doesn't support it. The long discussion between Alex and Kevin is what i am to understand how KVMGT works under vfio framework.

Quoted 53 lines · Kevin

Quoted 53 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:27 AM

Quoted 33 lines · Kevin

Quoted 28 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 19 lines

Quoted 10 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 8 lines

That sounds like a future optimization TBH. There's very strict layering between vfio and kvm. Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do. Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device. We can't simply take shortcuts to provide such direct access. Thanks,

Quoted 5 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 4 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place? Performance is important, but it's not an excuse to abandon designing interfaces between independent components. Thanks,

Quoted 5 lines

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
- KVMGT will expose a private control variable whether in-kernel
delivery is required;

But in-kernel delivery is never *required*. Wouldn't userspace want to deliver in-kernel any time it possibly could?

Quoted 6 lines

- when the variable is true, KVMGT will register in-kernel MMIO
emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
- when the variable is false, KVMGT will not register anything.
VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

Quoted 8 lines

Quoted 5 lines

No, that means the interface is entirely dependent on a backdoor through KVM. Why can't userspace (QEMU) do something like register an MMIO region with KVM handled via a provided file descriptor and offset, couldn't KVM then call the file ops without a kernel exit? Thanks,

Could you elaborate this thought? If it can achieve the purpose w/o a kernel exit definitely we can adapt to it. :-)

I only thought of it when replying to the last email and have been doing some research, but we already do quite a bit of synchronization through file descriptors. The kvm-vfio pseudo device uses a group file descriptor to ensure a user has access to a group, allowing some degree of interaction between modules. Eventfds and irqfds already make use of f_ops on file descriptors to poke data. So, if KVM had information that an MMIO region was backed by a file descriptor for which it already has a reference via fdget() (and verified access rights and whatnot), then it ought to be a simple matter to get to f_ops->read/write knowing the base offset of that MMIO region. Perhaps it could even simply use __vfs_read/write(). Then we've got a proper reference to the file descriptor for ownership purposes and we've transparently jumped across modules without any implicit knowledge of the other end. Could it work? Thanks,

Quoted 62 lines · Alex Williamson

Quoted 58 lines · Kevin

Quoted 52 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:27 AM

Quoted 32 lines · Kevin

Quoted 27 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 18 lines

Quoted 9 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 9 lines

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

Quoted 6 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 5 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place?  Performance is important, but it's not an excuse to abandon designing interfaces between independent components.  Thanks,

Quoted 6 lines

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
  - KVMGT will expose a private control variable whether in-kernel delivery is required;

But in-kernel delivery is never *required*.  Wouldn't userspace want to deliver in-kernel any time it possibly could?

Quoted 6 lines

  - when the variable is true, KVMGT will register in-kernel MMIO emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
  - when the variable is false, KVMGT will not register anything. VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

Quoted 6 lines

No, that means the interface is entirely dependent on a backdoor through KVM.  Why can't userspace (QEMU) do something like register an MMIO region with KVM handled via a provided file descriptor and offset, couldn't KVM then call the file ops without a kernel exit?  Thanks,

Quoted 3 lines

Could you elaborate this thought? If it can achieve the purpose w/o a kernel exit definitely we can adapt to it. :-)

Quoted 14 lines

I only thought of it when replying to the last email and have been doing some research, but we already do quite a bit of synchronization through file descriptors.  The kvm-vfio pseudo device uses a group file descriptor to ensure a user has access to a group, allowing some degree of interaction between modules.  Eventfds and irqfds already make use of f_ops on file descriptors to poke data.  So, if KVM had information that an MMIO region was backed by a file descriptor for which it already has a reference via fdget() (and verified access rights and whatnot), then it ought to be a simple matter to get to f_ops->read/write knowing the base offset of that MMIO region.  Perhaps it could even simply use __vfs_read/write().  Then we've got a proper reference to the file descriptor for ownership purposes and we've transparently jumped across modules without any implicit knowledge of the other end.  Could it work?

This is OK for KVMGT, from fops to vgpu device-model would always be simple. The only question is, how is KVM hypervisor supposed to get the fd on VM-exitings?

copy-and-paste the current implementation of vcpu_mmio_write(), seems nothing but GPA and len are provided:

  static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, const void *v)
  {
  int handled = 0;
  int n;

  do {
  n = min(len, 8);
  if (!(vcpu->arch.apic &&
        !kvm_iodevice_write(vcpu, &vcpu->arch.apic->dev, addr, n, v)) && kvm_io_bus_write(vcpu, KVM_MMIO_BUS, addr, n, v)) break;
  handled += n;
  addr += n;

  len -= n;
  v += n;
  } while (len);

  return handled;
  }

If we back a GPA range with a fd, this will also be a 'backdoor'?

Quoted 62 lines · Alex Williamson

Quoted 58 lines · Kevin

Quoted 52 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:27 AM

Quoted 32 lines · Kevin

Quoted 27 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 18 lines

Quoted 9 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 9 lines

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

Quoted 6 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 5 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place?  Performance is important, but it's not an excuse to abandon designing interfaces between independent components.  Thanks,

Quoted 6 lines

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
   - KVMGT will expose a private control variable whether in-kernel delivery is required;

But in-kernel delivery is never *required*.  Wouldn't userspace want to deliver in-kernel any time it possibly could?

Quoted 6 lines

   - when the variable is true, KVMGT will register in-kernel MMIO emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
   - when the variable is false, KVMGT will not register anything. VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

Quoted 6 lines

No, that means the interface is entirely dependent on a backdoor through KVM.  Why can't userspace (QEMU) do something like register an MMIO region with KVM handled via a provided file descriptor and offset, couldn't KVM then call the file ops without a kernel exit?  Thanks,

Quoted 3 lines

Could you elaborate this thought? If it can achieve the purpose w/o a kernel exit definitely we can adapt to it. :-)

Quoted 15 lines

I only thought of it when replying to the last email and have been doing some research, but we already do quite a bit of synchronization through file descriptors.  The kvm-vfio pseudo device uses a group file descriptor to ensure a user has access to a group, allowing some degree of interaction between modules.  Eventfds and irqfds already make use of f_ops on file descriptors to poke data.  So, if KVM had information that an MMIO region was backed by a file descriptor for which it already has a reference via fdget() (and verified access rights and whatnot), then it ought to be a simple matter to get to f_ops->read/write knowing the base offset of that MMIO region.  Perhaps it could even simply use __vfs_read/write().  Then we've got a proper reference to the file descriptor for ownership purposes and we've transparently jumped across modules without any implicit knowledge of the other end.  Could it work? Thanks,

ioeventfd is a good example.
As i known, all access to the MMIO of IGD is trapped into kernel. Also, the pci config space is emulated by Qemu. Same the for VGA, which is emulated too. I guest interrupt also is emulated(This means we cannot benifit from VT-d pi). The most important is that KVMGT doesn't required hardware IOMMU. As we known, VFIO is for the direct device assignment, but most of thing for KVMGT are emulated, why we should use VFIO for it?

Quoted 78 lines · Jike Song

Quoted 76 lines · Alex Williamson

Quoted 53 lines · Kevin

Quoted 53 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:27 AM

Quoted 33 lines · Kevin

Quoted 28 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 19 lines

Quoted 10 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 8 lines

That sounds like a future optimization TBH. There's very strict layering between vfio and kvm. Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do. Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device. We can't simply take shortcuts to provide such direct access. Thanks,

Quoted 5 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 4 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place? Performance is important, but it's not an excuse to abandon designing interfaces between independent components. Thanks,

Quoted 5 lines

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
- KVMGT will expose a private control variable whether in-kernel
delivery is required;

But in-kernel delivery is never *required*. Wouldn't userspace want to deliver in-kernel any time it possibly could?

Quoted 6 lines

- when the variable is true, KVMGT will register in-kernel MMIO
emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
- when the variable is false, KVMGT will not register anything.
VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

Quoted 8 lines

Quoted 5 lines

No, that means the interface is entirely dependent on a backdoor through KVM. Why can't userspace (QEMU) do something like register an MMIO region with KVM handled via a provided file descriptor and offset, couldn't KVM then call the file ops without a kernel exit? Thanks,

Could you elaborate this thought? If it can achieve the purpose w/o a kernel exit definitely we can adapt to it. :-)

I only thought of it when replying to the last email and have been doing some research, but we already do quite a bit of synchronization through file descriptors. The kvm-vfio pseudo device uses a group file descriptor to ensure a user has access to a group, allowing some degree of interaction between modules. Eventfds and irqfds already make use of f_ops on file descriptors to poke data. So, if KVM had information that an MMIO region was backed by a file descriptor for which it already has a reference via fdget() (and verified access rights and whatnot), then it ought to be a simple matter to get to f_ops->read/write knowing the base offset of that MMIO region. Perhaps it could even simply use __vfs_read/write(). Then we've got a proper reference to the file descriptor for ownership purposes and we've transparently jumped across modules without any implicit knowledge of the other end. Could it work?

Quoted 2 lines

This is OK for KVMGT, from fops to vgpu device-model would always be simple. The only question is, how is KVM hypervisor supposed to get the fd on VM-exitings?

Hi Jike,

Sorry, I don't understand "on VM-exiting". KVM would hold a reference to the fd via fdget(), so the vfio device wouldn't be closed until the VM exits and KVM releases that reference.

Quoted 2 lines

copy-and-paste the current implementation of vcpu_mmio_write(), seems nothing but GPA and len are provided:

I presume that an MMIO region is already registered with a GPA and length, the additional information necessary would be a file descriptor and offset into the file descriptor for the base of the MMIO space.

Quoted 21 lines

(quoted diff) +0 -0

 	static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len,
 				   const void *v)
 	{
 		int handled = 0;
 		int n;
 
 		do {
 			n = min(len, 8);
 			if (!(vcpu->arch.apic &&
 			      !kvm_iodevice_write(vcpu, &vcpu->arch.apic->dev, addr, n, v))
 			    && kvm_io_bus_write(vcpu, KVM_MMIO_BUS, addr, n, v))
 				break;
 			handled += n;
 			addr += n;
 			len -= n;
 			v += n;
 		} while (len);
 
 		return handled;
 	}

Quoted 1 line

If we back a GPA range with a fd, this will also be a 'backdoor'?

KVM would simply be able to service the MMIO access using the provided fd and offset. It's not a back door because we will have created an API for KVM to have a file descriptor and offset registered (by userspace) to handle the access. Also, KVM does not know the file descriptor is handled by a VFIO device and VFIO doesn't know the read/write accesses is initiated by KVM. Seems like the question is whether we can fit something like that into the existing KVM MMIO bus/device handlers in-kernel. Thanks,

Alex

Quoted 26 lines · Neo Jia

Quoted 1 line · Kevin

Quoted 1 line

From: Alex Williamson [mailto:[email protected]]

Hi Alex, Kevin and Jike,

(Seems I shouldn't use attachment, resend it again to the list, patches are inline at the end)

Thanks for adding me to this technical discussion, a great opportunity for us to design together which can bring both Intel and NVIDIA vGPU solution to KVM platform.

Instead of directly jumping to the proposal that we have been working on recently for NVIDIA vGPU on KVM, I think it is better for me to put out couple quick comments / thoughts regarding the existing discussions on this thread as fundamentally I think we are solving the same problem, DMA, interrupt and MMIO.

Then we can look at what we have, hopefully we can reach some consensus soon.

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group. The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Quoted 3 lines

Infact to keep vfio-vgpu to be more generic, vgpu device creation and management can be centralized and done in vfio-vgpu. That also include adding to IOMMU group and VFIO group.

Is this really a good idea? The concept of a vgpu is not unique to vfio, we want vfio to be a driver for a vgpu, not an integral part of the lifecycle of a vgpu. That certainly doesn't exclude adding infrastructure to make lifecycle management of a vgpu more consistent between drivers, but it should be done independently of vfio. I'll go back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfio does not create the VF, that's done in coordination with the PF making use of some PCI infrastructure for consistency between drivers.

It seems like we need to take more advantage of the class and driver core support to perhaps setup a vgpu bus and class with vfio-vgpu just being a driver for those devices.

Quoted 27 lines · Neo Jia

Quoted 12 lines · Alex Williamson

Quoted 4 lines

1.1 Under per-physical device sysfs:
----------------------------------------------------------------------------------

vgpu_supported_types - RO, list the current supported virtual GPU types and its

Quoted 8 lines

VGPU_ID. VGPU_ID - a vGPU type identifier returned from reads of "vgpu_supported_types".

vgpu_create - WO, input syntax <VM_UUID:idx:VGPU_ID>, create a virtual gpu device on a target physical GPU. idx: virtual device index inside a VM
vgpu_destroy - WO, input syntax <VM_UUID:idx>, destroy a virtual gpu device on a target physical GPU

Quoted 14 lines

I've noted in previous discussions that we need to separate user policy from kernel policy here, the kernel policy should not require a "VM UUID".  A UUID simply represents a set of one or more devices and an index picks the device within the set.  Whether that UUID matches a VM or is independently used is up to the user policy when creating the device.

Personally I'd also prefer to get rid of the concept of indexes within a UUID set of devices and instead have each device be independent.  This seems to be an imposition on the nvidia implementation into the kernel interface design.

Quoted 6 lines

Hi Alex,

I agree with you that we should not put UUID concept into a kernel API. At this point (without any prototyping), I am thinking of using a list of virtual devices instead of UUID.

Hi Neo,

A UUID is a perfectly fine name, so long as we let it be just a UUID and not the UUID matching some specific use case.

Quoted 17 lines

Quoted 10 lines

Quoted 10 lines

int vgpu_map_virtual_bar
(
    uint64_t virt_bar_addr,
    uint64_t phys_bar_addr,
    uint32_t len,
    uint32_t flags
)

EXPORT_SYMBOL(vgpu_map_virtual_bar);

Quoted 7 lines

Per the implementation provided, this needs to be implemented in the vfio device driver, not in the iommu interface.  Finding the DMA mapping of the device and replacing it is wrong.  It should be remapped at the vfio device file interface using vm_ops.

Quoted 6 lines

So you are basically suggesting that we are going to take a mmap fault and within that fault handler, we will go into vendor driver to look up the "pre-registered" mapping and remap there.

Is my understanding correct?

Essentially, hopefully the vendor driver will have already registered the backing for the mmap prior to the fault, but either way could work. I think the key though is that you want to remap it onto the vma accessing the vfio device file, not scanning it out of an IOVA mapping that might be dynamic and doing a vma lookup based on the point in time mapping of the BAR.  The latter doesn't give me much confidence that mappings couldn't change while the former should be a one time fault.

In case it's not clear to folks at Intel, the purpose of this is that a vGPU may directly map a segment of the physical GPU MMIO space, but we may not know what segment that is at setup time, when QEMU does an mmap of the vfio device file descriptor.  The thought is that we can create an invalid mapping when QEMU calls mmap(), knowing that it won't be accessed until later, then we can fault in the real mmap on demand.  Do you need anything similar?

Quoted 14 lines

Quoted 14 lines

Quoted 7 lines

int vgpu_dma_do_translate(dma_addr_t *gfn_buffer, uint32_t count)

EXPORT_SYMBOL(vgpu_dma_do_translate);

Still a lot to be added and modified, such as supporting multiple VMs and multiple virtual devices, tracking the mapped / pinned region within VGPU IOMMU kernel driver, error handling, roll-back and locked memory size per user, etc.

Particularly, handling of mapping changes is completely missing.  This cannot be a point in time translation, the user is free to remap addresses whenever they wish and device translations need to be updated accordingly.

Quoted 2 lines

When you say "user", do you mean the QEMU?

vfio is a generic userspace driver interface, QEMU is a very, very important user of the interface, but not the only user.  So for this conversation, we're mostly talking about QEMU as the user, but we should be careful about assuming QEMU is the only user.

Quoted 8 lines

Here, whenever the DMA that
the guest driver is going to launch will be first pinned within VM, and then registered to QEMU, therefore the IOMMU memory listener, eventually the pages will be pinned by the GPU or DMA engine.

Since we are keeping the upper level code same, thinking about passthru case, where the GPU has already put the real IOVA into his PTEs, I don't know how QEMU can change that mapping without causing an IOMMU fault on a active DMA device.

For the virtual BAR mapping above, it's easy to imagine that mapping a BAR to a given address is at the guest discretion, it may be mapped and unmapped, it may be mapped to different addresses at different points in time, the guest BIOS may choose to map it at yet another address, etc. So if somehow we were trying to setup a mapping for peer-to-peer, there are lots of ways that IOVA could change.  But even with RAM, we can support memory hotplug in a VM.  What was once a DMA target may be removed or may now be backed by something else.  Chipset configuration on the emulated platform may change how guest physical memory appears and that might change between VM boots.

Currently with physical device assignment the memory listener watches for both maps and unmaps and updates the iotlb to match.  Just like real hardware doing these same sorts of things, we rely on the guest to stop using memory that's going to be moved as a DMA target prior to moving it.

Quoted 96 lines

Quoted 81 lines

Quoted 8 lines

4. Modules
==================================================================================

Two new modules are introduced: vfio_iommu_type1_vgpu.ko and vgpu.ko
vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU TYPE1 v1 and v2 interface.

Depending on how intrusive it is, this can possibly by done within the existing type1 driver.  Either that or we can split out common code for use by a separate module.

Quoted 57 lines

vgpu.ko                  - provide registration interface and virtual device VFIO access.

5. QEMU note
==================================================================================

To allow us focus on the VGPU kernel driver prototyping, we have introduced a new VFIO class - vgpu inside QEMU, so we don't have to change the existing vfio/pci.c file and use it as a reference for our implementation. It is basically just a quick c & p from vfio/pci.c to quickly meet our needs.

Once this proposal is finalized, we will move to vfio/pci.c instead of a new class, and probably the only thing required is to have a new way to discover the device.

6. Examples
==================================================================================

On this server, we have two NVIDIA M60 GPUs.

[root@cjia-vgx-kvm ~]# lspci -d 10de:13f2
86:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1) 87:00.0 VGA compatible controller: NVIDIA Corporation Device 13f2 (rev a1)
After nvidia.ko gets initialized, we can query the supported vGPU type by accessing the "vgpu_supported_types" like following:

[root@cjia-vgx-kvm ~]# cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types 11:GRID M60-0B
12:GRID M60-0Q
13:GRID M60-1B
14:GRID M60-1Q
15:GRID M60-2B
16:GRID M60-2Q
17:GRID M60-4Q
18:GRID M60-8Q

For example the VM_UUID is c0b26072-dd1b-4340-84fe-bf338c510818, and we would like to create "GRID M60-4Q" VM on it.

echo "c0b26072-dd1b-4340-84fe-bf338c510818:0:17" > /sys/bus/pci/devices/0000\:86\:00.0/vgpu_create

Note: the number 0 here is for vGPU device index. So far the change is not tested for multiple vgpu devices yet, but we will support it.

At this moment, if you query the "vgpu_supported_types" it will still show all supported virtual GPU types as no virtual GPU resource is committed yet.
Starting VM:

echo "c0b26072-dd1b-4340-84fe-bf338c510818" > /sys/class/vgpu/vgpu_start
then, the supported vGPU type query will return:

[root@cjia-vgx-kvm /home/cjia]$

Quoted 1 line

cat /sys/bus/pci/devices/0000\:86\:00.0/vgpu_supported_types

Quoted 11 lines

17:GRID M60-4Q

So vgpu_supported_config needs to be called whenever a new virtual device gets created as the underlying HW might limit the supported types if there are any existing VM runnings.

Then, VM gets shutdown, writes to /sys/class/vgpu/vgpu_shutdown will info the GPU driver vendor to clean up resource.

Eventually, those virtual GPUs can be removed by writing to vgpu_destroy under device sysfs.

Quoted 15 lines

I'd like to hear Intel's thoughts on this interface.  Are there different vgpu capacities or priority classes that would necessitate different types of vcpus on Intel?

I think there are some gaps in translating from named vgpu types to indexes here, along with my previous mention of the UUID/set oddity.
Does Intel have a need for start and shutdown interfaces?

Neo, wasn't there at some point information about how many of each type could be supported through these interfaces?  How does a user know their capacity limits?

Quoted 10 lines

Thanks for reminding me that, I think we probably forget to put that *important* information as the output of "vgpu_supported_types".

Regarding the capacity, we can provide the frame buffer size as part of the "vgpu_supported_types" output as well, I would imagine those will be eventually show up on the openstack management interface or virt-mgr.

Basically, yes there would be a separate col to show the number of instance you can create for each type of VGPU on a specific physical GPU.

Ok, Thanks,

Alex

Quoted 30 lines · Alex Williamson

Quoted 25 lines · Neo Jia

Quoted 1 line · Kevin

Quoted 1 line

From: Alex Williamson [mailto:[email protected]]

Hi Alex, Kevin and Jike,

(Seems I shouldn't use attachment, resend it again to the list, patches are inline at the end)

Thanks for adding me to this technical discussion, a great opportunity for us to design together which can bring both Intel and NVIDIA vGPU solution to KVM platform.

Instead of directly jumping to the proposal that we have been working on recently for NVIDIA vGPU on KVM, I think it is better for me to put out couple quick comments / thoughts regarding the existing discussions on this thread as fundamentally I think we are solving the same problem, DMA, interrupt and MMIO.
Then we can look at what we have, hopefully we can reach some consensus soon.

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Quoted 4 lines

Infact to keep vfio-vgpu to be more generic, vgpu device creation and management can be centralized and done in vfio-vgpu. That also include adding to IOMMU group and VFIO group.

Quoted 12 lines

Is this really a good idea?  The concept of a vgpu is not unique to vfio, we want vfio to be a driver for a vgpu, not an integral part of the lifecycle of a vgpu.  That certainly doesn't exclude adding infrastructure to make lifecycle management of a vgpu more consistent between drivers, but it should be done independently of vfio.  I'll go back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfio does not create the VF, that's done in coordination with the PF making use of some PCI infrastructure for consistency between drivers.
It seems like we need to take more advantage of the class and driver core support to perhaps setup a vgpu bus and class with vfio-vgpu just being a driver for those devices.

For device passthrough or SR-IOV model, PCI devices are created by PCI bus driver and from the probe routine each device is added in vfio group.

For vgpu, there should be a common module that create vgpu device, say vgpu module, add vgpu device to an IOMMU group and then add it to vfio group.  This module can handle management of vgpus. Advantage of keeping this module a separate module than doing device creation in vendor modules is to have generic interface for vgpu management, for example, files /sys/class/vgpu/vgpu_start and  /sys/class/vgpu/vgpu_shudown and vgpu driver registration interface.

In the patch, vgpu_dev.c + vgpu_sysfs.c form such vgpu module and vgpu_vfio.c is for VFIO interface. Each vgpu device should be added to vfio group, so vgpu_group_init() from vgpu_vfio.c should be called per device. In the vgpu module, vgpu devices are created on request, so vgpu_group_init() should be called explicitly for per vgpu device. That’s why had merged the 2 modules, vgpu + vgpu_vfio to form one vgpu module.  Vgpu_vfio would remain separate entity but merged with vgpu module.

Quoted 63 lines · Alex Williamson

Quoted 27 lines · Neo Jia

Quoted 12 lines · Alex Williamson

Quoted 4 lines

1.1 Under per-physical device sysfs:
----------------------------------------------------------------------------------

vgpu_supported_types - RO, list the current supported virtual GPU types and its

Quoted 8 lines

VGPU_ID. VGPU_ID - a vGPU type identifier returned from reads of "vgpu_supported_types".

vgpu_create - WO, input syntax <VM_UUID:idx:VGPU_ID>, create a virtual gpu device on a target physical GPU. idx: virtual device index inside a VM
vgpu_destroy - WO, input syntax <VM_UUID:idx>, destroy a virtual gpu device on a target physical GPU

Quoted 14 lines

I've noted in previous discussions that we need to separate user policy from kernel policy here, the kernel policy should not require a "VM UUID".  A UUID simply represents a set of one or more devices and an index picks the device within the set.  Whether that UUID matches a VM or is independently used is up to the user policy when creating the device.

Personally I'd also prefer to get rid of the concept of indexes within a UUID set of devices and instead have each device be independent.  This seems to be an imposition on the nvidia implementation into the kernel interface design.

Quoted 6 lines

Hi Alex,

I agree with you that we should not put UUID concept into a kernel API. At this point (without any prototyping), I am thinking of using a list of virtual devices instead of UUID.

Hi Neo,

A UUID is a perfectly fine name, so long as we let it be just a UUID and not the UUID matching some specific use case.

Quoted 17 lines

Quoted 10 lines

Quoted 10 lines

int vgpu_map_virtual_bar
(
    uint64_t virt_bar_addr,
    uint64_t phys_bar_addr,
    uint32_t len,
    uint32_t flags
)

EXPORT_SYMBOL(vgpu_map_virtual_bar);

Quoted 7 lines

Per the implementation provided, this needs to be implemented in the vfio device driver, not in the iommu interface.  Finding the DMA mapping of the device and replacing it is wrong.  It should be remapped at the vfio device file interface using vm_ops.

Quoted 6 lines

So you are basically suggesting that we are going to take a mmap fault and within that fault handler, we will go into vendor driver to look up the "pre-registered" mapping and remap there.

Is my understanding correct?

Quoted 8 lines

Essentially, hopefully the vendor driver will have already registered the backing for the mmap prior to the fault, but either way could work. I think the key though is that you want to remap it onto the vma accessing the vfio device file, not scanning it out of an IOVA mapping that might be dynamic and doing a vma lookup based on the point in time mapping of the BAR.  The latter doesn't give me much confidence that mappings couldn't change while the former should be a one time fault.

Hi Alex,

The fact is that the vendor driver can only prevent such mmap fault by looking up the <iova, hva> mapping table that we have saved from IOMMU memory listerner

when the guest region gets programmed. Also, like you have mentioned below, such mapping between iova and hva shouldn't be changed as long as the SBIOS and guest OS are done with their job.

Yes, you are right it is one time fault, but the gpu work is heavily pipelined.

Probably we should just limit this interface to guest MMIO region and we can have some crosscheck between the VFIO driver who has monitored the config spcae access to make sure nothing getting moved around?

Quoted 22 lines

In case it's not clear to folks at Intel, the purpose of this is that a vGPU may directly map a segment of the physical GPU MMIO space, but we may not know what segment that is at setup time, when QEMU does an mmap of the vfio device file descriptor.  The thought is that we can create an invalid mapping when QEMU calls mmap(), knowing that it won't be accessed until later, then we can fault in the real mmap on demand.  Do you need anything similar?

Quoted 14 lines

Quoted 14 lines

Quoted 7 lines

int vgpu_dma_do_translate(dma_addr_t *gfn_buffer, uint32_t count)

EXPORT_SYMBOL(vgpu_dma_do_translate);

Still a lot to be added and modified, such as supporting multiple VMs and multiple virtual devices, tracking the mapped / pinned region within VGPU IOMMU kernel driver, error handling, roll-back and locked memory size per user, etc.

Particularly, handling of mapping changes is completely missing.  This cannot be a point in time translation, the user is free to remap addresses whenever they wish and device translations need to be updated accordingly.

Quoted 8 lines

Quoted 2 lines

When you say "user", do you mean the QEMU?

vfio is a generic userspace driver interface, QEMU is a very, very important user of the interface, but not the only user.  So for this conversation, we're mostly talking about QEMU as the user, but we should be careful about assuming QEMU is the only user.

Understood. I have to say that our focus at this moment is to support QEMU and KVM, but I know VFIO interface is much more than that, and that is why I think it is right to leverage this framework so we can together explore future use case in the userland.

Quoted 17 lines

Quoted 8 lines

Here, whenever the DMA that
the guest driver is going to launch will be first pinned within VM, and then registered to QEMU, therefore the IOMMU memory listener, eventually the pages will be pinned by the GPU or DMA engine.

Since we are keeping the upper level code same, thinking about passthru case, where the GPU has already put the real IOVA into his PTEs, I don't know how QEMU can change that mapping without causing an IOMMU fault on a active DMA device.

For the virtual BAR mapping above, it's easy to imagine that mapping a BAR to a given address is at the guest discretion, it may be mapped and unmapped, it may be mapped to different addresses at different points in time, the guest BIOS may choose to map it at yet another address, etc. So if somehow we were trying to setup a mapping for peer-to-peer, there are lots of ways that IOVA could change.  But even with RAM, we can support memory hotplug in a VM.  What was once a DMA target may be removed or may now be backed by something else.  Chipset configuration

Quoted 8 lines

on the emulated platform may change how guest physical memory appears and that might change between VM boots.

Currently with physical device assignment the memory listener watches for both maps and unmaps and updates the iotlb to match.  Just like real hardware doing these same sorts of things, we rely on the guest to stop using memory that's going to be moved as a DMA target prior to moving it.

Right,  you can only do that when the device is quiescent.

As long as this will be notified to the guest, I think we should be able to support it although the real implementation will depend on how the device gets into quiescent state.

This is definitely a very interesting feature we should explore, but I hope we probably can first focus on the most basic functionality.

Quoted 81 lines · Alex Williamson

Quoted 77 lines · Jike Song

Quoted 62 lines · Alex Williamson

Quoted 58 lines · Kevin

Quoted 52 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:27 AM

Quoted 32 lines · Kevin

Quoted 27 lines

From: Alex Williamson [mailto:[email protected]]
Sent: Wednesday, January 27, 2016 6:08 AM

Quoted 18 lines

Quoted 9 lines

Quoted 9 lines

Quoted 1 line

Today KVMGT (not using VFIO yet) registers I/O emulation callbacks to KVM, so VM MMIO access will be forwarded to KVMGT directly for emulation in kernel. If we reuse above R/W flags, the whole emulation path would be unnecessarily long with obvious performance impact. We either need a new flag here to indicate in-kernel emulation (bias from passthrough support), or just hide the region alternatively (let KVMGT to handle I/O emulation itself like today).

Quoted 9 lines

That sounds like a future optimization TBH.  There's very strict layering between vfio and kvm.  Physical device assignment could make use of it as well, avoiding a round trip through userspace when an ioread/write would do.  Userspace also needs to orchestrate those kinds of accelerators, there might be cases where userspace wants to see those transactions for debugging or manipulating the device.  We can't simply take shortcuts to provide such direct access.  Thanks,

Quoted 6 lines

But we have to balance such debugging flexibility and acceptable performance. To me the latter one is more important otherwise there'd be no real usage around this technique, while for debugging there are other alternative (e.g. ftrace) Consider some extreme case with 100k traps/second and then see how much impact a 2-3x longer emulation path can bring...

Quoted 5 lines

Are you jumping to the conclusion that it cannot be done with proper layering in place?  Performance is important, but it's not an excuse to abandon designing interfaces between independent components.  Thanks,

Quoted 6 lines

Two are not controversial. My point is to remove unnecessary long trip as possible. After another thought, yes we can reuse existing read/write flags:
  - KVMGT will expose a private control variable whether in-kernel delivery is required;

But in-kernel delivery is never *required*.  Wouldn't userspace want to deliver in-kernel any time it possibly could?

Quoted 6 lines

  - when the variable is true, KVMGT will register in-kernel MMIO emulation callbacks then VM MMIO request will be delivered to KVMGT directly;
  - when the variable is false, KVMGT will not register anything. VM MMIO request will then be delivered to Qemu and then ioread/write will be used to finally reach KVMGT emulation logic;

Quoted 6 lines

No, that means the interface is entirely dependent on a backdoor through KVM.  Why can't userspace (QEMU) do something like register an MMIO region with KVM handled via a provided file descriptor and offset, couldn't KVM then call the file ops without a kernel exit?  Thanks,

Quoted 3 lines

Could you elaborate this thought? If it can achieve the purpose w/o a kernel exit definitely we can adapt to it. :-)

Quoted 14 lines

I only thought of it when replying to the last email and have been doing some research, but we already do quite a bit of synchronization through file descriptors.  The kvm-vfio pseudo device uses a group file descriptor to ensure a user has access to a group, allowing some degree of interaction between modules.  Eventfds and irqfds already make use of f_ops on file descriptors to poke data.  So, if KVM had information that an MMIO region was backed by a file descriptor for which it already has a reference via fdget() (and verified access rights and whatnot), then it ought to be a simple matter to get to f_ops->read/write knowing the base offset of that MMIO region.  Perhaps it could even simply use __vfs_read/write().  Then we've got a proper reference to the file descriptor for ownership purposes and we've transparently jumped across modules without any implicit knowledge of the other end.  Could it work?

Quoted 3 lines

This is OK for KVMGT, from fops to vgpu device-model would always be simple. The only question is, how is KVM hypervisor supposed to get the fd on VM-exitings?

Quoted 7 lines

Hi Jike,

Sorry, I don't understand "on VM-exiting".  KVM would hold a reference to the fd via fdget(), so the vfio device wouldn't be closed until the VM exits and KVM releases that reference.

Sorry for my bad English, I meant VMEXIT, from non-root to kvm hypervisor.

Quoted 29 lines

Quoted 2 lines

copy-and-paste the current implementation of vcpu_mmio_write(), seems nothing but GPA and len are provided:

I presume that an MMIO region is already registered with a GPA and length, the additional information necessary would be a file descriptor and offset into the file descriptor for the base of the MMIO space.

Quoted 14 lines

  static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, const void *v)
  {
  int handled = 0;
  int n;

  do {
  n = min(len, 8);
  if (!(vcpu->arch.apic &&
        !kvm_iodevice_write(vcpu, &vcpu->arch.apic->dev, addr, n, v)) && kvm_io_bus_write(vcpu, KVM_MMIO_BUS, addr, n, v)) break;
  handled += n;
  addr += n;

Quoted 8 lines

  len -= n;
  v += n;
  } while (len);

  return handled;
  }

If we back a GPA range with a fd, this will also be a 'backdoor'?

Quoted 10 lines

KVM would simply be able to service the MMIO access using the provided fd and offset.  It's not a back door because we will have created an API for KVM to have a file descriptor and offset registered (by userspace) to handle the access.  Also, KVM does not know the file descriptor is handled by a VFIO device and VFIO doesn't know the read/write accesses is initiated by KVM.  Seems like the question is whether we can fit something like that into the existing KVM MMIO bus/device handlers in-kernel.  Thanks,

Had a look at eventfd, I would say yes, technically we are able to achieve the goal: introduce a fd, with fop->{read|write} defined in KVM, call into vgpu device-model, also an iodev registered for a MMIO GPA range to invoke the fop->{read|write}.  I just didn't understand why userspace can't register an iodev via API directly.

Besides, this doesn't necessarily require another thread, right? I guess it can be within the VCPU thread?

And this brought another question: except the vfio bus drvier and iommu backend (and the page_track ulitiy used for guest memory write-protection), is it KVMGT allowed to call into kvm.ko (or modify)? Though we are becoming less and less willing to do that with VFIO, it's still better to know that before going wrong.

Quoted 41 lines · Kirti Wankhede

Quoted 39 lines · Alex Williamson

Quoted 26 lines · Neo Jia

Quoted 1 line · Kevin

Quoted 1 line

From: Alex Williamson [mailto:[email protected]]

Hi Alex, Kevin and Jike,

(Seems I shouldn't use attachment, resend it again to the list, patches are inline at the end)

Thanks for adding me to this technical discussion, a great opportunity for us to design together which can bring both Intel and NVIDIA vGPU solution to KVM platform.

Instead of directly jumping to the proposal that we have been working on recently for NVIDIA vGPU on KVM, I think it is better for me to put out couple quick comments / thoughts regarding the existing discussions on this thread as fundamentally I think we are solving the same problem, DMA, interrupt and MMIO.

Then we can look at what we have, hopefully we can reach some consensus soon.

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group. The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Quoted 3 lines

Infact to keep vfio-vgpu to be more generic, vgpu device creation and management can be centralized and done in vfio-vgpu. That also include adding to IOMMU group and VFIO group.

Is this really a good idea? The concept of a vgpu is not unique to vfio, we want vfio to be a driver for a vgpu, not an integral part of the lifecycle of a vgpu. That certainly doesn't exclude adding infrastructure to make lifecycle management of a vgpu more consistent between drivers, but it should be done independently of vfio. I'll go back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfio does not create the VF, that's done in coordination with the PF making use of some PCI infrastructure for consistency between drivers.

Quoted 6 lines

Quoted 3 lines

It seems like we need to take more advantage of the class and driver core support to perhaps setup a vgpu bus and class with vfio-vgpu just being a driver for those devices.

For device passthrough or SR-IOV model, PCI devices are created by PCI bus driver and from the probe routine each device is added in vfio group.

An SR-IOV VF is created by the PF driver using standard interfaces provided by the PCI core. The IOMMU group for a VF is added by the IOMMU driver when the device is created on the pci_bus_type. The probe routine of the vfio bus driver (vfio-pci) is what adds the device into the vfio group.

Quoted 7 lines

For vgpu, there should be a common module that create vgpu device, say vgpu module, add vgpu device to an IOMMU group and then add it to vfio group. This module can handle management of vgpus. Advantage of keeping this module a separate module than doing device creation in vendor modules is to have generic interface for vgpu management, for example, files /sys/class/vgpu/vgpu_start and /sys/class/vgpu/vgpu_shudown and vgpu driver registration interface.

But you're suggesting something very different from the SR-IOV model. If we wanted to mimic that model, the GPU specific driver should create the vgpu using services provided by a common interface. For instance i915 could call a new vgpu_device_create() which creates the device, adds it to the vgpu class, etc. That vgpu device should not be assumed to be used with vfio though, that should happen via a separate probe using a vfio-vgpu driver. It's that vfio bus driver that will add the device to a vfio group.

Quoted 8 lines

In the patch, vgpu_dev.c + vgpu_sysfs.c form such vgpu module and vgpu_vfio.c is for VFIO interface. Each vgpu device should be added to vfio group, so vgpu_group_init() from vgpu_vfio.c should be called per device. In the vgpu module, vgpu devices are created on request, so vgpu_group_init() should be called explicitly for per vgpu device.
That’s why had merged the 2 modules, vgpu + vgpu_vfio to form one vgpu
module. Vgpu_vfio would remain separate entity but merged with vgpu module.

I disagree with this design, creation of a vgpu necessarily involves the GPU driver and should not be tied to use of the vgpu with vfio. vfio should be a driver for the device, maybe eventually not the only driver for the device. Thanks,

Alex

Quoted 72 lines · Neo Jia

Quoted 63 lines · Alex Williamson

Quoted 27 lines · Neo Jia

Quoted 12 lines · Alex Williamson

Quoted 4 lines

1.1 Under per-physical device sysfs:
----------------------------------------------------------------------------------

vgpu_supported_types - RO, list the current supported virtual GPU types and its

Quoted 8 lines

VGPU_ID. VGPU_ID - a vGPU type identifier returned from reads of "vgpu_supported_types".

vgpu_create - WO, input syntax <VM_UUID:idx:VGPU_ID>, create a virtual gpu device on a target physical GPU. idx: virtual device index inside a VM
vgpu_destroy - WO, input syntax <VM_UUID:idx>, destroy a virtual gpu device on a target physical GPU

Quoted 14 lines

I've noted in previous discussions that we need to separate user policy from kernel policy here, the kernel policy should not require a "VM UUID".  A UUID simply represents a set of one or more devices and an index picks the device within the set.  Whether that UUID matches a VM or is independently used is up to the user policy when creating the device.

Personally I'd also prefer to get rid of the concept of indexes within a UUID set of devices and instead have each device be independent.  This seems to be an imposition on the nvidia implementation into the kernel interface design.

Quoted 6 lines

Hi Alex,

I agree with you that we should not put UUID concept into a kernel API. At this point (without any prototyping), I am thinking of using a list of virtual devices instead of UUID.

Hi Neo,

A UUID is a perfectly fine name, so long as we let it be just a UUID and not the UUID matching some specific use case.

Quoted 17 lines

Quoted 10 lines

Quoted 10 lines

int vgpu_map_virtual_bar
(
    uint64_t virt_bar_addr,
    uint64_t phys_bar_addr,
    uint32_t len,
    uint32_t flags
)

EXPORT_SYMBOL(vgpu_map_virtual_bar);

Quoted 7 lines

Per the implementation provided, this needs to be implemented in the vfio device driver, not in the iommu interface.  Finding the DMA mapping of the device and replacing it is wrong.  It should be remapped at the vfio device file interface using vm_ops.

Quoted 6 lines

So you are basically suggesting that we are going to take a mmap fault and within that fault handler, we will go into vendor driver to look up the "pre-registered" mapping and remap there.

Is my understanding correct?

Quoted 8 lines

Essentially, hopefully the vendor driver will have already registered the backing for the mmap prior to the fault, but either way could work. I think the key though is that you want to remap it onto the vma accessing the vfio device file, not scanning it out of an IOVA mapping that might be dynamic and doing a vma lookup based on the point in time mapping of the BAR.  The latter doesn't give me much confidence that mappings couldn't change while the former should be a one time fault.

Quoted 5 lines

Hi Alex,

The fact is that the vendor driver can only prevent such mmap fault by looking up the <iova, hva> mapping table that we have saved from IOMMU memory listerner

Why do we need to prevent the fault?  We need to handle the fault when it occurs.

Quoted 3 lines

when the guest region gets programmed. Also, like you have mentioned below, such mapping between iova and hva shouldn't be changed as long as the SBIOS and guest OS are done with their job.

But you don't know they're done with their job.

Quoted 1 line

Yes, you are right it is one time fault, but the gpu work is heavily pipelined.

Why does that matter?  We're talking about the first time the VM accesses the range of the BAR that will be direct mapped to the physical GPU.  This isn't going to happen in the middle of a benchmark, it's going to happen during driver initialization in the guest.

Quoted 3 lines

Probably we should just limit this interface to guest MMIO region and we can have some crosscheck between the VFIO driver who has monitored the config spcae access to make sure nothing getting moved around?

No, the solution for the bar is very clear, map on fault to the vma accessing the mmap and be done with it for the remainder of this instance of the VM.

Quoted 62 lines

Quoted 22 lines

In case it's not clear to folks at Intel, the purpose of this is that a vGPU may directly map a segment of the physical GPU MMIO space, but we may not know what segment that is at setup time, when QEMU does an mmap of the vfio device file descriptor.  The thought is that we can create an invalid mapping when QEMU calls mmap(), knowing that it won't be accessed until later, then we can fault in the real mmap on demand.  Do you need anything similar?

Quoted 14 lines

Quoted 14 lines

Quoted 7 lines

int vgpu_dma_do_translate(dma_addr_t *gfn_buffer, uint32_t count)

EXPORT_SYMBOL(vgpu_dma_do_translate);

Still a lot to be added and modified, such as supporting multiple VMs and multiple virtual devices, tracking the mapped / pinned region within VGPU IOMMU kernel driver, error handling, roll-back and locked memory size per user, etc.

Particularly, handling of mapping changes is completely missing.  This cannot be a point in time translation, the user is free to remap addresses whenever they wish and device translations need to be updated accordingly.

Quoted 8 lines

Quoted 2 lines

When you say "user", do you mean the QEMU?

vfio is a generic userspace driver interface, QEMU is a very, very important user of the interface, but not the only user.  So for this conversation, we're mostly talking about QEMU as the user, but we should be careful about assuming QEMU is the only user.

Understood. I have to say that our focus at this moment is to support QEMU and KVM, but I know VFIO interface is much more than that, and that is why I think it is right to leverage this framework so we can together explore future use case in the userland.

Quoted 17 lines

Quoted 8 lines

Here, whenever the DMA that
the guest driver is going to launch will be first pinned within VM, and then registered to QEMU, therefore the IOMMU memory listener, eventually the pages will be pinned by the GPU or DMA engine.

Since we are keeping the upper level code same, thinking about passthru case, where the GPU has already put the real IOVA into his PTEs, I don't know how QEMU can change that mapping without causing an IOMMU fault on a active DMA device.

For the virtual BAR mapping above, it's easy to imagine that mapping a BAR to a given address is at the guest discretion, it may be mapped and unmapped, it may be mapped to different addresses at different points in time, the guest BIOS may choose to map it at yet another address, etc. So if somehow we were trying to setup a mapping for peer-to-peer, there are lots of ways that IOVA could change.  But even with RAM, we can support memory hotplug in a VM.  What was once a DMA target may be removed or may now be backed by something else.  Chipset configuration

Quoted 8 lines

on the emulated platform may change how guest physical memory appears and that might change between VM boots.

Currently with physical device assignment the memory listener watches for both maps and unmaps and updates the iotlb to match.  Just like real hardware doing these same sorts of things, we rely on the guest to stop using memory that's going to be moved as a DMA target prior to moving it.

Quoted 9 lines

Right,  you can only do that when the device is quiescent.

As long as this will be notified to the guest, I think we should be able to support it although the real implementation will depend on how the device gets into quiescent state.

This is definitely a very interesting feature we should explore, but I hope we probably can first focus on the most basic functionality.

If we only do a point-in-time translation and assume it never changes, that's good enough for a proof of concept, but it's not a complete solution.  I think this is  practical problem, not just an academic problem.  There needs to be a mechanism mappings to be invalidated based on VM memory changes.  Thanks,

Alex

Quoted 62 lines · Alex Williamson

Quoted 44 lines · Kirti Wankhede

Quoted 30 lines · Alex Williamson

Quoted 25 lines · Neo Jia

Quoted 1 line · Kevin

Quoted 1 line

From: Alex Williamson [mailto:[email protected]]

Hi Alex, Kevin and Jike,

(Seems I shouldn't use attachment, resend it again to the list, patches are inline at the end)

Thanks for adding me to this technical discussion, a great opportunity for us to design together which can bring both Intel and NVIDIA vGPU solution to KVM platform.

Instead of directly jumping to the proposal that we have been working on recently for NVIDIA vGPU on KVM, I think it is better for me to put out couple quick comments / thoughts regarding the existing discussions on this thread as fundamentally I think we are solving the same problem, DMA, interrupt and MMIO.
Then we can look at what we have, hopefully we can reach some consensus soon.

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group.  The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Quoted 4 lines

Infact to keep vfio-vgpu to be more generic, vgpu device creation and management can be centralized and done in vfio-vgpu. That also include adding to IOMMU group and VFIO group.

Quoted 12 lines

Is this really a good idea?  The concept of a vgpu is not unique to vfio, we want vfio to be a driver for a vgpu, not an integral part of the lifecycle of a vgpu.  That certainly doesn't exclude adding infrastructure to make lifecycle management of a vgpu more consistent between drivers, but it should be done independently of vfio.  I'll go back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfio does not create the VF, that's done in coordination with the PF making use of some PCI infrastructure for consistency between drivers.
It seems like we need to take more advantage of the class and driver core support to perhaps setup a vgpu bus and class with vfio-vgpu just being a driver for those devices.

Quoted 3 lines

For device passthrough or SR-IOV model, PCI devices are created by PCI bus driver and from the probe routine each device is added in vfio group.

An SR-IOV VF is created by the PF driver using standard interfaces provided by the PCI core.  The IOMMU group for a VF is added by the IOMMU driver when the device is created on the pci_bus_type.  The probe routine of the vfio bus driver (vfio-pci) is what adds the device into the vfio group.

Quoted 7 lines

For vgpu, there should be a common module that create vgpu device, say vgpu module, add vgpu device to an IOMMU group and then add it to vfio group.  This module can handle management of vgpus. Advantage of keeping this module a separate module than doing device creation in vendor modules is to have generic interface for vgpu management, for example, files /sys/class/vgpu/vgpu_start and  /sys/class/vgpu/vgpu_shudown and vgpu driver registration interface.

Quoted 10 lines

But you're suggesting something very different from the SR-IOV model. If we wanted to mimic that model, the GPU specific driver should create the vgpu using services provided by a common interface.  For instance i915 could call a new vgpu_device_create() which creates the device, adds it to the vgpu class, etc.  That vgpu device should not be assumed to be used with vfio though, that should happen via a separate probe using a vfio-vgpu driver.  It's that vfio bus driver that will add the device to a vfio group.

In that case vgpu driver should provide a driver registration interface to register vfio-vgpu driver.

struct vgpu_driver {
  const char *name;
  int (*probe) (struct vgpu_device *vdev);
  void (*remove) (struct vgpu_device *vdev);
}

int vgpu_register_driver(struct vgpu_driver *driver)
{
...
}
EXPORT_SYMBOL(vgpu_register_driver);

int vgpu_unregister_driver(struct vgpu_driver *driver)
{
...
}
EXPORT_SYMBOL(vgpu_unregister_driver);

vfio-vgpu driver registers to vgpu driver. Then from vgpu_device_create(), after creating the device it calls vgpu_driver->probe(vgpu_device) and vfio-vgpu driver adds the device to vfio group.

+--------------+    vgpu_register_driver()+---------------+
>     __init() +------------------------->+               |
>              |                          |               |
>              +<-------------------------+    vgpu.ko    |
> vfio_vgpu.ko |   probe()/remove()       |               |
>              |                +---------+               +---------+
+--------------+                |         +-------+-------+         |
                                 |                 ^                 |
                                 | callback        |                 |
                                 |         +-------+--------+        |
                                 |         |vgpu_register_device()   |
                                 |         |                |        |
                                 +---^-----+-----+    +-----+------+-+
                                     | nvidia.ko |    |  i915.ko   |
                                     |           |    |            |
                                     +-----------+    +------------+

Is my understanding correct?

Quoted 118 lines · Kirti Wankhede

Quoted 63 lines · Alex Williamson

Quoted 41 lines · Kirti Wankhede

Quoted 39 lines · Alex Williamson

Quoted 26 lines · Neo Jia

Quoted 1 line · Kevin

Quoted 1 line

From: Alex Williamson [mailto:[email protected]]

Hi Alex, Kevin and Jike,

(Seems I shouldn't use attachment, resend it again to the list, patches are inline at the end)

Thanks for adding me to this technical discussion, a great opportunity for us to design together which can bring both Intel and NVIDIA vGPU solution to KVM platform.

Instead of directly jumping to the proposal that we have been working on recently for NVIDIA vGPU on KVM, I think it is better for me to put out couple quick comments / thoughts regarding the existing discussions on this thread as fundamentally I think we are solving the same problem, DMA, interrupt and MMIO.

Then we can look at what we have, hopefully we can reach some consensus soon.

Quoted 6 lines

Yes, and since you're creating and destroying the vgpu here, this is where I'd expect a struct device to be created and added to an IOMMU group. The lifecycle management should really include links between the vGPU and physical GPU, which would be much, much easier to do with struct devices create here rather than at the point where we start doing vfio "stuff".

Quoted 3 lines

Infact to keep vfio-vgpu to be more generic, vgpu device creation and management can be centralized and done in vfio-vgpu. That also include adding to IOMMU group and VFIO group.

Is this really a good idea? The concept of a vgpu is not unique to vfio, we want vfio to be a driver for a vgpu, not an integral part of the lifecycle of a vgpu. That certainly doesn't exclude adding infrastructure to make lifecycle management of a vgpu more consistent between drivers, but it should be done independently of vfio. I'll go back to the SR-IOV model, vfio is often used with SR-IOV VFs, but vfio does not create the VF, that's done in coordination with the PF making use of some PCI infrastructure for consistency between drivers.

Quoted 6 lines

Quoted 3 lines

It seems like we need to take more advantage of the class and driver core support to perhaps setup a vgpu bus and class with vfio-vgpu just being a driver for those devices.

For device passthrough or SR-IOV model, PCI devices are created by PCI bus driver and from the probe routine each device is added in vfio group.

An SR-IOV VF is created by the PF driver using standard interfaces provided by the PCI core. The IOMMU group for a VF is added by the IOMMU driver when the device is created on the pci_bus_type. The probe routine of the vfio bus driver (vfio-pci) is what adds the device into the vfio group.

Quoted 7 lines

For vgpu, there should be a common module that create vgpu device, say vgpu module, add vgpu device to an IOMMU group and then add it to vfio group. This module can handle management of vgpus. Advantage of keeping this module a separate module than doing device creation in vendor modules is to have generic interface for vgpu management, for example, files /sys/class/vgpu/vgpu_start and /sys/class/vgpu/vgpu_shudown and vgpu driver registration interface.

Quoted 9 lines

But you're suggesting something very different from the SR-IOV model. If we wanted to mimic that model, the GPU specific driver should create the vgpu using services provided by a common interface. For instance i915 could call a new vgpu_device_create() which creates the device, adds it to the vgpu class, etc. That vgpu device should not be assumed to be used with vfio though, that should happen via a separate probe using a vfio-vgpu driver. It's that vfio bus driver that will add the device to a vfio group.

In that case vgpu driver should provide a driver registration interface to register vfio-vgpu driver.

struct vgpu_driver {
const char *name;
int (*probe) (struct vgpu_device *vdev); void (*remove) (struct vgpu_device *vdev);
}

int vgpu_register_driver(struct vgpu_driver *driver) {
...
}
EXPORT_SYMBOL(vgpu_register_driver);

int vgpu_unregister_driver(struct vgpu_driver *driver) {
...
}
EXPORT_SYMBOL(vgpu_unregister_driver);

vfio-vgpu driver registers to vgpu driver. Then from vgpu_device_create(), after creating the device it calls vgpu_driver->probe(vgpu_device) and vfio-vgpu driver adds the device to vfio group.

+--------------+    vgpu_register_driver()+---------------+
>      __init() +------------------------->+               |
>               |                          |               |
>               +<-------------------------+    vgpu.ko    |
> vfio_vgpu.ko |   probe()/remove()       |               |
>               |                +---------+               +---------+
+--------------+                |         +-------+-------+         |
                                  |                 ^                 |
                                  | callback        |                 |
                                  |         +-------+--------+        |
                                  |         |vgpu_register_device()   |
                                  |         |                |        |
                                  +---^-----+-----+    +-----+------+-+
                                      | nvidia.ko |    |  i915.ko   |
                                      |           |    |            |
                                      +-----------+    +------------+

Quoted 1 line

Is my understanding correct?

We have an entire driver core subsystem in Linux for the purpose of matching devices to drivers, I don't think we should be re-inventing that. That's why I'm suggesting that we should have infrastructure which facilitates GPU drivers to create vGPU devices in a common way, perhaps even placing the devices on a virtual vgpu bus, and then allow a vfio-vgpu driver to register as a driver for devices of that bus/class and use the existing driver callbacks. Thanks,

Alex

On Wed, 2016-01-27 at 13:43 +0800, Jike Song wrote:

Quoted 5 lines

Had a look at eventfd, I would say yes, technically we are able to achieve the goal: introduce a fd, with fop->{read|write} defined in KVM, call into vgpu device-model, also an iodev registered for a MMIO GPA range to invoke the fop->{read|write}. I just didn't understand why userspace can't register an iodev via API directly.

Please elaborate on how it would work via iodev.

Quoted 2 lines

Besides, this doesn't necessarily require another thread, right? I guess it can be within the VCPU thread?

I would think so too, the vcpu is blocked on the MMIO access, we should be able to service it in that context. I hope.

Quoted 5 lines

And this brought another question: except the vfio bus drvier and iommu backend (and the page_track ulitiy used for guest memory write-protection), is it KVMGT allowed to call into kvm.ko (or modify)? Though we are becoming less and less willing to do that with VFIO, it's still better to know that before going wrong.

kvm and vfio are separate modules, for the most part, they know nothing about each other and have no hard dependencies between them. We do have various accelerations we can use to avoid paths through userspace, but these are all via APIs that are agnostic of the party on the other end. For example, vfio signals interrups through eventfds and has no concept of whether that eventfd terminates in userspace or into an irqfd in KVM. vfio supports direct access to device MMIO regions via mmaps, but vfio has no idea if that mmap gets directly mapped into a VM address space. Even with posted interrupts, we've introduced an irq bypass manager allowing interrupt producers and consumers to register independently to form a connection without directly knowing anything about the other module. That sort or proper software layering needs to continue. It would be wrong for a vfio bus driver to assume KVM is the user and directly call into KVM interfaces. Thanks,

Alex

Quoted 1 line · Alex Williamson

On Wed, 2016-01-27 at 13:43 +0800, Jike Song wrote:

{snip}

Quoted 8 lines

Quoted 5 lines

Had a look at eventfd, I would say yes, technically we are able to achieve the goal: introduce a fd, with fop->{read|write} defined in KVM, call into vgpu device-model, also an iodev registered for a MMIO GPA range to invoke the fop->{read|write}.  I just didn't understand why userspace can't register an iodev via API directly.

Please elaborate on how it would work via iodev.

QEMU forwards BAR0 write to the bus driver, in the bus driver, if found that MEM bit is enabled, register an iodev to KVM: with an ops:

  const struct kvm_io_device_ops trap_mmio_ops = {
  .read = kvmgt_guest_mmio_read,
  .write = kvmgt_guest_mmio_write,
  };

I may not be able to illustrated it clearly with descriptions but this should not be a problem, thanks to your explanation, I can understand and adopt it for KVMGT.

Quoted 6 lines

Quoted 2 lines

Besides, this doesn't necessarily require another thread, right? I guess it can be within the VCPU thread?

I would think so too, the vcpu is blocked on the MMIO access, we should be able to service it in that context.  I hope.

Thanks for confirmation.

Quoted 5 lines

Quoted 5 lines

And this brought another question: except the vfio bus drvier and iommu backend (and the page_track ulitiy used for guest memory write-protection), is it KVMGT allowed to call into kvm.ko (or modify)? Though we are becoming less and less willing to do that with VFIO, it's still better to know that before going wrong.

Quoted 16 lines

kvm and vfio are separate modules, for the most part, they know nothing about each other and have no hard dependencies between them.  We do have various accelerations we can use to avoid paths through userspace, but these are all via APIs that are agnostic of the party on the other end. For example, vfio signals interrups through eventfds and has no concept of whether that eventfd terminates in userspace or into an irqfd in KVM. vfio supports direct access to device MMIO regions via mmaps, but vfio has no idea if that mmap gets directly mapped into a VM address space. Even with posted interrupts, we've introduced an irq bypass manager allowing interrupt producers and consumers to register independently to form a connection without directly knowing anything about the other module.  That sort or proper software layering needs to continue.  It would be wrong for a vfio bus driver to assume KVM is the user and directly call into KVM interfaces.  Thanks,

I understand and agree with your point, it's bad if the bus driver assume KVM is the user and/or call into KVM interfaces.

However, the vgpu device-model, in intel case also a part of i915 driver, will always need to call some hypervisor-specific interfaces.

For example, when a guest gfx driver submit GPU commands, the device-model may want to scan it for security or whatever-else purpose:

  - get a GPA (from GPU page tables)
  - want to read 16 bytes from that GPA
  - call hypervisor-specific read_gpa() method

  - for Xen, the GPA belongs to a foreign domain, it must find a way to map & read it - beyond our scope here;
  - for KVM, the GPA can converted to HVA, copy_from_user (if called from vcpu thread) or access_remote_vm (if called from other threads);

Please note that this is not from the vfio bus driver, but from the vgpu device-model; also this is not DMA addr from GPU talbes, but real GPA.

Quoted 22 lines · Jike Song

Quoted 1 line · Alex Williamson

On Wed, 2016-01-27 at 13:43 +0800, Jike Song wrote:

{snip}

Quoted 8 lines

Quoted 5 lines

Had a look at eventfd, I would say yes, technically we are able to achieve the goal: introduce a fd, with fop->{read|write} defined in KVM, call into vgpu device-model, also an iodev registered for a MMIO GPA range to invoke the fop->{read|write}. I just didn't understand why userspace can't register an iodev via API directly.

Please elaborate on how it would work via iodev.

QEMU forwards BAR0 write to the bus driver, in the bus driver, if found that MEM bit is enabled, register an iodev to KVM: with an ops:

(quoted diff) +0 -0

 	const struct kvm_io_device_ops trap_mmio_ops = {
 		.read	= kvmgt_guest_mmio_read,
 		.write	= kvmgt_guest_mmio_write,
 	};

Quoted 3 lines

I may not be able to illustrated it clearly with descriptions but this should not be a problem, thanks to your explanation, I can understand and adopt it for KVMGT.

You're still crossing modules with direct callbacks, right? What's the advantage versus using the file descriptor + offset approach which could offer the same performance and improve KVM overall by creating a new option for generically handling MMIO?

From KVMGT's side this is almost the same as what you suggested, I don't

Quoted 31 lines

Quoted 6 lines

Quoted 2 lines

Besides, this doesn't necessarily require another thread, right? I guess it can be within the VCPU thread?

I would think so too, the vcpu is blocked on the MMIO access, we should be able to service it in that context. I hope.

Thanks for confirmation.

Quoted 6 lines

Quoted 5 lines

And this brought another question: except the vfio bus drvier and iommu backend (and the page_track ulitiy used for guest memory write-protection), is it KVMGT allowed to call into kvm.ko (or modify)? Though we are becoming less and less willing to do that with VFIO, it's still better to know that before going wrong.

Quoted 15 lines

kvm and vfio are separate modules, for the most part, they know nothing about each other and have no hard dependencies between them. We do have various accelerations we can use to avoid paths through userspace, but these are all via APIs that are agnostic of the party on the other end. For example, vfio signals interrups through eventfds and has no concept of whether that eventfd terminates in userspace or into an irqfd in KVM. vfio supports direct access to device MMIO regions via mmaps, but vfio has no idea if that mmap gets directly mapped into a VM address space. Even with posted interrupts, we've introduced an irq bypass manager allowing interrupt producers and consumers to register independently to form a connection without directly knowing anything about the other module. That sort or proper software layering needs to continue. It would be wrong for a vfio bus driver to assume KVM is the user and directly call into KVM interfaces. Thanks,

Quoted 5 lines

I understand and agree with your point, it's bad if the bus driver assume KVM is the user and/or call into KVM interfaces.

However, the vgpu device-model, in intel case also a part of i915 driver, will always need to call some hypervisor-specific interfaces.

No, think differently.

Quoted 12 lines

For example, when a guest gfx driver submit GPU commands, the device-model may want to scan it for security or whatever-else purpose:

(quoted diff) +0 -0

 	- get a GPA (from GPU page tables)
 	- want to read 16 bytes from that GPA
 	- call hypervisor-specific read_gpa() method
 		- for Xen, the GPA belongs to a foreign domain, it must find
 		  a way to map & read it - beyond our scope here;
 		- for KVM, the GPA can converted to HVA, copy_from_user (if
 		  called from vcpu thread) or access_remote_vm (if called from
 		  other threads);

Quoted 2 lines

Please note that this is not from the vfio bus driver, but from the vgpu device-model; also this is not DMA addr from GPU talbes, but real GPA.

This is exactly why we're proposing that the vfio IOMMU interface be used as a database of guest translations.
The type1 IOMMU model in QEMU
maps all of guest memory through the IOMMU, in the vGPU model type1 is simply collecting these and they map GPA to process virtual memory.

When the GPU driver wants to get a GPA, it does so from this database. If it wants to read from it, it could get the mm and read from the virtual memory or pin the page for a GPA to HPA translation and read from the HPA. There is no reason to poke directly through to the hypervisor here. Let's design what you need into the vgpu version of the type1 IOMMU instead. Thanks,

Alex

This discussion becomes a little difficult for a newbie like me :(

Quoted 23 lines · Alex Williamson

Quoted 22 lines · Jike Song

Quoted 1 line · Alex Williamson

On Wed, 2016-01-27 at 13:43 +0800, Jike Song wrote:

{snip}

Quoted 8 lines

Quoted 5 lines

Had a look at eventfd, I would say yes, technically we are able to achieve the goal: introduce a fd, with fop->{read|write} defined in KVM, call into vgpu device-model, also an iodev registered for a MMIO GPA range to invoke the fop->{read|write}. I just didn't understand why userspace can't register an iodev via API directly.

Please elaborate on how it would work via iodev.

QEMU forwards BAR0 write to the bus driver, in the bus driver, if found that MEM bit is enabled, register an iodev to KVM: with an ops:

(quoted diff) +0 -0

 	const struct kvm_io_device_ops trap_mmio_ops = {
 		.read	= kvmgt_guest_mmio_read,
 		.write	= kvmgt_guest_mmio_write,
 	};

Quoted 9 lines

Quoted 3 lines

I may not be able to illustrated it clearly with descriptions but this should not be a problem, thanks to your explanation, I can understand and adopt it for KVMGT.

You're still crossing modules with direct callbacks, right? What's the advantage versus using the file descriptor + offset approach which could offer the same performance and improve KVM overall by creating a new option for generically handling MMIO?

Yes, the method I gave above is the current way: calling kvm_io_device_ops from KVM hypervisor, and then going to vgpu device-model directly.

From KVMGT's side this is almost the same as what you suggested, I don't think now we have a problem here. I will adopt your suggestion.

Quoted 51 lines

Quoted 31 lines

Quoted 6 lines

Quoted 2 lines

Besides, this doesn't necessarily require another thread, right? I guess it can be within the VCPU thread?

I would think so too, the vcpu is blocked on the MMIO access, we should be able to service it in that context. I hope.

Thanks for confirmation.

Quoted 6 lines

Quoted 5 lines

And this brought another question: except the vfio bus drvier and iommu backend (and the page_track ulitiy used for guest memory write-protection), is it KVMGT allowed to call into kvm.ko (or modify)? Though we are becoming less and less willing to do that with VFIO, it's still better to know that before going wrong.

Quoted 15 lines

kvm and vfio are separate modules, for the most part, they know nothing about each other and have no hard dependencies between them. We do have various accelerations we can use to avoid paths through userspace, but these are all via APIs that are agnostic of the party on the other end. For example, vfio signals interrups through eventfds and has no concept of whether that eventfd terminates in userspace or into an irqfd in KVM. vfio supports direct access to device MMIO regions via mmaps, but vfio has no idea if that mmap gets directly mapped into a VM address space. Even with posted interrupts, we've introduced an irq bypass manager allowing interrupt producers and consumers to register independently to form a connection without directly knowing anything about the other module. That sort or proper software layering needs to continue. It would be wrong for a vfio bus driver to assume KVM is the user and directly call into KVM interfaces. Thanks,

Quoted 5 lines

I understand and agree with your point, it's bad if the bus driver assume KVM is the user and/or call into KVM interfaces.

However, the vgpu device-model, in intel case also a part of i915 driver, will always need to call some hypervisor-specific interfaces.

No, think differently.

Quoted 12 lines

For example, when a guest gfx driver submit GPU commands, the device-model may want to scan it for security or whatever-else purpose:

(quoted diff) +0 -0

 	- get a GPA (from GPU page tables)
 	- want to read 16 bytes from that GPA
 	- call hypervisor-specific read_gpa() method
 		- for Xen, the GPA belongs to a foreign domain, it must find
 		  a way to map & read it - beyond our scope here;
 		- for KVM, the GPA can converted to HVA, copy_from_user (if
 		  called from vcpu thread) or access_remote_vm (if called from
 		  other threads);

Quoted 8 lines

Quoted 2 lines

Please note that this is not from the vfio bus driver, but from the vgpu device-model; also this is not DMA addr from GPU talbes, but real GPA.

This is exactly why we're proposing that the vfio IOMMU interface be used as a database of guest translations.
The type1 IOMMU model in QEMU
maps all of guest memory through the IOMMU, in the vGPU model type1 is simply collecting these and they map GPA to process virtual memory.

GPA to HVA mappings are maintained in KVM/QEMU, via memslots. Do you mean making type1 to duplicate the GPA <-> HVA/HPA translations from KVM? Even technically this could be done, how to synchronize it with KVM hypervisor? e.g. What is expected if guest hot-add a memslot?

What's more, GPA is totally a virtualization term. When VFIO is used for device assignment, it uses GPA as IOVA, maps it to HPA, that's true. But for KVMGT, since vGPU doesn't have its own DMA requester ID, VFIO won't call IOMMU-API, but DMA-API instead. GPAs from different guests may be identical, while IGD can only have 1 single IOMMU domain ...

Quoted 6 lines

When the GPU driver wants to get a GPA, it does so from this database. If it wants to read from it, it could get the mm and read from the virtual memory or pin the page for a GPA to HPA translation and read from the HPA. There is no reason to poke directly through to the hypervisor here. Let's design what you need into the vgpu version of the type1 IOMMU instead. Thanks,

For KVM, to access a GPA, having it translated to HVA is enough.

IIUC this may be the only remaining problem between us: where should a GPA be translated to HVA, KVM or VFIO?

Quoted 35 lines · Jike Song

This discussion becomes a little difficult for a newbie like me :(

Quoted 32 lines · Alex Williamson

Quoted 12 lines · Jike Song

Quoted 1 line · Alex Williamson

On Wed, 2016-01-27 at 13:43 +0800, Jike Song wrote:

{snip}

Quoted 8 lines

Quoted 5 lines

Had a look at eventfd, I would say yes, technically we are able to achieve the goal: introduce a fd, with fop->{read|write} defined in KVM, call into vgpu device-model, also an iodev registered for a MMIO GPA range to invoke the fop->{read|write}.  I just didn't understand why userspace can't register an iodev via API directly.

Please elaborate on how it would work via iodev.

Quoted 13 lines

QEMU forwards BAR0 write to the bus driver, in the bus driver, if found that MEM bit is enabled, register an iodev to KVM: with an ops:

  const struct kvm_io_device_ops trap_mmio_ops = {
  .read = kvmgt_guest_mmio_read,
  .write = kvmgt_guest_mmio_write,
  };

I may not be able to illustrated it clearly with descriptions but this should not be a problem, thanks to your explanation, I can understand and adopt it for KVMGT.

You're still crossing modules with direct callbacks, right?  What's the advantage versus using the file descriptor + offset approach which could offer the same performance and improve KVM overall by creating a new option for generically handling MMIO?

Quoted 6 lines

Yes, the method I gave above is the current way: calling kvm_io_device_ops from KVM hypervisor, and then going to vgpu device-model directly.
From KVMGT's side this is almost the same as what you suggested, I don't think now we have a problem here. I will adopt your suggestion.

Quoted 59 lines

Quoted 59 lines

Quoted 30 lines

Quoted 6 lines

Quoted 2 lines

Besides, this doesn't necessarily require another thread, right? I guess it can be within the VCPU thread?

I would think so too, the vcpu is blocked on the MMIO access, we should be able to service it in that context.  I hope.

Thanks for confirmation.

Quoted 5 lines

Quoted 5 lines

And this brought another question: except the vfio bus drvier and iommu backend (and the page_track ulitiy used for guest memory write-protection), is it KVMGT allowed to call into kvm.ko (or modify)? Though we are becoming less and less willing to do that with VFIO, it's still better to know that before going wrong.

Quoted 16 lines

kvm and vfio are separate modules, for the most part, they know nothing about each other and have no hard dependencies between them.  We do have various accelerations we can use to avoid paths through userspace, but these are all via APIs that are agnostic of the party on the other end. For example, vfio signals interrups through eventfds and has no concept of whether that eventfd terminates in userspace or into an irqfd in KVM. vfio supports direct access to device MMIO regions via mmaps, but vfio has no idea if that mmap gets directly mapped into a VM address space. Even with posted interrupts, we've introduced an irq bypass manager allowing interrupt producers and consumers to register independently to form a connection without directly knowing anything about the other module.  That sort or proper software layering needs to continue.  It would be wrong for a vfio bus driver to assume KVM is the user and directly call into KVM interfaces.  Thanks,

Quoted 6 lines

I understand and agree with your point, it's bad if the bus driver assume KVM is the user and/or call into KVM interfaces.

However, the vgpu device-model, in intel case also a part of i915 driver, will always need to call some hypervisor-specific interfaces.

No, think differently.

Quoted 6 lines

For example, when a guest gfx driver submit GPU commands, the device-model may want to scan it for security or whatever-else purpose:

  - get a GPA (from GPU page tables)
  - want to read 16 bytes from that GPA
  - call hypervisor-specific read_gpa() method

Quoted 8 lines

  - for Xen, the GPA belongs to a foreign domain, it must find a way to map & read it - beyond our scope here;
  - for KVM, the GPA can converted to HVA, copy_from_user (if called from vcpu thread) or access_remote_vm (if called from other threads);

Please note that this is not from the vfio bus driver, but from the vgpu device-model; also this is not DMA addr from GPU talbes, but real GPA.

This is exactly why we're proposing that the vfio IOMMU interface be used as a database of guest translations.
The type1 IOMMU model in QEMU
maps all of guest memory through the IOMMU, in the vGPU model type1 is simply collecting these and they map GPA to process virtual memory.

Quoted 5 lines

GPA to HVA mappings are maintained in KVM/QEMU, via memslots. Do you mean making type1 to duplicate the GPA <-> HVA/HPA translations from KVM? Even technically this could be done, how to synchronize it with KVM hypervisor? e.g. What is expected if guest hot-add a memslot?

Quoted 5 lines

What's more, GPA is totally a virtualization term. When VFIO is used for device assignment, it uses GPA as IOVA, maps it to HPA, that's true. But for KVMGT, since vGPU doesn't have its own DMA requester ID, VFIO won't call IOMMU-API, but DMA-API instead.  GPAs from different guests may be identical, while IGD can only have 1 single IOMMU domain ...

Quoted 11 lines

Quoted 6 lines

When the GPU driver wants to get a GPA, it does so from this database. If it wants to read from it, it could get the mm and read from the virtual memory or pin the page for a GPA to HPA translation and read from the HPA.  There is no reason to poke directly through to the hypervisor here.  Let's design what you need into the vgpu version of the type1 IOMMU instead.  Thanks,

For KVM, to access a GPA, having it translated to HVA is enough.
IIUC this may be the only remaining problem between us: where should a GPA be translated to HVA, KVM or VFIO?

Unfortunately it's not the only one. Another example is, device-model may want to write-protect a gfn (RAM). In case that this request goes to VFIO .. how it is supposed to reach KVM MMU?

Quoted 3 lines

Unfortunately it's not the only one. Another example is, device-model may want to write-protect a gfn (RAM). In case that this request goes to VFIO .. how it is supposed to reach KVM MMU?

Well, let's work through the problem. How is the GFN related to the device? Is this some sort of page table for device mappings with a base register in the vgpu hardware?

  Hi,

Quoted 7 lines

Quoted 3 lines

Unfortunately it's not the only one. Another example is, device-model may want to write-protect a gfn (RAM). In case that this request goes to VFIO .. how it is supposed to reach KVM MMU?

Well, let's work through the problem.  How is the GFN related to the device?  Is this some sort of page table for device mappings with a base register in the vgpu hardware?

IIRC this is needed to make sure the guest can't bypass execbuffer verification and works like this:

  (1) guest submits execbuffer.
  (2) host makes execbuffer readonly for the guest
  (3) verify the buffer (make sure it only accesses resources owned by the vm).
  (4) pass on execbuffer to the hardware.
  (5) when the gpu is done with it make the execbuffer writable again.

Alternatively can the vgpu
driver copy it to a private buffer and hardware can execute from that?

Quoted 9 lines · Gerd Hoffmann

  Hi,

Quoted 7 lines

Quoted 3 lines

Unfortunately it's not the only one. Another example is, device-model may want to write-protect a gfn (RAM). In case that this request goes to VFIO .. how it is supposed to reach KVM MMU?

Well, let's work through the problem.  How is the GFN related to the device?  Is this some sort of page table for device mappings with a base register in the vgpu hardware?

Quoted 10 lines

IIRC this is needed to make sure the guest can't bypass execbuffer verification and works like this:

  (1) guest submits execbuffer.
  (2) host makes execbuffer readonly for the guest
  (3) verify the buffer (make sure it only accesses resources owned by the vm).
  (4) pass on execbuffer to the hardware.
  (5) when the gpu is done with it make the execbuffer writable again.