vfio-pci: Allow write combining
Subsystems (2)
RFC PATCH 1/1 vfio-pci: Allow write combining
At the moment the protection in VFIO MMIO mappings is forced to _PAGE_NON_IDEMPOTENT which means that write combining is not really available to the userspace even for prefetchable 64bit MMIO BARs.
This replaces pgprot_noncached() with a platform specific phys_mem_access_prot() when available and depending on the platform the vm_page_prot may be set to _PAGE_TOLERANT allowing to exploit the write combining feature.
The guest drivers still have to use _wc versions of the ioremap/pci_ioremap API to get write combininig working.
---
This should allow DPDK and radix guests (x86, POWERPC, etc) to do write combining.
POWERPC hash guests should not be affected by this change, it should work even without this.
--- drivers/vfio/pci/vfio_pci.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci.c +6 -1
@@ -1156,8 +1156,13 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) } vma->vm_private_data = vdev; - vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); vma->vm_pgoff = (pci_resource_start (pdev, index) >>PAGE_SHIFT ) + pgoff; +#ifdef __HAVE_PHYS_MEM_ACCESS_PROT + vma->vm_page_prot =phys_mem_access_prot (NULL , vma->vm_pgoff, + req_len, vma->vm_page_prot); +#else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); +#endif return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, req_len, vma->vm_page_prot);
Signature · 1 line
2.11.0
Quoted 33 lines · On Mon, 9 Oct 2017 13:50:00 +1100 Alexey Kardashevskiy <[email protected]> wrote:
At the moment the protection in VFIO MMIO mappings is forced to
_PAGE_NON_IDEMPOTENTwhich means that write combining is not really available to the userspace even for prefetchable 64bit MMIO BARs.This replaces
pgprot_noncached()with a platform specificphys_mem_access_prot()when available and depending on the platform thevm_page_protmay be set to_PAGE_TOLERANTallowing to exploit the write combining feature.The guest drivers still have to use
_wcversions of the ioremap/pci_ioremap API to get write combininig working.Signed-off-by Alexey Kardashevskiy <[email protected]>---
This should allow DPDK and radix guests (x86, POWERPC, etc) to do write combining.
POWERPC hash guests should not be affected by this change, it should work even without this.
--- drivers/vfio/pci/vfio_pci.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)drivers/vfio/pci/vfio_pci.c +0
-1@@ -1156,8 +1156,13 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) } vma->vm_private_data = vdev; - vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot);
drivers/vfio/pci/vfio_pci.c static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) · line 1168
Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +6
-0vma->vm_pgoff = (pci_resource_start (pdev, index) >>PAGE_SHIFT ) + pgoff; +#ifdef __HAVE_PHYS_MEM_ACCESS_PROT + vma->vm_page_prot =phys_mem_access_prot (NULL , vma->vm_pgoff, + req_len, vma->vm_page_prot); +#else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); +#endif return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, req_len, vma->vm_page_prot);
Are you testing __HAVE_PHYS_MEM_ACCESS_PROT because the version of phys_mem_access_prot() defined in drivers/char/mem.c can dereference @file and we're hoping that platforms we care about won't both define __HAVE_PHYS_MEM_ACCESS_PROT and look at @file? Sketchy. Thanks,
Alex
Quoted 44 lines · Alex Williamson
Quoted 41 lines · On Mon, 9 Oct 2017 13:50:00 +1100 Alexey Kardashevskiy <[email protected]> wrote:
At the moment the protection in VFIO MMIO mappings is forced to
_PAGE_NON_IDEMPOTENTwhich means that write combining is not really available to the userspace even for prefetchable 64bit MMIO BARs.This replaces
pgprot_noncached()with a platform specificphys_mem_access_prot()when available and depending on the platform thevm_page_protmay be set to_PAGE_TOLERANTallowing to exploit the write combining feature.The guest drivers still have to use
_wcversions of the ioremap/pci_ioremap API to get write combininig working.Signed-off-by Alexey Kardashevskiy <[email protected]>---
This should allow DPDK and radix guests (x86, POWERPC, etc) to do write combining.
POWERPC hash guests should not be affected by this change, it should work even without this.
--- drivers/vfio/pci/vfio_pci.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)drivers/vfio/pci/vfio_pci.c +6
-1@@ -1156,8 +1156,13 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) } vma->vm_private_data = vdev; - vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); vma->vm_pgoff = (pci_resource_start (pdev, index) >>PAGE_SHIFT ) + pgoff; +#ifdef __HAVE_PHYS_MEM_ACCESS_PROT + vma->vm_page_prot =phys_mem_access_prot (NULL , vma->vm_pgoff, + req_len, vma->vm_page_prot); +#else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); +#endif
Quoted 7 lines
drivers/vfio/pci/vfio_pci.c static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) · line 1168
Quoted 2 lines
drivers/vfio/pci/vfio_pci.c +0
-0return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, req_len, vma->vm_page_prot);Are you testing
__HAVE_PHYS_MEM_ACCESS_PROTbecause the version ofphys_mem_access_prot()defined indrivers/char/mem.ccan dereference @file and we're hoping that platforms we care about won't both define__HAVE_PHYS_MEM_ACCESS_PROTand look at @file?
No.
That version in mem.c is static and not exported at all and I do not understand why it got this name. Every other instance of phys_mem_access_prot is accompanied by
#define __HAVE_PHYS_MEM_ACCESS_PROT
But 3 instances (ia64, x86, mips) are not exported (arm, arm64, ppc are) and v2 of this will come with 3 more single line patches, if we decide to proceed.
The only version which actually looks at @file is in arch/mips/loongson64/common/mem.c and I do not know what to do about it (can it do VFIO at all?), I could pass a file there but no actual code would use it anyway.
Quoted 4 lines
Sketchy. Thanks,
Alex
Signature · 1 line
Alexey
Quoted 53 lines · On Wed, 11 Oct 2017 13:05:00 +1100 Alexey Kardashevskiy <[email protected]> wrote:
Quoted 44 lines · Alex Williamson
Quoted 41 lines · On Mon, 9 Oct 2017 13:50:00 +1100 Alexey Kardashevskiy <[email protected]> wrote:
At the moment the protection in VFIO MMIO mappings is forced to
_PAGE_NON_IDEMPOTENTwhich means that write combining is not really available to the userspace even for prefetchable 64bit MMIO BARs.This replaces
pgprot_noncached()with a platform specificphys_mem_access_prot()when available and depending on the platform thevm_page_protmay be set to_PAGE_TOLERANTallowing to exploit the write combining feature.The guest drivers still have to use
_wcversions of the ioremap/pci_ioremap API to get write combininig working.Signed-off-by Alexey Kardashevskiy <[email protected]>---
This should allow DPDK and radix guests (x86, POWERPC, etc) to do write combining.
POWERPC hash guests should not be affected by this change, it should work even without this.
--- drivers/vfio/pci/vfio_pci.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)drivers/vfio/pci/vfio_pci.c +6
-1@@ -1156,8 +1156,13 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) } vma->vm_private_data = vdev; - vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); vma->vm_pgoff = (pci_resource_start (pdev, index) >>PAGE_SHIFT ) + pgoff; +#ifdef __HAVE_PHYS_MEM_ACCESS_PROT + vma->vm_page_prot =phys_mem_access_prot (NULL , vma->vm_pgoff, + req_len, vma->vm_page_prot); +#else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); +#endif Quoted 7 lines
drivers/vfio/pci/vfio_pci.c static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) · line 1168
Quoted 2 lines
drivers/vfio/pci/vfio_pci.c +0
-0return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, req_len, vma->vm_page_prot);Are you testing
__HAVE_PHYS_MEM_ACCESS_PROTbecause the version ofphys_mem_access_prot()defined indrivers/char/mem.ccan dereference @file and we're hoping that platforms we care about won't both define__HAVE_PHYS_MEM_ACCESS_PROTand look at @file?
Quoted 7 lines
No.
That version in
mem.cis static and not exported at all and I do not understand why it got this name. Every other instance ofphys_mem_access_protis accompanied by#define
__HAVE_PHYS_MEM_ACCESS_PROT
I did miss that mem.c was static there, but I think the point is still valid, file being NULL doesn't seem to be a universally expected option.
Quoted 8 lines
But 3 instances (ia64, x86, mips) are not exported (arm, arm64, ppc are) and v2 of this will come with 3 more single line patches, if we decide to proceed.
The only version which actually looks at @file is in
arch/mips/loongson64/common/mem.cand I do not know what to do about it (can it do VFIO at all?), I could pass a file there but no actual code would use it anyway.
The question is not whether this particular platform could use vfio, but instead is whether vfio is using the function correctly. That, I really don't know.
But also...
arch/arm64/mm/mmu.c:pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
unsigned long size, pgprot_t vma_prot)
{
if (!pfn_valid(pfn))
return pgprot_noncached(vma_prot);
else if (file->f_flags & O_SYNC)
return pgprot_writecombine(vma_prot);
return vma_prot;
}Why do we get to ignore dereferencing file on an arch that we definitely care about? Thanks,
Alex
Quoted 72 lines · Alex Williamson
Quoted 53 lines · On Wed, 11 Oct 2017 13:05:00 +1100 Alexey Kardashevskiy <[email protected]> wrote:
Quoted 44 lines · Alex Williamson
Quoted 41 lines · On Mon, 9 Oct 2017 13:50:00 +1100 Alexey Kardashevskiy <[email protected]> wrote:
At the moment the protection in VFIO MMIO mappings is forced to
_PAGE_NON_IDEMPOTENTwhich means that write combining is not really available to the userspace even for prefetchable 64bit MMIO BARs.This replaces
pgprot_noncached()with a platform specificphys_mem_access_prot()when available and depending on the platform thevm_page_protmay be set to_PAGE_TOLERANTallowing to exploit the write combining feature.The guest drivers still have to use
_wcversions of the ioremap/pci_ioremap API to get write combininig working.Signed-off-by Alexey Kardashevskiy <[email protected]>---
This should allow DPDK and radix guests (x86, POWERPC, etc) to do write combining.
POWERPC hash guests should not be affected by this change, it should work even without this.
--- drivers/vfio/pci/vfio_pci.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)drivers/vfio/pci/vfio_pci.c +6
-1@@ -1156,8 +1156,13 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) } vma->vm_private_data = vdev; - vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); vma->vm_pgoff = (pci_resource_start (pdev, index) >>PAGE_SHIFT ) + pgoff; +#ifdef __HAVE_PHYS_MEM_ACCESS_PROT + vma->vm_page_prot =phys_mem_access_prot (NULL , vma->vm_pgoff, + req_len, vma->vm_page_prot); +#else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); +#endif Quoted 7 lines
drivers/vfio/pci/vfio_pci.c static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma) · line 1168
Quoted 2 lines
drivers/vfio/pci/vfio_pci.c +0
-0return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, req_len, vma->vm_page_prot);Are you testing
__HAVE_PHYS_MEM_ACCESS_PROTbecause the version ofphys_mem_access_prot()defined indrivers/char/mem.ccan dereference @file and we're hoping that platforms we care about won't both define__HAVE_PHYS_MEM_ACCESS_PROTand look at @file?Quoted 7 lines
No.
That version in
mem.cis static and not exported at all and I do not understand why it got this name. Every other instance ofphys_mem_access_protis accompanied by#define
__HAVE_PHYS_MEM_ACCESS_PROTI did miss that
mem.cwas static there, but I think the point is still valid, file being NULL doesn't seem to be a universally expected option.Quoted 4 lines
But 3 instances (ia64, x86, mips) are not exported (arm, arm64, ppc are) and v2 of this will come with 3 more single line patches, if we decide to proceed.
Quoted 8 lines
Quoted 4 lines
The only version which actually looks at @file is in
arch/mips/loongson64/common/mem.cand I do not know what to do about it (can it do VFIO at all?), I could pass a file there but no actual code would use it anyway.The question is not whether this particular platform could use vfio, but instead is whether vfio is using the function correctly. That, I really don't know.
Who does? :) Let's cc: him.
Quoted 15 lines
But also...
arch/arm64/mm/mmu.c:pgprot_tphys_mem_access_prot(struct file *file, unsigned long pfn,
unsigned long size,pgprot_tvma_prot){ if (!pfn_valid(pfn)) return pgprot_noncached(vma_prot); else if (file->f_flags & O_SYNC) return pgprot_writecombine(vma_prot); return vma_prot; }
Quoted 2 lines
Why do we get to ignore dereferencing file on an arch that we definitely care about? Thanks,
Oopsie. This is because I overlooked it. Others do not use it. So I do need a file. But in the current scheme where all BARs share one fd - it won't work - I simply cannot allow WC on non-prefetchable BARs :-/
Signature · 1 line
Alexey
Quoted 3 lines · Alexey Kardashevskiy
Oopsie. This is because I overlooked it. Others do not use it. So I do need a file. But in the current scheme where all BARs share one fd - it won't work - I simply cannot allow WC on non-prefetchable BARs :-/
This is an oversight in the design of VFIO-PCI, it should have a way to specify write combine, either implicitely via such an arch hook, or explicitely via an ioctl prior to mapping the BARs for example.
Alex, what do you reckon is the best approach here ?
Cheers,
Ben.
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 3 lines · Alexey Kardashevskiy
Oopsie. This is because I overlooked it. Others do not use it. So I do need a file. But in the current scheme where all BARs share one fd - it won't work - I simply cannot allow WC on non-prefetchable BARs :-/
This is an oversight in the design of VFIO-PCI, it should have a way to specify write combine, either implicitely via such an arch hook, or explicitely via an ioctl prior to mapping the BARs for example.
Alex, what do you reckon is the best approach here ?
/me wonders if it is yet another issue for the dead issues bucket, just like the msix mapping one :)
Signature · 1 line
Alexey
Quoted 6 lines · Alexey Kardashevskiy
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 3 lines · Alexey Kardashevskiy
Oopsie. This is because I overlooked it. Others do not use it. So I do need a file. But in the current scheme where all BARs share one fd - it won't work - I simply cannot allow WC on non-prefetchable BARs :-/
Quoted 8 lines
Quoted 5 lines
This is an oversight in the design of VFIO-PCI, it should have a way to specify write combine, either implicitely via such an arch hook, or explicitely via an ioctl prior to mapping the BARs for example.
Alex, what do you reckon is the best approach here ?
/me wonders if it is yet another issue for the dead issues bucket, just like the msix mapping one :)
Maybe. Alexey, maybe you can make up a list of things that we (me, you, BenH) need to discuss with Alex W at KVM Forum?
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 11 lines · David Gibson
Quoted 10 lines · Alexey Kardashevskiy
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 3 lines · Alexey Kardashevskiy
Oopsie. This is because I overlooked it. Others do not use it. So I do need a file. But in the current scheme where all BARs share one fd - it won't work - I simply cannot allow WC on non-prefetchable BARs :-/
This is an oversight in the design of VFIO-PCI, it should have a way to specify write combine, either implicitely via such an arch hook, or explicitely via an ioctl prior to mapping the BARs for example.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Alex, what do you reckon is the best approach here ?
/me wonders if it is yet another issue for the dead issues bucket, just like the msix mapping one :)
Maybe. Alexey, maybe you can make up a list of things that we (me, you, BenH) need to discuss with Alex W at KVM Forum?
"you" - you meant me? I am not coming over there :(
The list is:
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like IOMMUMR::add_vfio_group() proposal or something.
Thanks!
Signature · 1 line
Alexey
Quoted 6 lines · Alexey Kardashevskiy
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
Why ? The "G" bit is entirely under control of the guest afaik. This would only affect qemu itself. It's still useful for things like dpdk using vfio.
Quoted 2 lines
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like
IOMMUMR::add_vfio_group()proposal or something.
Can you elaborate a bit ? I haven't followed this.
Cheers,
Ben.
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 6 lines · Alexey Kardashevskiy
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
Why ? The "G" bit is entirely under control of the guest afaik.
Yes, for hash guests. I am not sure sure about radix, Paul pointed me to the code in KVM which uses the VFIO's mapping VMA.
Quoted 3 lines
This
would only affect qemu itself. It's still useful for things like dpdk using vfio.
Correct, this is useful regardless KVM.
Quoted 4 lines
Quoted 2 lines
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like
IOMMUMR::add_vfio_group()proposal or something.Can you elaborate a bit ? I haven't followed this.
David knows :)
Signature · 1 line
Alexey
Quoted 5 lines · Alexey Kardashevskiy
Quoted 4 lines · Benjamin Herrenschmidt
Quoted 3 lines · Alexey Kardashevskiy
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
Why ? The "G" bit is entirely under control of the guest afaik.
Yes, for hash guests. I am not sure sure about radix, Paul pointed me to the code in KVM which uses the VFIO's mapping VMA.
With radix, the HW will honor the G bit set in the guest page tables.
Quoted 16 lines
Quoted 4 lines
This
would only affect qemu itself. It's still useful for things like dpdk using vfio.Correct, this is useful regardless KVM.
Quoted 4 lines
Quoted 2 lines
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like
IOMMUMR::add_vfio_group()proposal or something.Can you elaborate a bit ? I haven't followed this.
David knows :)
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 9 lines · Alexey Kardashevskiy
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 6 lines · Alexey Kardashevskiy
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Why ? The "G" bit is entirely under control of the guest afaik.
Yes, for hash guests. I am not sure sure about radix, Paul pointed me to the code in KVM which uses the VFIO's mapping VMA.
With radix, the HW will honor the G bit set in the guest page tables.
I am sure that you know better, Paul just pointed me to this:
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
This
would only affect qemu itself. It's still useful for things like dpdk using vfio.Correct, this is useful regardless KVM.
Quoted 4 lines
Quoted 2 lines
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like
IOMMUMR::add_vfio_group()proposal or something.Can you elaborate a bit ? I haven't followed this.
David knows :)
Signature · 1 line
Alexey
Quoted 7 lines · Alexey Kardashevskiy
Quoted 2 lines
With radix, the HW will honor the G bit set in the guest page tables.
I am sure that you know better, Paul just pointed me to this:
It's in the architecture somewhere, if you look at 3.0B in the radix section about nested radix, there's a table for the nesting of attributes.
Here you'll see that regardless of the G bit value set by the host, the HW will use the G bit value set by the guest.
However, we should still provide a way to exploit WC if anything for things like DPDK.
Cheers,
Ben.
Quoted 14 lines · Alexey Kardashevskiy
Quoted 13 lines · David Gibson
Quoted 12 lines · Alexey Kardashevskiy
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 3 lines · Alexey Kardashevskiy
Oopsie. This is because I overlooked it. Others do not use it. So I do need a file. But in the current scheme where all BARs share one fd - it won't work - I simply cannot allow WC on non-prefetchable BARs :-/
This is an oversight in the design of VFIO-PCI, it should have a way to specify write combine, either implicitely via such an arch hook, or explicitely via an ioctl prior to mapping the BARs for example.
Alex, what do you reckon is the best approach here ?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
/me wonders if it is yet another issue for the dead issues bucket, just like the msix mapping one :)
Maybe. Alexey, maybe you can make up a list of things that we (me, you, BenH) need to discuss with Alex W at KVM Forum?
"you" - you meant me? I am not coming over there :(
Oh.. I thought you were.
Quoted 10 lines
The list is:
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like
IOMMUMR::add_vfio_group()proposal or something.
Thanks.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 29 lines · David Gibson
Quoted 14 lines · Alexey Kardashevskiy
Quoted 13 lines · David Gibson
Quoted 12 lines · Alexey Kardashevskiy
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 3 lines · Alexey Kardashevskiy
Oopsie. This is because I overlooked it. Others do not use it. So I do need a file. But in the current scheme where all BARs share one fd - it won't work - I simply cannot allow WC on non-prefetchable BARs :-/
This is an oversight in the design of VFIO-PCI, it should have a way to specify write combine, either implicitely via such an arch hook, or explicitely via an ioctl prior to mapping the BARs for example.
Alex, what do you reckon is the best approach here ?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
/me wonders if it is yet another issue for the dead issues bucket, just like the msix mapping one :)
Maybe. Alexey, maybe you can make up a list of things that we (me, you, BenH) need to discuss with Alex W at KVM Forum?
"you" - you meant me? I am not coming over there :(
Oh.. I thought you were.
Quoted 4 lines
The list is:
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
Quoted 8 lines
Quoted 6 lines
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like
IOMMUMR::add_vfio_group()proposal or something.Thanks.
Any luck with these?
Ben, ping.
Signature · 1 line
Alexey
Quoted 22 lines · Alexey Kardashevskiy
Quoted 21 lines · David Gibson
Quoted 20 lines · Alexey Kardashevskiy
Quoted 18 lines · David Gibson
Quoted 6 lines · Alexey Kardashevskiy
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 3 lines · Alexey Kardashevskiy
Oopsie. This is because I overlooked it. Others do not use it. So I do need a file. But in the current scheme where all BARs share one fd - it won't work - I simply cannot allow WC on non-prefetchable BARs :-/
Quoted 8 lines
Quoted 5 lines
This is an oversight in the design of VFIO-PCI, it should have a way to specify write combine, either implicitely via such an arch hook, or explicitely via an ioctl prior to mapping the BARs for example.
Alex, what do you reckon is the best approach here ?
/me wonders if it is yet another issue for the dead issues bucket, just like the msix mapping one :)
Maybe. Alexey, maybe you can make up a list of things that we (me, you, BenH) need to discuss with Alex W at KVM Forum?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
"you" - you meant me? I am not coming over there :(
Oh.. I thought you were.
Quoted 3 lines
The list is:
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
We have a new plan on this - I'll discuss it over IRC.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
AIUI this isn't for radix, but for DPDK things that we need this. Ben talked about it a bit, but I don't know what the outcome was.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like
IOMMUMR::add_vfio_group()proposal or something.
This was discussed, and I'm still thinking about it. It's kind of curly.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 10 lines · David Gibson
Quoted 1 line
Quoted 1 line
Quoted 1 line
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
We have a new plan on this - I'll discuss it over IRC.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
AIUI this isn't for radix, but for DPDK things that we need this. Ben talked about it a bit, but I don't know what the outcome was.
So this is not a powerpc specific issue. Other archs similarily want to be able to do write combine mappings.
The way sysfs does it is that for prefetchable BARs, it exposes both a resourceN and a resourceN_wc file.
For VFIO it's a bit more tricky, maybe we need to game the offset using some of it as flags but that's very fishy, or maybe we do some kind of ioctl that selects the attributes used for that fd instance for subsequent mappings...
I'll let Alex chose what he feels most appropriate here.
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
3. what callback and where needs to be added to inform HV/PR KVM about VFIO group, like
IOMMUMR::add_vfio_group()proposal or something.This was discussed, and I'm still thinking about it. It's kind of curly.
Quoted 18 lines · On Tue, 14 Nov 2017 13:29:02 +1100 Benjamin Herrenschmidt…
Quoted 10 lines · David Gibson
Quoted 1 line
Quoted 1 line
Quoted 1 line
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
We have a new plan on this - I'll discuss it over IRC.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
AIUI this isn't for radix, but for DPDK things that we need this. Ben talked about it a bit, but I don't know what the outcome was.
So this is not a powerpc specific issue. Other archs similarily want to be able to do write combine mappings.
The way sysfs does it is that for prefetchable BARs, it exposes both a resourceN and a
resourceN_wcfile.
Quoted 6 lines
For VFIO it's a bit more tricky, maybe we need to game the offset using some of it as flags but that's very fishy, or maybe we do some kind of ioctl that selects the attributes used for that fd instance for subsequent mappings...
I'll let Alex chose what he feels most appropriate here.
My order of preference would be something like:
- mmap flags provide some way for the user to specify a wc mapping within existing regions
- some other mechanism of using the existing regions
- additional regions provided for use exclusively with wc attributes (generalizing PCI BAR wc regions within device specific regions)
- additional file descriptors provided for wc access
This isn't at the top of my priority list to figure out the solution, so whoever implements it will need to provide justification as they move down the list from more to less preferred solutions. Thanks,
Alex
Quoted 26 lines · Alex Williamson
Quoted 23 lines · On Tue, 14 Nov 2017 13:29:02 +1100 Benjamin Herrenschmidt…
Quoted 10 lines · David Gibson
Quoted 1 line
Quoted 1 line
Quoted 1 line
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
We have a new plan on this - I'll discuss it over IRC.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
AIUI this isn't for radix, but for DPDK things that we need this. Ben talked about it a bit, but I don't know what the outcome was.
So this is not a powerpc specific issue. Other archs similarily want to be able to do write combine mappings.
The way sysfs does it is that for prefetchable BARs, it exposes both a resourceN and a
resourceN_wcfile.For VFIO it's a bit more tricky, maybe we need to game the offset using some of it as flags but that's very fishy, or maybe we do some kind of ioctl that selects the attributes used for that fd instance for subsequent mappings...
Quoted 6 lines
Quoted 1 line
I'll let Alex chose what he feels most appropriate here.
My order of preference would be something like:
- mmap flags provide some way for the user to specify a wc mapping within existing regions
There are plenty of flags but none really matches, checked with Paul.
Quoted 1 line
- some other mechanism of using the existing regions
I can only think of madvise but it does not have appropriate flags either.
Quoted 2 lines
- additional regions provided for use exclusively with wc attributes (generalizing PCI BAR wc regions within device specific regions)
Adding VFIO_PCI_BAR0_WC_REGION_INDEX for VFIO_PCI_BAR0_REGION_INDEX (and so on for other BARs) seems a viable option.
However the comment for VFIO_PCI_xxx_REGION_INDEX says:
VFIO_PCI_NUM_REGIONS = 9 /* Fixed user ABI, region indexes >=9 use */
/* device specific cap to define content. */
which limits me in where I can add new indexes, I cannot just add new _WC indexes to that enum, can I? I cannot see any existing regions above 9 yet though.
Quoted 1 line
- additional file descriptors provided for wc access
It could be a capability + iocti(VFIO_DEVICE_GET_WC_RESOURCE) which would take a BAR index, check if the BAR is prefetchable and if so - return an fd which the userspace then could mmap(). This is won't break that ABI with 9 regions but it is the least favourable in the list...
Quoted 3 lines
This isn't at the top of my priority list to figure out the solution, so whoever implements it will need to provide justification as they move down the list from more to less preferred solutions. Thanks,
I am trying... I was really counting on you guys having this discussed in Prague :(
Signature · 1 line
Alexey
Quoted 27 lines · On Fri, 24 Nov 2017 15:58:09 +1100 Alexey Kardashevskiy <[email protected]> wrote:
Quoted 26 lines · Alex Williamson
Quoted 23 lines · On Tue, 14 Nov 2017 13:29:02 +1100 Benjamin Herrenschmidt…
Quoted 10 lines · David Gibson
Quoted 1 line
Quoted 1 line
Quoted 1 line
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
We have a new plan on this - I'll discuss it over IRC.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
AIUI this isn't for radix, but for DPDK things that we need this. Ben talked about it a bit, but I don't know what the outcome was.
So this is not a powerpc specific issue. Other archs similarily want to be able to do write combine mappings.
The way sysfs does it is that for prefetchable BARs, it exposes both a resourceN and a
resourceN_wcfile.For VFIO it's a bit more tricky, maybe we need to game the offset using some of it as flags but that's very fishy, or maybe we do some kind of ioctl that selects the attributes used for that fd instance for subsequent mappings...
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
I'll let Alex chose what he feels most appropriate here.
My order of preference would be something like:
- mmap flags provide some way for the user to specify a wc mapping within existing regions
There are plenty of flags but none really matches, checked with Paul.
Is MAP_NONBLOCK off the table? Why?
Quoted 3 lines
Quoted 1 line
- some other mechanism of using the existing regions
I can only think of madvise but it does not have appropriate flags either.
Is it worth the process to define something that is appropriate? Would either of the above be the obvious architectural/implementation choice if we could define a flag for it?
Quoted 9 lines
Quoted 2 lines
- additional regions provided for use exclusively with wc attributes (generalizing PCI BAR wc regions within device specific regions)
Adding
VFIO_PCI_BAR0_WC_REGION_INDEXforVFIO_PCI_BAR0_REGION_INDEX(and so on for other BARs) seems a viable option.However the comment for
VFIO_PCI_xxx_REGION_INDEXsays:
Quoted 7 lines
VFIO_PCI_NUM_REGIONS= 9 /* Fixed user ABI, region indexes >=9 use *//* device specific cap to define content. */which limits me in where I can add new indexes, I cannot just add new
_WCindexes to that enum, can I? I cannot see any existing regions above 9 yet though.
The comment explains how to do this, you'd add a device specific region with the type identifying it as a PCI MMIO WC region and the sub-type probably defining the BAR index.
Quoted 6 lines
Quoted 1 line
- additional file descriptors provided for wc access
It could be a capability +
iocti(VFIO_DEVICE_GET_WC_RESOURCE)which would take a BAR index, check if the BAR is prefetchable and if so - return an fd which the userspace then couldmmap(). This is won't break that ABI with 9 regions but it is the least favourable in the list...
Do the kernel mechanics require it to be a separate file descriptor? A separate fd is my last choice as well, but the interfaces your were attempting to use previously seemed to have fd granularity.
Quoted 6 lines
Quoted 3 lines
This isn't at the top of my priority list to figure out the solution, so whoever implements it will need to provide justification as they move down the list from more to less preferred solutions. Thanks,
I am trying... I was really counting on you guys having this discussed in Prague :(
Should have been there to push your agenda... Thanks,
Alex
Quoted 81 lines · Alex Williamson
Quoted 27 lines · On Fri, 24 Nov 2017 15:58:09 +1100 Alexey Kardashevskiy <[email protected]> wrote:
Quoted 26 lines · Alex Williamson
Quoted 23 lines · On Tue, 14 Nov 2017 13:29:02 +1100 Benjamin Herrenschmidt…
Quoted 10 lines · David Gibson
Quoted 1 line
Quoted 1 line
Quoted 1 line
1. Allow msix mapping to the userspace (to address non-64k-aligned msix bar)
We have a new plan on this - I'll discuss it over IRC.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
2. Allow write combining in vfio for the userspace (kvm guest is kinda special and may simply ignore mapping flags in some configs but PPC radix guests still rely on this)
AIUI this isn't for radix, but for DPDK things that we need this. Ben talked about it a bit, but I don't know what the outcome was.
So this is not a powerpc specific issue. Other archs similarily want to be able to do write combine mappings.
The way sysfs does it is that for prefetchable BARs, it exposes both a resourceN and a
resourceN_wcfile.For VFIO it's a bit more tricky, maybe we need to game the offset using some of it as flags but that's very fishy, or maybe we do some kind of ioctl that selects the attributes used for that fd instance for subsequent mappings...
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
I'll let Alex chose what he feels most appropriate here.
My order of preference would be something like:
- mmap flags provide some way for the user to specify a wc mapping within existing regions
There are plenty of flags but none really matches, checked with Paul.
Is
MAP_NONBLOCKoff the table? Why?Quoted 3 lines
Quoted 1 line
- some other mechanism of using the existing regions
I can only think of madvise but it does not have appropriate flags either.
Is it worth the process to define something that is appropriate? Would either of the above be the obvious architectural/implementation choice if we could define a flag for it?
Quoted 9 lines
Quoted 2 lines
- additional regions provided for use exclusively with wc attributes (generalizing PCI BAR wc regions within device specific regions)
Adding
VFIO_PCI_BAR0_WC_REGION_INDEXforVFIO_PCI_BAR0_REGION_INDEX(and so on for other BARs) seems a viable option.However the comment for
VFIO_PCI_xxx_REGION_INDEXsays:Quoted 7 lines
VFIO_PCI_NUM_REGIONS= 9 /* Fixed user ABI, region indexes >=9 use *//* device specific cap to define content. */which limits me in where I can add new indexes, I cannot just add new
_WCindexes to that enum, can I? I cannot see any existing regions above 9 yet though.The comment explains how to do this, you'd add a device specific region with the type identifying it as a PCI MMIO WC region and the sub-type probably defining the BAR index.
Quoted 6 lines
Quoted 1 line
- additional file descriptors provided for wc access
It could be a capability +
iocti(VFIO_DEVICE_GET_WC_RESOURCE)which would take a BAR index, check if the BAR is prefetchable and if so - return an fd which the userspace then couldmmap(). This is won't break that ABI with 9 regions but it is the least favourable in the list...Do the kernel mechanics require it to be a separate file descriptor? A separate fd is my last choice as well, but the interfaces your were attempting to use previously seemed to have fd granularity.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
This isn't at the top of my priority list to figure out the solution, so whoever implements it will need to provide justification as they move down the list from more to less preferred solutions. Thanks,
I am trying... I was really counting on you guys having this discussed in Prague :(
Should have been there to push your agenda... Thanks,
We discussed it briefly, BenH seemed to think there wasn't a big difficulty, IIRC, which is why we didn't spend much time on this (compared to the other issues). So, talk to him.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 4 lines · David Gibson
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
This isn't at the top of my priority list to figure out the solution, so whoever implements it will need to provide justification as they move down the list from more to less preferred solutions. Thanks,
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I am trying... I was really counting on you guys having this discussed in Prague :(
Should have been there to push your agenda... Thanks,
We discussed it briefly, BenH seemed to think there wasn't a big difficulty, IIRC, which is why we didn't spend much time on this (compared to the other issues). So, talk to him.
Well, first we established that this wasn't an issue for KVM, so the importance/urgency went down. It's still useful for DPDK.
Then we discussed quickly the various options, none of them is particularily *difficult* as in the implementation is rather trivial, the question is to chose which interface to provide userspace.
I don't have a strong opinion there. The mm guys might object to hijacking MAP_NONBLOCK, otherwise that seems like the best approach, so we should run that past them.
Cheers,
Ben.