vfio/pci: Support 8-byte PCI loads and stores
Subsystems (3)
PATCH 1/1 vfio/pci: Support 8-byte PCI loads and stores
From: Ben Segal <[email protected]>
Many PCI adapters can benefit or even require full 64bit read and write access to their registers. In order to enable work on user-space drivers for these devices add two new variations vfio_pci_core_io{read|write}64 of the existing access methods when the architecture supports 64-bit ioreads and iowrites.
---
Hi all,
we've successfully used this patch with a user-mode driver for a PCI device that requires 64bit register read/writes on s390. A quick grep showed that there are several other drivers for PCI devices in the kernel that use readq/writeq and eventually could use this too.
So we decided to propose this for general inclusion.
We've added conditional compiles for non-64bit architectures that produce graceful run-time errors. However, that path is just compile-tested.
Thank you,
Gerd Bayer
drivers/vfio/pci/vfio_pci_rdwr.c | 39 +++++++++++++++++++++++++++++++- include/linux/vfio_pci_core.h | 3 +++ 2 files changed, 41 insertions(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci_rdwr.c +38 -1
@@ -89,6 +89,9 @@ EXPORT_SYMBOL_GPL(vfio_pci_core_ioread##size); VFIO_IOREAD(8) VFIO_IOREAD(16) VFIO_IOREAD(32) +#ifdef ioread64 +VFIO_IOREAD(64) +#endif /* * Read or write from an __iomem region (MMIO or I/O port) with an excluded
@@ -114,7 +117,41 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,else fillable =0 ; -if (fillable >=4 && !(off %4 )) { +if (fillable >=8 && !(off %8 )) { +#if defined(ioread64) || defined(iowrite64) +u64 val; +#endif + +if (iswrite) { +#ifndef iowrite64 +pr_err_once ("vfio does not support iowrite64 on this arch" ); +return -EIO ; +#else +if (copy_from_user (&val, buf,8 )) +return -EFAULT ; + + ret =vfio_pci_core_iowrite64 (vdev, test_mem, + val, io + off); +if (ret) +return ret; +#endif + }else { +#ifndef ioread64 +pr_err_once ("vfio does not support ioread64 on this arch" ); +return -EIO ; +#else + ret =vfio_pci_core_ioread64 (vdev, test_mem, + &val, io + off); +if (ret) +return ret; + +if (copy_to_user (buf, &val,8 )) +return -EFAULT ; +#endif + } + + filled =8 ; + }else if (fillable >=4 && !(off %4 )) {u32 val;if (iswrite) {
include/linux/vfio_pci_core.h +3 -0
@@ -157,5 +157,8 @@ int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \ VFIO_IOREAD_DECLATION(8) VFIO_IOREAD_DECLATION(16) VFIO_IOREAD_DECLATION(32) +#ifdef ioread64 +VFIO_IOREAD_DECLATION(64) +#endif #endif /* VFIO_PCI_CORE_H */
Signature · 1 line
2.44.0
Quoted 11 lines · Gerd Bayer
From: Ben Segal <[email protected]>
Many PCI adapters can benefit or even require full 64bit read and write access to their registers. In order to enable work on user-space drivers for these devices add two new variations vfio_pci_core_io{read|write}64 of the existing access methods when the architecture supports 64-bit ioreads and iowrites.
Signed-off-by Ben Segal <[email protected]>Co-developed-by Gerd Bayer <[email protected]>Signed-off-by Gerd Bayer <[email protected]>
Quoted 6 lines
---
Hi all,
we've successfully used this patch with a user-mode driver for a PCI device that requires 64bit register read/writes on s390.
But why? S390 already has a system call for userspace to do the 64 bit write, and newer S390 has a userspace instruction to do it.
Why would you want to use a VFIO system call on the mmio emulation path?
mmap the registers and access them normally?
Quoted 5 lines
* Read or write from an
__iomemregion (MMIO or I/O port) with an excludeddrivers/vfio/pci/vfio_pci_rdwr.c +0
-0@@ -114,7 +117,41 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,else fillable =0 ;
drivers/vfio/pci/vfio_pci_rdwr.c ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem, · line 128
Quoted 10 lines
drivers/vfio/pci/vfio_pci_rdwr.c +9
-1-if (fillable >=4 && !(off %4 )) { +if (fillable >=8 && !(off %8 )) { +#if defined(ioread64) || defined(iowrite64) +u64 val; +#endif + +if (iswrite) { +#ifndef iowrite64 +pr_err_once ("vfio does not support iowrite64 on this arch" ); +return -EIO ;
can't do that you have to go back to what the old stuff did and do the 4 byte copy.
Jason
Quoted 19 lines · Jason Gunthorpe
Quoted 11 lines · Gerd Bayer
From: Ben Segal <[email protected]>
Many PCI adapters can benefit or even require full 64bit read and write access to their registers. In order to enable work on user-space drivers for these devices add two new variations vfio_pci_core_io{read|write}64 of the existing access methods when the architecture supports 64-bit ioreads and iowrites.
Signed-off-by Ben Segal <[email protected]>Co-developed-by Gerd Bayer <[email protected]>Signed-off-by Gerd Bayer <[email protected]>Quoted 6 lines
---
Hi all,
we've successfully used this patch with a user-mode driver for a PCI device that requires 64bit register read/writes on s390.
Quoted 7 lines
But why? S390 already has a system call for userspace to do the 64 bit write, and newer S390 has a userspace instruction to do it.
Why would you want to use a VFIO system call on the mmio emulation path?
mmap the registers and access them normally?
It's a very good point and digging into why this wasn't used by Benjamin. It turns out VFIO_PCI_MMAP is disabled for S390 which it really shouldn't be especially now that we have the user-space instructions. Before that though Benjamin turned to this interface which then lead him to this limitation. So yeah we'll definitely verify that it also works via VFIO_PCI_MMAP and send a patch to enable that.
That said I still think it's odd not to have the 8 byte case working here even if it isn't the right approach. Could still be useful for debug/testing without having to add the MIO instructions or the our special syscall.
Thanks,
Niklas
Quoted 26 lines · Niklas Schnelle
Quoted 25 lines · Jason Gunthorpe
Quoted 11 lines · Gerd Bayer
From: Ben Segal <[email protected]>
Many PCI adapters can benefit or even require full 64bit read and write access to their registers. In order to enable work on user-space drivers for these devices add two new variations vfio_pci_core_io{read|write}64 of the existing access methods when the architecture supports 64-bit ioreads and iowrites.
Signed-off-by Ben Segal <[email protected]>Co-developed-by Gerd Bayer <[email protected]>Signed-off-by Gerd Bayer <[email protected]>Quoted 6 lines
---
Hi all,
we've successfully used this patch with a user-mode driver for a PCI device that requires 64bit register read/writes on s390.
But why? S390 already has a system call for userspace to do the 64 bit write, and newer S390 has a userspace instruction to do it.
Why would you want to use a VFIO system call on the mmio emulation path?
Quoted 8 lines
Quoted 1 line
mmap the registers and access them normally?
It's a very good point and digging into why this wasn't used by Benjamin. It turns out
VFIO_PCI_MMAPis disabled for S390 which it really shouldn't be especially now that we have the user-space instructions. Before that though Benjamin turned to this interface which then lead him to this limitation. So yeah we'll definitely verify that it also works viaVFIO_PCI_MMAPand send a patch to enable that.
Make sense to me!
Quoted 4 lines
That said I still think it's odd not to have the 8 byte case working here even if it isn't the right approach. Could still be useful for debug/testing without having to add the MIO instructions or the our special syscall.
Yes, this also makes sense, but this patch needs some adjusting
Jason
Quoted 36 lines · On Fri, 19 Apr 2024 13:11:35 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 26 lines · Niklas Schnelle
Quoted 25 lines · Jason Gunthorpe
Quoted 11 lines · Gerd Bayer
From: Ben Segal <[email protected]>
Many PCI adapters can benefit or even require full 64bit read and write access to their registers. In order to enable work on user-space drivers for these devices add two new variations vfio_pci_core_io{read|write}64 of the existing access methods when the architecture supports 64-bit ioreads and iowrites.
Signed-off-by Ben Segal <[email protected]>Co-developed-by Gerd Bayer <[email protected]>Signed-off-by Gerd Bayer <[email protected]>Quoted 6 lines
---
Hi all,
we've successfully used this patch with a user-mode driver for a PCI device that requires 64bit register read/writes on s390.
But why? S390 already has a system call for userspace to do the 64 bit write, and newer S390 has a userspace instruction to do it.
Why would you want to use a VFIO system call on the mmio emulation path?
Quoted 8 lines
Quoted 1 line
mmap the registers and access them normally?
It's a very good point and digging into why this wasn't used by Benjamin. It turns out
VFIO_PCI_MMAPis disabled for S390 which it really shouldn't be especially now that we have the user-space instructions. Before that though Benjamin turned to this interface which then lead him to this limitation. So yeah we'll definitely verify that it also works viaVFIO_PCI_MMAPand send a patch to enable that.
Quoted 8 lines
Make sense to me!
Quoted 4 lines
That said I still think it's odd not to have the 8 byte case working here even if it isn't the right approach. Could still be useful for debug/testing without having to add the MIO instructions or the our special syscall.
Yes, this also makes sense, but this patch needs some adjusting
Yes, I think so too, falling back to 4-byte accesses of course if 8-byte is not available. Thanks,
Alex
Quoted 50 lines · Alex Williamson
Quoted 47 lines · On Fri, 19 Apr 2024 13:11:35 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 31 lines · Niklas Schnelle
Quoted 21 lines · Jason Gunthorpe
Quoted 12 lines · Gerd Bayer
From: Ben Segal <[email protected]>
Many PCI adapters can benefit or even require full 64bit read and write access to their registers. In order to enable work on
user-space drivers for these devices add two new variations vfio_pci_core_io{read|write}64 of the existing access methods when the architecture supports 64-bit ioreads and iowrites.Signed-off-by Ben Segal <[email protected]>Co-developed-by Gerd Bayer <[email protected]>Signed-off-by Gerd Bayer <[email protected]>Quoted 7 lines
---
Hi all,
we've successfully used this patch with a user-mode driver for a PCI device that requires 64bit register read/writes on s390.
Quoted 8 lines
But why? S390 already has a system call for userspace to do the 64 bit write, and newer S390 has a userspace instruction to do it.
Why would you want to use a VFIO system call on the mmio emulation path?
mmap the registers and access them normally?
Quoted 7 lines
It's a very good point and digging into why this wasn't used by Benjamin. It turns out
VFIO_PCI_MMAPis disabled for S390 which it really shouldn't be especially now that we have the user-space instructions. Before that though Benjamin turned to this interface which then lead him to this limitation. So yeah we'll definitely verify that it also works viaVFIO_PCI_MMAPand send a patch to enable that.Make sense to me!
Quoted 4 lines
That said I still think it's odd not to have the 8 byte case working here even if it isn't the right approach. Could still be useful for debug/testing without having to add the MIO instructions or the our special syscall.
Quoted 4 lines
Quoted 1 line
Yes, this also makes sense, but this patch needs some adjusting
Yes, I think so too, falling back to 4-byte accesses of course if 8-byte is not available. Thanks,
So I'll rework this to simply fall back to 32-bit if 64-bit is not available in a v2. And we'll investigate the VFIO_PCI_MMAP case separately.
Thank you,
Gerd