Skip to contentSkip to conversation

vfio/pci: Add iowrite64 and ioread64 support for vfio pci

PATCH Superseded Ramesh Thomas 3 participants 5 messages May 22, 2024 – Jun 4, 2024

Subsystems (3)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vfio/pci: Add iowrite64 and ioread64 support for vfio pci

ioread64 and iowrite64 macros called by vfio pci implementations are defined in asm/io.h if CONFIG_GENERIC_IOMAP is not defined. Include linux/io-64-nonatomic-lo-hi.h to define iowrite64 and ioread64 macros when they are not defined. io-64-nonatomic-lo-hi.h maps the macros to generic implementation in lib/iomap.c. The generic implementation does 64 bit rw if readq/writeq is defined for the architecture, otherwise it would do 32 bit back to back rw.

Note that there are two versions of the generic implementation that differs in the order the 32 bit words are written if 64 bit support is not present. This is not the little/big endian ordering, which is handled separately. This patch uses the lo followed by hi word ordering which is consistent with current back to back implementation in the vfio/pci code.

Refer patch series the requirement originated from:
https://lore.kernel.org/all/[email protected]/

---
 drivers/vfio/pci/vfio_pci_priv.h | 1 +
 1 file changed, 1 insertion(+)
drivers/vfio/pci/vfio_pci_priv.h +1 -0
index 5e4fa69aee16..5eab5abf2ff2 100644
@@ -3,6 +3,7 @@
 #define VFIO_PCI_PRIV_H
 
 #include <linux/vfio_pci_core.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
 
 /* Special capability IDs predefined access */
 #define PCI_CAP_ID_INVALID		0xFF	/* default raw access */
Signature · 1 line
2.34.1
#1 lore raw
Quoted 28 lines · Ramesh Thomas

ioread64 and iowrite64 macros called by vfio pci implementations are defined in asm/io.h if CONFIG_GENERIC_IOMAP is not defined. Include linux/io-64-nonatomic-lo-hi.h to define iowrite64 and ioread64 macros when they are not defined. io-64-nonatomic-lo-hi.h maps the macros to generic implementation in lib/iomap.c. The generic implementation does 64 bit rw if readq/writeq is defined for the architecture, otherwise it would do 32 bit back to back rw.

Note that there are two versions of the generic implementation that differs in the order the 32 bit words are written if 64 bit support is not present. This is not the little/big endian ordering, which is handled separately. This patch uses the lo followed by hi word ordering which is consistent with current back to back implementation in the vfio/pci code.

Refer patch series the requirement originated from:
https://lore.kernel.org/all/[email protected]/

---
 drivers/vfio/pci/vfio_pci_priv.h | 1 +
 1 file changed, 1 insertion(+)
drivers/vfio/pci/vfio_pci_priv.h +0 -0
index 5e4fa69aee16..5eab5abf2ff2 100644

drivers/vfio/pci/vfio_pci_priv.h line 6

Quoted 4 lines
drivers/vfio/pci/vfio_pci_priv.h +1 -0
@@ -3,6 +3,7 @@
 #define VFIO_PCI_PRIV_H
 
 #include <linux/vfio_pci_core.h>
+#include <linux/io-64-nonatomic-lo-hi.h>

Why include it here though?

It should go in vfio_pci_rdwr.c and this patch should remove all the "#ifdef iowrite64"'s from that file too.

But the idea looks right to me

Thanks,
Jason

#2 lore raw

Hi Jason,

Quoted 31 lines · Jason Gunthorpe
Quoted 30 lines · Ramesh Thomas

ioread64 and iowrite64 macros called by vfio pci implementations are defined in asm/io.h if CONFIG_GENERIC_IOMAP is not defined. Include linux/io-64-nonatomic-lo-hi.h to define iowrite64 and ioread64 macros when they are not defined. io-64-nonatomic-lo-hi.h maps the macros to generic implementation in lib/iomap.c. The generic implementation does 64 bit rw if readq/writeq is defined for the architecture, otherwise it would do 32 bit back to back rw.

Note that there are two versions of the generic implementation that differs in the order the 32 bit words are written if 64 bit support is not present. This is not the little/big endian ordering, which is handled separately. This patch uses the lo followed by hi word ordering which is consistent with current back to back implementation in the vfio/pci code.

Refer patch series the requirement originated from:
https://lore.kernel.org/all/[email protected]/

---
  drivers/vfio/pci/vfio_pci_priv.h | 1 +
  1 file changed, 1 insertion(+)
drivers/vfio/pci/vfio_pci_priv.h +0 -0
index 5e4fa69aee16..5eab5abf2ff2 100644
@@ -3,6 +3,7 @@
  #define VFIO_PCI_PRIV_H
 
Quoted 7 lines

drivers/vfio/pci/vfio_pci_priv.h line 6

Quoted 2 lines
drivers/vfio/pci/vfio_pci_priv.h +1 -0
  #include <linux/vfio_pci_core.h>
+#include <linux/io-64-nonatomic-lo-hi.h>

Why include it here though?

It should go in vfio_pci_rdwr.c and this patch should remove all the "#ifdef iowrite64"'s from that file too.

I was trying to make it future proof, but I agree it should be included only where iowrite64/ioread64 is getting called. I will make both the changes.

Thanks,
Ramesh

Quoted 5 lines

But the idea looks right to me

Thanks,
Jason

#3 lore raw

Hi Ramesh, hi Jason,

being back from a short vacation, I think I'm sold on enabling x86 for the 64bit accessors in vfio/pci.

Quoted 52 lines · Ramesh Thomas

Hi Jason,

Quoted 45 lines · Jason Gunthorpe
Quoted 38 lines · Ramesh Thomas

ioread64 and iowrite64 macros called by vfio pci implementations are
defined in asm/io.h if CONFIG_GENERIC_IOMAP is not defined. Include
linux/io-64-nonatomic-lo-hi.h to define iowrite64 and ioread64 macros
when they are not defined. io-64-nonatomic-lo-hi.h maps the macros to
generic implementation in lib/iomap.c. The generic implementation does 64 bit rw if readq/writeq is defined for the architecture, otherwise it would do 32 bit back to back rw.

Note that there are two versions of the generic implementation that
differs in the order the 32 bit words are written if 64 bit support is
not present. This is not the little/big endian ordering, which is handled separately. This patch uses the lo followed by hi word ordering
which is consistent with current back to back implementation in the
vfio/pci code.

Refer patch series the requirement originated from:
https://lore.kernel.org/all/[email protected]/

---
  drivers/vfio/pci/vfio_pci_priv.h | 1 +

  1 file changed, 1 insertion(+)
drivers/vfio/pci/vfio_pci_priv.h +0 -0
index 5e4fa69aee16..5eab5abf2ff2 100644
@@ -3,6 +3,7 @@
   #define VFIO_PCI_PRIV_H
Quoted 3 lines


  #include <linux/vfio_pci_core.h>
+#include <linux/io-64-nonatomic-lo-hi.h>

Why include it here though?

Quoted 2 lines

It should go in vfio_pci_rdwr.c and this patch should remove all the "#ifdef iowrite64"'s from that file too.

Quoted 12 lines

I was trying to make it future proof, but I agree it should be included only where iowrite64/ioread64 is getting called. I will make both the changes.

Thanks,
Ramesh

Quoted 5 lines

But the idea looks right to me

Thanks,
Jason

So how should we go about this?
To keep the scope that I can test manageable, my proposal would be:

I'll post a v5 of my series with the conditional compiles for "ioread64"/"iowrite64" (effectively still excluding x86) - and you Ramesh run this patch (add the include + change #ifdef's) as an explicit patch on top?

Thanks,
Gerd

#4 lore raw
Quoted 71 lines · Gerd Bayer

Hi Ramesh, hi Jason,

being back from a short vacation, I think I'm sold on enabling x86 for the 64bit accessors in vfio/pci.

Quoted 52 lines · Ramesh Thomas

Hi Jason,

Quoted 40 lines · Jason Gunthorpe
Quoted 39 lines · Ramesh Thomas

ioread64 and iowrite64 macros called by vfio pci implementations are
defined in asm/io.h if CONFIG_GENERIC_IOMAP is not defined. Include
linux/io-64-nonatomic-lo-hi.h to define iowrite64 and ioread64 macros
when they are not defined. io-64-nonatomic-lo-hi.h maps the macros to
generic implementation in lib/iomap.c. The generic implementation does 64 bit rw if readq/writeq is defined for the architecture, otherwise it would do 32 bit back to back rw.

Note that there are two versions of the generic implementation that
differs in the order the 32 bit words are written if 64 bit support is
not present. This is not the little/big endian ordering, which is handled separately. This patch uses the lo followed by hi word ordering
which is consistent with current back to back implementation in the
vfio/pci code.

Refer patch series the requirement originated from:
https://lore.kernel.org/all/[email protected]/

---
   drivers/vfio/pci/vfio_pci_priv.h | 1 +
   1 file changed, 1 insertion(+)
drivers/vfio/pci/vfio_pci_priv.h +0 -0
index 5e4fa69aee16..5eab5abf2ff2 100644
@@ -3,6 +3,7 @@
   #define VFIO_PCI_PRIV_H
 
Quoted 7 lines

drivers/vfio/pci/vfio_pci_priv.h line 6

Quoted 2 lines
drivers/vfio/pci/vfio_pci_priv.h +1 -0
   #include <linux/vfio_pci_core.h>
+#include <linux/io-64-nonatomic-lo-hi.h>

Why include it here though?

It should go in vfio_pci_rdwr.c and this patch should remove all the "#ifdef iowrite64"'s from that file too.

Quoted 12 lines

I was trying to make it future proof, but I agree it should be included only where iowrite64/ioread64 is getting called. I will make both the changes.

Thanks,
Ramesh

Quoted 5 lines

But the idea looks right to me

Thanks,
Jason

Quoted 7 lines

So how should we go about this?
To keep the scope that I can test manageable, my proposal would be:

I'll post a v5 of my series with the conditional compiles for "ioread64"/"iowrite64" (effectively still excluding x86) - and you Ramesh run this patch (add the include + change #ifdef's) as an explicit patch on top?

Hi Gerd,

Sounds good. Meanwhile I am also trying to get this tested in x86.

Thanks,
Ramesh

Quoted 4 lines

Thanks,
Gerd