Add io_sync stubs to generic IO memcpy/memset
PATCH 1/1 Add io_sync stubs to generic IO memcpy/memset
The recently added IO memcpy and memset functions lack support for barriers or other sync functions before and/or after the transaction. To convert more architectures to use the generic IO memcpy and memset functions, add empty __pre_io_sync and __post_io_sync defines that can be overwritten by individual architectures if needed.
--- lib/iomem_copy.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
lib/iomem_copy.c +20 -0
@@ -9,6 +9,14 @@#include <linux/types.h> #include <linux/unaligned.h> +#ifndef __pre_io_sync +#define __pre_io_sync +#endif + +#ifndef __post_io_sync +#define __post_io_sync +#endif +#ifndef memset_io/** * memset_io() - Set a range of I/O memory to a constant value
@@ -24,6 +32,8 @@ void memset_io(volatile void __iomem *addr, int val, size_t count) qc *= ~0UL /0xff ; + __pre_io_sync; +while (count && !IS_ALIGNED ((long )addr,sizeof (long ))) {__raw_writeb (val, addr); addr++;
@@ -46,6 +56,8 @@ void memset_io(volatile void __iomem *addr, int val, size_t count) addr++; count--; } + + __post_io_sync; }EXPORT_SYMBOL (memset_io);#endif
@@ -61,6 +73,8 @@ EXPORT_SYMBOL(memset_io);*/ void memcpy_fromio (void *dst,const volatile void __iomem *src,size_t count) { + __pre_io_sync; +while (count && !IS_ALIGNED ((long )src,sizeof (long ))) { *(u8 *)dst =__raw_readb (src); src++;
@@ -88,6 +102,8 @@ void memcpy_fromio(void *dst, const volatile void __iomem *src, size_t count) dst++; count--; } + +__post_io_sync }EXPORT_SYMBOL (memcpy_fromio );#endif
@@ -103,6 +119,8 @@ EXPORT_SYMBOL(memcpy_fromio);*/ void memcpy_toio (volatile void __iomem *dst,const void *src,size_t count) { + __pre_io_sync; +while (count && !IS_ALIGNED ((long )dst,sizeof (long ))) {__raw_writeb (*(u8 *)src, dst); src++;
@@ -129,6 +147,8 @@ void memcpy_toio(volatile void __iomem *dst, const void *src, size_t count) dst++; count--; } + + __post_io_sync; }EXPORT_SYMBOL (memcpy_toio);#endif
Signature · 1 line
2.34.1
Quoted 16 lines · Julian Vetter
The recently added IO memcpy and memset functions lack support for barriers or other sync functions before and/or after the transaction. To convert more architectures to use the generic IO memcpy and memset functions, add empty
__pre_io_syncand__post_io_syncdefines that can be overwritten by individual architectures if needed.Signed-off-by Julian Vetter <[email protected]>--- lib/iomem_copy.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)lib/iomem_copy.c +0
-0
lib/iomem_copy.c line 18
Quoted 10 lines
lib/iomem_copy.c +7
-0@@ -9,6 +9,14 @@#include <linux/types.h> #include <linux/unaligned.h> +#ifndef __pre_io_sync +#define __pre_io_sync +#endif + +#ifndef __post_io_sync +#define __post_io_sync +#endif
I think we should define what these barriers are supposed to do exactly, and how they relate to the __io_br/__io_ar/__io_bw/__io_aw ones include/asm-generic/io.h.
Depending on what the barriers are meant to do, we probably want to either use the existing ones directly or use a similar naming scheme.
Arnd
Quoted 20 lines · Arnd Bergmann
Quoted 19 lines · Julian Vetter
The recently added IO memcpy and memset functions lack support for barriers or other sync functions before and/or after the transaction. To convert more architectures to use the generic IO memcpy and memset functions, add empty
__pre_io_syncand__post_io_syncdefines that can be overwritten by individual architectures if needed.Signed-off-by Julian Vetter <[email protected]>--- lib/iomem_copy.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)lib/iomem_copy.c +0
-0@@ -9,6 +9,14 @@#include <linux/types.h> #include <linux/unaligned.h>
Quoted 12 lines
lib/iomem_copy.c line 18
Quoted 7 lines
lib/iomem_copy.c +7
-0+#ifndef __pre_io_sync +#define __pre_io_sync +#endif + +#ifndef __post_io_sync +#define __post_io_sync +#endif I think we should define what these barriers are supposed to do exactly, and how they relate to the __io_br/__io_ar/__io_bw/__io_aw ones
include/asm-generic/io.h.
Thank you for your quick reply. You're right, I was just going with the
naming used in the powerpc arch which has an io_sync define. I'm now
wondering if we can't simply use the read{l,q}/write{l,q} functions
(instead of the __raw_xxx version), there are already calls to __io_br
before and__io_ar after each read (and write). But this might have
performance implications on some architectures, depending what it
resolves to.
Otherwise I propose renaming the __pre_io_sync and __post_io_sync into a
single __io_mbr which is called before and after each loop. Looking at PowerPC and SuperH, both of them could be consolidated into the generic IO memcpy code when adding this. What do you think?
The existing ones, especially __io_br unfortunately don't resolve to the right define on these architectures. The __io_ar and __io_br resolve to the right mb() on SuperH and PowerPC as well, but this would again have implications on other architectures. Thank you!
Julian
Quoted 5 lines
Depending on what the barriers are meant to do, we probably want to either use the existing ones directly or use a similar naming scheme.
Arnd
Quoted 11 lines · Julian Vetter
Quoted 1 line · Arnd Bergmann
On Mon, Jan 27, 2025, at 11:04, Julian Vetter wrote:
Thank you for your quick reply. You're right, I was just going with the naming used in the powerpc arch which has an
io_syncdefine. I'm now wondering if we can't simply use the read{l,q}/write{l,q} functions (instead of the__raw_xxxversion), there are already calls to__io_brbeforeand__io_arafter each read (and write). But this might have performance implications on some architectures, depending what it resolves to.
Quoted 4 lines
Otherwise I propose renaming the
__pre_io_syncand__post_io_syncinto a single__io_mbrwhich is called before and after each loop. Looking at PowerPC and SuperH, both of them could be consolidated into the generic IO memcpy code when adding this. What do you think?
Having barriers between the accesses would be very expensive, and prevent the write-combining and prefetching that can otherwise happen (depending on mapping flags).
I suspect that the powerpc variant got this wrong for historic reasons, but that's hard to tell now. The ppc32 variant didn't have barriers at all originally, it was just memcpy/memset before it got combined with ppc64 into arch/powerpc.
Quoted 4 lines
The existing ones, especially
__io_brunfortunately don't resolve to the right define on these architectures. The__io_arand__io_brresolve to the rightmb()on SuperH and PowerPC as well, but this would again have implications on other architectures.
The barriers in the sh functions seem arbitrary, and I would expect them to be wrong.
ARnd
Quoted 12 lines · Arnd Bergmann
Quoted 11 lines · Julian Vetter
Quoted 1 line · Arnd Bergmann
On Mon, Jan 27, 2025, at 11:04, Julian Vetter wrote:
Thank you for your quick reply. You're right, I was just going with the naming used in the powerpc arch which has an
io_syncdefine. I'm now wondering if we can't simply use the read{l,q}/write{l,q} functions (instead of the__raw_xxxversion), there are already calls to__io_brbeforeand__io_arafter each read (and write). But this might have performance implications on some architectures, depending what it resolves to.
Quoted 8 lines
Quoted 4 lines
Otherwise I propose renaming the
__pre_io_syncand__post_io_syncinto a single__io_mbrwhich is called before and after each loop. Looking at PowerPC and SuperH, both of them could be consolidated into the generic IO memcpy code when adding this. What do you think?Having barriers between the accesses would be very expensive, and prevent the write-combining and prefetching that can otherwise happen (depending on mapping flags).
Yes, ok. I see.
Quoted 6 lines
I suspect that the powerpc variant got this wrong for historic reasons, but that's hard to tell now. The ppc32 variant didn't have barriers at all originally, it was just memcpy/memset before it got combined with ppc64 into
arch/powerpc.
hmmm... ok. I'm not sure what do make of this. But maybe I can just send a patch to the PowerPC mailinglist, without those "sync" calls and see what they have to say.
Quoted 8 lines
Quoted 4 lines
The existing ones, especially
__io_brunfortunately don't resolve to the right define on these architectures. The__io_arand__io_brresolve to the rightmb()on SuperH and PowerPC as well, but this would again have implications on other architectures.The barriers in the sh functions seem arbitrary, and I would expect them to be wrong.
Ok, same for SuperH, I will send a patch to the mailinglist without the 'mb()' before and after and see what they say. If they don't like it, I will come back to this.
Julian
Quoted 2 lines
ARnd
Quoted 10 lines · Julian Vetter
Quoted 5 lines · Arnd Bergmann
I suspect that the powerpc variant got this wrong for historic reasons, but that's hard to tell now. The ppc32 variant didn't have barriers at all originally, it was just memcpy/memset before it got combined with ppc64 into
arch/powerpc.hmmm... ok. I'm not sure what do make of this. But maybe I can just send a patch to the PowerPC mailinglist, without those "sync" calls and see what they have to say.
I just looked again and I see that the powerpc I/O memcpy/memset functions do use the same barriers as readl/writel -- sync before each one, twi/isync after read and sync after write, the only difference is the eieio in the middle, so you could start with a patch that removes the eieio. The question here is whether we want to allow or prevent combining and reordering accesses within the string operations, and my feeling is that allowing them makes more sense here, but there may be a powerpc specific reason we don't want that.
I'm still unsure about having barriers before/after the string operations, I can very much see that debate go either way, but I also feel like this should be done consistently across all architectures.
A possible answer may be that we declare these helpers to include the same barriers as readl_relaxed()/writel_relaxed(), i.e. ordering between I/O operations is enforced, but not the barriers for serializing against DMA and interrupts that is provided by the normal readl()/writel().
Most architectures assume that the relaxed variants don't need any barriers, so that is what asm-generic/io.h implements, but I think alpha and mips need barriers here and powerpc simply defines the relaxed accessors to be identical to the non-relaxed ones for simplicity.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
The existing ones, especially
__io_brunfortunately don't resolve to the right define on these architectures. The__io_arand__io_brresolve to the rightmb()on SuperH and PowerPC as well, but this would again have implications on other architectures.The barriers in the sh functions seem arbitrary, and I would expect them to be wrong.
Ok, same for SuperH, I will send a patch to the mailinglist without the '
mb()' before and after and see what they say. If they don't like it, I will come back to this.
I doubt there is anyone left who understands the history behind the exact implementation on sh, the only real options I see for it are to not touch it at all, or to remove the functions in favor of the generic ones.
sh also has custom readsl/writesl functions, which are related, but I see that it is completely missing the corresponding readsw/writesw and readsb/writesb interfaces and it prevents the use of the generic implementations.
Arnd