util: optimise flush_idcache_range when the ppc host has coherent icache
PATCH 1/1 util: optimise flush_idcache_range when the ppc host has coherent icache
dcache writeback and icache invalidate is not required when icache is coherent, a shorter fixed-length sequence can be used which just has to flush and re-fetch instructions that were in-flight.
---
I haven't been able to measure a significant performance difference with this, qemu isn't flushing large ranges frequently so the old sequence is not that slow.
include/qemu/cacheflush.h | 4 ++++ util/cacheflush.c | 9 +++++++++ util/cacheinfo.c | 16 ++++++++++++++-- 3 files changed, 27 insertions(+), 2 deletions(-)
include/qemu/cacheflush.h +4 -0
@@ -28,6 +28,10 @@ static inline void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)#else +#if defined(__powerpc__) +extern bool have_coherent_icache; +#endif +void flush_idcache_range (uintptr_t rx,uintptr_t rw,size_t len);#endif
util/cacheflush.c +9 -0
@@ -108,7 +108,16 @@ void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)size_t isize = qemu_icache_linesize; b = rw & ~(dsize -1 ); + +if (have_coherent_icache) { + asmvolatile ("sync" : : :"memory" ); + asmvolatile ("icbi 0,%0" : :"r" (b) :"memory" ); + asmvolatile ("isync" : : :"memory" ); +return ; + } + e = (rw + len + dsize -1 ) & ~(dsize -1 ); +for (p = b; p < e; p += dsize) { asmvolatile ("dcbst 0,%0" : :"r" (p) :"memory" ); }
util/cacheinfo.c +14 -2
@@ -133,18 +133,30 @@ static void arch_cache_info(int *isize, int *dsize) } } -#elif defined(_ARCH_PPC) && defined(__linux__) -# include "elf.h" +#elif defined(__powerpc__) +bool have_coherent_icache = false; + +# if defined(_ARCH_PPC) && defined(__linux__) +# include "elf.h" static void arch_cache_info (int *isize,int *dsize) { +# ifdef PPC_FEATURE_ICACHE_SNOOP +unsigned long hwcap =qemu_getauxval (AT_HWCAP ); +# endif +if (*isize ==0 ) { *isize =qemu_getauxval (AT_ICACHEBSIZE ); }if (*dsize ==0 ) { *dsize =qemu_getauxval (AT_DCACHEBSIZE ); } + +# ifdef PPC_FEATURE_ICACHE_SNOOP + have_coherent_icache = (hwcap &PPC_FEATURE_ICACHE_SNOOP ) !=0 ; +# endif } +# endif #else static void arch_cache_info (int *isize,int *dsize) { }
Signature · 1 line
2.35.1
Quoted 10 lines · Nicholas Piggin
dcache writeback and icache invalidate is not required when icache is coherent, a shorter fixed-length sequence can be used which just has to flush and re-fetch instructions that were in-flight.
Signed-off-by Nicholas Piggin <[email protected]>---
I haven't been able to measure a significant performance difference with this, qemu isn't flushing large ranges frequently so the old sequence is not that slow.
Yeah, we should be flushing smallish regions (< 1-4k), as we generate TranslationBlocks. And hopefully the translation cache is large enough that we spend more time executing blocks than re-compiling them. ;-)
include/qemu/cacheflush.h static inline void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) · line 33
Quoted 8 lines
+++
b/include/qemu/cacheflush.hinclude/qemu/cacheflush.h +3
-0@@ -28,6 +28,10 @@ static inline void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)#else +#if defined(__powerpc__) +extern bool have_coherent_icache; +#endif
Ug. I'm undecided where to put this. I'm tempted to say...
util/cacheflush.c void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)
Quoted 3 lines
util/cacheflush.c +0
-0@@ -108,7 +108,16 @@ void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)
... here in cacheflush.c, with a comment that the variable is defined and initialized in cacheinfo.c.
I'm even more tempted to merge the two files to put all of the machine-specific cache data in the same place, then this variable can be static. There's even an existing TODO comment in cacheflush.c for aarch64.
util/cacheflush.c void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) · line 117
Quoted 8 lines
util/cacheflush.c +7
-0b = rw & ~(dsize -1 ); + +if (have_coherent_icache) { + asmvolatile ("sync" : : :"memory" ); + asmvolatile ("icbi 0,%0" : :"r" (b) :"memory" ); + asmvolatile ("isync" : : :"memory" ); +return ; + }
Where can I find definitive rules on this?
Note that rx may not equal rw, and that we've got two virtual mappings for the same memory, one for "data" that is read-write and one for "execute" that is read-execute. (This split is enabled only for --enable-debug-tcg builds on linux, to make sure we don't regress apple m1, which requires the split all of the time.)
In particular, you're flushing one icache line with the dcache address, and that you're not flushing any of the other lines. Is the coherent icache thing really that we may simply skip the dcache flush step, but must still flush all of the icache lines?
Without docs, "icache snoop" to me would imply that we only need the two barriers and no flushes at all, just to make sure all memory writes complete before any new instructions are executed. This would be like the two AArch64 bits, IDC and DIC, which indicate that the two caches are coherent to Point of Unification, which leaves us with just the Instruction Sequence Barrier at the end of the function.
util/cacheflush.c void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) · line 116
Quoted 1 line
util/cacheflush.c +1
-0+bool have_coherent_icache = false;
scripts/checkpatch.pl should complain this is initialized to 0.
Quoted 6 lines
util/cacheflush.c +4
-0static void arch_cache_info (int *isize,int *dsize) { +# ifdef PPC_FEATURE_ICACHE_SNOOP +unsigned long hwcap =qemu_getauxval (AT_HWCAP ); +# endif +
util/cacheflush.c void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)
Quoted 10 lines
util/cacheflush.c +4
-0if (*isize ==0 ) { *isize =qemu_getauxval (AT_ICACHEBSIZE ); }if (*dsize ==0 ) { *dsize =qemu_getauxval (AT_DCACHEBSIZE ); } + +# ifdef PPC_FEATURE_ICACHE_SNOOP + have_coherent_icache = (hwcap &PPC_FEATURE_ICACHE_SNOOP ) !=0 ; +# endif
Better with only one ifdef, moving this second hunk up.
It would be nice if there were some kernel documentation for this...
r~
Excerpts from Richard Henderson's message of May 20, 2022 4:31 am:
Quoted 35 lines · Nicholas Piggin
Quoted 10 lines
dcache writeback and icache invalidate is not required when icache is coherent, a shorter fixed-length sequence can be used which just has to flush and re-fetch instructions that were in-flight.
Signed-off-by Nicholas Piggin <[email protected]>---
I haven't been able to measure a significant performance difference with this, qemu isn't flushing large ranges frequently so the old sequence is not that slow.
Yeah, we should be flushing smallish regions (< 1-4k), as we generate TranslationBlocks. And hopefully the translation cache is large enough that we spend more time executing blocks than re-compiling them. ;-)
include/qemu/cacheflush.h static inline void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) · line 33
Quoted 8 lines
+++
b/include/qemu/cacheflush.hinclude/qemu/cacheflush.h +3
-0@@ -28,6 +28,10 @@ static inline void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)#else +#if defined(__powerpc__) +extern bool have_coherent_icache; +#endif Ug. I'm undecided where to put this. I'm tempted to say...
util/cacheflush.c void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)
Quoted 3 lines
util/cacheflush.c +0
-0@@ -108,7 +108,16 @@ void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len)... here in
cacheflush.c, with a comment that the variable is defined and initialized incacheinfo.c.
Quoted 3 lines
I'm even more tempted to merge the two files to put all of the machine-specific cache data in the same place, then this variable can be static. There's even an existing TODO comment in
cacheflush.cfor aarch64.
That might be nice. Do you want me to look at doing that first?
Quoted 10 lines
util/cacheflush.c void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) · line 117
Quoted 8 lines
util/cacheflush.c +7
-0b = rw & ~(dsize -1 ); + +if (have_coherent_icache) { + asmvolatile ("sync" : : :"memory" ); + asmvolatile ("icbi 0,%0" : :"r" (b) :"memory" ); + asmvolatile ("isync" : : :"memory" ); +return ; + }Where can I find definitive rules on this?
In processor manuals (I don't know if there are any notes about this in the ISA, I would be tempted to say there should be since many processors implement it).
POWER9 UM, 4.6.2.2 Instruction Cache Block Invalidate (icbi)
https://ibm.ent.box.com/s/tmklq90ze7aj8f4n32er1mu3sy9u8k3k
Quoted 8 lines
Note that rx may not equal rw, and that we've got two virtual mappings for the same memory, one for "data" that is read-write and one for "execute" that is read-execute. (This split is enabled only for --enable-debug-tcg builds on linux, to make sure we don't regress apple m1, which requires the split all of the time.)
In particular, you're flushing one icache line with the dcache address, and that you're not flushing any of the other lines. Is the coherent icache thing really that we may simply skip the dcache flush step, but must still flush all of the icache lines?
Yeah it's just a funny sequence the processor implements. It treats icbi almost as a no-op except that it sets a flag such that the next isync will flush and refetch the pipeline. It doesn't do any cache flushing.
Quoted 22 lines
Without docs, "icache snoop" to me would imply that we only need the two barriers and no flushes at all, just to make sure all memory writes complete before any new instructions are executed. This would be like the two AArch64 bits, IDC and DIC, which indicate that the two caches are coherent to Point of Unification, which leaves us with just the Instruction Sequence Barrier at the end of the function.
util/cacheflush.c void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) · line 116
Quoted 1 line
util/cacheflush.c +1
-0+bool have_coherent_icache = false;
scripts/checkpatch.plshould complain this is initialized to 0.Quoted 10 lines
util/cacheflush.c +4
-0static void arch_cache_info (int *isize,int *dsize) { +# ifdef PPC_FEATURE_ICACHE_SNOOP +unsigned long hwcap =qemu_getauxval (AT_HWCAP ); +# endif +if (*isize ==0 ) { *isize =qemu_getauxval (AT_ICACHEBSIZE ); }if (*dsize ==0 ) {
Quoted 8 lines
util/cacheflush.c
Quoted 6 lines
*dsize = qemu_getauxval(AT_DCACHEBSIZE); }util/cacheflush.c +4
-0+ +# ifdef PPC_FEATURE_ICACHE_SNOOP + have_coherent_icache = (hwcap &PPC_FEATURE_ICACHE_SNOOP ) !=0 ; +# endif Better with only one ifdef, moving this second hunk up.
Will clean those bits up, thanks.
Quoted 1 line
It would be nice if there were some kernel documentation for this...
arm64 has kernel docs for hwcaps... powerpc probably should as well. Good point, I might do a patch for that.
Thanks,
Nick
Quoted 5 lines · Nicholas Piggin
Quoted 3 lines
I'm even more tempted to merge the two files to put all of the machine-specific cache data in the same place, then this variable can be static. There's even an existing TODO comment in
cacheflush.cfor aarch64.That might be nice. Do you want me to look at doing that first?
If you wouldn't mind, please do. I'll take care of aarch64 cleanup related to the TODO afterward.
Quoted 18 lines
Quoted 10 lines
util/cacheflush.c void flush_idcache_range(uintptr_t rx, uintptr_t rw, size_t len) · line 117
Quoted 8 lines
b = rw & ~(dsize - 1);util/cacheflush.c +7
-0+ +if (have_coherent_icache) { + asmvolatile ("sync" : : :"memory" ); + asmvolatile ("icbi 0,%0" : :"r" (b) :"memory" ); + asmvolatile ("isync" : : :"memory" ); +return ; + }Where can I find definitive rules on this?
In processor manuals (I don't know if there are any notes about this in the ISA, I would be tempted to say there should be since many processors implement it).
POWER9 UM, 4.6.2.2 Instruction Cache Block Invalidate (icbi)
...
Quoted 3 lines
Yeah it's just a funny sequence the processor implements. It treats icbi almost as a no-op except that it sets a flag such that the next isync will flush and refetch the pipeline. It doesn't do any cache flushing.
Thanks. A short comment in the code would be helpful here.
Also, since the docs say "any address", you might as well just use rx unmodified and sink the computation of 'b' back next to 'e'.
r~