accel/tcg: Optimize jump cache flush during tlb range flush
PATCH 1/2 accel/tcg: Optimize jump cache flush during tlb range flush
When the length of the range is large enough, clearing the whole cache is faster than iterating over the (possibly extremely large) set of pages contained in the range.
This mimics the pre-existing similar optimization done on the flush of the tlb itself.
--- accel/tcg/cputlb.c | 9 +++++++++ 1 file changed, 9 insertions(+)
accel/tcg/cputlb.c +9 -0
@@ -783,6 +783,15 @@ static void tlb_flush_range_by_mmuidx_async_0(CPUState *cpu, }qemu_spin_unlock (&env_tlb (env)->c.lock); +/* +* If the length is larger than the jump cache size, then it will take +* longer to clear each entry individually than it will to clear it all. +*/ +if (d.len >= (TARGET_PAGE_SIZE *TB_JMP_CACHE_SIZE )) { +cpu_tb_jmp_cache_clear (cpu); +return ; + } +for (target_ulong i =0 ; i < d.len; i +=TARGET_PAGE_SIZE ) {tb_flush_jmp_cache (cpu, d.addr + i); }
Signature · 1 line
2.33.1
PATCH 2/2 target/arm: Bail out early on 0-length tlb range invalidate
If the given range specifies no addresses to be flushed there's no reason to schedule a function on all CPUs that does nothing.
--- target/arm/helper.c | 4 ++++ 1 file changed, 4 insertions(+)
target/arm/helper.c +4 -0
@@ -4564,6 +4564,10 @@ static void do_rvae_write(CPUARMState *env, uint64_t value, length =tlbi_aa64_range_get_length (env, value); bits =tlbbits_for_regime (env, one_idx, baseaddr); +if (length ==0 ) { +return ; + } +if (synced) { tlb_flush_range_by_mmuidx_all_cpus_synced(env_cpu(env), baseaddr,
Signature · 1 line
2.33.1
Re: [PATCH 2/2] target/arm: Bail out early on 0-length tlb range invalidate
Quoted 14 lines · Idan Horowitz
If the given range specifies no addresses to be flushed there's no reason to schedule a function on all CPUs that does nothing.
Signed-off-by Idan Horowitz <[email protected]>--- target/arm/helper.c | 4 ++++ 1 file changed, 4 insertions(+)target/arm/helper.c +0
-0
target/arm/helper.c static void do_rvae_write(CPUARMState *env, uint64_t value, · line 4573
Quoted 10 lines
target/arm/helper.c +4
-0@@ -4564,6 +4564,10 @@ static void do_rvae_write(CPUARMState *env, uint64_t value, length =tlbi_aa64_range_get_length (env, value); bits =tlbbits_for_regime (env, one_idx, baseaddr); +if (length ==0 ) { +return ; + } +if (synced) { tlb_flush_range_by_mmuidx_all_cpus_synced(env_cpu(env), baseaddr,
Looks good. I guess we could sort the extractions above so that we do
length = ...;
if (length == 0) {
return;
}
addr = ...
bits = ...Either way,
r~
Re: [PATCH 2/2] target/arm: Bail out early on 0-length tlb range invalidate
Quoted 37 lines · Richard Henderson
Quoted 14 lines · Idan Horowitz
If the given range specifies no addresses to be flushed there's no reason to schedule a function on all CPUs that does nothing.
Signed-off-by Idan Horowitz <[email protected]>--- target/arm/helper.c | 4 ++++ 1 file changed, 4 insertions(+)target/arm/helper.c +0
-0target/arm/helper.c static void do_rvae_write(CPUARMState *env, uint64_t value, · line 4573
Quoted 10 lines
target/arm/helper.c +4
-0@@ -4564,6 +4564,10 @@ static void do_rvae_write(CPUARMState *env, uint64_t value, length =tlbi_aa64_range_get_length (env, value); bits =tlbbits_for_regime (env, one_idx, baseaddr); +if (length ==0 ) { +return ; + } +if (synced) { tlb_flush_range_by_mmuidx_all_cpus_synced(env_cpu(env), baseaddr,Looks good. I guess we could sort the extractions above so that we do
length = ...; if (length == 0) { return; } addr = ... bits = ...
Quoted 2 lines
Either way,
Reviewed-by Richard Henderson <[email protected]>
Should we handle this in the tlb_flush_* functions themselves, or is it just Arm that has to fix up a special case of "actually the length is zero" ?
-- PMM
Re: [PATCH 2/2] target/arm: Bail out early on 0-length tlb range invalidate
Quoted 39 lines · Peter Maydell
Quoted 37 lines · Richard Henderson
Quoted 14 lines · Idan Horowitz
If the given range specifies no addresses to be flushed there's no reason to schedule a function on all CPUs that does nothing.
Signed-off-by Idan Horowitz <[email protected]>--- target/arm/helper.c | 4 ++++ 1 file changed, 4 insertions(+)target/arm/helper.c +0
-0target/arm/helper.c static void do_rvae_write(CPUARMState *env, uint64_t value, · line 4573
Quoted 10 lines
target/arm/helper.c +4
-0@@ -4564,6 +4564,10 @@ static void do_rvae_write(CPUARMState *env, uint64_t value, length =tlbi_aa64_range_get_length (env, value); bits =tlbbits_for_regime (env, one_idx, baseaddr); +if (length ==0 ) { +return ; + } +if (synced) { tlb_flush_range_by_mmuidx_all_cpus_synced(env_cpu(env), baseaddr,Looks good. I guess we could sort the extractions above so that we do
length = ...; if (length == 0) { return; } addr = ... bits = ...
Quoted 6 lines
Quoted 2 lines
Either way,
Reviewed-by Richard Henderson <[email protected]>Should we handle this in the tlb_flush_* functions themselves, or is it just Arm that has to fix up a special case of "actually the length is zero" ?
Hmm. Probably should handle this in tlb_*, yes.
So far, Arm is the only user regardless.
r~
Quoted 8 lines · Idan Horowitz
When the length of the range is large enough, clearing the whole cache is faster than iterating over the (possibly extremely large) set of pages contained in the range.
This mimics the pre-existing similar optimization done on the flush of the tlb itself.
Signed-off-by Idan Horowitz <[email protected]>
For multi-patch series please include a cover letter which is the parent of all the patches. This is the default for git-send-email.
The code itself looks fine but what sort of improvements are we talking about here? What measurements have you taken and under what conditions?
Signature · 1 line
Alex Bennée
Quoted 5 lines · Alex Bennée
For multi-patch series please include a cover letter which is the parent of all the patches. This is the default for git-send-email.
Sorry, I will do so from now on.
Quoted 4 lines
The code itself looks fine but what sort of improvements are we talking about here? What measurements have you taken and under what conditions?
Execution time of the following assembly snippet in TCG aarch64-softmmu with icount (shift=10) enabled decreased from 4 minutes and 53 seconds to below 1 second:
movk x0, #0x0 movk x0, #0x0, lsl #16 movk x0, #0xff80, lsl #32 movk x0, #0x0, lsl #48 mov x9, #0x64 str x9, [x8] tlbi rvae1, x0 ldr x9, [x8]
sub x9, x9, #0x1
str x9, [x8]
cbnz x9, 0x5168abc8
Quoted 3 lines
--
Alex Bennée
Idan Horowitz
Quoted 3 lines · Idan Horowitz
cbnz x9,
0x5168abc8
I forgot to include the addresses of the instructions, making this jump undecipherable, here's the snippet again but with addresses this time:
0x5168abb0 movk x0, #0x0 0x5168abb4 movk x0, #0x0, lsl #16 0x5168abb8 movk x0, #0xff80, lsl #32 0x5168abbc movk x0, #0x0, lsl #48 0x5168abc0 mov x9, #0x64 0x5168abc4 str x9, [x8] 0x5168abc8 tlbi rvae1, x0 0x5168abcc ldr x9, [x8] 0x5168abd0 sub x9, x9, #0x1 0x5168abd4 str x9, [x8] 0x5168abd8 cbnz x9, 0x5168abc8
Quoted 2 lines
Idan Horowitz
Idan Horowitz
accel/tcg/cputlb.c static void tlb_flush_range_by_mmuidx_async_0(CPUState *cpu, · line 797
Quoted 13 lines · Idan Horowitz
accel/tcg/cputlb.c +9
-0+/* +* If the length is larger than the jump cache size, then it will take +* longer to clear each entry individually than it will to clear it all. +*/ +if (d.len >= (TARGET_PAGE_SIZE *TB_JMP_CACHE_SIZE )) { +cpu_tb_jmp_cache_clear (cpu); +return ; + } +for (target_ulong i =0 ; i < d.len; i +=TARGET_PAGE_SIZE ) {tb_flush_jmp_cache (cpu, d.addr + i); }
Hmm. Valid. It looks like we could reasonably rewrite tb_flush_jmp_cache to be more efficient, by passing down len. But this is a decent intermediate step.
Queued to tcg-next.
r~