Skip to contentSkip to conversation

accel/tcg: Optimize jump cache flush during tlb range flush

PATCH Applied Idan Horowitz 4 participants 9 messages Jan 10, 2022 – Jan 25, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 accel/tcg: Optimize jump cache flush during tlb range flush

When the length of the range is large enough, clearing the whole cache is faster than iterating over the (possibly extremely large) set of pages contained in the range.

This mimics the pre-existing similar optimization done on the flush of the tlb itself.

---
 accel/tcg/cputlb.c | 9 +++++++++
 1 file changed, 9 insertions(+)
accel/tcg/cputlb.c +9 -0
index 5e0d0eebc3..926d9a9192 100644
@@ -783,6 +783,15 @@ static void tlb_flush_range_by_mmuidx_async_0(CPUState *cpu,
     }
     qemu_spin_unlock(&env_tlb(env)->c.lock);
 
+    /*
+     * If the length is larger than the jump cache size, then it will take
+     * longer to clear each entry individually than it will to clear it all.
+     */
+    if (d.len >= (TARGET_PAGE_SIZE * TB_JMP_CACHE_SIZE)) {
+        cpu_tb_jmp_cache_clear(cpu);
+        return;
+    }
+
     for (target_ulong i = 0; i < d.len; i += TARGET_PAGE_SIZE) {
         tb_flush_jmp_cache(cpu, d.addr + i);
     }
Signature · 1 line
2.33.1
#1 lore raw

PATCH 2/2 target/arm: Bail out early on 0-length tlb range invalidate

If the given range specifies no addresses to be flushed there's no reason to schedule a function on all CPUs that does nothing.

---
 target/arm/helper.c | 4 ++++
 1 file changed, 4 insertions(+)
target/arm/helper.c +4 -0
index cfca0f5ba6..1e819835c2 100644
@@ -4564,6 +4564,10 @@ static void do_rvae_write(CPUARMState *env, uint64_t value,
     length = tlbi_aa64_range_get_length(env, value);
     bits = tlbbits_for_regime(env, one_idx, baseaddr);
 
+    if (length == 0) {
+        return;
+    }
+
     if (synced) {
         tlb_flush_range_by_mmuidx_all_cpus_synced(env_cpu(env),
                                                   baseaddr,
Signature · 1 line
2.33.1
Reviewed-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] target/arm: Bail out early on 0-length tlb range invalidate

Quoted 14 lines · Idan Horowitz

If the given range specifies no addresses to be flushed there's no reason to schedule a function on all CPUs that does nothing.

---
  target/arm/helper.c | 4 ++++
  1 file changed, 4 insertions(+)
target/arm/helper.c +0 -0
index cfca0f5ba6..1e819835c2 100644

target/arm/helper.c static void do_rvae_write(CPUARMState *env, uint64_t value, · line 4573

Quoted 10 lines
target/arm/helper.c +4 -0
@@ -4564,6 +4564,10 @@ static void do_rvae_write(CPUARMState *env, uint64_t value,
      length = tlbi_aa64_range_get_length(env, value);
      bits = tlbbits_for_regime(env, one_idx, baseaddr);
 
+    if (length == 0) {
+        return;
+    }
+
      if (synced) {
          tlb_flush_range_by_mmuidx_all_cpus_synced(env_cpu(env),
                                                    baseaddr,

Looks good. I guess we could sort the extractions above so that we do

    length = ...;
    if (length == 0) {
        return;
    }

    addr = ...
    bits = ...

Either way,

r~

#3 lore raw

Re: [PATCH 2/2] target/arm: Bail out early on 0-length tlb range invalidate

Quoted 37 lines · Richard Henderson
Quoted 14 lines · Idan Horowitz

If the given range specifies no addresses to be flushed there's no reason to schedule a function on all CPUs that does nothing.

---
  target/arm/helper.c | 4 ++++
  1 file changed, 4 insertions(+)
target/arm/helper.c +0 -0
index cfca0f5ba6..1e819835c2 100644

target/arm/helper.c static void do_rvae_write(CPUARMState *env, uint64_t value, · line 4573

Quoted 10 lines
target/arm/helper.c +4 -0
@@ -4564,6 +4564,10 @@ static void do_rvae_write(CPUARMState *env, uint64_t value,
      length = tlbi_aa64_range_get_length(env, value);
      bits = tlbbits_for_regime(env, one_idx, baseaddr);
 
+    if (length == 0) {
+        return;
+    }
+
      if (synced) {
          tlb_flush_range_by_mmuidx_all_cpus_synced(env_cpu(env),
                                                    baseaddr,

Looks good. I guess we could sort the extractions above so that we do

     length = ...;
     if (length == 0) {
         return;
     }

     addr = ...
     bits = ...
Quoted 2 lines

Either way,

Should we handle this in the tlb_flush_* functions themselves, or is it just Arm that has to fix up a special case of "actually the length is zero" ?

-- PMM

#4 lore raw

Re: [PATCH 2/2] target/arm: Bail out early on 0-length tlb range invalidate

Quoted 39 lines · Peter Maydell
Quoted 37 lines · Richard Henderson
Quoted 14 lines · Idan Horowitz

If the given range specifies no addresses to be flushed there's no reason to schedule a function on all CPUs that does nothing.

---
   target/arm/helper.c | 4 ++++
   1 file changed, 4 insertions(+)
target/arm/helper.c +0 -0
index cfca0f5ba6..1e819835c2 100644

target/arm/helper.c static void do_rvae_write(CPUARMState *env, uint64_t value, · line 4573

Quoted 10 lines
target/arm/helper.c +4 -0
@@ -4564,6 +4564,10 @@ static void do_rvae_write(CPUARMState *env, uint64_t value,
       length = tlbi_aa64_range_get_length(env, value);
       bits = tlbbits_for_regime(env, one_idx, baseaddr);
 
+    if (length == 0) {
+        return;
+    }
+
       if (synced) {
           tlb_flush_range_by_mmuidx_all_cpus_synced(env_cpu(env),
                                                     baseaddr,

Looks good. I guess we could sort the extractions above so that we do

      length = ...;
      if (length == 0) {
          return;
      }

      addr = ...
      bits = ...
Quoted 6 lines
Quoted 2 lines

Either way,

Should we handle this in the tlb_flush_* functions themselves, or is it just Arm that has to fix up a special case of "actually the length is zero" ?

Hmm. Probably should handle this in tlb_*, yes.
So far, Arm is the only user regardless.

r~

#5 lore raw
In reply to Idan Horowitz
Quoted 8 lines · Idan Horowitz

When the length of the range is large enough, clearing the whole cache is faster than iterating over the (possibly extremely large) set of pages contained in the range.

This mimics the pre-existing similar optimization done on the flush of the tlb itself.

For multi-patch series please include a cover letter which is the parent of all the patches. This is the default for git-send-email.

The code itself looks fine but what sort of improvements are we talking about here? What measurements have you taken and under what conditions?

Signature · 1 line
Alex Bennée
#6 lore raw
Quoted 5 lines · Alex Bennée

For multi-patch series please include a cover letter which is the parent of all the patches. This is the default for git-send-email.

Sorry, I will do so from now on.

Quoted 4 lines

The code itself looks fine but what sort of improvements are we talking about here? What measurements have you taken and under what conditions?

Execution time of the following assembly snippet in TCG aarch64-softmmu with icount (shift=10) enabled decreased from 4 minutes and 53 seconds to below 1 second:

movk    x0, #0x0
movk    x0, #0x0, lsl #16
movk    x0, #0xff80, lsl #32
movk    x0, #0x0, lsl #48
mov      x9, #0x64
str         x9, [x8]
tlbi        rvae1, x0
ldr         x9, [x8]

sub x9, x9, #0x1
str x9, [x8]
cbnz x9, 0x5168abc8

Quoted 3 lines

--
Alex Bennée

Idan Horowitz

#7 lore raw
Quoted 3 lines · Idan Horowitz

cbnz x9, 0x5168abc8

I forgot to include the addresses of the instructions, making this jump undecipherable, here's the snippet again but with addresses this time:

0x5168abb0 movk    x0, #0x0
0x5168abb4 movk    x0, #0x0, lsl #16
0x5168abb8 movk    x0, #0xff80, lsl #32
0x5168abbc movk    x0, #0x0, lsl #48
0x5168abc0 mov     x9, #0x64
0x5168abc4 str     x9, [x8]
0x5168abc8 tlbi    rvae1, x0
0x5168abcc ldr     x9, [x8]
0x5168abd0 sub     x9, x9, #0x1
0x5168abd4 str     x9, [x8]
0x5168abd8 cbnz    x9, 0x5168abc8
Quoted 2 lines

Idan Horowitz

Idan Horowitz

Reviewed-by on v1 1/2 Applied
#8 lore raw
In reply to Idan Horowitz

accel/tcg/cputlb.c static void tlb_flush_range_by_mmuidx_async_0(CPUState *cpu, · line 797

Quoted 13 lines · Idan Horowitz
accel/tcg/cputlb.c +9 -0
+    /*
+     * If the length is larger than the jump cache size, then it will take
+     * longer to clear each entry individually than it will to clear it all.
+     */
+    if (d.len >= (TARGET_PAGE_SIZE * TB_JMP_CACHE_SIZE)) {
+        cpu_tb_jmp_cache_clear(cpu);
+        return;
+    }
+
      for (target_ulong i = 0; i < d.len; i += TARGET_PAGE_SIZE) {
          tb_flush_jmp_cache(cpu, d.addr + i);
      }

Hmm. Valid. It looks like we could reasonably rewrite tb_flush_jmp_cache to be more efficient, by passing down len. But this is a decent intermediate step.

Queued to tcg-next.

r~