target/xtensa: rework zero overhead loops implementation

Don't invalidate TB with the end of zero overhead loop when LBEG or LEND change. Instead encode the distance from the TB start to the LEND in the TB flags and generate loopback code when offset of the next PC from the TB start equals that distance. Distance not greater than the maximal instruction length is encoded literally, greater distances are capped at the target page size and encoded as the maximal instruction length plus the greatest power of 2 that is not bigger than the distance.

Although this change adds dynamic TB search at the end of each zero overhead loop the resulting emulation speed is about 10% higher in uClibc-ng and LTP tests.

---
 dtc                       |  2 +-
 target/xtensa/cpu.h       | 11 ++++++++++
 target/xtensa/helper.h    |  2 --
 target/xtensa/op_helper.c | 24 ---------------------
 target/xtensa/translate.c | 53 ++++++++++++++++++++---------------------------
 5 files changed, 34 insertions(+), 58 deletions(-)

index 88f18909db73..e54388015af1 160000
@@ -1 +1 @@
-Subproject commit 88f18909db731a627456f26d779445f84e449536
+Subproject commit e54388015af1fb4bf04d0bca99caba1074d9cc42

index 34e5ccd9f1d6..ad76d75aadde 100644
@@ -694,6 +694,8 @@ static inline int cpu_mmu_index(CPUXtensaState *env, bool ifetch)
 #define XTENSA_TBFLAG_CWOE 0x40000
 #define XTENSA_TBFLAG_CALLINC_MASK 0x180000
 #define XTENSA_TBFLAG_CALLINC_SHIFT 19
+#define XTENSA_TBFLAG_LEND_MASK 0xfe00000
+#define XTENSA_TBFLAG_LEND_SHIFT 21
 
 static inline void cpu_get_tb_cpu_state(CPUXtensaState *env, target_ulong *pc,
         target_ulong *cs_base, uint32_t *flags)

@@ -706,6 +708,15 @@ static inline void cpu_get_tb_cpu_state(CPUXtensaState *env, target_ulong *pc,
     *flags |= xtensa_get_ring(env);
     if (env->sregs[PS] & PS_EXCM) {
         *flags |= XTENSA_TBFLAG_EXCM;
+    } else if (xtensa_option_enabled(env->config, XTENSA_OPTION_LOOP)) {
+        target_ulong lend_dist = env->sregs[LEND] - env->pc;
+
+        if (lend_dist > (1u << TARGET_PAGE_BITS)) {
+            lend_dist = MAX_INSN_LENGTH + 31 - TARGET_PAGE_BITS;
+        } else if (lend_dist > MAX_INSN_LENGTH) {
+            lend_dist = MAX_INSN_LENGTH + 31 - clz32(lend_dist);
+        }
+        *flags |= lend_dist << XTENSA_TBFLAG_LEND_SHIFT;
     }
     if (xtensa_option_enabled(env->config, XTENSA_OPTION_EXTENDED_L32R) &&
             (env->sregs[LITBASE] & 1)) {

index 10153c245360..2ebba0b2c2bf 100644
@@ -12,8 +12,6 @@ DEF_HELPER_2(rotw, void, env, i32)
 DEF_HELPER_3(window_check, noreturn, env, i32, i32)
 DEF_HELPER_1(restore_owb, void, env)
 DEF_HELPER_2(movsp, void, env, i32)
-DEF_HELPER_2(wsr_lbeg, void, env, i32)
-DEF_HELPER_2(wsr_lend, void, env, i32)
 #ifndef CONFIG_USER_ONLY
 DEF_HELPER_1(simcall, void, env)
 #endif

index e4b42ab3e56c..078aeb6c2c94 100644
@@ -107,13 +107,6 @@ static void tb_invalidate_virtual_addr(CPUXtensaState *env, uint32_t vaddr)
     }
 }
 
-#else
-
-static void tb_invalidate_virtual_addr(CPUXtensaState *env, uint32_t vaddr)
-{
-    tb_invalidate_phys_addr(vaddr);
-}
-
 #endif
 
 void HELPER(exception)(CPUXtensaState *env, uint32_t excp)

@@ -370,23 +363,6 @@ void HELPER(movsp)(CPUXtensaState *env, uint32_t pc)
     }
 }
 
-void HELPER(wsr_lbeg)(CPUXtensaState *env, uint32_t v)
-{
-    if (env->sregs[LBEG] != v) {
-        tb_invalidate_virtual_addr(env, env->sregs[LEND] - 1);
-        env->sregs[LBEG] = v;
-    }
-}
-
-void HELPER(wsr_lend)(CPUXtensaState *env, uint32_t v)
-{
-    if (env->sregs[LEND] != v) {
-        tb_invalidate_virtual_addr(env, env->sregs[LEND] - 1);
-        env->sregs[LEND] = v;
-        tb_invalidate_virtual_addr(env, env->sregs[LEND] - 1);
-    }
-}
-
 void HELPER(dump_state)(CPUXtensaState *env)
 {
     XtensaCPU *cpu = xtensa_env_get_cpu(env);

2.11.0

I thought about it some more and it looks like this is not going to work in general case in the presence of TB linking: a block with a big (and thus not precise) LEND distance may be linked to a block with a small (and thus precise) LEND distance. Then LEND may change so that next time it still goes to the first TB. In that case it shouldn't go from the first TB to the second, but with this scheme it will.

Thanks.
-- Max

Indeed. Perhaps think of ways in which LBEG and LEND can be represented relative to each other and PC and store than in the 32-bits you have available in the CS_BASE field.

Think first about how, if PC >= LEND or LEND - PC > PAGE_SIZE, that all of the loop stuff is irrelevant because we won't hit LEND within this TB.

Think second about how to represent the common case -- how LOOP sets LBEG and LEND together. That is, LEND - LBEG <= 256. So, usually, we can also have an exact representation of LBEG and have a direct link rather than an indirect link. But I presume that one can play games with special registers to create ranges that LOOP won't. So we need some setting that will indicate that.

Consider CS_BASE fields:

[12: 0] EDIF = LEND - PC, if PC < LEND && LEND - PC < 2*PAGE_SIZE, or 0. [20:13] BDIF = LEND - LBEG, if LEND - LBEG < 256, or 0.

So you can tell if advancing PC within a TB will exactly match LEND. You can tell what LBEG should be, except if BDIF == 0. In that, presumably rare case, you load LBEG at runtime as you did in this patch.

Note that if CS_BASE == 0, and thus EDIF == 0, looping is disabled for the TB.

I'll note that this also makes XTENSA_TBFLAG_EXCM redundant. Simply skip setting CS_BASE to a non-zero value instead.

r~

Hi Richard,

thank you for taking a look.

I tried similar thing as a followup to my original patch, recording entire LBEG in the cs_base -- surprisingly to me it made no measurable difference on my tests.

That has the same issue as my original patch: two TBs with PC > LEND in both may be linked together, and then LEND may change, so that PC1 > LEND and PC2 < LEND.
But I guess always storing low 13 bits of LEND - PC should work? I'll play with it...

Thanks.
-- Max

...when they're within two pages from each other...

Thanks.
-- Max