Skip to contentSkip to conversation

linux-user/mips: keep k0 in sync with CP0_UserLocal

PATCH In review James Hilliard 3 participants 9 messages Apr 8 – Apr 10

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 linux-user/mips: keep k0 in sync with CP0_UserLocal

MIPS glibc startup expects the thread pointer to be live in both CP0_UserLocal and k0. linux-user already updates CP0_UserLocal for cpu_set_tls() and TARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.

A guest can therefore successfully execute set_thread_area() and still reach __pthread_initialize_minimal() with k0 == 0, faulting on the first TLS-relative store.

Make cpu_set_tls() synchronize k0 with CP0_UserLocal, and route the MIPS set_thread_area syscall through cpu_set_tls() so all guest TLS updates follow the same path.

---
 linux-user/mips/target_cpu.h | 1 +
 linux-user/syscall.c         | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
linux-user/mips/target_cpu.h +1 -0
index c375616c55..0a000ac8f3 100644
@@ -36,6 +36,7 @@ static inline void cpu_clone_regs_parent(CPUMIPSState *env, unsigned flags)
 static inline void cpu_set_tls(CPUMIPSState *env, target_ulong newtls)
 {
     env->active_tc.CP0_UserLocal = newtls;
+    env->active_tc.gpr[26] = newtls;
 }
 
 static inline abi_ulong get_sp_from_cpustate(CPUMIPSState *state)
linux-user/syscall.c +1 -1
index f4b74ad350..8e96cc26db 100644
@@ -13216,7 +13216,7 @@ static abi_long do_syscall1(CPUArchState *cpu_env, int num, abi_long arg1,
 #ifdef TARGET_NR_set_thread_area
     case TARGET_NR_set_thread_area:
 #if defined(TARGET_MIPS)
-      cpu_env->active_tc.CP0_UserLocal = arg1;
+      cpu_set_tls(cpu_env, arg1);
       return 0;
 #elif defined(TARGET_I386) && defined(TARGET_ABI32)
       return do_set_thread_area(cpu_env, arg1);
Signature · 1 line
2.53.0
#1 lore raw
Quoted 6 lines · James Hilliard

MIPS glibc startup expects the thread pointer to be live in both CP0_UserLocal and k0. linux-user already updates CP0_UserLocal for cpu_set_tls() and TARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.

Quoted 7 lines

A guest can therefore successfully execute set_thread_area() and still reach __pthread_initialize_minimal() with k0 == 0, faulting on the first TLS-relative store.

Make cpu_set_tls() synchronize k0 with CP0_UserLocal, and route the MIPS set_thread_area syscall through cpu_set_tls() so all guest TLS updates follow the same path.

Where does the kernel do this? Looking at the mips set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it sets CP0_UserLocal plus the tp_value field in the thread_info struct; and I can't immediately see anywhere where it would read either CP0_UserLocal or tp_value and put it into k0.

I also couldn't find the MIPS glibc code that uses k0, but I didn't look very hard there as I don't know my way around the glibc sources -- do you have a pointer to it on that end?

thanks
-- PMM

#2 lore raw
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard

MIPS glibc startup expects the thread pointer to be live in both CP0_UserLocal and k0. linux-user already updates CP0_UserLocal for cpu_set_tls() and TARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.

Quoted 7 lines

A guest can therefore successfully execute set_thread_area() and still reach __pthread_initialize_minimal() with k0 == 0, faulting on the first TLS-relative store.

Make cpu_set_tls() synchronize k0 with CP0_UserLocal, and route the MIPS set_thread_area syscall through cpu_set_tls() so all guest TLS updates follow the same path.

Quoted 5 lines

Where does the kernel do this? Looking at the mips set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it sets CP0_UserLocal plus the tp_value field in the thread_info struct; and I can't immediately see anywhere where it would read either CP0_UserLocal or tp_value and put it into k0.

So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.

I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html

It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96

Quoted 3 lines

I also couldn't find the MIPS glibc code that uses k0, but I didn't look very hard there as I don't know my way around the glibc sources -- do you have a pointer to it on that end?

From Octeon SDK glibc/ports/sysdeps/mips/nptl/tls.h:
/* Note: rd must be $v1 to be ABI-conformant. */

# if !_MIPS_ARCH_OCTEON
# define READ_THREAD_POINTER() \
    ({ void *__result;                                  \
       asm volatile (".set\tpush\n\t.set\tmips32r2\n\t"                  \
             "rdhwr\t%0, $29\n\t.set\tpop" : "=v" (__result));          \
       __result; })
# else /* OCTEON */
/* The Kernel stores the value of "rdhwr v1,$29" in k0 ($26) register. And
   it is the Kernel's responsibility to always have the correct value in
   k0.  Replacing rdhwr instruction with k0, as this instruction needs to
   be emulated by the Kernel.  */
# define READ_THREAD_POINTER() ( { register void *__result asm
("$26"); __result; } )
# endif /* OCTEON */

From Octeon SDK glibc/ports/sysdeps/mips/tls-macros.h:

# if !_MIPS_ARCH_OCTEON
# define TLS_IE(x)                                      \
  ({ void *__result, *__tmp;                            \
     asm (".set push\n\t.set mips32r2\n\t"              \
          "rdhwr\t%0,$29\n\t.set pop"                   \
          : "=v" (__result));                           \
     asm (LOAD_GP LW " $3,%%gottprel(" #x ")($28)\n\t"  \
          ADDU " %0,%0,$3"                              \
          UNLOAD_GP                                     \
          : "+r" (__result), [tmp] "=&r" (__tmp)        \
          : : "$3");                                    \
     __result; })
# define TLS_LE(x)                                      \
  ({ void *__result;                                    \
     asm (".set push\n\t.set mips32r2\n\t"              \
          "rdhwr\t%0,$29\n\t.set pop"                   \
          : "=v" (__result));                           \
     asm ("lui $3,%%tprel_hi(" #x ")\n\t"               \
          "addiu $3,$3,%%tprel_lo(" #x ")\n\t"          \
          ADDU " %0,%0,$3"                              \
          : "+r" (__result) : : "$3");                  \
     __result; })
# else /* OCTEON */
/* On Octeon the kernel stores the value of the thread pointer in the k0 ($26)
   register.  */
# define TLS_IE(x)                                      \
  ({ void *__result, *__tmp;                            \
     asm (LOAD_GP LW " $3,%%gottprel(" #x ")($28)\n\t"  \
          ADDU " %0,$26,$3"                             \
          UNLOAD_GP                                     \
          : "+r" (__result), [tmp] "=&r" (__tmp)        \
          : : "$3");                                    \
     __result; })
# define TLS_LE(x)                                      \
  ({ void *__result;                                    \
     asm ("lui $3,%%tprel_hi(" #x ")\n\t"               \
          "addiu $3,$3,%%tprel_lo(" #x ")\n\t"          \
          ADDU " %0,$26,$3"                             \
          : "+r" (__result) : : "$3");                  \
     __result; })
# endif /* OCTEON */
Quoted 3 lines

thanks
-- PMM

#3 lore raw
Quoted 24 lines · James Hilliard
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard

MIPS glibc startup expects the thread pointer to be live in both CP0_UserLocal and k0. linux-user already updates CP0_UserLocal for cpu_set_tls() and TARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.

Quoted 7 lines

A guest can therefore successfully execute set_thread_area() and still reach __pthread_initialize_minimal() with k0 == 0, faulting on the first TLS-relative store.

Make cpu_set_tls() synchronize k0 with CP0_UserLocal, and route the MIPS set_thread_area syscall through cpu_set_tls() so all guest TLS updates follow the same path.

Quoted 5 lines

Where does the kernel do this? Looking at the mips set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it sets CP0_UserLocal plus the tp_value field in the thread_info struct; and I can't immediately see anywhere where it would read either CP0_UserLocal or tp_value and put it into k0.

Quoted 5 lines

So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.

I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html

Ah, nice find.

Quoted 2 lines

It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96

Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.

-- PMM

#4 lore raw
Quoted 32 lines · Peter Maydell
Quoted 24 lines · James Hilliard
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard

MIPS glibc startup expects the thread pointer to be live in both CP0_UserLocal and k0. linux-user already updates CP0_UserLocal for cpu_set_tls() and TARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.

Quoted 7 lines

A guest can therefore successfully execute set_thread_area() and still reach __pthread_initialize_minimal() with k0 == 0, faulting on the first TLS-relative store.

Make cpu_set_tls() synchronize k0 with CP0_UserLocal, and route the MIPS set_thread_area syscall through cpu_set_tls() so all guest TLS updates follow the same path.

Quoted 5 lines

Where does the kernel do this? Looking at the mips set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it sets CP0_UserLocal plus the tp_value field in the thread_info struct; and I can't immediately see anywhere where it would read either CP0_UserLocal or tp_value and put it into k0.

Quoted 5 lines

So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.

I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html

Quoted 7 lines

Ah, nice find.

Quoted 2 lines

It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96

Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.

Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?

Quoted 2 lines

-- PMM

#5 lore raw
Quoted 38 lines · James Hilliard
Quoted 37 lines · Peter Maydell
Quoted 24 lines · James Hilliard
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard

MIPS glibc startup expects the thread pointer to be live in both CP0_UserLocal and k0. linux-user already updates CP0_UserLocal for cpu_set_tls() and TARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.

Quoted 7 lines

A guest can therefore successfully execute set_thread_area() and still reach __pthread_initialize_minimal() with k0 == 0, faulting on the first TLS-relative store.

Make cpu_set_tls() synchronize k0 with CP0_UserLocal, and route the MIPS set_thread_area syscall through cpu_set_tls() so all guest TLS updates follow the same path.

Quoted 5 lines

Where does the kernel do this? Looking at the mips set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it sets CP0_UserLocal plus the tp_value field in the thread_info struct; and I can't immediately see anywhere where it would read either CP0_UserLocal or tp_value and put it into k0.

Quoted 5 lines

So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.

I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html

Ah, nice find.

Quoted 2 lines

It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96

Quoted 8 lines
Quoted 2 lines

Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.

Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?

What about (untested):

-- >8 --

linux-user/mips/target_cpu.h +3 -2
index 0a000ac8f35..c4aa08df4a3 100644
@@ -36,7 +36,9 @@ static inline void cpu_clone_regs_parent(CPUMIPSState *env, unsigned flags)
 static inline void cpu_set_tls(CPUMIPSState *env, target_ulong newtls)
 {
     env->active_tc.CP0_UserLocal = newtls;
-    env->active_tc.gpr[26] = newtls;
+    if (env->insn_flags & INSN_OCTEON) { /* Octeon1 */
+        env->active_tc.gpr[26] = newtls;
+    }
 }
---
 
#6 lore raw
Quoted 57 lines · Philippe Mathieu-Daudé
Quoted 38 lines · James Hilliard
Quoted 37 lines · Peter Maydell
Quoted 24 lines · James Hilliard
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard

MIPS glibc startup expects the thread pointer to be live in both CP0_UserLocal and k0. linux-user already updates CP0_UserLocal for cpu_set_tls() and TARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.

Quoted 7 lines

A guest can therefore successfully execute set_thread_area() and still reach __pthread_initialize_minimal() with k0 == 0, faulting on the first TLS-relative store.

Make cpu_set_tls() synchronize k0 with CP0_UserLocal, and route the MIPS set_thread_area syscall through cpu_set_tls() so all guest TLS updates follow the same path.

Quoted 5 lines

Where does the kernel do this? Looking at the mips set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it sets CP0_UserLocal plus the tp_value field in the thread_info struct; and I can't immediately see anywhere where it would read either CP0_UserLocal or tp_value and put it into k0.

Quoted 5 lines

So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.

I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html

Ah, nice find.

Quoted 2 lines

It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96

Quoted 8 lines
Quoted 2 lines

Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.

Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?

What about (untested):

-- >8 --

linux-user/mips/target_cpu.h +0 -0
index 0a000ac8f35..c4aa08df4a3 100644

linux-user/mips/target_cpu.h static inline void cpu_clone_regs_parent(CPUMIPSState · line 40

Quoted 6 lines
linux-user/mips/target_cpu.h +1 -1
@@ -36,7 +36,9 @@ static inline void cpu_clone_regs_parent(CPUMIPSState
 *env, unsigned flags)
  static inline void cpu_set_tls(CPUMIPSState *env, target_ulong newtls)
  {
      env->active_tc.CP0_UserLocal = newtls;
-    env->active_tc.gpr[26] = newtls;
+    if (env->insn_flags & INSN_OCTEON) { /* Octeon1 */

INST_OCTEON isn't OCTEON1 specific, but we could do something like this:
https://lore.kernel.org/qemu-devel/[email protected]/

Quoted 4 lines
linux-user/mips/target_cpu.h +2 -1
+        env->active_tc.gpr[26] = newtls;
+    }
  }
---
#7 lore raw
In reply to James Hilliard
Quoted 15 lines · James Hilliard
Quoted 13 lines · Peter Maydell
Quoted 5 lines · James Hilliard

So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.

I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html

Ah, nice find.

Quoted 2 lines

It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96

Quoted 8 lines
Quoted 2 lines

Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.

Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?

We didn't add this 15 years ago, and my feeling is we still don't want to, because we track the mainline kernel. If an actual host kernel doesn't have a quirk to let these binaries run, we don't need one either.

-- PMM

#8 lore raw
Quoted 26 lines · Peter Maydell
Quoted 15 lines · James Hilliard
Quoted 13 lines · Peter Maydell
Quoted 5 lines · James Hilliard

So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.

I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html

Ah, nice find.

Quoted 2 lines

It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96

Quoted 8 lines
Quoted 2 lines

Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.

Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?

Quoted 4 lines

We didn't add this 15 years ago, and my feeling is we still don't want to, because we track the mainline kernel. If an actual host kernel doesn't have a quirk to let these binaries run, we don't need one either.

I'm wondering, is there any actual downside to enabling this quirk for Octeon 1 binaries? It wouldn't actually interfere with binaries that don't require the quirk right?

Quoted 2 lines

-- PMM