linux-user/mips: keep k0 in sync with CP0_UserLocal
PATCH 1/1 linux-user/mips: keep k0 in sync with CP0_UserLocal
MIPS glibc startup expects the thread pointer to be live in both CP0_UserLocal and k0. linux-user already updates CP0_UserLocal for cpu_set_tls() and TARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.
A guest can therefore successfully execute set_thread_area() and still reach __pthread_initialize_minimal() with k0 == 0, faulting on the first TLS-relative store.
Make cpu_set_tls() synchronize k0 with CP0_UserLocal, and route the MIPS set_thread_area syscall through cpu_set_tls() so all guest TLS updates follow the same path.
--- linux-user/mips/target_cpu.h | 1 + linux-user/syscall.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
linux-user/mips/target_cpu.h +1 -0
@@ -36,6 +36,7 @@ static inline void cpu_clone_regs_parent(CPUMIPSState *env, unsigned flags)static inline void cpu_set_tls (CPUMIPSState *env,target_ulong newtls) { env->active_tc.CP0_UserLocal = newtls; + env->active_tc.gpr[26 ] = newtls; }static inline abi_ulong get_sp_from_cpustate (CPUMIPSState *state)
linux-user/syscall.c +1 -1
@@ -13216,7 +13216,7 @@ static abi_long do_syscall1(CPUArchState *cpu_env, int num, abi_long arg1,#ifdef TARGET_NR_set_thread_areacase TARGET_NR_set_thread_area:#if defined(TARGET_MIPS ) - cpu_env->active_tc.CP0_UserLocal = arg1; +cpu_set_tls (cpu_env, arg1);return 0 ;#elif defined(TARGET_I386 ) && defined(TARGET_ABI32 )return do_set_thread_area (cpu_env, arg1);
Signature · 1 line
2.53.0
Quoted 6 lines · James Hilliard
MIPS glibc startup expects the thread pointer to be live in both
CP0_UserLocaland k0. linux-user already updatesCP0_UserLocalforcpu_set_tls()andTARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.
Quoted 7 lines
A guest can therefore successfully execute
set_thread_area()and still reach__pthread_initialize_minimal()with k0 == 0, faulting on the first TLS-relative store.Make
cpu_set_tls()synchronize k0 withCP0_UserLocal, and route the MIPSset_thread_areasyscall throughcpu_set_tls()so all guest TLS updates follow the same path.
Where does the kernel do this? Looking at the mips set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it sets CP0_UserLocal plus the tp_value field in the thread_info struct; and I can't immediately see anywhere where it would read either CP0_UserLocal or tp_value and put it into k0.
I also couldn't find the MIPS glibc code that uses k0, but I didn't look very hard there as I don't know my way around the glibc sources -- do you have a pointer to it on that end?
thanks
-- PMM
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard
MIPS glibc startup expects the thread pointer to be live in both
CP0_UserLocaland k0. linux-user already updatesCP0_UserLocalforcpu_set_tls()andTARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.Quoted 7 lines
A guest can therefore successfully execute
set_thread_area()and still reach__pthread_initialize_minimal()with k0 == 0, faulting on the first TLS-relative store.Make
cpu_set_tls()synchronize k0 withCP0_UserLocal, and route the MIPSset_thread_areasyscall throughcpu_set_tls()so all guest TLS updates follow the same path.
Quoted 5 lines
Where does the kernel do this? Looking at the mips
set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it setsCP0_UserLocalplus thetp_valuefield in thethread_infostruct; and I can't immediately see anywhere where it would read eitherCP0_UserLocalortp_valueand put it into k0.
So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.
I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html
It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96
Quoted 3 lines
I also couldn't find the MIPS glibc code that uses k0, but I didn't look very hard there as I don't know my way around the glibc sources -- do you have a pointer to it on that end?
From Octeon SDK glibc/ports/sysdeps/mips/nptl/tls.h:
/* Note: rd must be $v1 to be ABI-conformant. */
# if !_MIPS_ARCH_OCTEON
# define READ_THREAD_POINTER() \
({ void *__result; \
asm volatile (".set\tpush\n\t.set\tmips32r2\n\t" \
"rdhwr\t%0, $29\n\t.set\tpop" : "=v" (__result)); \
__result; })
# else /* OCTEON */
/* The Kernel stores the value of "rdhwr v1,$29" in k0 ($26) register. And
it is the Kernel's responsibility to always have the correct value in
k0. Replacing rdhwr instruction with k0, as this instruction needs to
be emulated by the Kernel. */
# define READ_THREAD_POINTER() ( { register void *__result asm
("$26"); __result; } )
# endif /* OCTEON */From Octeon SDK glibc/ports/sysdeps/mips/tls-macros.h:
# if !_MIPS_ARCH_OCTEON
# define TLS_IE(x) \
({ void *__result, *__tmp; \
asm (".set push\n\t.set mips32r2\n\t" \
"rdhwr\t%0,$29\n\t.set pop" \
: "=v" (__result)); \
asm (LOAD_GP LW " $3,%%gottprel(" #x ")($28)\n\t" \
ADDU " %0,%0,$3" \
UNLOAD_GP \
: "+r" (__result), [tmp] "=&r" (__tmp) \
: : "$3"); \
__result; })
# define TLS_LE(x) \
({ void *__result; \
asm (".set push\n\t.set mips32r2\n\t" \
"rdhwr\t%0,$29\n\t.set pop" \
: "=v" (__result)); \
asm ("lui $3,%%tprel_hi(" #x ")\n\t" \
"addiu $3,$3,%%tprel_lo(" #x ")\n\t" \
ADDU " %0,%0,$3" \
: "+r" (__result) : : "$3"); \
__result; })# else /* OCTEON */
/* On Octeon the kernel stores the value of the thread pointer in the k0 ($26)
register. */
# define TLS_IE(x) \
({ void *__result, *__tmp; \
asm (LOAD_GP LW " $3,%%gottprel(" #x ")($28)\n\t" \
ADDU " %0,$26,$3" \
UNLOAD_GP \
: "+r" (__result), [tmp] "=&r" (__tmp) \
: : "$3"); \
__result; })
# define TLS_LE(x) \
({ void *__result; \
asm ("lui $3,%%tprel_hi(" #x ")\n\t" \
"addiu $3,$3,%%tprel_lo(" #x ")\n\t" \
ADDU " %0,$26,$3" \
: "+r" (__result) : : "$3"); \
__result; })
# endif /* OCTEON */Quoted 3 lines
thanks
-- PMM
Quoted 24 lines · James Hilliard
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard
MIPS glibc startup expects the thread pointer to be live in both
CP0_UserLocaland k0. linux-user already updatesCP0_UserLocalforcpu_set_tls()andTARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.Quoted 7 lines
A guest can therefore successfully execute
set_thread_area()and still reach__pthread_initialize_minimal()with k0 == 0, faulting on the first TLS-relative store.Make
cpu_set_tls()synchronize k0 withCP0_UserLocal, and route the MIPSset_thread_areasyscall throughcpu_set_tls()so all guest TLS updates follow the same path.Quoted 5 lines
Where does the kernel do this? Looking at the mips
set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it setsCP0_UserLocalplus thetp_valuefield in thethread_infostruct; and I can't immediately see anywhere where it would read eitherCP0_UserLocalortp_valueand put it into k0.
Quoted 5 lines
So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.
I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html
Ah, nice find.
Quoted 2 lines
It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96
Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.
-- PMM
Quoted 32 lines · Peter Maydell
Quoted 24 lines · James Hilliard
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard
MIPS glibc startup expects the thread pointer to be live in both
CP0_UserLocaland k0. linux-user already updatesCP0_UserLocalforcpu_set_tls()andTARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.Quoted 7 lines
A guest can therefore successfully execute
set_thread_area()and still reach__pthread_initialize_minimal()with k0 == 0, faulting on the first TLS-relative store.Make
cpu_set_tls()synchronize k0 withCP0_UserLocal, and route the MIPSset_thread_areasyscall throughcpu_set_tls()so all guest TLS updates follow the same path.Quoted 5 lines
Where does the kernel do this? Looking at the mips
set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it setsCP0_UserLocalplus thetp_valuefield in thethread_infostruct; and I can't immediately see anywhere where it would read eitherCP0_UserLocalortp_valueand put it into k0.Quoted 5 lines
So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.
I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.html
Quoted 7 lines
Ah, nice find.
Quoted 2 lines
It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.
Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?
Quoted 2 lines
-- PMM
Quoted 38 lines · James Hilliard
Quoted 37 lines · Peter Maydell
Quoted 24 lines · James Hilliard
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard
MIPS glibc startup expects the thread pointer to be live in both
CP0_UserLocaland k0. linux-user already updatesCP0_UserLocalforcpu_set_tls()andTARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.Quoted 7 lines
A guest can therefore successfully execute
set_thread_area()and still reach__pthread_initialize_minimal()with k0 == 0, faulting on the first TLS-relative store.Make
cpu_set_tls()synchronize k0 withCP0_UserLocal, and route the MIPSset_thread_areasyscall throughcpu_set_tls()so all guest TLS updates follow the same path.Quoted 5 lines
Where does the kernel do this? Looking at the mips
set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it setsCP0_UserLocalplus thetp_valuefield in thethread_infostruct; and I can't immediately see anywhere where it would read eitherCP0_UserLocalortp_valueand put it into k0.Quoted 5 lines
So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.
I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.htmlAh, nice find.
Quoted 2 lines
It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96
Quoted 8 lines
Quoted 2 lines
Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.
Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?
What about (untested):
-- >8 --
linux-user/mips/target_cpu.h +3 -2
@@ -36,7 +36,9 @@ static inline void cpu_clone_regs_parent(CPUMIPSState *env, unsigned flags)static inline void cpu_set_tls (CPUMIPSState *env,target_ulong newtls) { env->active_tc.CP0_UserLocal = newtls; - env->active_tc.gpr[26 ] = newtls; +if (env->insn_flags &INSN_OCTEON ) {/* Octeon1 */ + env->active_tc.gpr[26 ] = newtls; + } } ---
Quoted 57 lines · Philippe Mathieu-Daudé
Quoted 38 lines · James Hilliard
Quoted 37 lines · Peter Maydell
Quoted 24 lines · James Hilliard
Quoted 16 lines · Peter Maydell
Quoted 6 lines · James Hilliard
MIPS glibc startup expects the thread pointer to be live in both
CP0_UserLocaland k0. linux-user already updatesCP0_UserLocalforcpu_set_tls()andTARGET_NR_set_thread_area, but it leaves gpr[26] unchanged.Quoted 7 lines
A guest can therefore successfully execute
set_thread_area()and still reach__pthread_initialize_minimal()with k0 == 0, faulting on the first TLS-relative store.Make
cpu_set_tls()synchronize k0 withCP0_UserLocal, and route the MIPSset_thread_areasyscall throughcpu_set_tls()so all guest TLS updates follow the same path.Quoted 5 lines
Where does the kernel do this? Looking at the mips
set_thread_area()
https://elixir.bootlin.com/linux/v6.19.11/source/arch/mips/kernel/syscall.c#L87
it looks like it setsCP0_UserLocalplus thetp_valuefield in thethread_infostruct; and I can't immediately see anywhere where it would read eitherCP0_UserLocalortp_valueand put it into k0.Quoted 5 lines
So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.
I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.htmlAh, nice find.
Quoted 2 lines
It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96Quoted 8 lines
Quoted 2 lines
Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.
Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?
What about (untested):
-- >8 --
linux-user/mips/target_cpu.h +0
-0
linux-user/mips/target_cpu.h static inline void cpu_clone_regs_parent(CPUMIPSState · line 40
Quoted 6 lines
linux-user/mips/target_cpu.h +1
-1@@ -36,7 +36,9 @@ static inline void cpu_clone_regs_parent(CPUMIPSState *env, unsigned flags)static inline void cpu_set_tls (CPUMIPSState *env,target_ulong newtls) { env->active_tc.CP0_UserLocal = newtls; - env->active_tc.gpr[26 ] = newtls; +if (env->insn_flags &INSN_OCTEON ) {/* Octeon1 */
INST_OCTEON isn't OCTEON1 specific, but we could do something like this:
https://lore.kernel.org/qemu-devel/[email protected]/
Quoted 4 lines
linux-user/mips/target_cpu.h +2
-1+ env->active_tc.gpr[26 ] = newtls; + } } ---
Quoted 15 lines · James Hilliard
Quoted 13 lines · Peter Maydell
Quoted 5 lines · James Hilliard
So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.
I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.htmlAh, nice find.
Quoted 2 lines
It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96
Quoted 8 lines
Quoted 2 lines
Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.
Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?
We didn't add this 15 years ago, and my feeling is we still don't want to, because we track the mainline kernel. If an actual host kernel doesn't have a quirk to let these binaries run, we don't need one either.
-- PMM
Quoted 26 lines · Peter Maydell
Quoted 15 lines · James Hilliard
Quoted 13 lines · Peter Maydell
Quoted 5 lines · James Hilliard
So this seems to be something used by the vendor SDK for Octeon1 but not Octeon2 and Octeon3 targets, kinda strange/annoying.
I also found this:
https://lists.gnu.org/archive/html/qemu-devel/2011-08/msg02026.htmlAh, nice find.
Quoted 2 lines
It's in the Octeon vendor kernel here looks like:
https://github.com/MarvellEmbeddedProcessors/Octeon-Linux-kernel-4.14/blob/5c48e5031fbd86361e8185f6c530ffcfa5c9a60d/arch/mips/kernel/syscall.c#L94-L96Quoted 8 lines
Quoted 2 lines
Mmm. We emulate the upstream kernel, not vendor forks, so we wouldn't want to set k0 here I think.
Yeah, I was trying to see if there was some sort of way to trivially identify binaries that require this k0 support and autoenable a quirk but I didn't find anything, maybe it would make sense to have a separate runtime target for enabling this k0 quirk since it seems a lot of octeon binaries require it?
Quoted 4 lines
We didn't add this 15 years ago, and my feeling is we still don't want to, because we track the mainline kernel. If an actual host kernel doesn't have a quirk to let these binaries run, we don't need one either.
I'm wondering, is there any actual downside to enabling this quirk for Octeon 1 binaries? It wouldn't actually interfere with binaries that don't require the quirk right?
Quoted 2 lines
-- PMM