Cannot unwind through MIPS signal frames with ICACHE_REFILLS_WORKAROUND_WAR
With the current kernel (2.6.23.1) in my R5000 based O2 it seems impossible for GCC's exception unwinding machinery to unwind through signal frames. The cause of the problems is the ICACHE_REFILLS_WORKAROUND_WAR which puts the sigcontext at an almost impossible to determine offset from the signal return trampoline. The unwinder depends on being able to find the sigcontext given a known location of the trampoline.
It seems there are a couple of possible solutions:
1) The comments in war.h indicate the problem only exists in R7000 and E9000 processors. We could turn off the workaround if the kernel is configured for R5000. That would help me, but not those with the effected systems.
2) In the non-workaround case, the siginfo immediately follows the trampoline and the first member is the signal number. For the workaround case the first word following the trampoline is zero. We could replace this with the offset to the sigcontext which is always a small negative value. The unwinder could then distinguish the two cases (signal numbers are positive and the offset negative). If we did this, the change would have to be coordinated with GCC's unwinder (in libgcc_s.so.1).
Thoughts?
David Daney
Quoted 24 lines · David Daney
With the current kernel (2.6.23.1) in my R5000 based O2 it seems impossible for GCC's exception unwinding machinery to unwind through signal frames. The cause of the problems is the
ICACHE_REFILLS_WORKAROUND_WARwhich puts the sigcontext at an almost impossible to determine offset from the signal return trampoline. The unwinder depends on being able to find the sigcontext given a known location of the trampoline.It seems there are a couple of possible solutions:
1) The comments in
war.hindicate the problem only exists in R7000 and E9000 processors. We could turn off the workaround if the kernel is configured for R5000. That would help me, but not those with the effected systems.2) In the non-workaround case, the siginfo immediately follows the trampoline and the first member is the signal number. For the workaround case the first word following the trampoline is zero. We could replace this with the offset to the sigcontext which is always a small negative value. The unwinder could then distinguish the two cases (signal numbers are positive and the offset negative). If we did this, the change would have to be coordinated with GCC's unwinder (in libgcc_s.so.1).
Quoted 1 line
Thoughts?
The best solution is to put the unwinder info in the kernel. Does MIPS use a vDSO ?
Andrew.
Quoted 25 lines · Andrew Haley
Quoted 24 lines · David Daney
With the current kernel (2.6.23.1) in my R5000 based O2 it seems impossible for GCC's exception unwinding machinery to unwind through signal frames. The cause of the problems is the
ICACHE_REFILLS_WORKAROUND_WARwhich puts the sigcontext at an almost impossible to determine offset from the signal return trampoline. The unwinder depends on being able to find the sigcontext given a known location of the trampoline.It seems there are a couple of possible solutions:
1) The comments in
war.hindicate the problem only exists in R7000 and E9000 processors. We could turn off the workaround if the kernel is configured for R5000. That would help me, but not those with the effected systems.2) In the non-workaround case, the siginfo immediately follows the trampoline and the first member is the signal number. For the workaround case the first word following the trampoline is zero. We could replace this with the offset to the sigcontext which is always a small negative value. The unwinder could then distinguish the two cases (signal numbers are positive and the offset negative). If we did this, the change would have to be coordinated with GCC's unwinder (in libgcc_s.so.1).
Quoted 4 lines
Quoted 1 line
Thoughts?
The best solution is to put the unwinder info in the kernel. Does MIPS use a vDSO ?
No though we should.
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush operation to be performed on all processors which is expensive. Having the trampoline in a vDSO would solve that.
I need to look into it, not sure what it would take.
Ralf
Quoted 33 lines · Ralf Baechle
Quoted 25 lines · Andrew Haley
Quoted 24 lines · David Daney
With the current kernel (2.6.23.1) in my R5000 based O2 it seems impossible for GCC's exception unwinding machinery to unwind through signal frames. The cause of the problems is the
ICACHE_REFILLS_WORKAROUND_WARwhich puts the sigcontext at an almost impossible to determine offset from the signal return trampoline. The unwinder depends on being able to find the sigcontext given a known location of the trampoline.It seems there are a couple of possible solutions:
1) The comments in
war.hindicate the problem only exists in R7000 and E9000 processors. We could turn off the workaround if the kernel is configured for R5000. That would help me, but not those with the effected systems.2) In the non-workaround case, the siginfo immediately follows the trampoline and the first member is the signal number. For the workaround case the first word following the trampoline is zero. We could replace this with the offset to the sigcontext which is always a small negative value. The unwinder could then distinguish the two cases (signal numbers are positive and the offset negative). If we did this, the change would have to be coordinated with GCC's unwinder (in libgcc_s.so.1).
Quoted 4 lines
Quoted 1 line
Thoughts?
The best solution is to put the unwinder info in the kernel. Does MIPS use a vDSO ?
Quoted 7 lines
No though we should.
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush operation to be performed on all processors which is expensive. Having the trampoline in a vDSO would solve that.
And the stack wouldn't need to have exec permission anymore.
Quoted 2 lines
I need to look into it, not sure what it would take.
I started to add vdso support for MIPS a couple months ago, but it's in a very early stage and I unfortunately haven't time to finish it. I can send it to you if you want.
Signature · 1 line
Franck
Quoted 32 lines · Franck Bui-Huu
Quoted 32 lines
Quoted 25 lines
Quoted 24 lines · David Daney
With the current kernel (2.6.23.1) in my R5000 based O2 it seems impossible for GCC's exception unwinding machinery to unwind through signal frames. The cause of the problems is the
ICACHE_REFILLS_WORKAROUND_WARwhich puts the sigcontext at an almost impossible to determine offset from the signal return trampoline. The unwinder depends on being able to find the sigcontext given a known location of the trampoline.It seems there are a couple of possible solutions:
1) The comments in
war.hindicate the problem only exists in R7000 and E9000 processors. We could turn off the workaround if the kernel is configured for R5000. That would help me, but not those with the effected systems.2) In the non-workaround case, the siginfo immediately follows the trampoline and the first member is the signal number. For the workaround case the first word following the trampoline is zero. We could replace this with the offset to the sigcontext which is always a small negative value. The unwinder could then distinguish the two cases (signal numbers are positive and the offset negative). If we did this, the change would have to be coordinated with GCC's unwinder (in libgcc_s.so.1).
Quoted 4 lines
Quoted 1 line
Thoughts?
The best solution is to put the unwinder info in the kernel. Does MIPS use a vDSO ?
No though we should.
Quoted 7 lines
Quoted 5 lines
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush operation to be performed on all processors which is expensive. Having the trampoline in a vDSO would solve that.
And the stack wouldn't need to have exec permission anymore.
Oh?
extern void frob(void (*)(void));
int foo(void)
{
int x;
void bar(void)
{
x++;
}
frob(&bar);
print("x is %d\n", x);
}Compile and enjoy.
Ralf
Quoted 60 lines · Ralf Baechle
Quoted 32 lines · Franck Bui-Huu
Quoted 32 lines
Quoted 25 lines
Quoted 24 lines · David Daney
With the current kernel (2.6.23.1) in my R5000 based O2 it seems impossible for GCC's exception unwinding machinery to unwind through signal frames. The cause of the problems is the
ICACHE_REFILLS_WORKAROUND_WARwhich puts the sigcontext at an almost impossible to determine offset from the signal return trampoline. The unwinder depends on being able to find the sigcontext given a known location of the trampoline.It seems there are a couple of possible solutions:
1) The comments in
war.hindicate the problem only exists in R7000 and E9000 processors. We could turn off the workaround if the kernel is configured for R5000. That would help me, but not those with the effected systems.2) In the non-workaround case, the siginfo immediately follows the trampoline and the first member is the signal number. For the workaround case the first word following the trampoline is zero. We could replace this with the offset to the sigcontext which is always a small negative value. The unwinder could then distinguish the two cases (signal numbers are positive and the offset negative). If we did this, the change would have to be coordinated with GCC's unwinder (in libgcc_s.so.1).
Quoted 4 lines
Quoted 1 line
Thoughts?
The best solution is to put the unwinder info in the kernel. Does MIPS use a vDSO ?
No though we should.
Quoted 7 lines
Quoted 5 lines
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush operation to be performed on all processors which is expensive. Having the trampoline in a vDSO would solve that.
And the stack wouldn't need to have exec permission anymore.
Oh?
extern void
frob(void (*)(void));int foo(void)
{ int x; void bar(void) { x++; } frob(&bar); print("x is %d\n", x); }
Quoted 2 lines
Compile and enjoy.
Sorry Ralf, I missed your point.
Signature · 1 line
Franck
Quoted 15 lines · Franck Bui-Huu
Quoted 15 lines
Quoted 1 line
And the stack wouldn't need to have exec permission anymore.
Oh?
extern void
frob(void (*)(void));int foo(void)
{ int x; void bar(void) { x++; }
Quoted 8 lines
Quoted 6 lines
frob(&bar); print("x is %d\n", x); }Compile and enjoy.
Sorry Ralf, I missed your point.
This piece of code compiles to something that copies a trampoline to the stack. The address of that trampoline is what is then passed as argument to frob().
Old versions of glibc were probable the most notorious users of trampolines. Objective C also generates them. Since a cacheflush that is a syscall is required performance is less than great.
Which means the libc() cacheflush() function is another candidate for a vDSO, it can be optimized by using SYNCI on some configurations.
Ralf
Quoted 3 lines · Ralf Baechle
Old versions of glibc were probable the most notorious users of trampolines. Objective C also generates them. Since a cacheflush that is a syscall is required performance is less than great.
No Objective-C does not generate them. Objective-C returns the exact function pointer back. Now libffi generates trampolines.
-- Pinski
Franck a dit:
Quoted 7 lines
Quoted 5 lines
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush operation to be performed on all processors which is expensive. Having the trampoline in a vDSO would solve that.
And the stack wouldn't need to have exec permission anymore.
True, though it should perhaps be noted that currently it's only on 4KSc/Sd systems (which I know you work on) where it's even possible for the stack *not* to have exec permissions, since the classical MIPS MMU gives execute permission to any page that is readable.
Regards,
Kevin K.
Quoted 7 lines · Kevin D. Kissell
Franck a dit:
Quoted 6 lines
Quoted 5 lines
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush operation to be performed on all processors which is expensive. Having the trampoline in a vDSO would solve that.
Quoted 7 lines
Quoted 1 line
And the stack wouldn't need to have exec permission anymore.
True, though it should perhaps be noted that currently it's only on 4KSc/Sd systems (which I know you work on) where it's even possible for the stack *not* to have exec permissions, since the classical MIPS MMU gives execute permission to any page that is readable.
Well, the noexec stack is pretty usefull I think. I can't believe it will be limited to these 2 systems in the near future...
Signature · 1 line
Franck
Quoted 4 lines · Kevin D. Kissell
True, though it should perhaps be noted that currently it's only on 4KSc/Sd systems (which I know you work on) where it's even possible for the stack *not* to have exec permissions, since the classical MIPS MMU gives execute permission to any page that is readable.
Disabling PROT_EXEC on a mapping tells the kernel it doesn't need to take care of I-cache coherency. So it's somewhat beneficial even in absence of a protection bit in the actual TLB hardware.
Some of these performance optimizations are impossible because the kernel can't have definate knowledge that certain addresses have never entered the I-cache.
Ralf
Re: Cannot unwind through MIPS signal frames withICACHE_REFILLS_WORKAROUND_WAR
Quoted 8 lines
Quoted 4 lines
True, though it should perhaps be noted that currently it's only on 4KSc/Sd systems (which I know you work on) where it's even possible for the stack *not* to have exec permissions, since the classical MIPS MMU gives execute permission to any page that is readable.
Disabling
PROT_EXECon a mapping tells the kernel it doesn't need to take care of I-cache coherency. So it's somewhat beneficial even in absence of a protection bit in the actual TLB hardware.
That depends on just what the consequences of I-cache incoherence might be. Without help from the MMU, the kernel cannot *know* that a given location isn't in the I-cache, because a program can always compute a pointer-to-function to an arbitrary address and dereference it successfully so long as the location is readable. If it's only the user who does this that will suffer as a result of I-cache incoherence, one can argue that it serves him right. But if it can screw up the execution of the kernel, or other user processes, I think we have to assume the worst.
Quoted 3 lines
Some of these performance optimizations are impossible because the kernel can't have definate knowledge that certain addresses have never entered the I-cache.
Sad but true.
Regards,
Kevin K.
VDSO on mips (was Re: Cannot unwind through MIPS signal frames with ICACHE_REFILLS_WORKAROUND_WAR)
In reply to Franck Bui-HuuQuoted 5 lines · Franck Bui-Huu
I started to add vdso support for MIPS a couple months ago, but it's in a very early stage and I unfortunately haven't time to finish it. I can send it to you if you want.
Here it is. As I said it far from complete but it might help.
Franck
--- 8< ---
arch/mips/kernel/Makefile +3 -1
@@ -6,11 +6,13 @@ extra-y := head.o init_task.o vmlinux.lds obj-y += cpu-probe.o branch.o entry.o genex.o irq.o process.o \ ptrace.o reset.o semaphore.o setup.o signal.o syscall.o \ - time.o topology.o traps.o unaligned.o + time.o topology.o traps.o unaligned.o vdso.o binfmt_irix-objs := irixelf.o irixinv.o irixioctl.o irixsig.o \ irix5sys.o sysirix.o +obj-$(CONFIG_32BIT) += vdso32/ + obj-$(CONFIG_STACKTRACE) += stacktrace.o obj-$(CONFIG_MODULES) += mips_ksyms.o module.o
arch/mips/kernel/vdso.c new file +52 -0
52 lines not on this page · raw message
arch/mips/kernel/vdso32/.gitignore new file +1 -0
@@ -0,0 +1 @@ +vdso32.lds
arch/mips/kernel/vdso32/Makefile new file +35 -0
@@ -0,0 +1,35 @@ +# List of files in the vdso +obj-vdso32 = sigtramp.o + +# Build rules +targets := $(obj-vdso32) vdso32.so +obj-vdso32 := $(addprefix $(obj)/, $(obj-vdso32)) + + +EXTRA_CFLAGS := -shared -s -fno-common -fno-builtin +EXTRA_CFLAGS += -nostdlib -Wl,-soname=linux-vdso32.so.1 +EXTRA_CFLAGS += $(call ld-option, -Wl$(comma)--hash-style=sysv) + +EXTRA_AFLAGS := -D__VDSO32__ -s + +obj-y += vdso32.o +extra-y += vdso32.lds +CPPFLAGS_vdso32.lds += -P -C -U$(ARCH) + +# kbuild does not track this dependency due to usage of .incbin +$(obj)/vdso32.o : $(obj)/vdso32.so + +# link rule for the .so file, .lds has to be first +$(obj)/vdso32.so: $(src)/vdso32.lds $(obj-vdso32) + $(call if_changed,vdso32ld) + +# assembly rules for the .S files +$(obj-vdso32): %.o: %.S + $(call if_changed_dep,vdso32as) + +# actual build commands +quiet_cmd_vdso32ld = VDSO32_LD $@ + cmd_vdso32ld = $(CC) $(c_flags) -Wl,-T $^ -o $@ +quiet_cmd_vdso32as = VDSO32_AS $@ + cmd_vdso32as = $(CC) $(a_flags) -c -o $@ $< +
arch/mips/kernel/vdso32/sigtramp.S new file +13 -0
@@ -0,0 +1,13 @@ +#include <asm/asm.h> +#include <asm/regdef.h> +#include <asm/unistd.h> + +LEAF(__kernel_sigtramp_32) + li v0, __NR_sigreturn + syscall +END(__kernel_sigtramp_32) + +LEAF(__kernel_sigtramp_rt32) + li v0, __NR_rt_sigreturn + syscall +END(__kernel_sigtramp_rt32)
arch/mips/kernel/vdso32/vdso32.S new file +11 -0
@@ -0,0 +1,11 @@ +#include <linux/init.h> + +__INITDATA + + .globl vdso32_start + .globl vdso32_end +vdso32_start: + .incbin "arch/mips/kernel/vdso32/vdso32.so" +vdso32_end: + +__FINIT
arch/mips/kernel/vdso32/vdso32.lds.S new file +73 -0
73 lines not on this page · raw message
Quoted 2 lines · Ralf Baechle
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush
BTW, could we get rid of the trampoline so easily ? I mean won't we have to keep it for backward compatibility reasons ?
Franck
Quoted 6 lines · Franck Bui-Huu
Quoted 2 lines · Ralf Baechle
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush
BTW, could we get rid of the trampoline so easily ? I mean won't we have to keep it for backward compatibility reasons ?
The trampolines are an implementation detail. Little software needs to know about it, so while I expect some slight colateral damage from getting rid of trampolines it's not going to be painful. GDB is the primary piece of software that will need to change.
Some of the other architectures have an sa_restorer field in struct sigaction but we don't have that on MIPS.
One way to deal with this would be to do a similar as IRIX where the sigaction(2) takes a 4th argument which takes the role of sa_restorer. For backward compatibility an SA_RESTORER field. So if the SA_RESTORER is clear we'd be using a classic trampoline, if it's set the value of the 4th argument.
Or slightly crazier, put a kernel address into the $ra register of the invoked signal handler. So the signal handler will cause an address error exception which then can be trapped. Additional advantage - some of the "Don't let your children do this ..." sections of code can go away ;-)
Ralf
Quoted 7 lines · Ralf Baechle
Quoted 5 lines · Franck Bui-Huu
Quoted 2 lines · Ralf Baechle
Another reason is to get rid of the classic trampoline the kernel installs on the stack. On some multiprocessor systems it requires a cacheflush
BTW, could we get rid of the trampoline so easily ? I mean won't we have to keep it for backward compatibility reasons ?
There has to be some way for user code to unwind the call stack through signal handler frames. For Linux/MIPS systems that use GCC as their system compiler, this is done by code in libgcc which is part of the GCC runtime.
Currently the unwinder examines the code at the signal handler return address and if it detects one of the several know trampoline code sequences, it makes assumptions about the way the kernel lays out the stack. In particular, it assumes that there is a struct sigcontext at a
well known offset from the stack pointer at entry to the signal handler.
The code that handles this is in gcc/config/mips/linux-unwind.h
As of GCC-4.3 the location of the trampoline is unimportant, but the location of the sigcontext relative to the value in $sp is important and should not be changed unless there really is no other choice.
Quoted 5 lines
The trampolines are an implementation detail. Little software needs to know about it, so while I expect some slight colateral damage from getting rid of trampolines it's not going to be painful. GDB is the primary piece of software that will need to change.
And any user code that uses SIGSEGV to detect and handle dereferencing null pointers.
Quoted 10 lines
Some of the other architectures have an
sa_restorerfield in struct sigaction but we don't have that on MIPS.One way to deal with this would be to do a similar as IRIX where the sigaction(2) takes a 4th argument which takes the role of
sa_restorer. For backward compatibility anSA_RESTORERfield. So if theSA_RESTORERis clear we'd be using a classic trampoline, if it's set the value of the 4th argument.
Quoted 4 lines
Or slightly crazier, put a kernel address into the $ra register of the invoked signal handler. So the signal handler will cause an address error exception which then can be trapped. Additional advantage - some of the "Don't let your children do this ..." sections of code can go away ;-)
I am liking the idea of putting the trampoline code in the (as of yet non-existent vdso). This eliminates the need to flush the icache, but maintains the ability of user code to unwind through signal frames.
Putting a .eh_frame section in the vdso would be even better as that would eliminate the need for libgcc to do code reading and let the kernel use any means desired to return from signal handlers.
David Daney
Quoted 8 lines · David Daney
I am liking the idea of putting the trampoline code in the (as of yet non-existent vdso). This eliminates the need to flush the icache, but maintains the ability of user code to unwind through signal frames.
Putting a .eh_frame section in the vdso would be even better as that would eliminate the need for libgcc to do code reading and let the kernel use any means desired to return from signal handlers.
You're very welcome to improve the ld script for VDSO generation, I sent a couple days ago. I would be happy to integrate your improvements in my patch for the future version.
thanks,
Franck
Quoted 8 lines · David Daney
With the current kernel (2.6.23.1) in my R5000 based O2 it seems impossible for GCC's exception unwinding machinery to unwind through signal frames. The cause of the problems is the
ICACHE_REFILLS_WORKAROUND_WARwhich puts the sigcontext at an almost impossible to determine offset from the signal return trampoline. The unwinder depends on being able to find the sigcontext given a known location of the trampoline.It seems there are a couple of possible solutions:
1) The comments in
war.hindicate the problem only exists in R7000 and E9000 processors. We could turn off the workaround if the kernel is configured for R5000. That would help me, but not those with the effected systems.2) In the non-workaround case, the siginfo immediately follows the trampoline and the first member is the signal number. For the workaround case the first word following the trampoline is zero. We could replace this with the offset to the sigcontext which is always a small negative value. The unwinder could then distinguish the two cases (signal numbers are positive and the offset negative). If we did this, the change would have to be coordinated with GCC's unwinder (in libgcc_s.so.1).
I think I have found a solution that doesn't require kernel changes.
The CFA (i.e. the stack pointer of the signal handler) of the the context when calling mips_fallback_frame_state is at a constant offset from the sigcontext. I can just use the CFA instead of the trampoline's address.
Does this seem plausible?
Thanks,
David Daney