Skip to contentSkip to conversation

sigaltstack and RBS

Linux-ia64 Applied Matt Chapman 3 participants 11 messages Feb 9, 2003 – Mar 7, 2003

lore
Share & tools
lore
#0 lore raw

I'm having some difficulty "demand paging" register backing store from userspace (i.e. using SIGSEGV to map pages in on demand).

The problem is that even when using sigaltstack, the original backing store (which caused the fault) is still touched when returning to the signal trampoline, before it switches to the alternate RBS. Thus I get recursive faulting before it gets to the signal handler.

Ideally, signal handling on an alternate RBS/stack wouldn't touch the original RBS/stack at all.

Any suggestions how to deal with this?

Matt

#1 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Matt Chapman

Matt> I'm having some difficulty "demand paging" register backing Matt> store from userspace (i.e. using SIGSEGV to map pages in on Matt> demand).

Matt> The problem is that even when using sigaltstack, the original Matt> backing store (which caused the fault) is still touched when Matt> returning to the signal trampoline, before it switches to the Matt> alternate RBS. Thus I get recursive faulting before it gets Matt> to the signal handler.

Matt> Ideally, signal handling on an alternate RBS/stack wouldn't Matt> touch the original RBS/stack at all.

Matt> Any suggestions how to deal with this?

It sounds like you're using an old kernel. I don't recall exactly when this was fixed, but recent kernels will put the dirty partition on the _new_ stack, not the old one.

You can check gate.S:ia64_sigtramp(). If it branches to "setup_rbs" _before_ the first "alloc" instruction, you should be fine.

	--david
#2 lore raw
In reply to Matt Chapman
Quoted 2 lines · David Mosberger

It sounds like you're using an old kernel.

Sorry, I should have given more information. I'm using 2.5.59-ia64-030124.

Quoted 6 lines

I don't recall exactly
when this was fixed, but recent kernels will put the dirty partition on the _new_ stack, not the old one.

You can check gate.S:ia64_sigtramp(). If it branches to "setup_rbs" _before_ the first "alloc" instruction, you should be fine.

Yes, it does. Hmm, the faults I'm getting are at ia64_sigtramp (exactly). The first instruction (add r2@,r12) doesn't look like it could possibly be the cause, so I presume it is the RFI that is forcing a frame to be loaded (?).

Matt

#3 lore raw
In reply to Matt Chapman

Here's a small test program.

Matt

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/mman.h>
#include <asm/page.h>

#define RBS_BASE (void *)0x60000fff80000000UL
#define RBS_SIZE PAGE_SIZE

static void
mprotect_rbs(int flags)

{
	mprotect(RBS_BASE, RBS_SIZE, flags);
}

static void
SIGSEGV_handler(int sig, siginfo_t *info, void *uc) {

#if 1
	abort();
#endif
	mprotect_rbs(PROT_READ|PROT_WRITE);
}

int
main(void)

{
	struct sigaction sa;
	stack_t newstack;
	static unsigned long stack[SIGSTKSZ/sizeof(unsigned long)] __attribute__((aligned(PAGE_SIZE))) /* paranoia */;

	newstack.ss_sp = stack;
	newstack.ss_flags = 0;
	newstack.ss_size = sizeof(stack);
	if (sigaltstack(&newstack, NULL) == -1)
	{
		perror("sigaltstack");
		return 1;
	}

	sigemptyset(&sa.sa_mask);
	sa.sa_flags = SA_ONSTACK | SA_SIGINFO;
	sa.sa_sigaction = SIGSEGV_handler;
	if (sigaction(SIGSEGV, &sa, NULL) == -1)
	{
		perror("sigaction");
		return 1;
	}

	mprotect_rbs(0);
	printf("Still alive!\n");
	return 0;
}
#4 lore raw
In reply to Matt Chapman
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Matt Chapman

Matt> Here's a small test program.

Hmmh, the test program doesn't test backing-store _overflow_, it tests what happens when you _remove_ a formerly valid mapping. The program fails because the "rfi" that gets executed when returning from mprotect() may end up trying to restore registers that got spilled to the backing store before the call mprotect(), the mprotect() then removes access permission and hence the "rfi" can never finish execution (effectively, the mprotect() makes the contents of the spilled stacked registers disappear for good).

The current sigaltstack implementation isn't designed to handle such a case. And I'm not sure whether it should. Is there a particular reason you want to do this sort of thing?

	--david
#5 lore raw
In reply to Matt Chapman
Quoted 4 lines · David Mosberger

The current sigaltstack implementation isn't designed to handle such a case. And I'm not sure whether it should. Is there a particular reason you want to do this sort of thing?

I'll explain the context. I've written a virtual machine monitor which currently (for ease of prototyping) runs completely in userspace. e.g. itc does an mmap, ptc does an munmap, changing RID unmaps a whole region, SIGSEGV delivers a TLB miss to the "guest" kernel.

Now after a flush or RID change the guest kernel returns to its userspace with ar.bspstore pointing off to somewhere that isn't mapped, expecting to get a fault eventually. This is where the problem occurs. A mandatory RSE load faults as expected and the kernel tries to deliver SIGSEGV. But then the RFI to the signal trampoline repeats the same RSE load that caused the fault in the first place, before the signal handler can deal with it.

Is there any reason that the signal trampoline needs to see the original frame, or would it suffice to give it an empty frame? (Hmm, presumably this would mean filling out sc_cfm in the kernel... how to do that if we're in a syscall and haven't done the cover?)

Matt

#6 lore raw
In reply to Matt Chapman
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Matt Chapman

Matt> I'll explain the context. I've written a virtual machine Matt> monitor which currently (for ease of prototyping) runs Matt> completely in userspace. e.g. itc does an mmap, ptc does an Matt> munmap, changing RID unmaps a whole region, SIGSEGV delivers a Matt> TLB miss to the "guest" kernel.

Matt> Now after a flush or RID change the guest kernel returns to Matt> its userspace with ar.bspstore pointing off to somewhere that Matt> isn't mapped, expecting to get a fault eventually. This is Matt> where the problem occurs. A mandatory RSE load faults as Matt> expected and the kernel tries to deliver SIGSEGV. But then Matt> the RFI to the signal trampoline repeats the same RSE load Matt> that caused the fault in the first place, before the signal Matt> handler can deal with it.

Ah, that does sound interesting.

Actually, my explanation from yesterday can't be right: the current register frame as of the time of the mandatory RSE fault by definition is NOT on the user backing store, so the "rfi" shouldn't trigger any mandatory loads (the user's current frame is restored by the "loadrs" in the kernel's exit path).

Matt> Is there any reason that the signal trampoline needs to see Matt> the original frame, or would it suffice to give it an empty Matt> frame? (Hmm, presumably this would mean filling out sc_cfm in Matt> the kernel... how to do that if we're in a syscall and Matt> haven't done the cover?)

Someone needs to take care of backing the user's dirty partition. Since the user's original stack may not be valid, it needs to be backed by the alternate signal stack. That's most easily done by the signal trampoline after switching to the new backing store.

I'm not sure off-hand what's wrong. I'll take a look Monday. For what it's worth, the test program does seem to work fine on 2.5.52.

	--david
#7 lore raw
In reply to Matt Chapman
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Matt Chapman

Matt> I'm having some difficulty "demand paging" register backing Matt> store from userspace (i.e. using SIGSEGV to map pages in on Matt> demand).

Matt> The problem is that even when using sigaltstack, the original Matt> backing store (which caused the fault) is still touched when Matt> returning to the signal trampoline, before it switches to the Matt> alternate RBS. Thus I get recursive faulting before it gets Matt> to the signal handler.

Matt> Ideally, signal handling on an alternate RBS/stack wouldn't Matt> touch the original RBS/stack at all.

Matt> Any suggestions how to deal with this?

OK, I looked into this. The patch below should fix the problem. Bjorn, you might want to consider applying the same patch for 2.4.xx (with some extra testing).

The problem was that the signal delivery code wasn't prepared to handle incomplete register frames. Such frames are the result of mandatory RSE loads which fault. This situation is a corner-case because when a mandatory RSE load faults, the triggering instruction has been executed already (except for the restoration of the current frame). The patch below fixes the problem by making sure that we don't try to restore any registers from the user-level backing store when invoking the signal handler. The dirty partition of course still gets preserved, as it should be.

	--david
arch/ia64/kernel/entry.S +4 -2
diff -Nru a/arch/ia64/kernel/entry.S b/arch/ia64/kernel/entry.S
@@ -904,13 +904,14 @@
 	mov r9=ar.unat
 	mov loc0=rp				// save return address
 	mov out0=0				// there is no "oldset"
-	adds out1=0,sp				// out1=&sigscratch
+	adds out1=8,sp				// out1=&sigscratch->ar_pfs
 (pSys)	mov out2=1				// out2=1 => we're in a syscall
 	;;
 (pNonSys) mov out2=0				// out2=0 => not a syscall
 	.fframe 16
 	.spillpsp ar.unat, 16			// (note that offset is relative to psp+0x10!)
 	st8 [sp]=r9,-16				// allocate space for ar.unat and save it
+	st8 [out1]=loc1,-8			// save ar.pfs, out1=&sigscratch
 	.body
 	br.call.sptk.many rp=do_notify_resume_user
 .ret15:	.restore sp
@@ -931,11 +932,12 @@
 	mov loc0=rp				// save return address
 	mov out0=in0				// mask
 	mov out1=in1				// sigsetsize
-	adds out2=0,sp				// out2=&sigscratch
+	adds out2=8,sp				// out2=&sigscratch->ar_pfs
 	;;
 	.fframe 16
 	.spillpsp ar.unat, 16			// (note that offset is relative to psp+0x10!)
 	st8 [sp]=r9,-16				// allocate space for ar.unat and save it
+	st8 [out2]=loc1,-8			// save ar.pfs, out2=&sigscratch
 	.body
 	br.call.sptk.many rp=ia64_rt_sigsuspend
 .ret17:	.restore sp
arch/ia64/kernel/gate.S +9 -16
diff -Nru a/arch/ia64/kernel/gate.S b/arch/ia64/kernel/gate.S
@@ -145,11 +145,12 @@
 	 */
 
 #define SIGTRAMP_SAVES										\
-	.unwabi @svr4, 's'		// mark this as a sigtramp handler (saves scratch regs)	\
-	.savesp ar.unat, UNAT_OFF+SIGCONTEXT_OFF						\
-	.savesp ar.fpsr, FPSR_OFF+SIGCONTEXT_OFF						\
-	.savesp pr, PR_OFF+SIGCONTEXT_OFF							\
-	.savesp rp, RP_OFF+SIGCONTEXT_OFF							\
+	.unwabi @svr4, 's';	/* mark this as a sigtramp handler (saves scratch regs) */	\
+	.savesp ar.unat, UNAT_OFF+SIGCONTEXT_OFF;						\
+	.savesp ar.fpsr, FPSR_OFF+SIGCONTEXT_OFF;						\
+	.savesp pr, PR_OFF+SIGCONTEXT_OFF;     							\
+	.savesp rp, RP_OFF+SIGCONTEXT_OFF;							\
+	.savesp ar.pfs, CFM_OFF+SIGCONTEXT_OFF;							\
 	.vframesp SP_OFF+SIGCONTEXT_OFF
 
 GLOBAL_ENTRY(ia64_sigtramp)
@@ -173,9 +174,7 @@
 	.spillsp.p p8, ar.rnat, RNAT_OFF+SIGCONTEXT_OFF
 (p8)	br.cond.spnt setup_rbs		// yup -> (clobbers r14, r15, and r16)
 back_from_setup_rbs:
-
-	.spillreg ar.pfs, r8
-	alloc r8=ar.pfs,0,0,3,0		// get CFM0, EC0, and CPL0 into r8
+	alloc r8=ar.pfs,0,0,3,0
 	ld8 out0=[base0],16		// load arg0 (signum)
 	adds base1=(ARG1_OFF-(RBS_BASE_OFF+SIGCONTEXT_OFF)),base1
 	;;
@@ -184,17 +183,12 @@
 	;;
 	ld8 out2=[base0]		// load arg2 (sigcontextp)
 	ld8 gp=[r17]			// get signal handler's global pointer
-
 	adds base0=(BSP_OFF+SIGCONTEXT_OFF),sp
 	;;
 	.spillsp ar.bsp, BSP_OFF+SIGCONTEXT_OFF
-	st8 [base0]=r9,(CFM_OFF-BSP_OFF)	// save sc_ar_bsp
-	dep r8=0,r8,38,26			// clear EC0, CPL0 and reserved bits
-	adds base1=(FR6_OFF+16+SIGCONTEXT_OFF),sp
-	;;
-	.spillsp ar.pfs, CFM_OFF+SIGCONTEXT_OFF
-	st8 [base0]=r8				// save CFM0
+	st8 [base0]=r9			// save sc_ar_bsp
 	adds base0=(FR6_OFF+SIGCONTEXT_OFF),sp
+	adds base1=(FR6_OFF+16+SIGCONTEXT_OFF),sp
 	;;
 	stf.spill [base0]ö,32
 	stf.spill [base1]÷,32
@@ -217,7 +211,6 @@
 	ld8 r15=[base0],(CFM_OFF-BSP_OFF)	// fetch sc_ar_bsp and advance to CFM_OFF
 	mov r14=ar.bsp
 	;;
-	ld8 r8=[base0]				// restore (perhaps modified) CFM0, EC0, and CPL0
 	cmp.ne p8,p0=r14,r15			// do we need to restore the rbs?
 (p8)	br.cond.spnt restore_rbs		// yup -> (clobbers r14-r18, f6 & f7)
 	;;
arch/ia64/kernel/sigframe.h +1 -1
diff -Nru a/arch/ia64/kernel/sigframe.h b/arch/ia64/kernel/sigframe.h
@@ -1,6 +1,6 @@
 struct sigscratch {
 	unsigned long scratch_unat;	/* ar.unat for the general registers saved in pt */
-	unsigned long pad;
+	unsigned long ar_pfs;		/* for syscalls, the user-level function-state  */
 	struct pt_regs pt;
 };
 
arch/ia64/kernel/signal.c +14 -2
diff -Nru a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c
@@ -315,7 +315,7 @@
 static long
 setup_sigcontext (struct sigcontext *sc, sigset_t *mask, struct sigscratch *scr)
 {
-	unsigned long flags = 0, ifs, nat;
+	unsigned long flags = 0, ifs, cfm, nat;
 	long err;
 
 	ifs = scr->pt.cr_ifs;
@@ -325,7 +325,9 @@
 	if ((ifs & (1UL << 63)) = 0) {
 		/* if cr_ifs isn't valid, we got here through a syscall */
 		flags |= IA64_SC_FLAG_IN_SYSCALL;
-	}
+		cfm = scr->ar_pfs & ((1UL << 38) - 1);
+	} else
+		cfm = ifs & ((1UL << 38) - 1);
 	ia64_flush_fph(current);
 	if ((current->thread.flags & IA64_THREAD_FPH_VALID)) {
 		flags |= IA64_SC_FLAG_FPH_VALID;
@@ -344,6 +346,7 @@
 
 	err |= __put_user(nat, &sc->sc_nat);
 	err |= PUT_SIGSET(mask, &sc->sc_mask);
+	err |= __put_user(cfm, &sc->sc_cfm);
 	err |= __put_user(scr->pt.cr_ipsr & IA64_PSR_UM, &sc->sc_um);
 	err |= __put_user(scr->pt.ar_rsc, &sc->sc_ar_rsc);
 	err |= __put_user(scr->pt.ar_ccv, &sc->sc_ar_ccv);
@@ -422,6 +425,15 @@
 	scr->pt.ar_fpsr = FPSR_DEFAULT;			/* reset fpsr for signal handler */
 	scr->pt.cr_iip = tramp_addr;
 	ia64_psr(&scr->pt)->ri = 0;			/* start executing in first slot */
+	/*
+	 * Force the interruption function mask to zero.  This has no effect when a
+	 * system-call got interrupted by a signal (since, in that case, scr->pt_cr_ifs is
+	 * ignored), but it has the desirable effect of making it possible to deliver a
+	 * signal with an incomplete register frame (which happens when a mandatory RSE
+	 * load faults).  Furthermore, it has no negative effect on the getting the user's
+	 * dirty partition preserved, because that's governed by scr->pt.loadrs.
+	 */
+	scr->pt.cr_ifs = (1UL << 63);
 
 	/*
 	 * Note: this affects only the NaT bits of the scratch regs (the ones saved in
#9 lore raw
In reply to Matt Chapman
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Matt Chapman

Matt> On Mon, Feb 10, 2003 at 07:06:54PM -0800, David Mosberger

Quoted 1 line · Matt> wrote:
Quoted 1 line

OK, I looked into this. The patch below should fix the problem.

Matt> Thanks! That does the trick.

Well, thanks for the nice test case. That always helps a great deal.

	--david
#10 lore raw
In reply to Matt Chapman
Quoted 9 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Matt Chapman

Matt> I'm having some difficulty "demand paging" register backing Matt> store from userspace (i.e. using SIGSEGV to map pages in on Matt> demand).

OK, I looked into this. The patch below should fix the problem. Bjorn, you might want to consider applying the same patch for 2.4.xx (with some extra testing).

I applied this patch to 2.4.

Bjorn