Intel P6 vs P7 system call performance

You just have to make the NMI a task gate with an own TSS, then the microcode will set up an own stack for you.

The only issue afterwards is that "current" does not work, but that can be worked around by being a bit careful in the handler. It has to run with interrupts off too to avoid a race with an timer interrupt which uses current (or alternatively the timer interrupt could check for the "in nmi condition" - I don't think any other interrupts access current except when they crash)

[in theory it would be also possible to align the NMI stacks to 8K and put a "pseudo" task into that stack, but it would look a bit inelegant for me]

Using a task gate would be a good idea for kernel stack faults and double faults too, then it would be at least possible to get an oops for them, not the usual double fault.

[x86-64 does it similarly, except that it uses ISTs instead of task gates and avoids the current problem by using an explicit base register]

I cannot implement SYSENTER for x86-64/32bit emulation, but I think I can change the vsyscall code to use SYSCALL, not SYSENTER. The only issue is that I cannot easily use a fixmap to map into 32bit processes, because the kernel fixmap are way up into the 48bit address space and not reachable from compatibility mode.
I suspect a similar trick as with the lazy vmallocs - map it in the page fault handler on demand will work. I hope there won't be much more of these special cases though, do_page_fault is getting awfully complicated.

-Andi

Actually, I came up with a much simpler solution (which I didn't yet implement, but should be just a few lines).

The simpler solution is to just make the temporary ESP stack _look_ like it's a real process - ie make it 8kB per CPU (instead of the current 4kB) and put a fake "thread_info" at the bottom of it with the right CPU number etc. That way if an NMI comes in (in the _extremely_ tiny window), it will still see a sane picture of the system. It will basically think that we had a micro-task-switch between two instructions.

It's also entirely possible that the NMI window may not actually even exist, since I'm not even sure that Intel checks for pending interrupt before the first instruction of a trap handler.

We can't get stack faults without degrading performance horribly (they require you to set up the stack segment in magic ways that gcc doesn't even support). For double-faults, yes, but quite frankly, if you ever get a double fault things are _so_ screwed up that it's not very funny any more.

Right. The point of my patches is that user-level really _cannot_ use sysenter directly, because the sysenter semantics are just not useful for user land. So as far as user land is concerned, it really _is_ just a "call 0xfffff000", and then the kernel can do whatever is appropriate for that CPU.

Linus

How about this patch? Instead of making a per-cpu trampoline, write to the msr during each context switch. This means that the stack pointer is valid at all times, and also saves memory and a cache line bounce. I also included some misc cleanups.

Tested on an Athlon XP.

sysenter: 158.854423 cycles
int80:    273.658134 cycles

--
Brian Gerst

diff -urN linux-2.5.52-bk2/arch/i386/kernel/cpu/common.c linux/arch/i386/kernel/cpu/common.c
@@ -487,7 +487,7 @@
 		BUG();
 	enter_lazy_tlb(&init_mm, current, cpu);
 
-	t->esp0 = thread->esp0;
+	load_esp0(t, thread->esp0);
 	set_tss_desc(cpu,t);
 	cpu_gdt_table[cpu][GDT_ENTRY_TSS].b &= 0xfffffdff;
 	load_TR_desc();

diff -urN linux-2.5.52-bk2/arch/i386/kernel/process.c linux/arch/i386/kernel/process.c
@@ -440,7 +440,7 @@
 	/*
 	 * Reload esp0, LDT and the page table pointer:
 	 */
-	tss->esp0 = next->esp0;
+	load_esp0(tss, next->esp0);
 
 	/*
 	 * Load the per-thread Thread-Local Storage descriptor.

diff -urN linux-2.5.52-bk2/arch/i386/kernel/sysenter.c linux/arch/i386/kernel/sysenter.c
@@ -20,22 +20,12 @@
 
 static void __init enable_sep_cpu(void *info)
 {
-	unsigned long page = __get_free_page(GFP_ATOMIC);
 	int cpu = get_cpu();
-	unsigned long *esp0_ptr = &(init_tss + cpu)->esp0;
-	unsigned long rel32;
+	struct tss_struct *tss = init_tss + cpu;
 
-	rel32 = (unsigned long) sysenter_entry - (page+11);
-
-	
-	*(short *) (page+0) = 0x258b;		/* movl xxxxx,%esp */
-	*(long **) (page+2) = esp0_ptr;
-	*(char *)  (page+6) = 0xe9;		/* jmp rl32 */
-	*(long *)  (page+7) = rel32;
-
-	wrmsr(0x174, __KERNEL_CS, 0);		/* SYSENTER_CS_MSR */
-	wrmsr(0x175, page+PAGE_SIZE, 0);	/* SYSENTER_ESP_MSR */
-	wrmsr(0x176, page, 0);			/* SYSENTER_EIP_MSR */
+	wrmsr(MSR_IA32_SYSENTER_CS, __KERNEL_CS, 0);
+	wrmsr(MSR_IA32_SYSENTER_ESP, tss->esp0, 0);
+	wrmsr(MSR_IA32_SYSENTER_EIP, (unsigned long) sysenter_entry, 0);
 
 	printk("Enabling SEP on CPU %d\n", cpu);
 	put_cpu();	

@@ -60,14 +50,15 @@
 	};
 	unsigned long page = get_zeroed_page(GFP_ATOMIC);
 
+	if (cpu_has_sep) {
+		memcpy((void *) page, sysent, sizeof(sysent));
+		enable_sep_cpu(NULL);
+		smp_call_function(enable_sep_cpu, NULL, 1, 1);
+	} else
+		memcpy((void *) page, int80, sizeof(int80));
+
 	__set_fixmap(FIX_VSYSCALL, __pa(page), PAGE_READONLY);
-	memcpy((void *) page, int80, sizeof(int80));
-	if (!boot_cpu_has(X86_FEATURE_SEP))
-		return 0;
-
-	memcpy((void *) page, sysent, sizeof(sysent));
-	enable_sep_cpu(NULL);
-	smp_call_function(enable_sep_cpu, NULL, 1, 1);
+
 	return 0;
 }
 

diff -urN linux-2.5.52-bk2/arch/i386/kernel/vm86.c linux/arch/i386/kernel/vm86.c
@@ -113,7 +113,7 @@
 		do_exit(SIGSEGV);
 	}
 	tss = init_tss + smp_processor_id();
-	tss->esp0 = current->thread.esp0 = current->thread.saved_esp0;
+	load_esp0(tss, current->thread.saved_esp0);
 	current->thread.saved_esp0 = 0;
 	ret = KVM86->regs32;
 	return ret;

@@ -283,7 +283,8 @@
 	info->regs32->eax = 0;
 	tsk->thread.saved_esp0 = tsk->thread.esp0;
 	tss = init_tss + smp_processor_id();
-	tss->esp0 = tsk->thread.esp0 = (unsigned long) &info->VM86_TSS_ESP0;
+	tsk->thread.esp0 = (unsigned long) &info->VM86_TSS_ESP0;
+	load_esp0(tss, tsk->thread.esp0);
 
 	tsk->thread.screen_bitmap = info->screen_bitmap;
 	if (info->flags & VM86_SCREEN_BITMAP)

diff -urN linux-2.5.52-bk2/include/asm-i386/cpufeature.h linux/include/asm-i386/cpufeature.h
@@ -7,6 +7,8 @@
 #ifndef __ASM_I386_CPUFEATURE_H
 #define __ASM_I386_CPUFEATURE_H
 
+#include <linux/bitops.h>
+
 #define NCAPINTS	4	/* Currently we have 4 32-bit words worth of info */
 
 /* Intel-defined CPU features, CPUID level 0x00000001, word 0 */

@@ -74,6 +76,7 @@
 #define cpu_has_pae		boot_cpu_has(X86_FEATURE_PAE)
 #define cpu_has_pge		boot_cpu_has(X86_FEATURE_PGE)
 #define cpu_has_apic		boot_cpu_has(X86_FEATURE_APIC)
+#define cpu_has_sep		boot_cpu_has(X86_FEATURE_SEP)
 #define cpu_has_mtrr		boot_cpu_has(X86_FEATURE_MTRR)
 #define cpu_has_mmx		boot_cpu_has(X86_FEATURE_MMX)
 #define cpu_has_fxsr		boot_cpu_has(X86_FEATURE_FXSR)

diff -urN linux-2.5.52-bk2/include/asm-i386/msr.h linux/include/asm-i386/msr.h
@@ -53,6 +53,10 @@
 
 #define MSR_IA32_BBL_CR_CTL		0x119
 
+#define MSR_IA32_SYSENTER_CS		0x174
+#define MSR_IA32_SYSENTER_ESP		0x175
+#define MSR_IA32_SYSENTER_EIP		0x176
+
 #define MSR_IA32_MCG_CAP		0x179
 #define MSR_IA32_MCG_STATUS		0x17a
 #define MSR_IA32_MCG_CTL		0x17b

diff -urN linux-2.5.52-bk2/include/asm-i386/processor.h linux/include/asm-i386/processor.h
@@ -14,6 +14,7 @@
 #include <asm/types.h>
 #include <asm/sigcontext.h>
 #include <asm/cpufeature.h>
+#include <asm/msr.h>
 #include <linux/cache.h>
 #include <linux/config.h>
 #include <linux/threads.h>

@@ -416,6 +417,13 @@
 	{~0, } /* ioperm */					\
 }
 
+static inline void load_esp0(struct tss_struct *tss, unsigned long esp0)
+{
+	tss->esp0 = esp0;
+	if (cpu_has_sep)
+		wrmsr(MSR_IA32_SYSENTER_ESP, esp0, 0);
+}
+
 #define start_thread(regs, new_eip, new_esp) do {		\
 	__asm__("movl %0,%%fs ; movl %0,%%gs": :"r" (0));	\
 	set_fs(USER_DS);					\

I wanted to avoid slowing down the context switch, but I didn't actually time how much the MSR write hurts you (it needs to be conditional, though, I think).

Linus

this is the solution i took in the original vsyscall patches. The syscall entry cost is at least one factor more important than the context-switch cost. The MSR write was not all that slow when i measured it (it was on the range of 20 cycles), and it's definitely something the chip makers should keep fast.

Ingo

I think it would be better to make NMI and Debug trap use task gates, if that does actually work, and let the NMI and debug handlers fix the stack if they trap in the entry paths. They are much rarer after all.

My unreliable memory recalls about 40 cycles for an MSR write on my Celeron.

However, I think you still need MSR writes _sometimes_ in the context switch to disable sysenter for vm86-mode tasks.

Could the context switch be written like this?:

	if (unlikely((prev_task->flags | next_task->flags) & PF_SLOW_SWITCH)) {
		// Do rare stuff including debug registers
		// and sysenter/syscall MSR change for vm86.
	}

	// Other stuff.

-- Jamie

Just a little bit of benchmarking:
(little testprogram by Linus out of this thread)
(on a AMD Duron 750)

2.5.52-bk2+sysenter-1 (Brian Gerst):

igor3:~# ./a.out
187.894946 cycles    (call 0xfffff000)
299.155075 cycles    (int $0x80)

2.5.52-bk6:

igor3:~# ./a.out
202.134535 cycles    (call 0xffffe000)
299.117583 cycles    (int $0x80)

Not really much, but the difference is there. (I don't about other side effects)

Christian Leber