Implement per-processor data areas for i386.

This patch implements per-processor data areas by using %gs as the base segment of the per-processor memory. This has two principle advantages:

- It allows very simple direct access to per-processor data by effectively using an effective address of the form %gs:offset, where offset is the offset into struct i386_pda. These sequences are faster and smaller than the current mechanism using current_thread_info().

- It also allows per-CPU data to be allocated as each CPU is brought up, rather than statically allocating it based on the maximum number of CPUs which could be brought up.

I haven't measured performance yet, but when using the PDA for "current" and "smp_processor_id", I see a 5715 byte reduction in .text segment size for my kernel.

Unfortunately, these patches don't actually work yet. I'm not sure why; I'm hoping review will turn something up.

Some background for people unfamiliar with x86 segmentation:

This uses the x86 segmentation stuff in a way similar to NPTL's way of implementing Thread-Local Storage. It relies on the fact that each CPU has its own Global Descriptor Table (GDT), which is basically an array of base-length pairs (with some extra stuff). When a segment register is loaded with a descriptor (approximately, an index in the GDT), and you use that segment register for memory access, the address has the base added to it, and the resulting address is used.

In other words, if you imagine the GDT containing an entry:

	Index	Offset
	123:	0xc0211000 (allocated PDA)
and you load %gs with this selector:
	mov $123, %gs
and then use GS later on:
	mov %gs:4, %eax
This has the effect of
	mov 0xc0211004, %eax

and because the GDT is per-CPU, the offset (= 0xc0211000 = memory allocated for this CPU's PDA) can be a CPU-specific value while leaving everything else constant.

This means that something like "current" or "smp_processor_id()" can collapse to a single instruction:
mov %gs:PDA_current, %reg

TODO:
- Make it work. It works UP on a test QEMU machine, but it doesn't yet work on real hardware, or SMP (though not working SMP on QEMU is more likely to be a QEMU problem). Not sure what the problem is yet; I'm hoping review will reveal something.
- Measure performance impact. The patch adds a segment register save/restore on entry/exit to the kernel. This expense should be offset by savings in using the PDA while in the kernel, but I haven't measured this yet. Space savings are already appealing though.
- Modify more things to use the PDA. The more that uses it, the more the cost of the %gs save/restore is amortized. smp_processor_id and current are the obvious first choices, which are implemented in this series.
- Make it a config option? UP systems don't need to do any of this, other than having a single pre-allocated PDA. Unfortunately, it gets a bit messy to do this given the changes needed in handling %gs.
--

This patch has the basic definitions of struct i386_pda, and the segment selector in the GDT.

asm-i386/pda.h is more or less a direct copy of asm-x86_64/pda.h. The most interesting difference is the use of _proxy_pda, which is used to give gcc a model for the actual memory operations on the real pda structure. No actual reference is ever made to _proxy_pda, so it is never defined.

---
 arch/i386/kernel/asm-offsets.c |    4 ++
 arch/i386/kernel/head.S        |    2 -
 include/asm-i386/pda.h         |   67 ++++++++++++++++++++++++++++++++++++++++
 include/asm-i386/segment.h     |    5 ++
 4 files changed, 76 insertions(+), 2 deletions(-)

===================================================================

@@ -15,6 +15,7 @@
 #include <asm/processor.h>
 #include <asm/thread_info.h>
 #include <asm/elf.h>
+#include <asm/pda.h>
 
 #define DEFINE(sym, val) \
         asm volatile("\n->" #sym " %0 " #val : : "i" (val))

@@ -74,4 +75,7 @@ void foo(void)
 	DEFINE(VDSO_PRELINK, VDSO_PRELINK);
 
 	OFFSET(crypto_tfm_ctx_offset, crypto_tfm, __crt_ctx);
+
+	BLANK();
+	OFFSET(PDA_pcurrent, i386_pda, pcurrent);
 }

===================================================================

@@ -591,7 +591,7 @@ ENTRY(cpu_gdt_table)
 	.quad 0x004092000000ffff	/* 0xc8 APM DS    data */
 
 	.quad 0x0000920000000000	/* 0xd0 - ESPFIX 16-bit SS */
-	.quad 0x0000000000000000	/* 0xd8 - unused */
+	.quad 0x0000000000000000	/* 0xd8 - PDA */
 	.quad 0x0000000000000000	/* 0xe0 - unused */
 	.quad 0x0000000000000000	/* 0xe8 - unused */
 	.quad 0x0000000000000000	/* 0xf0 - unused */

===================================================================

@@ -39,7 +39,7 @@
  *  25 - APM BIOS support 
  *
  *  26 - ESPFIX small SS
- *  27 - unused
+ *  27 - PDA				[ per-cpu private data area ]
  *  28 - unused
  *  29 - unused
  *  30 - unused

@@ -73,6 +73,9 @@
 
 #define GDT_ENTRY_ESPFIX_SS		(GDT_ENTRY_KERNEL_BASE + 14)
 #define __ESPFIX_SS (GDT_ENTRY_ESPFIX_SS * 8)
+
+#define GDT_ENTRY_PDA			(GDT_ENTRY_KERNEL_BASE + 15)
+#define __KERNEL_PDA (GDT_ENTRY_PDA * 8)
 
 #define GDT_ENTRY_DOUBLEFAULT_TSS	31
 

===================================================================

--

When a CPU is brought up, a PDA and GDT are allocated for it. The GDT's __KERNEL_PDA entry is pointed to the allocated PDA memory, so that all references using this segment selector will refer to the PDA.

---
 arch/i386/kernel/cpu/common.c |   49 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 46 insertions(+), 3 deletions(-)

===================================================================

@@ -18,6 +18,7 @@
 #include <asm/apic.h>
 #include <mach_apic.h>
 #endif
+#include <asm/pda.h>
 
 #include "cpu.h"
 

@@ -26,6 +27,9 @@ EXPORT_PER_CPU_SYMBOL(cpu_gdt_descr);
 
 DEFINE_PER_CPU(unsigned char, cpu_16bit_stack[CPU_16BIT_STACK_SIZE]);
 EXPORT_PER_CPU_SYMBOL(cpu_16bit_stack);
+
+struct i386_pda *_cpu_pda[NR_CPUS] __read_mostly;
+EXPORT_SYMBOL(_cpu_pda);
 
 static int cachesize_override __cpuinitdata = -1;
 static int disable_x86_fxsr __cpuinitdata;

@@ -582,6 +586,26 @@ void __init early_cpu_init(void)
 	disable_pse = 1;
 #endif
 }
+
+static __cpuinit void pda_init(int cpu)
+{
+	struct i386_pda *pda = cpu_pda(cpu);
+
+	memset(pda, 0, sizeof(*pda));
+
+	pda->cpu_number = cpu;
+	pda->pcurrent = current;
+
+	printk("cpu %d current %p\n", cpu, current);
+}
+
+struct pt_regs * __devinit idle_regs(struct pt_regs *regs)
+{
+	memset(regs, 0, sizeof(struct pt_regs));
+	regs->xgs = __KERNEL_PDA;
+	return regs;
+}
+
 /*
  * cpu_init() initializes state that is per-CPU. Some data is already
  * initialized (naturally) in the bootstrap process, such as the GDT

@@ -594,6 +618,7 @@ void __cpuinit cpu_init(void)
 	struct tss_struct * t = &per_cpu(init_tss, cpu);
 	struct thread_struct *thread = &current->thread;
 	struct desc_struct *gdt;
+	struct i386_pda *pda;
 	__u32 stk16_off = (__u32)&per_cpu(cpu_16bit_stack, cpu);
 	struct Xgt_desc_struct *cpu_gdt_descr = &per_cpu(cpu_gdt_descr, cpu);
 

@@ -615,6 +640,10 @@ void __cpuinit cpu_init(void)
 	/* The CPU hotplug case */
 	if (cpu_gdt_descr->address) {
 		gdt = (struct desc_struct *)cpu_gdt_descr->address;
+		pda = cpu_pda(cpu);
+
+		BUG_ON(pda == NULL);
+
 		memset(gdt, 0, PAGE_SIZE);
 		goto old_gdt;
 	}

@@ -625,13 +654,17 @@ void __cpuinit cpu_init(void)
 	 * CPUs, when bootmem will have gone away
 	 */
 	if (NODE_DATA(0)->bdata->node_bootmem_map) {
-		gdt = (struct desc_struct *)alloc_bootmem_pages(PAGE_SIZE);
+		gdt = alloc_bootmem_pages(PAGE_SIZE);
+		pda = alloc_bootmem(sizeof(*pda));
+
 		/* alloc_bootmem_pages panics on failure, so no check */
 		memset(gdt, 0, PAGE_SIZE);
 	} else {
 		gdt = (struct desc_struct *)get_zeroed_page(GFP_KERNEL);
-		if (unlikely(!gdt)) {
-			printk(KERN_CRIT "CPU%d failed to allocate GDT\n", cpu);
+		pda = kmalloc_node(sizeof(*pda), GFP_KERNEL, cpu_to_node(cpu));
+
+		if (unlikely(!gdt || !pda)) {
+			printk(KERN_CRIT "CPU%d failed to allocate GDT or PDA\n", cpu);
 			for (;;)
 				local_irq_enable();
 		}

@@ -651,6 +684,16 @@ old_gdt:
 
 	cpu_gdt_descr->size = GDT_SIZE - 1;
  	cpu_gdt_descr->address = (unsigned long)gdt;
+
+	/* Set up the PDA in this CPU's GDT */
+	cpu_pda(cpu) = pda;
+
+	pda_init(cpu);
+
+	pack_descriptor((u32 *)&gdt[GDT_ENTRY_PDA].a,
+			(u32 *)&gdt[GDT_ENTRY_PDA].b,
+			(unsigned long)pda, sizeof(*pda) - 1,
+			0x80 | DESCTYPE_S | 0x2, 0); /* present read-write data segment */
 
 	load_gdt(cpu_gdt_descr);
 	load_idt(&idt_descr);

--

This patch is the meat of the PDA change. This patch makes several related changes:

1: Most significantly, %gs is now used in the kernel. This means that on
entry, the old value of %gs is saved away, and it is reloaded with __KERNEL_PDA.

2: entry.S constructs the stack in the shape of struct pt_regs, and this
is passed around the kernel so that the process's saved register state can be accessed.

Unfortunately struct pt_regs doesn't currently have space for %gs (or %fs). This patch extends pt_regs to add space for fs and gs, though only gs is actually saved/restored from it (reserving space for fs makes simplifies the changes to ptrace).

3: Because %gs is now saved on the stack like %ds, %es and the integer
registers, there are a number of places where it no longer needs to be handled specially.

4: And since kernel threads run in kernel space and call normal kernel
code, they need to be created with their %gs == __KERNEL_PDA.

---
 arch/i386/kernel/cpu/common.c  |    7 ++-
 arch/i386/kernel/entry.S       |   92 +++++++++++++++++++++++++++-------------
 arch/i386/kernel/process.c     |   27 +++++------
 arch/i386/kernel/signal.c      |    6 --
 arch/i386/kernel/vm86.c        |    4 -
 include/asm-i386/mmu_context.h |    4 -
 include/asm-i386/pda.h         |    2 
 include/asm-i386/processor.h   |    4 +
 include/asm-i386/ptrace-abi.h  |    2 
 9 files changed, 93 insertions(+), 55 deletions(-)

===================================================================

@@ -717,8 +717,11 @@ old_gdt:
 	__set_tss_desc(cpu, GDT_ENTRY_DOUBLEFAULT_TSS, &doublefault_tss);
 #endif
 
-	/* Clear %fs and %gs. */
-	asm volatile ("movl %0, %%fs; movl %0, %%gs" : : "r" (0));
+	/* Clear %fs. */
+	asm volatile ("mov %0, %%fs" : : "r" (0));
+
+	/* Set %gs for this CPU's PDA */
+	asm volatile ("mov %0, %%gs" : : "r" (__KERNEL_PDA));
 
 	/* Clear all 6 debug registers: */
 	set_debugreg(0, 0);

===================================================================

===================================================================

@@ -56,6 +56,7 @@
 
 #include <asm/tlbflush.h>
 #include <asm/cpu.h>
+#include <asm/pda.h>
 
 asmlinkage void ret_from_fork(void) __asm__("ret_from_fork");
 

@@ -336,6 +337,7 @@ int kernel_thread(int (*fn)(void *), voi
 
 	regs.xds = __USER_DS;
 	regs.xes = __USER_DS;
+	regs.xgs = __KERNEL_PDA;
 	regs.orig_eax = -1;
 	regs.eip = (unsigned long) kernel_thread_helper;
 	regs.xcs = __KERNEL_CS | get_kernel_rpl();

@@ -421,7 +423,6 @@ int copy_thread(int nr, unsigned long cl
 	p->thread.eip = (unsigned long) ret_from_fork;
 
 	savesegment(fs,p->thread.fs);
-	savesegment(gs,p->thread.gs);
 
 	tsk = current;
 	if (unlikely(test_tsk_thread_flag(tsk, TIF_IO_BITMAP))) {

@@ -652,16 +653,16 @@ struct task_struct fastcall * __switch_t
 	load_esp0(tss, next);
 
 	/*
-	 * Save away %fs and %gs. No need to save %es and %ds, as
-	 * those are always kernel segments while inside the kernel.
-	 * Doing this before setting the new TLS descriptors avoids
-	 * the situation where we temporarily have non-reloadable
-	 * segments in %fs and %gs.  This could be an issue if the
-	 * NMI handler ever used %fs or %gs (it does not today), or
-	 * if the kernel is running inside of a hypervisor layer.
+	 * Save away %fs. No need to save %gs, as it was saved on the
+	 * stack on entry.  No need to save %es and %ds, as those are
+	 * always kernel segments while inside the kernel.  Doing this
+	 * before setting the new TLS descriptors avoids the situation
+	 * where we temporarily have non-reloadable segments in %fs
+	 * and %gs.  This could be an issue if the NMI handler ever
+	 * used %fs or %gs (it does not today), or if the kernel is
+	 * running inside of a hypervisor layer.
 	 */
 	savesegment(fs, prev->fs);
-	savesegment(gs, prev->gs);
 
 	/*
 	 * Load the per-thread Thread-Local Storage descriptor.

@@ -669,16 +670,14 @@ struct task_struct fastcall * __switch_t
 	load_TLS(next, cpu);
 
 	/*
-	 * Restore %fs and %gs if needed.
+	 * Restore %fs if needed.
 	 *
-	 * Glibc normally makes %fs be zero, and %gs is one of
-	 * the TLS segments.
+	 * Glibc normally makes %fs be zero.
 	 */
 	if (unlikely(prev->fs | next->fs))
 		loadsegment(fs, next->fs);
 
-	if (prev->gs | next->gs)
-		loadsegment(gs, next->gs);
+	write_pda(pcurrent, next_p);
 
 	/*
 	 * Restore IOPL if needed.

===================================================================

@@ -128,7 +128,7 @@ restore_sigcontext(struct pt_regs *regs,
 			 X86_EFLAGS_TF | X86_EFLAGS_SF | X86_EFLAGS_ZF | \
 			 X86_EFLAGS_AF | X86_EFLAGS_PF | X86_EFLAGS_CF)
 
-	GET_SEG(gs);
+	COPY_SEG(gs);
 	GET_SEG(fs);
 	COPY_SEG(es);
 	COPY_SEG(ds);

@@ -244,9 +244,7 @@ setup_sigcontext(struct sigcontext __use
 {
 	int tmp, err = 0;
 
-	tmp = 0;
-	savesegment(gs, tmp);
-	err |= __put_user(tmp, (unsigned int __user *)&sc->gs);
+	err |= __put_user(regs->xgs, (unsigned int __user *)&sc->gs);
 	savesegment(fs, tmp);
 	err |= __put_user(tmp, (unsigned int __user *)&sc->fs);
 

===================================================================

@@ -294,7 +294,7 @@ static void do_sys_vm86(struct kernel_vm
 	info->regs32->eax = 0;
 	tsk->thread.saved_esp0 = tsk->thread.esp0;
 	savesegment(fs, tsk->thread.saved_fs);
-	savesegment(gs, tsk->thread.saved_gs);
+	tsk->thread.saved_gs = info->regs32->xgs;
 
 	tss = &per_cpu(init_tss, get_cpu());
 	tsk->thread.esp0 = (unsigned long) &info->VM86_TSS_ESP0;

@@ -306,7 +306,7 @@ static void do_sys_vm86(struct kernel_vm
 	tsk->thread.screen_bitmap = info->screen_bitmap;
 	if (info->flags & VM86_SCREEN_BITMAP)
 		mark_screen_rdonly(tsk->mm);
-	__asm__ __volatile__("xorl %eax,%eax; movl %eax,%fs; movl %eax,%gs\n\t");
+	__asm__ __volatile__("movl %0,%%fs\n\t" : : "r" (0));
 	__asm__ __volatile__("movl %%eax, %0\n" :"=r"(eax));
 
 	/*call audit_syscall_exit since we do not exit via the normal paths */

===================================================================

@@ -62,8 +62,8 @@ static inline void switch_mm(struct mm_s
 #endif
 }
 
-#define deactivate_mm(tsk, mm) \
-	asm("movl %0,%%fs ; movl %0,%%gs": :"r" (0))
+#define deactivate_mm(tsk, mm)			\
+	asm("movl %0,%%fs": :"r" (0));
 
 #define activate_mm(prev, next) \
 	switch_mm((prev),(next),NULL)

===================================================================

@@ -471,6 +471,7 @@ struct thread_struct {
 	.vm86_info = NULL,						\
 	.sysenter_cs = __KERNEL_CS,					\
 	.io_bitmap_ptr = NULL,						\
+	.gs = __KERNEL_PDA,						\
 }
 
 /*

@@ -498,7 +499,8 @@ static inline void load_esp0(struct tss_
 }
 
 #define start_thread(regs, new_eip, new_esp) do {		\
-	__asm__("movl %0,%%fs ; movl %0,%%gs": :"r" (0));	\
+	__asm__("movl %0,%%fs": :"r" (0));			\
+	regs->xgs = 0;						\
 	set_fs(USER_DS);					\
 	regs->xds = __USER_DS;					\
 	regs->xes = __USER_DS;					\

===================================================================

@@ -33,6 +33,8 @@ struct pt_regs {
 	long eax;
 	int  xds;
 	int  xes;
+	int  xfs;
+	int  xgs;
 	long orig_eax;
 	long eip;
 	int  xcs;

--

...

Yeah, I was considering that. I'll do a patch.

J

movl is recommended in 32bit mode

Yep. And the "m" args in the pda asm isn't quite right for rmw PDA ops (not that there are any at the moment).

OK. I thought the assembler objected to me about it.

The sigcontext contains the userspace register state at the time of the signal. Since userspace %gs is stored in the on-stack pt_regs, that should be where it fetches it from to fill out the sigcontext, rather than the kernel's internal value of %gs - in other words, it should be the same as ds and es. Or am I missing something?

Yep. But there seems to be some other very dubious code in there as well (the asm("mov %%eax,%0" : "=r" (eax)) sequence). I was wondering about what it all does...

J

arch/i386/kernel/entry.S: Assembler messages:
arch/i386/kernel/entry.S:334: Error: suffix or operands invalid for `mov'

J

Looks like a gas bug to me.

-Andi

So the suffixing (MOVW vs MOVL) does not depend on the source operand? %gs is only 16 bit wide so MOVW seems reasonable. If the destination operand was a 32 bit location, something like MOVZX would be needed, would not it?

Jan Engelhardt

Well, it generates the 32-bit form of the instruction (no 0x66 prefix) anyway.

J

This was an intentional change (by H.J. if I recall right) as using movl with segment registers gives the incorrect impression that one gets a 32-bit memory access (especially for stores this is important, since there's really nothing stored to the upper 16 bits). One should always use suffix-less 'mov' for segment register accesses.

Jan

mov will generate the optimal opcode with old and new assemblers.

H.J.

Not so much a bug as a misfeature. They changed the meaning of "movw" and "movl" on segment registers between versions.

Either way, just "mov" should work.

-hpa

There are a few places where the change in struct pt_regs and the use of %gs affect the userspace ABI. These are primarily debugging interfaces where thread state can be inspected or extracted.

---
 arch/i386/kernel/process.c |    6 +++---
 arch/i386/kernel/ptrace.c  |    8 +-------
 include/asm-i386/elf.h     |    2 +-
 include/asm-i386/unwind.h  |    1 +
 4 files changed, 6 insertions(+), 11 deletions(-)

===================================================================

@@ -305,8 +305,8 @@ void show_regs(struct pt_regs * regs)
 		regs->eax,regs->ebx,regs->ecx,regs->edx);
 	printk("ESI: %08lx EDI: %08lx EBP: %08lx",
 		regs->esi, regs->edi, regs->ebp);
-	printk(" DS: %04x ES: %04x\n",
-		0xffff & regs->xds,0xffff & regs->xes);
+	printk(" DS: %04x ES: %04x GS: %04x\n",
+	       0xffff & regs->xds,0xffff & regs->xes, 0xffff & regs->xgs);
 
 	cr0 = read_cr0();
 	cr2 = read_cr2();

@@ -500,7 +500,7 @@ void dump_thread(struct pt_regs * regs, 
 	dump->regs.ds = regs->xds;
 	dump->regs.es = regs->xes;
 	savesegment(fs,dump->regs.fs);
-	savesegment(gs,dump->regs.gs);
+	dump->regs.gs = regs->xgs;
 	dump->regs.orig_eax = regs->orig_eax;
 	dump->regs.eip = regs->eip;
 	dump->regs.cs = regs->xcs;

===================================================================

@@ -94,13 +94,9 @@ static int putreg(struct task_struct *ch
 				return -EIO;
 			child->thread.fs = value;
 			return 0;
-		case GS:
-			if (value && (value & 3) != 3)
-				return -EIO;
-			child->thread.gs = value;
-			return 0;
 		case DS:
 		case ES:
+		case GS:
 			if (value && (value & 3) != 3)
 				return -EIO;
 			value &= 0xffff;

@@ -132,8 +128,6 @@ static unsigned long getreg(struct task_
 			retval = child->thread.fs;
 			break;
 		case GS:
-			retval = child->thread.gs;
-			break;
 		case DS:
 		case ES:
 		case SS:

===================================================================

@@ -88,7 +88,7 @@ typedef struct user_fxsr_struct elf_fpxr
 	pr_reg[7] = regs->xds;				\
 	pr_reg[8] = regs->xes;				\
 	savesegment(fs,pr_reg[9]);			\
-	savesegment(gs,pr_reg[10]);			\
+	pr_reg[10] = regs->xgs;				\
 	pr_reg[11] = regs->orig_eax;			\
 	pr_reg[12] = regs->eip;				\
 	pr_reg[13] = regs->xcs;				\

===================================================================

@@ -64,6 +64,7 @@ static inline void arch_unw_init_blocked
 	info->regs.xss = __KERNEL_DS;
 	info->regs.xds = __USER_DS;
 	info->regs.xes = __USER_DS;
+	info->regs.xgs = __KERNEL_PDA;
 }
 
 extern asmlinkage int arch_unwind_init_running(struct unwind_frame_info *,

--

Yup. I'll recheck.

J

Use the cpu_number in the PDA to implement raw_smp_processor_id. This is a little simpler than using thread_info, though the cpu field in thread_info cannot be removed since it is used for things other than getting the current CPU in common code.

The slightly subtle part of this patch is dealing with very early uses of smp_processor_id(). This is handled on the boot CPU by setting up a very early PDA, which is later replaced when cpu_init() is called on the boot CPU. For other CPUs, it uses the thread_info cpu field until the PDA has been set up.

This is more or less an example of using the PDA, and to give it a proper exercising.

---
 arch/i386/kernel/asm-offsets.c |    2 +-
 arch/i386/kernel/cpu/common.c  |   16 +++++++++++++++-
 include/asm-i386/smp.h         |    3 ++-
 3 files changed, 18 insertions(+), 3 deletions(-)

===================================================================

@@ -52,7 +52,6 @@ void foo(void)
 	OFFSET(TI_exec_domain, thread_info, exec_domain);
 	OFFSET(TI_flags, thread_info, flags);
 	OFFSET(TI_status, thread_info, status);
-	OFFSET(TI_cpu, thread_info, cpu);
 	OFFSET(TI_preempt_count, thread_info, preempt_count);
 	OFFSET(TI_addr_limit, thread_info, addr_limit);
 	OFFSET(TI_restart_block, thread_info, restart_block);

@@ -78,4 +77,5 @@ void foo(void)
 
 	BLANK();
 	OFFSET(PDA_pcurrent, i386_pda, pcurrent);
+	OFFSET(PDA_cpu, i386_pda, cpu_number);
 }

===================================================================

@@ -606,6 +606,20 @@ struct pt_regs * __devinit idle_regs(str
 	return regs;
 }
 
+/* Set up a very early PDA for the boot CPU so that smp_processor_id will work */
+void __init smp_setup_processor_id(void)
+{
+	static const __initdata struct i386_pda boot_pda;
+
+	pack_descriptor((u32 *)&cpu_gdt_table[GDT_ENTRY_PDA].a,
+			(u32 *)&cpu_gdt_table[GDT_ENTRY_PDA].b,
+			(unsigned long)&boot_pda, sizeof(struct i386_pda) - 1,
+			0x80 | DESCTYPE_S | 0x2, 0); /* present read-write data segment */
+
+	/* Set %gs for this CPU's PDA */
+	asm volatile ("mov %0, %%gs" : : "r" (__KERNEL_PDA));
+}
+
 /*
  * cpu_init() initializes state that is per-CPU. Some data is already
  * initialized (naturally) in the bootstrap process, such as the GDT

@@ -614,7 +628,7 @@ struct pt_regs * __devinit idle_regs(str
  */
 void __cpuinit cpu_init(void)
 {
-	int cpu = smp_processor_id();
+	int cpu = current_thread_info()->cpu; /* need to use this until the PDA is set up */
 	struct tss_struct * t = &per_cpu(init_tss, cpu);
 	struct thread_struct *thread = &current->thread;
 	struct desc_struct *gdt;

===================================================================

@@ -8,6 +8,7 @@
 #include <linux/kernel.h>
 #include <linux/threads.h>
 #include <linux/cpumask.h>
+#include <asm/pda.h>
 #endif
 
 #ifdef CONFIG_X86_LOCAL_APIC

@@ -58,7 +59,7 @@ extern void cpu_uninit(void);
  * from the initial startup. We map APIC_BASE very early in page_setup(),
  * so this is correct in the x86 case.
  */
-#define raw_smp_processor_id() (current_thread_info()->cpu)
+#define raw_smp_processor_id() (read_pda(cpu_number))
 
 extern cpumask_t cpu_callout_map;
 extern cpumask_t cpu_callin_map;

--

Use the pcurrent field in the PDA to implement the "current" macro. This ends up compiling down to a single instruction to get the current task.

The slightly tricky part about this patch is that cpu_init() uses current very early, before the PDA has been set up. In this instance, it uses current_thread_info()->task to get the current task.

Also, the very early PDA set up for the boot cpu contains the initial task pointer so current works from a very early stage.

--

 arch/i386/kernel/cpu/common.c |   27 ++++++++++++++++-----------
 include/asm-i386/current.h    |   11 ++---------
 2 files changed, 18 insertions(+), 20 deletions(-)

===================================================================

@@ -1,15 +1,8 @@
 #ifndef _I386_CURRENT_H
 #define _I386_CURRENT_H
 
-#include <linux/thread_info.h>
+#include <asm/pda.h>
 
-struct task_struct;
-
-static __always_inline struct task_struct * get_current(void)
-{
-	return current_thread_info()->task;
-}
- 
-#define current get_current()
+#define current (read_pda(pcurrent))
 
 #endif /* !(_I386_CURRENT_H) */

===================================================================

@@ -587,16 +587,16 @@ void __init early_cpu_init(void)
 #endif
 }
 
-static __cpuinit void pda_init(int cpu)
+static __cpuinit void pda_init(int cpu, struct task_struct *curr)
 {
 	struct i386_pda *pda = cpu_pda(cpu);
 
 	memset(pda, 0, sizeof(*pda));
 
 	pda->cpu_number = cpu;
-	pda->pcurrent = current;
-
-	printk("cpu %d current %p\n", cpu, current);
+	pda->pcurrent = curr;
+
+	printk("cpu %d current %p\n", cpu, curr);
 }
 
 struct pt_regs * __devinit idle_regs(struct pt_regs *regs)

@@ -609,7 +609,7 @@ struct pt_regs * __devinit idle_regs(str
 /* Set up a very early PDA for the boot CPU so that smp_processor_id will work */
 void __init smp_setup_processor_id(void)
 {
-	static const __initdata struct i386_pda boot_pda;
+	static __initdata struct i386_pda boot_pda;
 
 	pack_descriptor((u32 *)&cpu_gdt_table[GDT_ENTRY_PDA].a,
 			(u32 *)&cpu_gdt_table[GDT_ENTRY_PDA].b,

@@ -618,6 +618,8 @@ void __init smp_setup_processor_id(void)
 
 	/* Set %gs for this CPU's PDA */
 	asm volatile ("mov %0, %%gs" : : "r" (__KERNEL_PDA));
+
+	boot_pda.pcurrent = current_thread_info()->task;
 }
 
 /*

@@ -628,9 +630,12 @@ void __init smp_setup_processor_id(void)
  */
 void __cpuinit cpu_init(void)
 {
-	int cpu = current_thread_info()->cpu; /* need to use this until the PDA is set up */
+	/* Need to use thread_info for cpu and current until the PDA is set up */
+	int cpu = current_thread_info()->cpu;
+	struct task_struct *curr = current_thread_info()->task;
+
 	struct tss_struct * t = &per_cpu(init_tss, cpu);
-	struct thread_struct *thread = &current->thread;
+	struct thread_struct *thread = &curr->thread;
 	struct desc_struct *gdt;
 	struct i386_pda *pda;
 	__u32 stk16_off = (__u32)&per_cpu(cpu_16bit_stack, cpu);

@@ -702,7 +707,7 @@ old_gdt:
 	/* Set up the PDA in this CPU's GDT */
 	cpu_pda(cpu) = pda;
 
-	pda_init(cpu);
+	pda_init(cpu, curr);
 
 	pack_descriptor((u32 *)&gdt[GDT_ENTRY_PDA].a,
 			(u32 *)&gdt[GDT_ENTRY_PDA].b,

@@ -716,10 +721,10 @@ old_gdt:
 	 * Set up and load the per-CPU TSS and LDT
 	 */
 	atomic_inc(&init_mm.mm_count);
-	current->active_mm = &init_mm;
-	if (current->mm)
+	curr->active_mm = &init_mm;
+	if (curr->mm)
 		BUG();
-	enter_lazy_tlb(&init_mm, current);
+	enter_lazy_tlb(&init_mm, curr);
 
 	load_esp0(t, thread);
 	set_tss_desc(cpu,t);

--

With that you could remove the code in do_IRQ now that setups up a fake thread_info for interrupt stacks.

-Andi

Oh, OK. I was wondering what that was for.

J

this will be interesting; x86-64 has a nice instruction to help with this; 32 bit does not... so far conventional wisdom has been that without the instruction it's not going to be worth it.

When you're benchmarking this please use multiple CPU generations from different vendors; I suspect this is one of those things that vary greatly between models

A config option for this is a mistake imo. Not every patch is worth a config option! It's good or it's not, if it's good it should be there always, if it's not.... Something this fundamental to the core doesn't really have a "but it's optional" argument going for it, unlike individual drivers or subsystems...

if you want to mail me at work (you don't), use arjan (at) linux.intel.com

Hm, swapgs may be quick, but it isn't very easy to use since it doesn't stack, and so requires careful handling for recursive kernel entries, which involves extra tests and conditional jumps. I tried doing something similar with my earlier patches, but it got all too messy. Stacking %gs like the other registers turns out pretty cleanly.

Hm, it seems to me that unless the existing %ds/%es register save/restores are a significant part of the existing cost of going through entry.S, adding %gs to the set shouldn't make too much difference. And I'm not sure about the relative cost of using a %gs override vs. the normal current_task_info() masking, but I'm assuming they're at worst equal, with the %gs override having a code-size advantage.

But yes, it definitely needs measurement.

J

iirc the %fs one is at least. But it has been a while since I've looked at this part of the kernel via performance traces.

your worst case scenario would be if the segment override would make it a "complex" instruction, so not parallel decodable. That'd mean it would basically cost you 6 or 7 instruction slots that can't be filled... while an and and such at least run nicely in parallel with other stuff. I don't know which if any processors actually do this, but it's rare/new enough that I'd not be surprised if there are some.

if you want to mail me at work (you don't), use arjan (at) linux.intel.com

On AMD K7/K8 a segment register prefix is a single cycle penalty.

I couldn't find anything in the Intel optimization manuals on it, but I assume it's also not dramatic.

-Andi

All I could find was:

* avoid multiple prefixes (which was the least important guideline in instruction selection)
* avoid using multiple segment registers (the pentium M only has one level of segment register renaming)
* avoid prefixes which take the instruction length over 7 bytes

None of these apply to the use of %gs to access PDA.

Most of the discussion about prefixes is in avoiding the 0x66 16-bit prefix.

J

Very cool.

I bet qemu doesn't have a real descriptor cache unlike real CPUs. So likely it is some disconnect between changing the backing GDT and referencing the register. Reload %gs more aggressively?

Comparing with SimNow! (which should behave more like a real CPU) might be also interesting.

per cpu data would be the prime candidate. It is pretty simple.

Please don't.

(weak point:)

- The stack protector code might work one day on i386 too.

-Andi

The GDT only gets touched once in cpu_init(), and %gs is reloaded on every kernel entry, so I don't think that's it. I seems to have interrupt issues with SMP.

And either way, it still doesn't work on real hardware...

Yeah, I'll have to try that out.

Well, it has to be arch-specific per-cpu data, since the PDA is arch specific. But there should be various pieces of interrupt state that adapt well to it.

Yeah, that wasn't really a serious thought...

J

Hi,

Yess!!
Something like that had to be done eventually about the inefficient current_thread_info() mechanism, but I wasn't sure what exactly.

This is interesting, since even by doing a non-elegant current->... --> struct task_struct *tsk = current;
replacement for excessive uses of current, I was able to gain almost 10kB within a single file already!
I guess it's due to having tried that on an older installation with gcc 3.2, which probably does less efficient opcode merging of current_thread_info() requests compared to a current gcc version.
IOW, .text segment reduction could be quite a bit higher for older gcc:s.

Not a problem for more daring user-space apps (i.e. Wine), I hope?

Andreas Mohr

No programming skills!? Why not help translate many Linux applications! 
https://launchpad.ubuntu.com/rosetta
(or alternatively buy nicely packaged Linux distros/OSS software to help
support Linux developers creating shiny new things for you?)

That doesn't sound likely. current_thread_info() is only about 2 instructions. Are you looking at the .text segment size, or the actual file size? The latter can be very misleading in the presence of debug info.

No. The LDT is still available for userspace use.

J

Hi,

Got me. File size only. And that was 10kB of around 170kB, BTW.

Andreas Mohr

Inefficient? It's two fast instructions. I won't call that inefficient.

gcc normally doesn't merge inline assembly at all.

-Andi

Hi,

And that AGI stall?

Depends on use of volatile, right?

OK, so probably there was no merging of separate requests, but opcode intermingling could have played a role.

Andreas Mohr

What AGI stall?

[btw AGI stall is an outdated concept on modern x86 CPUs]

No. It can only merge statements it knows anything about, and it doesn't about inline assembly.

It seems to make some difference if it's able to move asm around and if they don't have memory clobbers. memory clobbers really seem to cause much worse code in the whole function.

But current_thread_info didn't have that.

-Andi