Skip to contentSkip to conversation

PowerPC context switch optimisations

PATCH In review Anton Blanchard 3 participants 13 messages Jul 14, 2009 – Oct 4, 2009 1 message not in the archive PowerPC

lore
Share & tools
lore
#0 lore raw

cover PowerPC context switch optimisations

Here are a few optimisations that improve context switch performance.

#1 lore raw

PATCH 1/3 powerpc: Move 64bit VDSO to improve context switch performance

On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO is position independent we can remove the hint and let get_unmapped_area pick an area. This will mean the vdso will be near other mmaps and will share an SLB entry:

10000000-10001000 r-xp 00000000 08:06 5778459        /root/context_switch_64
10010000-10011000 r--p 00000000 08:06 5778459        /root/context_switch_64
10011000-10012000 rw-p 00001000 08:06 5778459        /root/context_switch_64
fffa92ae000-fffa92b0000 rw-p 00000000 00:00 0 
fffa92b0000-fffa9453000 r-xp 00000000 08:06 4334051  /lib64/power6/libc-2.9.so
fffa9453000-fffa9462000 ---p 001a3000 08:06 4334051  /lib64/power6/libc-2.9.so
fffa9462000-fffa9466000 r--p 001a2000 08:06 4334051  /lib64/power6/libc-2.9.so
fffa9466000-fffa947c000 rw-p 001a6000 08:06 4334051  /lib64/power6/libc-2.9.so

fffa947c000-fffa9480000 rw-p 00000000 00:00 0 fffa9480000-fffa94a8000 r-xp 00000000 08:06 4333852 /lib64/ld-2.9.so fffa94b3000-fffa94b4000 rw-p 00000000 00:00 0

fffa94b4000-fffa94b7000 r-xp 00000000 00:00 0 [vdso] <----- here I am

fffa94b7000-fffa94b8000 r--p 00027000 08:06 4333852 /lib64/ld-2.9.so fffa94b8000-fffa94bb000 rw-p 00028000 08:06 4333852 /lib64/ld-2.9.so fffa94bb000-fffa94bc000 rw-p 00000000 00:00 0
fffe4c10000-fffe4c25000 rw-p 00000000 00:00 0 [stack]

On a microbenchmark that bounces a token between two 64bit processes over pipes and calls gettimeofday each iteration (to access the VDSO), our context switch rate goes from 268k to 277k ctx switches/sec (tested on a 4GHz POWER6).

---

arch/powerpc/include/asm/vdso.h +1 -2
Index: linux.trees.git/arch/powerpc/include/asm/vdso.h
===================================================================
@@ -7,9 +7,8 @@
 #define VDSO32_LBASE	0x100000
 #define VDSO64_LBASE	0x100000
 
-/* Default map addresses */
+/* Default map addresses for 32bit vDSO */
 #define VDSO32_MBASE	VDSO32_LBASE
-#define VDSO64_MBASE	VDSO64_LBASE
 
 #define VDSO_VERSION_STRING	LINUX_2.6.15
 
arch/powerpc/kernel/vdso.c +6 -1
Index: linux.trees.git/arch/powerpc/kernel/vdso.c
===================================================================
@@ -203,7 +203,12 @@
 	} else {
 		vdso_pagelist = vdso64_pagelist;
 		vdso_pages = vdso64_pages;
-		vdso_base = VDSO64_MBASE;
+		/*
+		 * On 64bit we don't have a preferred map address. This
+		 * allows get_unmapped_area to find an area near other mmaps
+		 * and most likely share a SLB entry.
+		 */
+		vdso_base = 0;
 	}
 #else
 	vdso_pagelist = vdso32_pagelist;
#2 lore raw

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

Quoted 5 lines · Anton Blanchard

plain text document attachment (move_vdso_v2)
On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO is position independent we can remove the hint and let get_unmapped_area pick an area. This will mean the vdso will be near other mmaps and will share an SLB entry:

Don't we lose randomization ? Or do we randomize the whole mem map nowadays ?

Cheers,
Ben.

#3 lore raw

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

Hi Ben,

Quoted 2 lines

Don't we lose randomization ? Or do we randomize the whole mem map nowadays ?

The start of the top down mmap region is randomized, so the VDSO will be in a different position each time. A quick example:

run 1:

fffb01f6000-fffb01f9000 r-xp 00000000 00:00 0         [vdso]
fffb01f9000-fffb01fa000 r--p 00027000 08:06 4333852   /lib64/ld-2.9.so
fffb01fa000-fffb01fd000 rw-p 00028000 08:06 4333852   /lib64/ld-2.9.so

fffb01fd000-fffb01fe000 rw-p 00000000 00:00 0
ffff7c6f000-ffff7c84000 rw-p 00000000 00:00 0 [stack]

run 2:

fff9a094000-fff9a097000 r-xp 00000000 00:00 0         [vdso]  
fff9a097000-fff9a098000 r--p 00027000 08:06 4333852   /lib64/ld-2.9.so
fff9a098000-fff9a09b000 rw-p 00028000 08:06 4333852   /lib64/ld-2.9.so

fff9a09b000-fff9a09c000 rw-p 00000000 00:00 0
fffea0a6000-fffea0bb000 rw-p 00000000 00:00 0 [stack]

You will notice we aren't randomising each mmap, so the relative offset between ld.so and the vdso will be consistent. I just checked and it looks like x86 does the same.

It might make sense to add a small amount of randomness between mmaps on both x86 and PowerPC, at least for 64bit applications where we have enough address space.

Anton

#4 lore raw

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

In reply to Anton Blanchard
Quoted 3 lines · Anton Blanchard

On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO is position independent we can remove the hint and let get_unmapped_area pick an area.

This breaks gdb. The section table in the VDSO image when mapped into the process no longer contains meaningful values, and gdb rejects it.

Andreas.

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

Quoted 8 lines · Andreas Schwab
Quoted 3 lines · Anton Blanchard

On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO is position independent we can remove the hint and let get_unmapped_area pick an area.

This breaks gdb. The section table in the VDSO image when mapped into the process no longer contains meaningful values, and gdb rejects it.

The problem is that the load segment requires 64k alignment, but the page allocater of course only provides PAGE_SIZE alignment, causing the image to be unaligned in memory.

Andreas.

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#7 lore raw

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

Quoted 8 lines · Andreas Schwab
Quoted 6 lines · Andreas Schwab
Quoted 3 lines · Anton Blanchard

On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO is position independent we can remove the hint and let get_unmapped_area pick an area.

Quoted 6 lines
Quoted 2 lines

This breaks gdb. The section table in the VDSO image when mapped into the process no longer contains meaningful values, and gdb rejects it.

The problem is that the load segment requires 64k alignment, but the page allocater of course only provides PAGE_SIZE alignment, causing the image to be unaligned in memory.

Here is a patch. The disadvantage is that now gdb complains that the vdso does not contain section headers, because they are no longer covered by the load segment.

Andreas.
---

Quoted 1 line

From 003c323c753d8717e58220c9560329882bcfa2c2 Mon Sep 17 00:00:00 2001

From: Andreas Schwab <[email protected]>
Date: Sat, 3 Oct 2009 16:46:45 +0200
Subject: [PATCH] powerpc: fix unaligned load segment in vdso64

Since the page allocator can only guarantee PAGE_SIZE alignment the load segment in the vdso object cannot claim more than that alignment. Use 4k alignment which is the minimum possible page size.

---
 arch/powerpc/kernel/vdso64/Makefile |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
arch/powerpc/kernel/vdso64/Makefile +2 -0
index 79da65d..d4be303 100644
@@ -10,7 +10,9 @@ obj-vdso64 := $(addprefix $(obj)/, $(obj-vdso64))
 GCOV_PROFILE := n
 
 EXTRA_CFLAGS := -shared -fno-common -fno-builtin
+# Force maximum page size of 4k so that it works with any page size
 EXTRA_CFLAGS += -nostdlib -Wl,-soname=linux-vdso64.so.1 \
+		-Wl,-z,max-page-size=4096 \
 		$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
 EXTRA_AFLAGS := -D__VDSO64__ -s
 
Signature · 7 lines
1.6.4.4


-- 
Andreas Schwab, [email protected]
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#8 lore raw

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

Quoted 13 lines · Andreas Schwab
Quoted 11 lines · Andreas Schwab
Quoted 8 lines · Andreas Schwab
Quoted 3 lines · Anton Blanchard

On 64bit applications the VDSO is the only thing in segment 0. Since the VDSO is position independent we can remove the hint and let get_unmapped_area pick an area.

This breaks gdb. The section table in the VDSO image when mapped into the process no longer contains meaningful values, and gdb rejects it.

Quoted 7 lines
Quoted 3 lines

The problem is that the load segment requires 64k alignment, but the page allocater of course only provides PAGE_SIZE alignment, causing the image to be unaligned in memory.

Here is a patch. The disadvantage is that now gdb complains that the vdso does not contain section headers, because they are no longer covered by the load segment.

Maybe a better fix is to force alignment in the kernel by requesting size + 64k - 4k and aligning it.

Ben.

Quoted 34 lines

Andreas.
---

Quoted 1 line

From 003c323c753d8717e58220c9560329882bcfa2c2 Mon Sep 17 00:00:00 2001

From: Andreas Schwab <[email protected]>
Date: Sat, 3 Oct 2009 16:46:45 +0200
Subject: [PATCH] powerpc: fix unaligned load segment in vdso64

Since the page allocator can only guarantee PAGE_SIZE alignment the load segment in the vdso object cannot claim more than that alignment. Use 4k alignment which is the minimum possible page size.

---
 arch/powerpc/kernel/vdso64/Makefile |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
arch/powerpc/kernel/vdso64/Makefile +2 -0
index 79da65d..d4be303 100644
@@ -10,7 +10,9 @@ obj-vdso64 := $(addprefix $(obj)/, $(obj-vdso64))
 GCOV_PROFILE := n
 
 EXTRA_CFLAGS := -shared -fno-common -fno-builtin
+# Force maximum page size of 4k so that it works with any page size
 EXTRA_CFLAGS += -nostdlib -Wl,-soname=linux-vdso64.so.1 \
+		-Wl,-z,max-page-size=4096 \
 		$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
 EXTRA_AFLAGS := -D__VDSO64__ -s
 
Signature · 1 line
1.6.4.4
#9 lore raw

Re: [patch 1/3] powerpc: Move 64bit VDSO to improve context switch performance

Quoted 2 lines · Benjamin Herrenschmidt

Maybe a better fix is to force alignment in the kernel by requesting size + 64k - 4k and aligning it.

Sure you are right.

Andreas.

Quoted 1 line

From b9441a3d2148d439e2730def3222a7b70dccc432 Mon Sep 17 00:00:00 2001

From: Andreas Schwab <[email protected]>
Date: Sun, 4 Oct 2009 14:29:04 +0200
Subject: [PATCH] powerpc: align vDSO base address

---
 arch/powerpc/kernel/vdso.c |   11 ++++++++++-
 1 files changed, 10 insertions(+), 1 deletions(-)
arch/powerpc/kernel/vdso.c +10 -1
index 94e2df3..137dc22 100644
@@ -50,6 +50,9 @@
 /* Max supported size for symbol names */
 #define MAX_SYMNAME	64
 
+/* The alignment of the vDSO */
+#define VDSO_ALIGNMENT	(1 << 16)
+
 extern char vdso32_start, vdso32_end;
 static void *vdso32_kbase = &vdso32_start;
 static unsigned int vdso32_pages;
@@ -231,15 +234,21 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
 	 * pick a base address for the vDSO in process space. We try to put it
 	 * at vdso_base which is the "natural" base for it, but we might fail
 	 * and end up putting it elsewhere.
+	 * Add enough to the size so that the result can be aligned.
 	 */
 	down_write(&mm->mmap_sem);
 	vdso_base = get_unmapped_area(NULL, vdso_base,
-				      vdso_pages << PAGE_SHIFT, 0, 0);
+				      (vdso_pages << PAGE_SHIFT) +
+				      ((VDSO_ALIGNMENT - 1) & PAGE_MASK),
+				      0, 0);
 	if (IS_ERR_VALUE(vdso_base)) {
 		rc = vdso_base;
 		goto fail_mmapsem;
 	}
 
+	/* Add required alignment. */
+	vdso_base = ALIGN(vdso_base, VDSO_ALIGNMENT);
+
 	/*
 	 * Put vDSO base into mm struct. We need to do this before calling
 	 * install_special_mapping or the perf counter mmap tracking code
Signature · 7 lines
1.6.4.4


-- 
Andreas Schwab, [email protected]
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#10 lore raw

PATCH 2/3 powerpc: Rearrange SLB preload code

In reply to Anton Blanchard

With the new top down layout it is likely that the pc and stack will be in the same segment, because the pc is most likely in a library allocated via a top down mmap. Right now we bail out early if these segments match.

Rearrange the SLB preload code to sanity check all SLB preload addresses are not in the kernel, then check all addresses for conflicts.

---

arch/powerpc/mm/slb.c +8 -13
Index: linux.trees.git/arch/powerpc/mm/slb.c
===================================================================
@@ -225,23 +225,18 @@
 	else
 		unmapped_base = TASK_UNMAPPED_BASE_USER64;
 
-	if (is_kernel_addr(pc))
-		return;
-	slb_allocate(pc);
-
-	if (esids_match(pc,stack))
+	if (is_kernel_addr(pc) || is_kernel_addr(stack) ||
+	    is_kernel_addr(unmapped_base))
 		return;
 
-	if (is_kernel_addr(stack))
-		return;
-	slb_allocate(stack);
+	slb_allocate(pc);
 
-	if (esids_match(pc,unmapped_base) || esids_match(stack,unmapped_base))
-		return;
+	if (!esids_match(pc, stack))
+		slb_allocate(stack);
 
-	if (is_kernel_addr(unmapped_base))
-		return;
-	slb_allocate(unmapped_base);
+	if (!esids_match(pc, unmapped_base) &&
+	    !esids_match(stack, unmapped_base))
+		slb_allocate(unmapped_base);
 }
 
 static inline void patch_slb_encoding(unsigned int *insn_addr,
#11 lore raw

PATCH 3/3 powerpc: Preload application text segment instead of TASK_UNMAPPED_BASE

In reply to Anton Blanchard

TASK_UNMAPPED_BASE is not used with the new top down mmap layout. We can reuse this preload slot by loading in the segment at 0x10000000, where almost all PowerPC binaries are linked at.

On a microbenchmark that bounces a token between two 64bit processes over pipes and calls gettimeofday each iteration (to access the VDSO), both the 32bit and 64bit context switch rate improves (tested on a 4GHz POWER6):

32bit: 273k/sec -> 283k/sec
64bit: 277k/sec -> 284k/sec

---

arch/powerpc/mm/slb.c +8 -9
Index: linux.trees.git/arch/powerpc/mm/slb.c
===================================================================
@@ -191,7 +191,7 @@
 	unsigned long slbie_data = 0;
 	unsigned long pc = KSTK_EIP(tsk);
 	unsigned long stack = KSTK_ESP(tsk);
-	unsigned long unmapped_base;
+	unsigned long exec_base;
 
 	if (!cpu_has_feature(CPU_FTR_NO_SLBIE_B) &&
 	    offset <= SLB_CACHE_ENTRIES) {
@@ -219,14 +219,13 @@
 
 	/*
 	 * preload some userspace segments into the SLB.
+	 * Almost all 32 and 64bit PowerPC executables are linked at
+	 * 0x10000000 so it makes sense to preload this segment.
 	 */
-	if (test_tsk_thread_flag(tsk, TIF_32BIT))
-		unmapped_base = TASK_UNMAPPED_BASE_USER32;
-	else
-		unmapped_base = TASK_UNMAPPED_BASE_USER64;
+	exec_base = 0x10000000;
 
 	if (is_kernel_addr(pc) || is_kernel_addr(stack) ||
-	    is_kernel_addr(unmapped_base))
+	    is_kernel_addr(exec_base))
 		return;
 
 	slb_allocate(pc);
@@ -234,9 +233,9 @@
 	if (!esids_match(pc, stack))
 		slb_allocate(stack);
 
-	if (!esids_match(pc, unmapped_base) &&
-	    !esids_match(stack, unmapped_base))
-		slb_allocate(unmapped_base);
+	if (!esids_match(pc, exec_base) &&
+	    !esids_match(stack, exec_base))
+		slb_allocate(exec_base);
 }
 
 static inline void patch_slb_encoding(unsigned int *insn_addr,
#12 lore raw

Re: [patch 3/3] powerpc: Preload application text segment instead of TASK_UNMAPPED_BASE

Quoted 11 lines · Anton Blanchard

plain text document attachment (preload_0x10000000) TASK_UNMAPPED_BASE is not used with the new top down mmap layout. We can reuse this preload slot by loading in the segment at 0x10000000, where almost all PowerPC binaries are linked at.

On a microbenchmark that bounces a token between two 64bit processes over pipes and calls gettimeofday each iteration (to access the VDSO), both the 32bit and 64bit context switch rate improves (tested on a 4GHz POWER6):

32bit: 273k/sec -> 283k/sec
64bit: 277k/sec -> 284k/sec

Any chance you can put that little test program online somewhere ?

Cheers,
Ben.