Skip to contentSkip to conversation

Improved VSID allocation algorithm

PPC64 David Gibson 3 participants 10 messages Sep 13, 2004 – Sep 30, 2004

lore
Share & tools
lore
#0 lore raw

Andrew, please apply. This patch has been tested both on SLB and segment table machines. This new approach is far from the final word in VSID/context allocation, but it's a noticeable improvement on the old method.

Replace the VSID allocation algorithm. The new algorithm first generates a 36-bit "proto-VSID" (with 0xfffffffff reserved). For kernel addresses this is equal to the ESID (address >> 28), for user addresses it is:

	(context << 15) | (esid & 0x7fff)

These are distinguishable from kernel proto-VSIDs because the top bit is clear. Proto-VSIDs with the top two bits equal to 0b10 are reserved for now.

The proto-VSIDs are then scrambled into real VSIDs with the multiplicative hash:

	VSID = (proto-VSID * VSID_MULTIPLIER) % VSID_MODULUS
	where	VSID_MULTIPLIER = 268435399 = 0xFFFFFC7
		VSID_MODULUS = 2^36-1 = 0xFFFFFFFFF

This scramble is 1:1, because VSID_MULTIPLIER and VSID_MODULUS are co-prime since VSID_MULTIPLIER is prime (the largest 28-bit prime, in fact).

This scheme has a number of advantages over the old one:
- We now have VSIDs for every kernel address (i.e. everything
above 0xC000000000000000), except the very top segment. That simplifies a number of things.
- We allow for 15 significant bits of ESID for user addresses
with 20 bits of context. i.e. 8T (43 bits) of address space for up to 1M contexts, significantly more than the old method (although we will need changes in the hash path and context allocation to take advantage of this).
- Because we use a real multiplicative hash function, we have
better and more robust hash scattering with this VSID algorithm (at least based on some initial results).

Because the MODULUS is 2^n-1 we can use a trick to compute it efficiently without a divide or extra multiply. This makes the new algorithm barely slower than the old one.

include/asm-ppc64/mmu_context.h +77 -36
Index: working-2.6/include/asm-ppc64/mmu_context.h
===================================================================
@@ -34,7 +34,7 @@
 }
 
 #define NO_CONTEXT		0
-#define FIRST_USER_CONTEXT	0x10    /* First 16 reserved for kernel */
+#define FIRST_USER_CONTEXT	1
 #define LAST_USER_CONTEXT	0x8000  /* Same as PID_MAX for now... */
 #define NUM_USER_CONTEXT	(LAST_USER_CONTEXT-FIRST_USER_CONTEXT)
 
@@ -181,46 +181,87 @@
 	local_irq_restore(flags);
 }
 
-/* This is only valid for kernel (including vmalloc, imalloc and bolted) EA's
+/* VSID allocation
+ * ===============
+ *
+ * We first generate a 36-bit "proto-VSID".  For kernel addresses this
+ * is equal to the ESID, for user addresses it is:
+ *	(context << 15) | (esid & 0x7fff)
+ *
+ * The two forms are distinguishable because the top bit is 0 for user
+ * addresses, whereas the top two bits are 1 for kernel addresses.
+ * Proto-VSIDs with the top two bits equal to 0b10 are reserved for
+ * now.
+ *
+ * The proto-VSIDs are then scrambled into real VSIDs with the
+ * multiplicative hash:
+ *
+ *	VSID = (proto-VSID * VSID_MULTIPLIER) % VSID_MODULUS
+ *	where	VSID_MULTIPLIER = 268435399 = 0xFFFFFC7
+ *		VSID_MODULUS = 2^36-1 = 0xFFFFFFFFF
+ *
+ * This scramble is only well defined for proto-VSIDs below
+ * 0xFFFFFFFFF, so both proto-VSID and actual VSID 0xFFFFFFFFF are
+ * reserved.  VSID_MULTIPLIER is prime (the largest 28-bit prime, in
+ * fact), so in particular it is co-prime to VSID_MODULUS, making this
+ * a 1:1 scrambling function.  Because the modulus is 2^n-1 we can
+ * compute it efficiently without a divide or extra multiply (see
+ * below).
+ *
+ * This scheme has several advantages over older methods:
+ *
+ * 	- We have VSIDs allocated for every kernel address
+ * (i.e. everything above 0xC000000000000000), except the very top
+ * segment, which simplifies several things.
+ *
+ * 	- We allow for 15 significant bits of ESID and 20 bits of
+ * context for user addresses.  i.e. 8T (43 bits) of address space for
+ * up to 1M contexts (although the page table structure and context
+ * allocation will need changes to take advantage of this).
+ *
+ * 	- The scramble function gives robust scattering in the hash
+ * table (at least based on some initial results).  The previous
+ * method was more susceptible to pathological cases giving excessive
+ * hash collisions.
  */
-static inline unsigned long
-get_kernel_vsid( unsigned long ea )
-{
-	unsigned long ordinal, vsid;
-	
-	ordinal = (((ea >> 28) & 0x1fff) * LAST_USER_CONTEXT) | (ea >> 60);
-	vsid = (ordinal * VSID_RANDOMIZER) & VSID_MASK;
-
-#ifdef HTABSTRESS
-	/* For debug, this path creates a very poor vsid distribuition.
-	 * A user program can access virtual addresses in the form
-	 * 0x0yyyyxxxx000 where yyyy = xxxx to cause multiple mappings
-	 * to hash to the same page table group.
-	 */
-	ordinal = ((ea >> 28) & 0x1fff) | (ea >> 44);
-	vsid = ordinal & VSID_MASK;
-#endif /* HTABSTRESS */
 
-	return vsid;
-} 
+/*
+ * WARNING - If you change these you must make sure the asm
+ * implementations in slb_allocate(), do_stab_bolted and mmu.h
+ * (ASM_VSID_SCRAMBLE macro) are changed accordingly.
+ *
+ * You'll also need to change the precomputed VSID values in head.S
+ * which are used by the iSeries firmware.
+ */
+
+static inline unsigned long vsid_scramble(unsigned long protovsid)
+{
+#if 0
+	/* The code below is equivalent to this function for arguments
+	 * < 2^VSID_BITS, which is all this should ever be called
+	 * with.  However gcc is not clever enough to compute the
+	 * modulus (2^n-1) without a second multiply. */
+	return ((protovsid * VSID_MULTIPLIER) % VSID_MODULUS);
+#else /* 1 */
+	unsigned long x;
+
+	x = protovsid * VSID_MULTIPLIER;
+	x = (x >> VSID_BITS) + (x & VSID_MODULUS);
+	return (x + ((x+1) >> VSID_BITS)) & VSID_MODULUS;
+#endif /* 1 */
+}
 
-/* This is only valid for user EA's (user EA's do not exceed 2^41 (EADDR_SIZE))
- */
-static inline unsigned long
-get_vsid( unsigned long context, unsigned long ea )
+/* This is only valid for addresses >= KERNELBASE */
+static inline unsigned long get_kernel_vsid(unsigned long ea)
 {
-	unsigned long ordinal, vsid;
-
-	ordinal = (((ea >> 28) & 0x1fff) * LAST_USER_CONTEXT) | context;
-	vsid = (ordinal * VSID_RANDOMIZER) & VSID_MASK;
-
-#ifdef HTABSTRESS
-	/* See comment above. */
-	ordinal = ((ea >> 28) & 0x1fff) | (context << 16);
-	vsid = ordinal & VSID_MASK;
-#endif /* HTABSTRESS */
+	return vsid_scramble(ea >> SID_SHIFT);
+} 
 
-	return vsid;
+/* This is only valid for user addresses (which are below 2^41) */
+static inline unsigned long get_vsid(unsigned long context, unsigned long ea)
+{
+	return vsid_scramble((context << USER_ESID_BITS)
+			     | (ea >> SID_SHIFT));
 }
 
 #endif /* __PPC64_MMU_CONTEXT_H */
include/asm-ppc64/mmu.h +39 -6

57 lines not on this page · raw message

arch/ppc64/mm/slb_low.S +21 -32

87 lines not on this page · raw message

arch/ppc64/kernel/head.S +8 -17
Index: working-2.6/arch/ppc64/kernel/head.S
===================================================================
@@ -548,15 +548,15 @@
 	.llong	0		/* Reserved */
 	.llong	0		/* Reserved */
 	.llong	0		/* Reserved */
-	.llong	0xc00000000	/* KERNELBASE ESID */
-	.llong	0x6a99b4b14	/* KERNELBASE VSID */
+	.llong	(KERNELBASE>>SID_SHIFT)
+	.llong	0x40bffffd5	/* KERNELBASE VSID */
 	/* We have to list the bolted VMALLOC segment here, too, so that it
 	 * will be restored on shared processor switch */
-	.llong	0xd00000000	/* VMALLOCBASE ESID */
-	.llong	0x08d12e6ab	/* VMALLOCBASE VSID */
+	.llong	(VMALLOCBASE>>SID_SHIFT)
+	.llong	0xb0cffffd1	/* VMALLOCBASE VSID */
 	.llong	8192		/* # pages to map (32 MB) */
 	.llong	0		/* Offset from start of loadarea to start of map */
-	.llong	0x0006a99b4b140000	/* VPN of first page to map */
+	.llong	0x40bffffd50000	/* VPN of first page to map */
 
 	. = 0x6100
 
@@ -1064,18 +1064,9 @@
 	rldimi	r10,r11,7,52	/* r10 = first ste of the group */
 
 	/* Calculate VSID */
-	/* (((ea >> 28) & 0x1fff) << 15) | (ea >> 60) */
-	rldic	r11,r11,15,36
-	ori	r11,r11,0xc
-
-	/* VSID_RANDOMIZER */
-	li	r9,9
-	sldi	r9,r9,32
-	oris	r9,r9,58231
-	ori	r9,r9,39831
-
-	mulld	r9,r11,r9
-	rldic	r9,r9,12,16	/* r9 = vsid << 12 */
+	/* This is a kernel address, so protovsid = ESID */
+	ASM_VSID_SCRAMBLE(r11, r9)
+	rldic	r9,r11,12,16	/* r9 = vsid << 12 */
 
 	/* Search the primary group for a free entry */
 1:	ld	r11,0(r10)	/* Test valid bit of the current ste	*/
arch/ppc64/mm/stab.c +1 -5
Index: working-2.6/arch/ppc64/mm/stab.c
===================================================================
@@ -115,15 +115,11 @@
 	unsigned char stab_entry;
 	unsigned long offset;
 
-	/* Check for invalid effective addresses. */
-	if (!IS_VALID_EA(ea))
-		return 1;
-
 	/* Kernel or user address? */
 	if (ea >= KERNELBASE) {
 		vsid = get_kernel_vsid(ea);
 	} else {
-		if (! mm)
+		if ((ea >= TASK_SIZE_USER64) || (! mm))
 			return 1;
 
 		vsid = get_vsid(mm->context.id, ea);
include/asm-ppc64/pgtable.h +10 -4
Index: working-2.6/include/asm-ppc64/pgtable.h
===================================================================
@@ -45,10 +45,16 @@
                     PGD_INDEX_SIZE + PAGE_SHIFT) 
 
 /*
+ * Size of EA range mapped by our pagetables.
+ */
+#define PGTABLE_EA_BITS	41
+#define PGTABLE_EA_MASK	((1UL<<PGTABLE_EA_BITS)-1)
+
+/*
  * Define the address range of the vmalloc VM area.
  */
 #define VMALLOC_START (0xD000000000000000ul)
-#define VMALLOC_END   (VMALLOC_START + VALID_EA_BITS)
+#define VMALLOC_END   (VMALLOC_START + PGTABLE_EA_MASK)
 
 /*
  * Define the address range of the imalloc VM area.
@@ -58,19 +64,19 @@
 #define IMALLOC_VMADDR(x) ((unsigned long)(x))
 #define PHBS_IO_BASE  	  (0xE000000000000000ul)	/* Reserve 2 gigs for PHBs */
 #define IMALLOC_BASE      (0xE000000080000000ul)  
-#define IMALLOC_END       (IMALLOC_BASE + VALID_EA_BITS)
+#define IMALLOC_END       (IMALLOC_BASE + PGTABLE_EA_MASK)
 
 /*
  * Define the address range mapped virt <-> physical
  */
 #define KRANGE_START KERNELBASE
-#define KRANGE_END   (KRANGE_START + VALID_EA_BITS)
+#define KRANGE_END   (KRANGE_START + PGTABLE_EA_MASK)
 
 /*
  * Define the user address range
  */
 #define USER_START (0UL)
-#define USER_END   (USER_START + VALID_EA_BITS)
+#define USER_END   (USER_START + PGTABLE_EA_MASK)
 
 
 /*
arch/ppc64/mm/hash_utils.c +5 -5
Index: working-2.6/arch/ppc64/mm/hash_utils.c
===================================================================
@@ -253,24 +253,24 @@
 	int local = 0;
 	cpumask_t tmp;
 
-	/* Check for invalid addresses. */
-	if (!IS_VALID_EA(ea))
-		return 1;
-
  	switch (REGION_ID(ea)) {
 	case USER_REGION_ID:
 		user_region = 1;
 		mm = current->mm;
-		if (mm == NULL)
+		if ((ea > USER_END) || (! mm))
 			return 1;
 
 		vsid = get_vsid(mm->context.id, ea);
 		break;
 	case IO_REGION_ID:
+		if (ea > IMALLOC_END)
+			return 1;
 		mm = &ioremap_mm;
 		vsid = get_kernel_vsid(ea);
 		break;
 	case VMALLOC_REGION_ID:
+		if (ea > VMALLOC_END)
+			return 1;
 		mm = &init_mm;
 		vsid = get_kernel_vsid(ea);
 		break;
include/asm-ppc64/page.h +0 -11
Index: working-2.6/include/asm-ppc64/page.h
===================================================================
@@ -212,17 +212,6 @@
 #define USER_REGION_ID     (0UL)
 #define REGION_ID(X)	   (((unsigned long)(X))>>REGION_SHIFT)
 
-/*
- * Define valid/invalid EA bits (for all ranges)
- */
-#define VALID_EA_BITS   (0x000001ffffffffffUL)
-#define INVALID_EA_BITS (~(REGION_MASK|VALID_EA_BITS))
-
-#define IS_VALID_REGION_ID(x) \
-        (((x) == USER_REGION_ID) || ((x) >= KERNEL_REGION_ID))
-#define IS_VALID_EA(x) \
-        ((!((x) & INVALID_EA_BITS)) && IS_VALID_REGION_ID(REGION_ID(x)))
-
 #define __bpn_to_ba(x) ((((unsigned long)(x))<<PAGE_SHIFT) + KERNELBASE)
 #define __ba_to_bpn(x) ((((unsigned long)(x)) & ~REGION_MASK) >> PAGE_SHIFT)
 
Signature · 4 lines
David Gibson			| For every complex problem there is a
david AT gibson.dropbear.id.au	| solution which is simple, neat and
				| wrong.
http://www.ozlabs.org/people/dgibson
#1 lore raw
Quoted 4 lines · David Gibson

Andrew, please apply. This patch has been tested both on SLB and segment table machines. This new approach is far from the final word in VSID/context allocation, but it's a noticeable improvement on the old method.

This patch went into 2.6.9-rc2-bk2, and my p640 does not boot anymore. Hangs after 'returning from prom_init', wants a power cycle.

Signature · 3 lines
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG
#2 lore raw
Quoted 9 lines · Olaf Hering
Quoted 4 lines · David Gibson

Andrew, please apply. This patch has been tested both on SLB and segment table machines. This new approach is far from the final word in VSID/context allocation, but it's a noticeable improvement on the old method.

This patch went into 2.6.9-rc2-bk2, and my p640 does not boot anymore. Hangs after 'returning from prom_init', wants a power cycle.

Have you isolated the problem to the VSID allocation patch? I think there may have been a number of ppc64 changes which went into 2.6.9-rc2-bk2.

Signature · 4 lines
David Gibson			| For every complex problem there is a
david AT gibson.dropbear.id.au	| solution which is simple, neat and
				| wrong.
http://www.ozlabs.org/people/dgibson
#3 lore raw
Quoted 8 lines · David Gibson
Quoted 7 lines · Olaf Hering
Quoted 4 lines · David Gibson

Andrew, please apply. This patch has been tested both on SLB and segment table machines. This new approach is far from the final word in VSID/context allocation, but it's a noticeable improvement on the old method.

Quoted 6 lines
Quoted 2 lines

This patch went into 2.6.9-rc2-bk2, and my p640 does not boot anymore. Hangs after 'returning from prom_init', wants a power cycle.

Have you isolated the problem to the VSID allocation patch? I think there may have been a number of ppc64 changes which went into 2.6.9-rc2-bk2.

Yes, rc2 does not boot on power3 with that patch.

Signature · 3 lines
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG
#4 lore raw
Quoted 10 lines · Olaf Hering
Quoted 8 lines · David Gibson
Quoted 7 lines · Olaf Hering
Quoted 4 lines · David Gibson

Andrew, please apply. This patch has been tested both on SLB and segment table machines. This new approach is far from the final word in VSID/context allocation, but it's a noticeable improvement on the old method.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

This patch went into 2.6.9-rc2-bk2, and my p640 does not boot anymore. Hangs after 'returning from prom_init', wants a power cycle.

Have you isolated the problem to the VSID allocation patch? I think there may have been a number of ppc64 changes which went into 2.6.9-rc2-bk2.

Yes, rc2 does not boot on power3 with that patch.

Hrm.. current BK works on the 270 here, so it's not all power3s.

Signature · 4 lines
David Gibson			| For every complex problem there is a
david AT gibson.dropbear.id.au	| solution which is simple, neat and
				| wrong.
http://www.ozlabs.org/people/dgibson
#5 lore raw
In reply to Olaf Hering
Quoted 3 lines

This patch went into 2.6.9-rc2-bk2, and my p640 does not boot anymore. Hangs after 'returning from prom_init', wants a power cycle.

How much memory do you have? We might be filling up a hpte bucket completely with certain amounts of memory.

Anton

#6 lore raw
Quoted 6 lines · Anton Blanchard
Quoted 3 lines

This patch went into 2.6.9-rc2-bk2, and my p640 does not boot anymore. Hangs after 'returning from prom_init', wants a power cycle.

How much memory do you have? We might be filling up a hpte bucket completely with certain amounts of memory.

It has 4gig.

Signature · 3 lines
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG
#7 lore raw
In reply to Anton Blanchard
Quoted 6 lines · Anton Blanchard
Quoted 3 lines

This patch went into 2.6.9-rc2-bk2, and my p640 does not boot anymore. Hangs after 'returning from prom_init', wants a power cycle.

How much memory do you have? We might be filling up a hpte bucket completely with certain amounts of memory.

Bugger, bugger, bugger bugger. That's it. Just ran 4GB linear mapping with 4k pages through by hash scattering simulator - max bucket load is 2 with the old algo and 16 with the new one. Well, we just found the first case where the new algorithm scatters significantly worse than the old one. It would be something this dire, wouldn't it.

Signature · 4 lines
David Gibson			| For every complex problem there is a
david AT gibson.dropbear.id.au	| solution which is simple, neat and
				| wrong.
http://www.ozlabs.org/people/dgibson
#8 lore raw
Quoted 8 lines · David Gibson
Quoted 6 lines · Anton Blanchard
Quoted 3 lines

This patch went into 2.6.9-rc2-bk2, and my p640 does not boot anymore. Hangs after 'returning from prom_init', wants a power cycle.

How much memory do you have? We might be filling up a hpte bucket completely with certain amounts of memory.

Quoted 6 lines

Bugger, bugger, bugger bugger. That's it. Just ran 4GB linear mapping with 4k pages through by hash scattering simulator - max bucket load is 2 with the old algo and 16 with the new one. Well, we just found the first case where the new algorithm scatters significantly worse than the old one. It would be something this dire, wouldn't it.

Ok, after some more investigations, I think I've come to a theoretical understanding of why this is happening. The problem is that VSID_MULTIPLIER is too close to (1<<28), i.e. too close to all-1s. That means the differences between the VSIDs for the 16 segments in the linear mapping are all either in the high bits - where they get truncated my the size of the hash table - or in the low bits, where they get masked by the vpn component of the hash (it cycles through every possible value in the low 16 bits). We crucially need the bits in the middle to be different - with an order 19 hash table, we only have three significant bits to play with..

Fortunately, I think it's not too hard to fix. Olaf, can you try changing VSID_MULTIPLIER in include/asm-ppc64/mmu.h to 200730139, instead of the current value. According to my hash simulator that should fix the problem for you (and work out to larger amounts of RAM, too).

I'll push a patch for this tomorrow - the fact that this has come up suggest to me that I need to think a little deeper about the rationale for picking VSID_MULTIPLIER.

NB, I'm assuming this is a pSeries machine we're talking about - just changing VSID_MULTIPLIER is not sufficient on an iSeries machine.

Signature · 4 lines
David Gibson			| For every complex problem there is a
david AT gibson.dropbear.id.au	| solution which is simple, neat and
				| wrong.
http://www.ozlabs.org/people/dgibson
#9 lore raw
Quoted 4 lines · David Gibson

changing VSID_MULTIPLIER in include/asm-ppc64/mmu.h to 200730139, instead of the current value. According to my hash simulator that should fix the problem for you (and work out to larger amounts of RAM, too).

Yes, that number works, tested on rc2 + the vsid patch.

Signature · 3 lines
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG