Skip to contentSkip to conversation

Enable hashdist by default on PowerPC

PATCH In review Anton Blanchard 3 participants 9 messages Feb 18, 2009 – Mar 3, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Enable hashdist by default on PowerPC

On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):

Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%

If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:

Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%

For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.

---

include/linux/bootmem.h +3 -2
index 95837bf..c0c63ee 100644
@@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,
 #define HASH_EARLY	0x00000001	/* Allocating during early boot? */
 
 /* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+ * IA64, x86_64 and PowerPC have sufficient vmalloc space.
  */
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64) || \
+	defined(CONFIG_PPC64))
 #define HASHDIST_DEFAULT 1
 #else
 #define HASHDIST_DEFAULT 0
#1 lore raw
Quoted 7 lines

For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.

---

You have numbers ? :-) I'm asking mostly because I've been wondering whether it offsets the 16M pages vs. 4K or 64K pages in term of TLB/ERAT impact.

Cheers,
Ben.

Quoted 21 lines
include/linux/bootmem.h +3 -2
index 95837bf..c0c63ee 100644
@@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,
 #define HASH_EARLY	0x00000001	/* Allocating during early boot? */
 
 /* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+ * IA64, x86_64 and PowerPC have sufficient vmalloc space.
  */
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64) || \
+	defined(CONFIG_PPC64))
 #define HASHDIST_DEFAULT 1
 #else
 #define HASHDIST_DEFAULT 0

_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc-dev

#2 lore raw

Hi Ben,

Quoted 3 lines

You have numbers ? :-) I'm asking mostly because I've been wondering whether it offsets the 16M pages vs. 4K or 64K pages in term of TLB/ERAT impact.

The speedup is application dependent. Things like linpack usually improve when you throw more memmory at them.

The potential slowdown will be in heavy dcache use (eg fileserving). We originally added the large boot time hash code when we were benchmarking SPECsfs (an NFS benchmark).

We can go back to the old behaviour with the hashdist=0 boot option, so it's mostly a question of what the default should be.

Anton

#3 lore raw
In reply to Anton Blanchard

From: Anton Blanchard <[email protected]>
Date: Wed, 18 Feb 2009 16:11:12 +1100

include/linux/bootmem.h extern void *alloc_large_system_hash(const char *tablename, · line 154

Quoted 13 lines
include/linux/bootmem.h +3 -2
@@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,
 #define HASH_EARLY	0x00000001	/* Allocating during early boot? */
 
 /* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+ * IA64, x86_64 and PowerPC have sufficient vmalloc space.
  */
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64) || \
+	defined(CONFIG_PPC64))
 #define HASHDIST_DEFAULT 1
 #else
 #define HASHDIST_DEFAULT 0

I should probably do this on sparc64 too.

Why don't we just change this thing to CONFIG_64BIT?

#4 lore raw

Hi David,

Quoted 3 lines

I should probably do this on sparc64 too.

Why don't we just change this thing to CONFIG_64BIT?

I agree. How does this look?

Anton
--

On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):

Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%

If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:

Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%

For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.

Since all 64bit NUMA capable architectures should have sufficient vmalloc space, it makes sense to enable it via CONFIG_64BIT.

---

include/linux/bootmem.h +3 -3
index 95837bf..aa861d5 100644
@@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,
 
 #define HASH_EARLY	0x00000001	/* Allocating during early boot? */
 
-/* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+/* Only NUMA needs hash distribution. 64bit NUMA architectures have
+ * sufficient vmalloc space.
  */
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_64BIT)
 #define HASHDIST_DEFAULT 1
 #else
 #define HASHDIST_DEFAULT 0
#5 lore raw

From: Anton Blanchard <[email protected]>
Date: Fri, 20 Feb 2009 16:19:56 +1100

Quoted 8 lines

Hi David,

Quoted 3 lines

I should probably do this on sparc64 too.

Why don't we just change this thing to CONFIG_64BIT?

I agree. How does this look?

Hmmm... my bad, I think you need to keep the CONFIG_NUMA there too as there is a TLB usage penalty for non-NUMA systems if you only use CONFIG_64BIT there.

#6 lore raw

PATCH 1/1 Enable hashdist by default on 64bit NUMA

Hi David,

Quoted 3 lines

Hmmm... my bad, I think you need to keep the CONFIG_NUMA there too as there is a TLB usage penalty for non-NUMA systems if you only use CONFIG_64BIT there.

Sorry that was my screwup, here's a fixed version.

Anton

--

On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):

Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%

If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:

Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%

For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.

Since all 64bit NUMA capable architectures should have sufficient vmalloc space, it makes sense to enable it via CONFIG_64BIT.

---

include/linux/bootmem.h +3 -3
index 95837bf..0c4d4b7 100644
@@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,
 
 #define HASH_EARLY	0x00000001	/* Allocating during early boot? */
 
-/* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+/* Only NUMA needs hash distribution. 64bit NUMA architectures have
+ * sufficient vmalloc space.
  */
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_NUMA) && defined(CONFIG_64BIT)
 #define HASHDIST_DEFAULT 1
 #else
 #define HASHDIST_DEFAULT 0
Acked-by on v1 1/1
#7 lore raw

Re: [PATCH] Enable hashdist by default on 64bit NUMA

From: Anton Blanchard <[email protected]>
Date: Thu, 26 Feb 2009 22:24:32 +1100

Quoted 19 lines

On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):

Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%

If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:

Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%

Quoted 8 lines

For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.

Since all 64bit NUMA capable architectures should have sufficient vmalloc space, it makes sense to enable it via CONFIG_64BIT.

Acked-by on v1 1/1
#8 lore raw

Re: [PATCH] Enable hashdist by default on 64bit NUMA

In reply to Anton Blanchard
Quoted 7 lines · Anton Blanchard

Hi David,

Quoted 3 lines

Hmmm... my bad, I think you need to keep the CONFIG_NUMA there too as there is a TLB usage penalty for non-NUMA systems if you only use CONFIG_64BIT there.

Sorry that was my screwup, here's a fixed version.

Sounds good, how do we proceed for merging that ? Andrew ? Should it hop by linux-mm ?

---

Quoted 55 lines

Anton

--

On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):

Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%

If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:

Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%

For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.

Since all 64bit NUMA capable architectures should have sufficient vmalloc space, it makes sense to enable it via CONFIG_64BIT.

---

include/linux/bootmem.h +3 -3
index 95837bf..0c4d4b7 100644
@@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,
 
 #define HASH_EARLY	0x00000001	/* Allocating during early boot? */
 
-/* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+/* Only NUMA needs hash distribution. 64bit NUMA architectures have
+ * sufficient vmalloc space.
  */
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_NUMA) && defined(CONFIG_64BIT)
 #define HASHDIST_DEFAULT 1
 #else
 #define HASHDIST_DEFAULT 0

_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc-dev