Enable hashdist by default on PowerPC
PATCH 1/1 Enable hashdist by default on PowerPC
On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):
Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%
If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:
Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%
For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.
---
include/linux/bootmem.h +3 -2
@@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,#define HASH_EARLY 0x00000001/* Allocating during early boot? */ /* Only NUMA needs hash distribution. -* IA64 and x86_64 have sufficient vmalloc space. +* IA64, x86_64 and PowerPC have sufficient vmalloc space. */ -#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 )) +#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 ) || \ + defined(CONFIG_PPC64 ))#define HASHDIST_DEFAULT 1#else #define HASHDIST_DEFAULT 0
Quoted 7 lines
For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.
Signed-off-by Anton Blanchard <[email protected]>---
You have numbers ? :-) I'm asking mostly because I've been wondering whether it offsets the 16M pages vs. 4K or 64K pages in term of TLB/ERAT impact.
Cheers,
Ben.
Quoted 21 lines
include/linux/bootmem.h +3
-2@@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,#define HASH_EARLY 0x00000001/* Allocating during early boot? */ /* Only NUMA needs hash distribution. -* IA64 and x86_64 have sufficient vmalloc space. +* IA64, x86_64 and PowerPC have sufficient vmalloc space. */ -#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 )) +#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 ) || \ + defined(CONFIG_PPC64 ))#define HASHDIST_DEFAULT 1#else #define HASHDIST_DEFAULT 0_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc-dev
Hi Ben,
Quoted 3 lines
You have numbers ? :-) I'm asking mostly because I've been wondering whether it offsets the 16M pages vs. 4K or 64K pages in term of TLB/ERAT impact.
The speedup is application dependent. Things like linpack usually improve when you throw more memmory at them.
The potential slowdown will be in heavy dcache use (eg fileserving). We originally added the large boot time hash code when we were benchmarking SPECsfs (an NFS benchmark).
We can go back to the old behaviour with the hashdist=0 boot option, so it's mostly a question of what the default should be.
Anton
From: Anton Blanchard <[email protected]>
Date: Wed, 18 Feb 2009 16:11:12 +1100
include/linux/bootmem.h extern void *alloc_large_system_hash(const char *tablename, · line 154
Quoted 13 lines
include/linux/bootmem.h +3
-2@@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,#define HASH_EARLY 0x00000001/* Allocating during early boot? */ /* Only NUMA needs hash distribution. -* IA64 and x86_64 have sufficient vmalloc space. +* IA64, x86_64 and PowerPC have sufficient vmalloc space. */ -#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 )) +#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 ) || \ + defined(CONFIG_PPC64 ))#define HASHDIST_DEFAULT 1#else #define HASHDIST_DEFAULT 0
I should probably do this on sparc64 too.
Why don't we just change this thing to CONFIG_64BIT?
Hi David,
Quoted 3 lines
I should probably do this on sparc64 too.
Why don't we just change this thing to
CONFIG_64BIT?
I agree. How does this look?
Anton
--
On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):
Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%
If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:
Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%
For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.
Since all 64bit NUMA capable architectures should have sufficient vmalloc space, it makes sense to enable it via CONFIG_64BIT.
---
include/linux/bootmem.h +3 -3
@@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,#define HASH_EARLY 0x00000001/* Allocating during early boot? */ -/* Only NUMA needs hash distribution. -* IA64 and x86_64 have sufficient vmalloc space. +/* Only NUMA needs hash distribution. 64bit NUMA architectures have +* sufficient vmalloc space. */ -#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 )) +#if defined(CONFIG_64BIT )#define HASHDIST_DEFAULT 1#else #define HASHDIST_DEFAULT 0
From: Anton Blanchard <[email protected]>
Date: Fri, 20 Feb 2009 16:19:56 +1100
Quoted 8 lines
Hi David,
Quoted 3 lines
I should probably do this on sparc64 too.
Why don't we just change this thing to
CONFIG_64BIT?I agree. How does this look?
Hmmm... my bad, I think you need to keep the CONFIG_NUMA there too as there is a TLB usage penalty for non-NUMA systems if you only use CONFIG_64BIT there.
PATCH 1/1 Enable hashdist by default on 64bit NUMA
Hi David,
Quoted 3 lines
Hmmm... my bad, I think you need to keep the
CONFIG_NUMAthere too as there is a TLB usage penalty for non-NUMA systems if you only useCONFIG_64BITthere.
Sorry that was my screwup, here's a fixed version.
Anton
--
On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):
Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%
If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:
Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%
For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.
Since all 64bit NUMA capable architectures should have sufficient vmalloc space, it makes sense to enable it via CONFIG_64BIT.
---
include/linux/bootmem.h +3 -3
@@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,#define HASH_EARLY 0x00000001/* Allocating during early boot? */ -/* Only NUMA needs hash distribution. -* IA64 and x86_64 have sufficient vmalloc space. +/* Only NUMA needs hash distribution. 64bit NUMA architectures have +* sufficient vmalloc space. */ -#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 )) +#if defined(CONFIG_NUMA ) && defined(CONFIG_64BIT )#define HASHDIST_DEFAULT 1#else #define HASHDIST_DEFAULT 0
Re: [PATCH] Enable hashdist by default on 64bit NUMA
From: Anton Blanchard <[email protected]>
Date: Thu, 26 Feb 2009 22:24:32 +1100
Quoted 19 lines
On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):
Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:
Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%
Quoted 8 lines
For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.
Since all 64bit NUMA capable architectures should have sufficient vmalloc space, it makes sense to enable it via
CONFIG_64BIT.Signed-off-by Anton Blanchard <[email protected]>
Re: [PATCH] Enable hashdist by default on 64bit NUMA
In reply to Anton BlanchardQuoted 7 lines · Anton Blanchard
Hi David,
Quoted 3 lines
Hmmm... my bad, I think you need to keep the
CONFIG_NUMAthere too as there is a TLB usage penalty for non-NUMA systems if you only useCONFIG_64BITthere.Sorry that was my screwup, here's a fixed version.
Sounds good, how do we proceed for merging that ? Andrew ? Should it hop by linux-mm ?
---
Quoted 55 lines
Anton
--
On PowerPC we allocate large boot time hashes on node 0. This leads to an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):
Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%If we switch to using vmalloc (like ia64 and x86-64) things are more balanced:
Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%For many HPC applications we are limited by the free available memory on the smallest node, so even though the same amount of memory is used the better balancing helps.
Since all 64bit NUMA capable architectures should have sufficient vmalloc space, it makes sense to enable it via
CONFIG_64BIT.Signed-off-by Anton Blanchard <[email protected]>---
include/linux/bootmem.h +3
-3@@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,#define HASH_EARLY 0x00000001/* Allocating during early boot? */ -/* Only NUMA needs hash distribution. -* IA64 and x86_64 have sufficient vmalloc space. +/* Only NUMA needs hash distribution. 64bit NUMA architectures have +* sufficient vmalloc space. */ -#if defined(CONFIG_NUMA ) && (defined(CONFIG_IA64 ) || defined(CONFIG_X86_64 )) +#if defined(CONFIG_NUMA ) && defined(CONFIG_64BIT )#define HASHDIST_DEFAULT 1#else #define HASHDIST_DEFAULT 0_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc-dev