powerpc: Move 64bit heap above 1TB on machines with 1TB segments
If we are using 1TB segments and we are allowed to randomise the heap, we can put it above 1TB so it is backed by a 1TB segment. Otherwise the heap will be in the bottom 1TB which always uses 256MB segments and this may result in a performance penalty.
This functionality is disabled when heap randomisation is turned off:
echo 1 > /proc/sys/kernel/randomize_va_space
which may be useful when trying to allocate the maximum amount of 16M or 16G pages.
On a microbenchmark that repeatedly touches 32GB of memory with a stride of 256MB + 4kB (designed to stress 256MB segments while still mapping nicely into the L1 cache), we see the improvement:
Force malloc to use heap all the time:
# export MALLOC_MMAP_MAX_=0 MALLOC_TRIM_THRESHOLD_=-1
Disable heap randomization:
# echo 1 > /proc/sys/kernel/randomize_va_space # time ./test 12.51s
Enable heap randomization:
# echo 2 > /proc/sys/kernel/randomize_va_space # time ./test 1.70s
---
I've cc-ed Mel on this one. As you can see it definitely helps the base page size performance, but I'm a bit worried of the impact of taking away another of our 1TB slices.
arch/powerpc/kernel/process.c +16 -1
@@ -1165,7 +1165,22 @@ static inline unsigned long brk_rnd(voidunsigned long arch_randomize_brk (struct mm_struct *mm) { -unsigned long ret =PAGE_ALIGN (mm->brk +brk_rnd ()); +unsigned long base = mm->brk; +unsigned long ret; + +#ifdef CONFIG_PPC64 +/* +* If we are using 1TB segments and we are allowed to randomise +* the heap, we can put it above 1TB so it is backed by a 1TB +* segment. Otherwise the heap will be in the bottom 1TB +* which always uses 256MB segments and this may result in a +* performance penalty. +*/ +if (!is_32bit_task () && (mmu_highuser_ssize ==MMU_SEGSIZE_1T )) + base = max_t(unsigned long, mm->brk, 1UL << SID_SHIFT_1T); +#endif + + ret =PAGE_ALIGN (base +brk_rnd ());if (ret < mm->brk)return mm->brk;
Anton Blanchard <[email protected]> wrote on 22/09/2009 03:52:35:
Quoted 1 line
If we are using 1TB segments and we are allowed to randomise the heap, we
can
Quoted 1 line
put it above 1TB so it is backed by a 1TB segment. Otherwise the heap
will be
Quoted 1 line
in the bottom 1TB which always uses 256MB segments and this may result in
a
Quoted 7 lines
performance penalty.
This functionality is disabled when heap randomisation is turned off:
echo 1 >
/proc/sys/kernel/randomize_va_spacewhich may be useful when trying to allocate the maximum amount of 16M or
16G
Quoted 3 lines
pages.
On a microbenchmark that repeatedly touches 32GB of memory with a stride
of
Quoted 1 line
256MB + 4kB (designed to stress 256MB segments while still mapping nicely
into
Quoted 15 lines
the L1 cache), we see the improvement:
Force malloc to use heap all the time:
# export MALLOC_MMAP_MAX_=0 MALLOC_TRIM_THRESHOLD_=-1Disable heap randomization:
# echo 1 > /proc/sys/kernel/randomize_va_space # time ./test 12.51sEnable heap randomization:
# echo 2 > /proc/sys/kernel/randomize_va_space # time ./test 1.70s
Quoted 7 lines
Signed-off-by Anton Blanchard <[email protected]>---
I've cc-ed Mel on this one. As you can see it definitely helps the base page size performance, but I'm a bit worried of the impact of taking away another of our 1TB slices.
Unfortunately, I am not sensitive to issues surrounding 1TB segments or how they are currently being used. However, as this clearly helps performance for large amounts of memory, is it worth providing an option to libhugetlbfs to locate 16MB pages above 1TB when they are otherwise being unused?
Quoted 30 lines
Index:
linux.trees.git/arch/powerpc/kernel/process.c
===================================================================
---linux.trees.git.orig/arch/powerpc/kernel/process.c2009-09-17
15:47:46.000000000 +1000
+++linux.trees.git/arch/powerpc/kernel/process.c2009-09-17 15:
49:11.000000000 +1000arch/powerpc/kernel/process.c +16
-1@@ -1165,7 +1165,22 @@ static inline unsigned long brk_rnd(voidunsigned long arch_randomize_brk (struct mm_struct *mm) { -unsigned long ret =PAGE_ALIGN (mm->brk +brk_rnd ()); +unsigned long base = mm->brk; +unsigned long ret; + +#ifdef CONFIG_PPC64 +/* +* If we are using 1TB segments and we are allowed to randomise +* the heap, we can put it above 1TB so it is backed by a 1TB +* segment. Otherwise the heap will be in the bottom 1TB +* which always uses 256MB segments and this may result in a +* performance penalty. +*/ +if (!is_32bit_task () && (mmu_highuser_ssize ==MMU_SEGSIZE_1T )) + base = max_t(unsigned long, mm->brk, 1UL << SID_SHIFT_1T); +#endif + + ret =PAGE_ALIGN (base +brk_rnd ());if (ret < mm->brk)return mm->brk;
Quoted 6 lines
Unfortunately, I am not sensitive to issues surrounding 1TB segments or how they are currently being used. However, as this clearly helps performance for large amounts of memory, is it worth providing an option to libhugetlbfs to locate 16MB pages above 1TB when they are otherwise being unused?
AFAIK, that is already the case, at least the kernel will hand out pages above 1T preferentially iirc.
There were talks about making huge pages below 1T not even come up untily you ask for them with MAP_FIXED, dunno where that went.
Cheers,
Ben.
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 6 lines
Unfortunately, I am not sensitive to issues surrounding 1TB segments or how they are currently being used. However, as this clearly helps performance for large amounts of memory, is it worth providing an option to libhugetlbfs to locate 16MB pages above 1TB when they are otherwise being unused?
Quoted 5 lines
AFAIK, that is already the case, at least the kernel will hand out pages above 1T preferentially iirc.
There were talks about making huge pages below 1T not even come up untily you ask for them with
MAP_FIXED, dunno where that went.
That was already the case as far as I remember. But it's just possible that changed when the general slice handling code came in,
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Benjamin Herrenschmidt <[email protected]> wrote on 22/09/2009 22:08:22:
Quoted 1 line
Quoted 1 line
Unfortunately, I am not sensitive to issues surrounding 1TB segments or
how
Quoted 1 line
Quoted 1 line
they are currently being used. However, as this clearly helps
performance
Quoted 2 lines
Quoted 2 lines
for large amounts of memory, is it worth providing an option to libhugetlbfs to locate 16MB pages above 1TB when they are otherwise
being
Quoted 8 lines
Quoted 1 line
unused?
AFAIK, that is already the case, at least the kernel will hand out pages above 1T preferentially iirc.
There were talks about making huge pages below 1T not even come up untily you ask for them with
MAP_FIXED, dunno where that went.
Confirmed, huge pages are already being placed above the 1TB mark. I hadn't given thought previously to where hugepages were being placed except within segment boundaries. The patch works as advertised and doesn't appear to collide with huge pages in any obvious way as far as I can tell.