mm/slub: allocate sheaves on local memory nodes
Subsystems (2)
Hao Li
reviewerPATCH v2 1/1 mm/slub: allocate sheaves on local memory nodes
Sheaf structs are exchanged through node-local barns. Since barn structs are already allocated from their local NUMA node, this patch aims to allocate sheaf structs from their local memory nodes as well.
To achieve this, the obvious choice would be using cpu_to_mem(). However, init_percpu_sheaves() and bootstrap_cache_sheaves() iterate through possible CPUs, whereas cpu_to_mem() is only initialized for online CPUs. Therefore, we cannot use cpu_to_mem() and instead need to use local_memory_node(cpu_to_node(cpu)), similar to what __build_all_zonelists() does.
The primary goal of this patch is to improve NUMA node locality. Although the actual performance impact is minor, it still yields a ~1% improvement on a 192-core, 8-NUMA-node system when testing with the will-it-scale mmap test case.
---
Changes in v2:
- Make init_percpu_sheaves() use a NUMA-aware sheaf struct allocation too. (Thanks Harry)
- Rebase on latest code.
v1: https://lore.kernel.org/linux-mm/[email protected]/
--- mm/slub.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
mm/slub.c +19 -8
@@ -2757,7 +2757,7 @@ static inline void *setup_object(struct kmem_cache *s, void *object) }static struct slab_sheaf *__alloc_empty_sheaf (struct kmem_cache *s,gfp_t gfp, -unsigned int capacity) +unsigned int capacity,int node) {struct slab_sheaf *sheaf;size_t sheaf_size;
@@ -2771,7 +2771,7 @@ static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp, gfp |= __GFP_NO_OBJ_EXT; sheaf_size =struct_size (sheaf, objects, capacity); - sheaf =kzalloc (sheaf_size, gfp); + sheaf =kzalloc_node (sheaf_size, gfp, node);if (unlikely (!sheaf))return NULL ;
@@ -2791,7 +2791,7 @@ static inline struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp &= ~OBJCGS_CLEAR_MASK ; -return __alloc_empty_sheaf (s, gfp, s->sheaf_capacity); +return __alloc_empty_sheaf (s, gfp, s->sheaf_capacity,numa_mem_id ()); }static void free_empty_sheaf (struct kmem_cache *s,struct slab_sheaf *sheaf)
@@ -5014,7 +5014,7 @@ kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)if (unlikely (size > s->sheaf_capacity)) { - sheaf =__alloc_empty_sheaf (s, gfp, size); + sheaf =__alloc_empty_sheaf (s, gfp, size,numa_mem_id ());if (!sheaf)return NULL ;
@@ -7575,6 +7575,7 @@ static int init_percpu_sheaves(struct kmem_cache *s)for_each_possible_cpu (cpu) {struct slub_percpu_sheaves *pcs; +int mem_node; pcs =per_cpu_ptr (s->cpu_sheaves, cpu);
@@ -7598,10 +7599,13 @@ static int init_percpu_sheaves(struct kmem_cache *s)* For kmalloc caches it's used temporarily during the initial * bootstrap. */ -if (!s->sheaf_capacity) +if (!s->sheaf_capacity) { pcs->main = &bootstrap_sheaf; -else - pcs->main =alloc_empty_sheaf (s,GFP_KERNEL ); + }else { + mem_node =local_memory_node (cpu_to_node (cpu)); + pcs->main =__alloc_empty_sheaf (s,GFP_KERNEL , + s->sheaf_capacity, mem_node); + }if (!pcs->main)return -ENOMEM ;
@@ -8465,10 +8469,17 @@ static void __init bootstrap_cache_sheaves(struct kmem_cache *s)for_each_possible_cpu (cpu) {struct slub_percpu_sheaves *pcs; +int mem_node; pcs =per_cpu_ptr (s->cpu_sheaves, cpu); - pcs->main =__alloc_empty_sheaf (s,GFP_KERNEL , capacity); +/* +* Cannot use cpu_to_mem() here because it's only initialized +* for online CPUs at this point (see __build_all_zonelists), +* while we need to allocate sheaves for all possible CPUs. +*/ + mem_node =local_memory_node (cpu_to_node (cpu)); + pcs->main =__alloc_empty_sheaf (s,GFP_KERNEL , capacity, mem_node);if (!pcs->main) { failed = true;
Signature · 1 line
2.54.0
Harry Yoo
maintainerQuoted 11 lines · Hao Li
Sheaf structs are exchanged through node-local barns. Since barn structs are already allocated from their local NUMA node, this patch aims to allocate sheaf structs from their local memory nodes as well.
To achieve this, the obvious choice would be using
cpu_to_mem(). However,init_percpu_sheaves()andbootstrap_cache_sheaves()iterate through possible CPUs, whereascpu_to_mem()is only initialized for online CPUs. Therefore, we cannot usecpu_to_mem()and instead need to use local_memory_node(cpu_to_node(cpu)), similar to what__build_all_zonelists()does.
Quoted 4 lines
The primary goal of this patch is to improve NUMA node locality. Although the actual performance impact is minor, it still yields a ~1% improvement on a 192-core, 8-NUMA-node system when testing with the will-it-scale mmap test case.
Oh, nice :)
I have a question though...
I wonder if would be better to handle this by e.g.) not returning empty sheaves back to barn and freeing them if the node id doesn't match and it's not a memoryless node.
init_percpu_sheaves() and bootstrap_cache_sheaves() are not the only places that can allocate sheaves from remote nodes; sheaves allocation could fall back to other nodes and then SLUB could keep reusing those sheaves from remote nodes even after memory is reclaimed.
If this works well, we probably don't need to handle it in init_percpu_sheaves() and bootstrap_cache_sheaves() at all as they will eventually be freed, while covering the other case too?
Quoted 8 lines
Signed-off-by Hao Li <[email protected]>---
Changes in v2:
- Makeinit_percpu_sheaves()use a NUMA-aware sheaf struct allocation too. (Thanks Harry)
- Rebase on latest code.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 21 lines · Harry Yoo
Quoted 11 lines · Hao Li
Sheaf structs are exchanged through node-local barns. Since barn structs are already allocated from their local NUMA node, this patch aims to allocate sheaf structs from their local memory nodes as well.
To achieve this, the obvious choice would be using
cpu_to_mem(). However,init_percpu_sheaves()andbootstrap_cache_sheaves()iterate through possible CPUs, whereascpu_to_mem()is only initialized for online CPUs. Therefore, we cannot usecpu_to_mem()and instead need to use local_memory_node(cpu_to_node(cpu)), similar to what__build_all_zonelists()does.Quoted 4 lines
The primary goal of this patch is to improve NUMA node locality. Although the actual performance impact is minor, it still yields a ~1% improvement on a 192-core, 8-NUMA-node system when testing with the will-it-scale mmap test case.
Oh, nice :)
I have a question though...
Quoted 8 lines
I wonder if would be better to handle this by e.g.) not returning empty sheaves back to barn and freeing them if the node id doesn't match and it's not a memoryless node.
init_percpu_sheaves()andbootstrap_cache_sheaves()are not the only places that can allocate sheaves from remote nodes; sheaves allocation could fall back to other nodes and then SLUB could keep reusing those sheaves from remote nodes even after memory is reclaimed.
This is a good catch. In addition to the fallback mechanism, task migration between CPUs in __pcs_replace_empty_main() and __pcs_replace_full_main() can also mix up sheaf structs across different barns. So yeah, changing allocation locality is not a silver bullet.
Quoted 4 lines
If this works well, we probably don't need to handle it in
init_percpu_sheaves()andbootstrap_cache_sheaves()at all as they will eventually be freed, while covering the other case too?
freeing the empty sheaf if the NUMA node mismatches instead of putting it back into the barn is indeed a good idea. I like it. But unfortunately, my testing didn't show a clear performance improvement, though there was no noticeable degradation either. :-(
I also did some more testing on my patch too. Under CONFIG_PREEMPT_LAZY, the improvement is only about 0.5% (sometimes 1%). And when switching to CONFIG_PREEMPT, the patch doesn't seem to yield statistically significant benefits, likely because sheaves get mixed during task migration.
So, perhaps the performance gain just isn't worth the extra complexity. It's a bit frustrating, but maybe we should just abandon this direction and keep things as they are... :(
Thanks for the feedback anyway!
Signature · 2 lines
Thanks, Hao
Harry Yoo
maintainerQuoted 36 lines · Hao Li
Quoted 21 lines · Harry Yoo
Quoted 11 lines · Hao Li
Sheaf structs are exchanged through node-local barns. Since barn structs are already allocated from their local NUMA node, this patch aims to allocate sheaf structs from their local memory nodes as well.
To achieve this, the obvious choice would be using
cpu_to_mem(). However,init_percpu_sheaves()andbootstrap_cache_sheaves()iterate through possible CPUs, whereascpu_to_mem()is only initialized for online CPUs. Therefore, we cannot usecpu_to_mem()and instead need to use local_memory_node(cpu_to_node(cpu)), similar to what__build_all_zonelists()does.Quoted 4 lines
The primary goal of this patch is to improve NUMA node locality. Although the actual performance impact is minor, it still yields a ~1% improvement on a 192-core, 8-NUMA-node system when testing with the will-it-scale mmap test case.
Oh, nice :)
I have a question though...
Quoted 8 lines
I wonder if would be better to handle this by e.g.) not returning empty sheaves back to barn and freeing them if the node id doesn't match and it's not a memoryless node.
init_percpu_sheaves()andbootstrap_cache_sheaves()are not the only places that can allocate sheaves from remote nodes; sheaves allocation could fall back to other nodes and then SLUB could keep reusing those sheaves from remote nodes even after memory is reclaimed.This is a good catch. In addition to the fallback mechanism, task migration between CPUs in
__pcs_replace_empty_main()and__pcs_replace_full_main()can also mix up sheaf structs across different barns. So yeah, changing allocation locality is not a silver bullet.
Quoted 8 lines
Quoted 3 lines
If this works well, we probably don't need to handle it in
init_percpu_sheaves()andbootstrap_cache_sheaves()at all as they will eventually be freed, while covering the other case too?freeing the empty sheaf if the NUMA node mismatches instead of putting it back into the barn is indeed a good idea. I like it. But unfortunately, my testing didn't show a clear performance improvement, though there was no noticeable degradation either. :-(
Hmm... the idea still makes sense to me, but yeah, it's not late to fix it when we have data to back up.
Quoted 8 lines
I also did some more testing on my patch too. Under
CONFIG_PREEMPT_LAZY, the improvement is only about 0.5% (sometimes 1%). And when switching toCONFIG_PREEMPT, the patch doesn't seem to yield statistically significant benefits, likely because sheaves get mixed during task migration.So, perhaps the performance gain just isn't worth the extra complexity. It's a bit frustrating, but maybe we should just abandon this direction and keep things as they are... :(
Ouch. Don't let it get you down, it's completely normal for experiments not to go as planned.... ;)
Quoted 1 line
Thanks for the feedback anyway!
No problem!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerQuoted 43 lines · Harry Yoo
Quoted 40 lines · Hao Li
Quoted 21 lines · Harry Yoo
Quoted 11 lines · Hao Li
Sheaf structs are exchanged through node-local barns. Since barn structs are already allocated from their local NUMA node, this patch aims to allocate sheaf structs from their local memory nodes as well.
To achieve this, the obvious choice would be using
cpu_to_mem(). However,init_percpu_sheaves()andbootstrap_cache_sheaves()iterate through possible CPUs, whereascpu_to_mem()is only initialized for online CPUs. Therefore, we cannot usecpu_to_mem()and instead need to use local_memory_node(cpu_to_node(cpu)), similar to what__build_all_zonelists()does.Quoted 4 lines
The primary goal of this patch is to improve NUMA node locality. Although the actual performance impact is minor, it still yields a ~1% improvement on a 192-core, 8-NUMA-node system when testing with the will-it-scale mmap test case.
Oh, nice :)
I have a question though...
Quoted 8 lines
I wonder if would be better to handle this by e.g.) not returning empty sheaves back to barn and freeing them if the node id doesn't match and it's not a memoryless node.
init_percpu_sheaves()andbootstrap_cache_sheaves()are not the only places that can allocate sheaves from remote nodes; sheaves allocation could fall back to other nodes and then SLUB could keep reusing those sheaves from remote nodes even after memory is reclaimed.This is a good catch. In addition to the fallback mechanism, task migration between CPUs in
__pcs_replace_empty_main()and__pcs_replace_full_main()can also mix up sheaf structs across different barns. So yeah, changing allocation locality is not a silver bullet.Quoted 3 lines
If this works well, we probably don't need to handle it in
init_percpu_sheaves()andbootstrap_cache_sheaves()at all as they will eventually be freed, while covering the other case too?
Quoted 7 lines
Quoted 4 lines
freeing the empty sheaf if the NUMA node mismatches instead of putting it back into the barn is indeed a good idea. I like it. But unfortunately, my testing didn't show a clear performance improvement, though there was no noticeable degradation either. :-(
Hmm... the idea still makes sense to me, but yeah, it's not late to fix it when we have data to back up.
I think if that approach doesn't complicate things unreasonably, it makes sense to do it even if there are no clear wins (as long as there are no regressions).
Harry Yoo
maintainerQuoted 47 lines · Vlastimil Babka (SUSE)
Quoted 46 lines · Harry Yoo
Quoted 44 lines · Hao Li
Quoted 21 lines · Harry Yoo
Quoted 11 lines · Hao Li
Sheaf structs are exchanged through node-local barns. Since barn structs are already allocated from their local NUMA node, this patch aims to allocate sheaf structs from their local memory nodes as well.
To achieve this, the obvious choice would be using
cpu_to_mem(). However,init_percpu_sheaves()andbootstrap_cache_sheaves()iterate through possible CPUs, whereascpu_to_mem()is only initialized for online CPUs. Therefore, we cannot usecpu_to_mem()and instead need to use local_memory_node(cpu_to_node(cpu)), similar to what__build_all_zonelists()does.Quoted 4 lines
The primary goal of this patch is to improve NUMA node locality. Although the actual performance impact is minor, it still yields a ~1% improvement on a 192-core, 8-NUMA-node system when testing with the will-it-scale mmap test case.
Oh, nice :)
I have a question though...
Quoted 8 lines
I wonder if would be better to handle this by e.g.) not returning empty sheaves back to barn and freeing them if the node id doesn't match and it's not a memoryless node.
init_percpu_sheaves()andbootstrap_cache_sheaves()are not the only places that can allocate sheaves from remote nodes; sheaves allocation could fall back to other nodes and then SLUB could keep reusing those sheaves from remote nodes even after memory is reclaimed.This is a good catch. In addition to the fallback mechanism, task migration between CPUs in
__pcs_replace_empty_main()and__pcs_replace_full_main()can also mix up sheaf structs across different barns. So yeah, changing allocation locality is not a silver bullet.Quoted 3 lines
If this works well, we probably don't need to handle it in
init_percpu_sheaves()andbootstrap_cache_sheaves()at all as they will eventually be freed, while covering the other case too?freeing the empty sheaf if the NUMA node mismatches instead of putting it back into the barn is indeed a good idea. I like it. But unfortunately, my testing didn't show a clear performance improvement, though there was no noticeable degradation either. :-(
Quoted 6 lines
Quoted 2 lines
Hmm... the idea still makes sense to me, but yeah, it's not late to fix it when we have data to back up.
I think if that approach doesn't complicate things unreasonably, it makes sense to do it even if there are no clear wins (as long as there are no regressions).
I'm fine with that unless it introduces regressions or the implementation complicates things too much :)
What do you say, Hao?
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 53 lines · Harry Yoo
Quoted 50 lines · Vlastimil Babka (SUSE)
Quoted 48 lines · Harry Yoo
Quoted 36 lines · Hao Li
Quoted 21 lines · Harry Yoo
Quoted 11 lines · Hao Li
Sheaf structs are exchanged through node-local barns. Since barn structs are already allocated from their local NUMA node, this patch aims to allocate sheaf structs from their local memory nodes as well.
To achieve this, the obvious choice would be using
cpu_to_mem(). However,init_percpu_sheaves()andbootstrap_cache_sheaves()iterate through possible CPUs, whereascpu_to_mem()is only initialized for online CPUs. Therefore, we cannot usecpu_to_mem()and instead need to use local_memory_node(cpu_to_node(cpu)), similar to what__build_all_zonelists()does.Quoted 4 lines
The primary goal of this patch is to improve NUMA node locality. Although the actual performance impact is minor, it still yields a ~1% improvement on a 192-core, 8-NUMA-node system when testing with the will-it-scale mmap test case.
Oh, nice :)
I have a question though...
Quoted 8 lines
I wonder if would be better to handle this by e.g.) not returning empty sheaves back to barn and freeing them if the node id doesn't match and it's not a memoryless node.
init_percpu_sheaves()andbootstrap_cache_sheaves()are not the only places that can allocate sheaves from remote nodes; sheaves allocation could fall back to other nodes and then SLUB could keep reusing those sheaves from remote nodes even after memory is reclaimed.This is a good catch. In addition to the fallback mechanism, task migration between CPUs in
__pcs_replace_empty_main()and__pcs_replace_full_main()can also mix up sheaf structs across different barns. So yeah, changing allocation locality is not a silver bullet.Quoted 8 lines
Quoted 3 lines
If this works well, we probably don't need to handle it in
init_percpu_sheaves()andbootstrap_cache_sheaves()at all as they will eventually be freed, while covering the other case too?freeing the empty sheaf if the NUMA node mismatches instead of putting it back into the barn is indeed a good idea. I like it. But unfortunately, my testing didn't show a clear performance improvement, though there was no noticeable degradation either. :-(
Hmm... the idea still makes sense to me, but yeah, it's not late to fix it when we have data to back up.
Quoted 9 lines
Quoted 3 lines
I think if that approach doesn't complicate things unreasonably, it makes sense to do it even if there are no clear wins (as long as there are no regressions).
I'm fine with that unless it introduces regressions or the implementation complicates things too much :)
What do you say, Hao?
Sure thing, let me test it thoroughly!
Signature · 2 lines
Thanks, Hao
Hao Li
reviewerQuoted 49 lines · Vlastimil Babka (SUSE)
Quoted 48 lines · Harry Yoo
Quoted 44 lines · Hao Li
Quoted 21 lines · Harry Yoo
Quoted 11 lines · Hao Li
Sheaf structs are exchanged through node-local barns. Since barn structs are already allocated from their local NUMA node, this patch aims to allocate sheaf structs from their local memory nodes as well.
To achieve this, the obvious choice would be using
cpu_to_mem(). However,init_percpu_sheaves()andbootstrap_cache_sheaves()iterate through possible CPUs, whereascpu_to_mem()is only initialized for online CPUs. Therefore, we cannot usecpu_to_mem()and instead need to use local_memory_node(cpu_to_node(cpu)), similar to what__build_all_zonelists()does.Quoted 4 lines
The primary goal of this patch is to improve NUMA node locality. Although the actual performance impact is minor, it still yields a ~1% improvement on a 192-core, 8-NUMA-node system when testing with the will-it-scale mmap test case.
Oh, nice :)
I have a question though...
Quoted 8 lines
I wonder if would be better to handle this by e.g.) not returning empty sheaves back to barn and freeing them if the node id doesn't match and it's not a memoryless node.
init_percpu_sheaves()andbootstrap_cache_sheaves()are not the only places that can allocate sheaves from remote nodes; sheaves allocation could fall back to other nodes and then SLUB could keep reusing those sheaves from remote nodes even after memory is reclaimed.This is a good catch. In addition to the fallback mechanism, task migration between CPUs in
__pcs_replace_empty_main()and__pcs_replace_full_main()can also mix up sheaf structs across different barns. So yeah, changing allocation locality is not a silver bullet.Quoted 3 lines
If this works well, we probably don't need to handle it in
init_percpu_sheaves()andbootstrap_cache_sheaves()at all as they will eventually be freed, while covering the other case too?freeing the empty sheaf if the NUMA node mismatches instead of putting it back into the barn is indeed a good idea. I like it. But unfortunately, my testing didn't show a clear performance improvement, though there was no noticeable degradation either. :-(
Quoted 6 lines
Quoted 2 lines
Hmm... the idea still makes sense to me, but yeah, it's not late to fix it when we have data to back up.
I think if that approach doesn't complicate things unreasonably, it makes sense to do it even if there are no clear wins (as long as there are no regressions).
This make sense to me. let me double check the performance impact. Thanks!
Signature · 2 lines
Thanks, Hao