fix numa spreading for large hash tables
Subsystems (5)
v2 cover fix numa spreading for large hash tables
[PATCH v2 1/2] fix numa spreading problem
[PATCH v2 2/2] optimization about performance
v1 ==> v2:
1. do a minimal fix in [PATCH v2 1/2]
2. add some comments
Chen Wandun (2):
mm/vmalloc: fix numa spreading for large hash tables mm/vmalloc: introduce alloc_pages_bulk_array_mempolicy to accelerate
memory allocation
include/linux/gfp.h | 4 +++ mm/mempolicy.c | 82 +++++++++++++++++++++++++++++++++++++++++++++ mm/vmalloc.c | 28 ++++++++++++---- 3 files changed, 108 insertions(+), 6 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/2 mm/vmalloc: fix numa spreading for large hash tables
Eric Dumazet reported a strange numa spreading info in [1], and found commit 121e6f3258fe ("mm/vmalloc: hugepage vmalloc mappings") introduced this issue [2].
Dig into the difference before and after this patch, page allocation has some difference:
before:alloc_large_system_hash
__vmalloc
__vmalloc_node(..., NUMA_NO_NODE, ...)
__vmalloc_node_range
__vmalloc_area_node
alloc_page /* because NUMA_NO_NODE, so choose alloc_page branch */
alloc_pages_current
alloc_page_interleave /* can be proved by print policy mode */
after:alloc_large_system_hash
__vmalloc
__vmalloc_node(..., NUMA_NO_NODE, ...)
__vmalloc_node_range
__vmalloc_area_node
alloc_pages_node /* choose nid by nuam_mem_id() */
__alloc_pages_node(nid, ....)So after commit 121e6f3258fe ("mm/vmalloc: hugepage vmalloc mappings"), it will allocate memory in current node instead of interleaving allocate memory.
[1]
https://lore.kernel.org/linux-mm/CANn89iL6AAyWhfxdHO+jaT075iOa3XcYn9k6JJc7JR2XYn6k_Q@mail.gmail.com/
[2]
https://lore.kernel.org/linux-mm/CANn89iLofTR=AK-QOZY87RdUZENCZUT4O6a0hvhu3_EwRMerOg@mail.gmail.com/
--- mm/vmalloc.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
mm/vmalloc.c +10 -6
@@ -2816,6 +2816,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid, unsigned int order, unsigned int nr_pages, struct page **pages) { unsigned int nr_allocated = 0; +struct page *page; +int i;/* * For order-0 pages we make use of bulk allocator, if
@@ -2823,7 +2825,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,* to fails, fallback to a single page allocator that is * more permissive. */ -if (!order) { +if (!order && nid !=NUMA_NO_NODE ) {while (nr_allocated < nr_pages) {unsigned int nr, nr_pages_request;
@@ -2848,7 +2850,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,if (nr != nr_pages_request)break ; } - }else + }else if (order)/* * Compound pages required for remap_vmalloc_page if * high-order pages.
@@ -2856,11 +2858,13 @@ vm_area_alloc_pages(gfp_t gfp, int nid, gfp |= __GFP_COMP;/* High-order pages or fallback path if "bulk" fails. */ -while (nr_allocated < nr_pages) { -struct page *page; -int i; - page =alloc_pages_node (nid, gfp, order); + page =NULL ; +while (nr_allocated < nr_pages) { +if (nid ==NUMA_NO_NODE ) + page =alloc_pages (gfp, order); +else + page =alloc_pages_node (nid, gfp, order);if (unlikely (!page))break ;
Signature · 1 line
2.25.1
Re: [PATCH v2 1/2] mm/vmalloc: fix numa spreading for large hash tables
Quoted 3 lines · Chen Wandun
Eric Dumazet reported a strange numa spreading info in [1], and found commit
121e6f3258fe("mm/vmalloc: hugepage vmalloc mappings") introduced this issue [2].
I think the root problem here is that we have two meanings for NUMA_NO_NODE. I tend to read it as "The memory can be allocated from any node", but here it's used to mean "The memory should be spread over every node". Should we split those out as -1 and -2?
Re: [PATCH v2 1/2] mm/vmalloc: fix numa spreading for large hash tables
Quoted 9 lines · Matthew Wilcox
Quoted 3 lines · Chen Wandun
Eric Dumazet reported a strange numa spreading info in [1], and found commit
121e6f3258fe("mm/vmalloc: hugepage vmalloc mappings") introduced this issue [2].I think the root problem here is that we have two meanings for
NUMA_NO_NODE. I tend to read it as "The memory can be allocated from any node", but here it's used to mean "The memory should be spread over every node". Should we split those out as -1 and -2?
Yes, the intent of NUMA_NO_NODE some time is confused.
Besides,I think NUMA_NO_NODE should consider mempolicy in most cases in the kernel unless it point out explicitly memory can be allocated without considering mempolicy.
Quoted 2 lines
.
Re: [PATCH v2 1/2] mm/vmalloc: fix numa spreading for large hash tables
In reply to Matthew WilcoxQuoted 10 lines · Matthew Wilcox
Quoted 3 lines · Chen Wandun
Eric Dumazet reported a strange numa spreading info in [1], and found commit
121e6f3258fe("mm/vmalloc: hugepage vmalloc mappings") introduced this issue [2].I think the root problem here is that we have two meanings for
NUMA_NO_NODE. I tend to read it as "The memory can be allocated from any node", but here it's used to mean "The memory should be spread over every node". Should we split those out as -1 and -2?
I agree with Willy's suggestion to make it more explicit but as a followup work. This patch needs a backport, so keep this simple.
Re: [PATCH v2 1/2] mm/vmalloc: fix numa spreading for large hash tables
Quoted 8 lines · Shakeel Butt
Quoted 6 lines · Matthew Wilcox
Quoted 3 lines · Chen Wandun
Eric Dumazet reported a strange numa spreading info in [1], and found commit
121e6f3258fe("mm/vmalloc: hugepage vmalloc mappings") introduced this issue [2].
Quoted 7 lines
Quoted 4 lines
I think the root problem here is that we have two meanings for
NUMA_NO_NODE. I tend to read it as "The memory can be allocated from any node", but here it's used to mean "The memory should be spread over every node". Should we split those out as -1 and -2?I agree with Willy's suggestion to make it more explicit but as a followup work. This patch needs a backport, so keep this simple.
NUMA_NO_NODE in process context also meant :
Please follow current thread NUMA policies.
One could hope for instance, that whenever large BPF maps are allocated, current thread could set non default NUMA policies.
Uladzislau Rezki
maintainerRe: [PATCH v2 1/2] mm/vmalloc: fix numa spreading for large hash tables
In reply to Chen WandunQuoted 75 lines
Eric Dumazet reported a strange numa spreading info in [1], and found commit
121e6f3258fe("mm/vmalloc: hugepage vmalloc mappings") introduced this issue [2].Dig into the difference before and after this patch, page allocation has some difference:
before:
alloc_large_system_hash
__vmalloc
__vmalloc_node(..., NUMA_NO_NODE, ...)
__vmalloc_node_range
__vmalloc_area_node
alloc_page/* becauseNUMA_NO_NODE, so choosealloc_pagebranch */
alloc_pages_current
alloc_page_interleave/* can be proved by print policy mode */after:
alloc_large_system_hash__vmalloc __vmalloc_node(..., NUMA_NO_NODE, ...) __vmalloc_node_range __vmalloc_area_node alloc_pages_node /* choose nid by nuam_mem_id() */ __alloc_pages_node(nid, ....)So after commit
121e6f3258fe("mm/vmalloc: hugepage vmalloc mappings"), it will allocate memory in current node instead of interleaving allocate memory.[1]
https://lore.kernel.org/linux-mm/CANn89iL6AAyWhfxdHO+jaT075iOa3XcYn9k6JJc7JR2XYn6k_Q@mail.gmail.com/[2]
https://lore.kernel.org/linux-mm/CANn89iLofTR=AK-QOZY87RdUZENCZUT4O6a0hvhu3_EwRMerOg@mail.gmail.com/Fixes 121e6f3258fe ("mm/vmalloc: hugepage vmalloc mappings")Reported-by Eric Dumazet <[email protected]>Signed-off-by Chen Wandun <[email protected]>--- mm/vmalloc.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)mm/vmalloc.c +4
-2@@ -2816,6 +2816,8 @@ vm_area_alloc_pages(gfp_t gfp, int nid, unsigned int order, unsigned int nr_pages, struct page **pages) { unsigned int nr_allocated = 0; +struct page *page; +int i;/* * For order-0 pages we make use of bulk allocator, if @@ -2823,7 +2825,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,* to fails, fallback to a single page allocator that is * more permissive. */ -if (!order) { +if (!order && nid !=NUMA_NO_NODE ) {while (nr_allocated < nr_pages) {unsigned int nr, nr_pages_request;@@ -2848,7 +2850,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,if (nr != nr_pages_request)break ; } - }else + }else if (order)/* * Compound pages required for remap_vmalloc_page if * high-order pages.
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 2863
Quoted 10 lines
mm/vmalloc.c +1
-4@@ -2856,11 +2858,13 @@ vm_area_alloc_pages(gfp_t gfp, int nid, gfp |= __GFP_COMP;/* High-order pages or fallback path if "bulk" fails. */ -while (nr_allocated < nr_pages) { -struct page *page; -int i; - page =alloc_pages_node (nid, gfp, order); + page =NULL ;
Why do you need to set page to NULL here?
Signature · 1 line
Vlad Rezki
Re: [PATCH v2 1/2] mm/vmalloc: fix numa spreading for large hash tables
In reply to Chen WandunQuoted 1 line · Chen Wandun
[...]
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 2862
Quoted 8 lines
/* High-order pages or fallback path if "bulk" fails. */mm/vmalloc.c +1
-4-while (nr_allocated < nr_pages) { -struct page *page; -int i; - page =alloc_pages_node (nid, gfp, order); + page =NULL ;
No need for the above NULL assignment.
After removing this, you can add:
Quoted 11 lines
mm/vmalloc.c +5
-1+while (nr_allocated < nr_pages) { +if (nid ==NUMA_NO_NODE ) + page =alloc_pages (gfp, order); +else + page =alloc_pages_node (nid, gfp, order);if (unlikely (!page))break ; --2.25.1
PATCH v2 2/2 mm/vmalloc: introduce alloc_pages_bulk_array_mempolicy to accelerate memory allocation
In reply to Chen WandunIt will cause significant performance regressions in some situations as Andrew mentioned in [1]. The main situation is vmalloc, vmalloc will allocate pages with NUMA_NO_NODE by default, that will result in alloc page one by one;
In order to solve this, __alloc_pages_bulk and mempolicy should be considered at the same time.
1) If node is specified in memory allocation request, it will alloc all pages by __alloc_pages_bulk.
2) If interleaving allocate memory, it will cauculate how many pages should be allocated in each node, and use __alloc_pages_bulk to alloc pages in each node.
--- include/linux/gfp.h | 4 +++ mm/mempolicy.c | 82 +++++++++++++++++++++++++++++++++++++++++++++ mm/vmalloc.c | 20 ++++++++--- 3 files changed, 102 insertions(+), 4 deletions(-)
include/linux/gfp.h +4 -0
@@ -535,6 +535,10 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid,struct list_head *page_list, struct page **page_array); +unsigned long alloc_pages_bulk_array_mempolicy (gfp_t gfp, +unsigned long nr_pages, +struct page **page_array); +/* Bulk allocate order-0 pages */ static inline unsigned long alloc_pages_bulk_list (gfp_t gfp,unsigned long nr_pages,struct list_head *list)
mm/mempolicy.c +82 -0
88 lines not on this page · raw message
mm/vmalloc.c +16 -4
@@ -2825,7 +2825,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,* to fails, fallback to a single page allocator that is * more permissive. */ -if (!order && nid !=NUMA_NO_NODE ) { +if (!order) {while (nr_allocated < nr_pages) {unsigned int nr, nr_pages_request;
@@ -2837,8 +2837,20 @@ vm_area_alloc_pages(gfp_t gfp, int nid,*/ nr_pages_request =min (100U , nr_pages - nr_allocated); - nr =alloc_pages_bulk_array_node (gfp, nid, - nr_pages_request, pages + nr_allocated); +/* memory allocation should consider mempolicy, we cant +* wrongly use nearest node when nid == NUMA_NO_NODE, +* otherwise memory may be allocated in only one node, +* but mempolcy want to alloc memory by interleaving. +*/ +if (nid ==NUMA_NO_NODE ) + nr =alloc_pages_bulk_array_mempolicy (gfp, + nr_pages_request, + pages + nr_allocated); + +else + nr =alloc_pages_bulk_array_node (gfp, nid, + nr_pages_request, + pages + nr_allocated); nr_allocated += nr;cond_resched ();
@@ -2850,7 +2862,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid,if (nr != nr_pages_request)break ; } - }else if (order) + }else /* * Compound pages required for remap_vmalloc_page if * high-order pages.
Signature · 1 line
2.25.1