mm/vmalloc: Switch to bulk allocator in __vmalloc_area_node()
Subsystems (3)
Uladzislau Rezki (Sony)
maintainerPATCH v2 1/2 mm/vmalloc: Switch to bulk allocator in __vmalloc_area_node()
Recently there has been introduced a page bulk allocator for users which need to get number of pages per one call request.
For order-0 pages switch to __alloc_pages_bulk() instead of alloc_pages_node(), the reason is the former is not capable of allocating set of pages, thus a one call is per one page.
Second, according to my tests the bulk allocator uses less cycles even for scenarios when only one page is requested. Running the "perf" on same test case shows below difference:
<default>
- 45.18% __vmalloc_node
- __vmalloc_node_range
- 35.60% __alloc_pages
- get_page_from_freelist
3.36% __list_del_entry_valid
3.00% check_preemption_disabled
1.42% prep_new_page
<default>
<patch>
- 31.00% __vmalloc_node
- __vmalloc_node_range
- 14.48% __alloc_pages_bulk
3.22% __list_del_entry_valid
- 0.83% __alloc_pages
get_page_from_freelist
<patch>
The "test_vmalloc.sh" also shows performance improvements:
fix_size_alloc_test_4MB loops: 1000000 avg: 89105095 usecfix_size_alloc_test loops: 1000000 avg: 513672 usecfull_fit_alloc_test loops: 1000000 avg: 748900 useclong_busy_list_alloc_test loops: 1000000 avg: 8043038 usecrandom_size_alloc_test loops: 1000000 avg: 4028582 usecfix_align_alloc_test loops: 1000000 avg: 1457671 usec
fix_size_alloc_test_4MB loops: 1000000 avg: 62083711 usecfix_size_alloc_test loops: 1000000 avg: 449207 usecfull_fit_alloc_test loops: 1000000 avg: 735985 useclong_busy_list_alloc_test loops: 1000000 avg: 5176052 usecrandom_size_alloc_test loops: 1000000 avg: 2589252 usecfix_align_alloc_test loops: 1000000 avg: 1365009 usec
For example 4MB allocations illustrates ~30% gain, all the rest is also better.
--- mm/vmalloc.c | 74 ++++++++++++++++++++++++++++------------------------ 1 file changed, 40 insertions(+), 34 deletions(-)
mm/vmalloc.c +40 -34
@@ -2766,8 +2766,6 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,unsigned long array_size;unsigned int nr_small_pages = size >>PAGE_SHIFT ;unsigned int page_order; -struct page **pages; -unsigned int i; array_size = (unsigned long )nr_small_pages *sizeof (struct page *); gfp_mask |= __GFP_NOWARN;
@@ -2776,13 +2774,13 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,/* Please note that the recursion is strictly bounded. */ if (array_size >PAGE_SIZE ) { - pages =__vmalloc_node (array_size,1 , nested_gfp, node, + area->pages =__vmalloc_node (array_size,1 , nested_gfp, node, area->caller); }else { - pages =kmalloc_node (array_size, nested_gfp, node); + area->pages =kmalloc_node (array_size, nested_gfp, node); } -if (!pages) { +if (!area->pages) {free_vm_area (area);warn_alloc (gfp_mask,NULL , "vmalloc size %lu allocation failure: "
@@ -2791,43 +2789,51 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,return NULL ; } - area->pages = pages; - area->nr_pages = nr_small_pages;set_vm_area_page_order (area, page_shift -PAGE_SHIFT ); - page_order =vm_area_page_order (area); -/* -* Careful, we allocate and map page_order pages, but tracking is done -* per PAGE_SIZE page so as to keep the vm_struct APIs independent of -* the physical/mapped size. -*/ -for (i =0 ; i < area->nr_pages; i +=1U << page_order) { -struct page *page; -int p; - -/* Compound pages required for remap_vmalloc_page */ - page =alloc_pages_node (node, gfp_mask | __GFP_COMP, page_order); -if (unlikely (!page)) { -/* Successfully allocated i pages, free them in __vfree() */ - area->nr_pages = i; -atomic_long_add (area->nr_pages, &nr_vmalloc_pages); -warn_alloc (gfp_mask,NULL , -"vmalloc size %lu allocation failure: " -"page order %u allocation failed" , - area->nr_pages *PAGE_SIZE , page_order); - goto fail; - } +if (!page_order) { + area->nr_pages =__alloc_pages_bulk (gfp_mask, node, +NULL , nr_small_pages,NULL , area->pages); + }else { +/* +* Careful, we allocate and map page_order pages, but tracking is done +* per PAGE_SIZE page so as to keep the vm_struct APIs independent of +* the physical/mapped size. +*/ +for (area->nr_pages =0 ; area->nr_pages < nr_small_pages; + area->nr_pages +=1U << page_order) { +struct page *page; +int i; + +/* Compound pages required for remap_vmalloc_page */ + page =alloc_pages_node (node, gfp_mask | __GFP_COMP, page_order); +if (unlikely (!page)) +break ; -for (p =0 ; p < (1U << page_order); p++) - area->pages[i + p] = page + p; +for (i =0 ; i < (1U << page_order); i++) + area->pages[area->nr_pages + i] = page + i; -if (gfpflags_allow_blocking (gfp_mask)) -cond_resched (); +if (gfpflags_allow_blocking (gfp_mask)) +cond_resched (); + } } +atomic_long_add (area->nr_pages, &nr_vmalloc_pages); -if (vmap_pages_range (addr, addr + size, prot, pages, page_shift) <0 ) { +/* +* If not enough pages were obtained to accomplish an +* allocation request, free them via __vfree() if any. +*/ +if (area->nr_pages != nr_small_pages) { +warn_alloc (gfp_mask,NULL , +"vmalloc size %lu allocation failure: " +"page order %u allocation failed" , + area->nr_pages *PAGE_SIZE , page_order); + goto fail; + } + +if (vmap_pages_range (addr, addr + size, prot, area->pages, page_shift) <0 ) { warn_alloc(gfp_mask, NULL, "vmalloc size %lu allocation failure: " "failed to map pages",
Signature · 1 line
2.20.1
Uladzislau Rezki (Sony)
maintainerPATCH v2 2/2 mm/vmalloc: Print a warning message first on failure
When a memory allocation for array of pages are not succeed emit a warning message as a first step and then perform the further cleanup.
--- Makefile | 2 +- mm/vmalloc.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
Makefile +1 -1
@@ -430,7 +430,7 @@ HOSTCXX = g++ endif export KBUILD_USERCFLAGS := -Wall -Wmissing-prototypes -Wstrict-prototypes \ - -O2 -fomit-frame-pointer -std=gnu89 + -O0 -g -fomit-frame-pointer -std=gnu89 export KBUILD_USERLDFLAGS := KBUILD_HOSTCFLAGS := $(KBUILD_USERCFLAGS) $(HOST_LFS_CFLAGS) $(HOSTCFLAGS)
mm/vmalloc.c +1 -1
@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }if (!area->pages) { -free_vm_area (area);warn_alloc (gfp_mask,NULL ,"vmalloc size %lu allocation failure: " "page array size %lu allocation failed" , nr_small_pages *PAGE_SIZE , array_size); +free_vm_area (area);return NULL ; }
Signature · 1 line
2.20.1
Uladzislau Rezki
maintainerRe: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
Quoted 43 lines
When a memory allocation for array of pages are not succeed emit a warning message as a first step and then perform the further cleanup.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- Makefile | 2 +- mm/vmalloc.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)Makefile +1
-1@@ -430,7 +430,7 @@ HOSTCXX = g++ endif export KBUILD_USERCFLAGS := -Wall -Wmissing-prototypes -Wstrict-prototypes \ - -O2 -fomit-frame-pointer -std=gnu89 + -O0 -g -fomit-frame-pointer -std=gnu89 export KBUILD_USERLDFLAGS := KBUILD_HOSTCFLAGS := $(KBUILD_USERCFLAGS) $(HOST_LFS_CFLAGS) $(HOSTCFLAGS)mm/vmalloc.c +1
-1@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }if (!area->pages) { -free_vm_area (area);warn_alloc (gfp_mask,NULL ,"vmalloc size %lu allocation failure: " "page array size %lu allocation failed" , nr_small_pages *PAGE_SIZE , array_size); +free_vm_area (area);return NULL ; }Signature · 1 line
2.20.1
Please consider a V3 of this patch. Apparently a modification of Makefile was included in the patch. What is wrong :)
From 9c484d83a630c4430dcb055a26c879d8e3c5cae1 Mon Sep 17 00:00:00 2001
From: "Uladzislau Rezki (Sony)" <[email protected]>
Date: Sat, 8 May 2021 23:41:21 +0200
Subject: [PATCH v3 2/2] mm/vmalloc: Print a warning message first on failure
When a memory allocation for array of pages are not succeed emit a warning message as a first step and then perform the further cleanup.
--- mm/vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmalloc.c +1 -1
@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }if (!area->pages) { -free_vm_area (area);warn_alloc (gfp_mask,NULL ,"vmalloc size %lu allocation failure: " "page array size %lu allocation failed" , nr_small_pages *PAGE_SIZE , array_size); +free_vm_area (area);return NULL ; }
Signature · 4 lines
2.20.1 -- Vlad Rezki
Re: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
In reply to Uladzislau Rezki (Sony)Makefile HOSTCXX = g++ · line 433
Quoted 3 lines · Uladzislau Rezki (Sony)
export
KBUILD_USERCFLAGS:= -Wall -Wmissing-prototypes -Wstrict-prototypes \Makefile +1
-1- -O2 -fomit-frame-pointer -std=gnu89 + -O0 -g -fomit-frame-pointer -std=gnu89
You clearly didn't intend to submit this portion ...
mm/vmalloc.c static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, · line 2790
Quoted 13 lines
+++
b/mm/vmalloc.cmm/vmalloc.c +1
-1@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }if (!area->pages) { -free_vm_area (area);warn_alloc (gfp_mask,NULL ,"vmalloc size %lu allocation failure: " "page array size %lu allocation failed" , nr_small_pages *PAGE_SIZE , array_size); +free_vm_area (area);return NULL ; }
I think this is a bad idea. We're clearly low on memory (a memory allocation just failed). We should free the memory we have allocated to improve the chances of the warning message making it out.
Uladzislau Rezki
maintainerRe: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
Quoted 7 lines · Matthew Wilcox
Makefile HOSTCXX = g++ · line 433
Quoted 3 lines · Uladzislau Rezki (Sony)
export
KBUILD_USERCFLAGS:= -Wall -Wmissing-prototypes -Wstrict-prototypes \Makefile +1
-1- -O2 -fomit-frame-pointer -std=gnu89 + -O0 -g -fomit-frame-pointer -std=gnu89You clearly didn't intend to submit this portion ...
I sent a v3 that does not include it. That was added accidentally.
Quoted 4 lines
Quoted 4 lines
+++
b/mm/vmalloc.c(quoted diff) +0
-0@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }
Quoted 13 lines
mm/vmalloc.c static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, · line 2790
Quoted 9 lines
mm/vmalloc.c +1
-1if (!area->pages) { -free_vm_area (area);warn_alloc (gfp_mask,NULL ,"vmalloc size %lu allocation failure: " "page array size %lu allocation failed" , nr_small_pages *PAGE_SIZE , array_size); +free_vm_area (area);return NULL ; }I think this is a bad idea. We're clearly low on memory (a memory allocation just failed). We should free the memory we have allocated to improve the chances of the warning message making it out.
Not sure if i fully follow you here. We do free the memory. The intention was to print a warning message first because, if, potentially, the free_vm_area(area) also does some prints it would be a bit messy from the point what has been broken first.
So, could you please clarify what was your concern?
--
Vlad Rezki
Re: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
Quoted 18 lines · Uladzislau Rezki
Quoted 4 lines · Matthew Wilcox
Quoted 4 lines
+++
b/mm/vmalloc.c(quoted diff) +0
-0@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }Quoted 13 lines
mm/vmalloc.c static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, · line 2790
Quoted 9 lines
mm/vmalloc.c +1
-1if (!area->pages) { -free_vm_area (area);warn_alloc (gfp_mask,NULL ,"vmalloc size %lu allocation failure: " "page array size %lu allocation failed" , nr_small_pages *PAGE_SIZE , array_size); +free_vm_area (area);return NULL ; }I think this is a bad idea. We're clearly low on memory (a memory allocation just failed). We should free the memory we have allocated to improve the chances of the warning message making it out.
Quoted 6 lines
Not sure if i fully follow you here. We do free the memory. The intention was to print a warning message first because, if, potentially, the
free_vm_area(area)also does some prints it would be a bit messy from the point what has been broken first.So, could you please clarify what was your concern?
We may need to allocate memory in order to emit the error message. Your commit message didn't mention the potential confusion, and I think that is worth adding for a v4.
Uladzislau Rezki
maintainerRe: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
Quoted 24 lines · Matthew Wilcox
Quoted 23 lines · Uladzislau Rezki
Quoted 4 lines · Matthew Wilcox
Quoted 4 lines
+++
b/mm/vmalloc.c(quoted diff) +0
-0@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }Quoted 13 lines
mm/vmalloc.c static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, · line 2790
Quoted 9 lines
mm/vmalloc.c +1
-1if (!area->pages) { -free_vm_area (area);warn_alloc (gfp_mask,NULL ,"vmalloc size %lu allocation failure: " "page array size %lu allocation failed" , nr_small_pages *PAGE_SIZE , array_size); +free_vm_area (area);return NULL ; }I think this is a bad idea. We're clearly low on memory (a memory allocation just failed). We should free the memory we have allocated to improve the chances of the warning message making it out.
Not sure if i fully follow you here. We do free the memory. The intention was to print a warning message first because, if, potentially, the
free_vm_area(area)also does some prints it would be a bit messy from the point what has been broken first.
Quoted 6 lines
Quoted 1 line
So, could you please clarify what was your concern?
We may need to allocate memory in order to emit the error message.
Your commit message didn't mention the potential confusion, and I think that is worth adding for a v4.
I agree that the commit message should be updated in regard of potential confusion, because it was the main intention of this patch.
I will upload a v4 tomorrow.
--
Vlad Rezki
Uladzislau Rezki
maintainerRe: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
Quoted 27 lines · Uladzislau Rezki
Quoted 26 lines · Matthew Wilcox
Quoted 24 lines · Uladzislau Rezki
Quoted 4 lines · Matthew Wilcox
Quoted 4 lines
+++
b/mm/vmalloc.c(quoted diff) +0
-0@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }Quoted 13 lines
Review static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, · line 2790
Quoted 9 lines
(quoted diff) +1
-1if (!area->pages) { - free_vm_area(area); warn_alloc(gfp_mask, NULL, "vmalloc size %lu allocation failure: " "page array size %lu allocation failed", nr_small_pages * PAGE_SIZE, array_size); + free_vm_area(area); return NULL; }I think this is a bad idea. We're clearly low on memory (a memory allocation just failed). We should free the memory we have allocated to improve the chances of the warning message making it out.
Not sure if i fully follow you here. We do free the memory. The intention was to print a warning message first because, if, potentially, the
free_vm_area(area)also does some prints it would be a bit messy from the point what has been broken first.So, could you please clarify what was your concern?
Quoted 9 lines
Quoted 4 lines
We may need to allocate memory in order to emit the error message.
Your commit message didn't mention the potential confusion, and I think that is worth adding for a v4.
I agree that the commit message should be updated in regard of potential confusion, because it was the main intention of this patch.
I will upload a v4 tomorrow.
Please find the v4 version of the patch that is in question:
From 7e27e4ac8f299ae244e9e0e90e0292ae2c08d37d Mon Sep 17 00:00:00 2001
From: "Uladzislau Rezki (Sony)" <[email protected]>
Date: Sat, 8 May 2021 23:41:21 +0200
Subject: [PATCH v4 1/1] mm/vmalloc: Print a warning message first on failure
When a memory allocation for array of pages are not succeed emit a warning message as a first step and then perform the further cleanup.
The reason it should be done in a right order is the clean up function which is free_vm_area() can potentially also follow its error paths what can lead to confusion what was broken first.
--- mm/vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmalloc.c +1 -1
@@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, }if (!area->pages) { -free_vm_area (area);warn_alloc (gfp_mask,NULL ,"vmalloc size %lu allocation failure: " "page array size %lu allocation failed" , nr_small_pages *PAGE_SIZE , array_size); +free_vm_area (area);return NULL ; }
Signature · 4 lines
2.20.1 -- Vlad Rezki
Andrew Morton
maintainerRe: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
Quoted 6 lines · On Mon, 10 May 2021 12:33:42 +0200 Uladzislau Rezki <[email protected]> wrote:
Please find the v4 version of the patch that is in question:
Quoted 1 line
From
7e27e4ac8f299ae244e9e0e90e0292ae2c08d37dMon Sep 17 00:00:00 2001From: "Uladzislau Rezki (Sony)" <[email protected]>
Date: Sat, 8 May 2021 23:41:21 +0200
Subject: [PATCH v4 1/1] mm/vmalloc: Print a warning message first on failure
Added, thanks.
Matthew has a point of course, but I do think that any console driver which tries to allocate memory within the cotext of printk() is so pathetic that it isn't worth compromising core code to cater for it...
Re: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
Quoted 9 lines · Andrew Morton
Quoted 6 lines · On Mon, 10 May 2021 12:33:42 +0200 Uladzislau Rezki <[email protected]> wrote:
Please find the v4 version of the patch that is in question:
Quoted 1 line
From
7e27e4ac8f299ae244e9e0e90e0292ae2c08d37dMon Sep 17 00:00:00 2001From: "Uladzislau Rezki (Sony)" <[email protected]>
Date: Sat, 8 May 2021 23:41:21 +0200
Subject: [PATCH v4 1/1] mm/vmalloc: Print a warning message first on failure
Quoted 5 lines
Added, thanks.
Matthew has a point of course, but I do think that any console driver which tries to allocate memory within the cotext of
printk()is so pathetic that it isn't worth compromising core code to cater for it...
I'm fine with v4 of this patch, fwiw. I saw no advantage to the earlier version, but now that the advantage has been explained, the advantage outweighs the disadvantage.
Uladzislau Rezki
maintainerRe: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure
Quoted 12 lines · Matthew Wilcox
Quoted 11 lines · Andrew Morton
Quoted 6 lines · On Mon, 10 May 2021 12:33:42 +0200 Uladzislau Rezki <[email protected]> wrote:
Please find the v4 version of the patch that is in question:
Quoted 1 line
From
7e27e4ac8f299ae244e9e0e90e0292ae2c08d37dMon Sep 17 00:00:00 2001From: "Uladzislau Rezki (Sony)" <[email protected]>
Date: Sat, 8 May 2021 23:41:21 +0200
Subject: [PATCH v4 1/1] mm/vmalloc: Print a warning message first on failureAdded, thanks.
Quoted 7 lines
Quoted 3 lines
Matthew has a point of course, but I do think that any console driver which tries to allocate memory within the cotext of
printk()is so pathetic that it isn't worth compromising core code to cater for it...I'm fine with v4 of this patch, fwiw. I saw no advantage to the earlier version, but now that the advantage has been explained, the advantage outweighs the disadvantage.
Thanks!
The point about an extra memory for printk is correct. From the other hand i am not able to recall issues related to lack of memory for printk() to emit a message. At least on our mobile devices i have not seen such scenarios.
--
Vlad Rezki