mm/vmalloc: request large order pages from buddy allocator
Subsystems (2)
RFC PATCH 1/1 mm/vmalloc: request large order pages from buddy allocator
Sometimes, vm_area_alloc_pages() will want many pages from the buddy allocator. Rather than making requests to the buddy allocator for at most 100 pages at a time, we can eagerly request large order pages a smaller number of times.
We still split the large order pages down to order-0 as the rest of the vmalloc code (and some callers) depend on it. We still defer to the bulk allocator and fallback path in case of order-0 pages or failure.
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s
10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259s
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671s
This is an RFC, comments and thoughts are welcomed. There is a clear benefit to be had for large allocations, but there is some regression for smaller allocations.
--- mm/vmalloc.c | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)
mm/vmalloc.c +33 -1
@@ -3621,6 +3621,38 @@ vm_area_alloc_pages(gfp_t gfp, int nid,unsigned int nr_allocated =0 ;struct page *page;int i; +gfp_t large_gfp = (gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +unsigned int large_order =ilog2 (nr_pages - nr_allocated); + +/* +* Initially, attempt to have the page allocator give us large order +* pages. Do not attempt allocating smaller than order chunks since +* __vmap_pages_range() expects physically contigous pages of exactly +* order long chunks. +*/ +while (large_order > order && nr_allocated < nr_pages) { +/* +* High-order nofail allocations are really expensive and +* potentially dangerous (pre-mature OOM, disruptive reclaim +* and compaction etc. +*/ +if (gfp & __GFP_NOFAIL) +break ; +if (nid ==NUMA_NO_NODE ) + page =alloc_pages_noprof (large_gfp, large_order); +else + page =alloc_pages_node_noprof (nid, large_gfp, large_order); + +if (unlikely (!page)) +break ; + +split_page (page, large_order); +for (i =0 ; i < (1U << large_order); i++) + pages[nr_allocated + i] = page + i; + + nr_allocated +=1U << large_order; + large_order =ilog2 (nr_pages - nr_allocated); + }/* * For order-0 pages we make use of bulk allocator, if
@@ -3665,7 +3697,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, } } -/* High-order pages or fallback path if "bulk" fails. */ +/* High-order arch pages or fallback path if "bulk" fails. */ while (nr_allocated < nr_pages) {if (!(gfp & __GFP_NOFAIL) &&fatal_signal_pending (current))break ;
Signature · 1 line
2.51.0
Quoted 14 lines · Vishal Moola (Oracle)
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259s
Quoted 5 lines
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671s
I'd be more confident in the 20kB numbers if you'd done 10x more iterations.
Also, I think 20kB is probably an _interesting_ number, but it's not going to display your change to its best advantage. A 32kB allocation will look much better, for example.
Also, can you go into more detail of the test? Based on our off-list conversation, we were talking about allocating something like 100MB of memory (in these various sizes) then freeing it, just to be sure that we're measuring the performance of the buddy allocator and not the PCP list.
Quoted 3 lines
This is an RFC, comments and thoughts are welcomed. There is a clear benefit to be had for large allocations, but there is some regression for smaller allocations.
Also we think that there's probably a later win to be had by not splitting the page we allocated.
At some point, we should also start allocating frozen pages for vmalloc. That's going to be interesting for the users which map vmalloc pages to userspace.
Quoted 15 lines
mm/vmalloc.c +7
-0@@ -3621,6 +3621,38 @@ vm_area_alloc_pages(gfp_t gfp, int nid,unsigned int nr_allocated =0 ;struct page *page;int i; +gfp_t large_gfp = (gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +unsigned int large_order =ilog2 (nr_pages - nr_allocated); + +/* +* Initially, attempt to have the page allocator give us large order +* pages. Do not attempt allocating smaller than order chunks since +* __vmap_pages_range() expects physically contigous pages of exactly
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3640
Quoted 10 lines
mm/vmalloc.c +10
-0+* order long chunks. +*/ +while (large_order > order && nr_allocated < nr_pages) { +/* +* High-order nofail allocations are really expensive and +* potentially dangerous (pre-mature OOM, disruptive reclaim +* and compaction etc. +*/ +if (gfp & __GFP_NOFAIL) +break ;
sure, but we could just clear NOFAIL from the large_gfp flags instead of giving up on this path so quickly?
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3647
Quoted 7 lines
mm/vmalloc.c +7
-0+if (nid ==NUMA_NO_NODE ) + page =alloc_pages_noprof (large_gfp, large_order); +else + page =alloc_pages_node_noprof (nid, large_gfp, large_order); + +if (unlikely (!page)) +break ;
I'm not entirely convinced here. We might want to fall back to the next larger size. eg if we try to allocate an order-6 page, and there's not one readily available, perhaps we should try to allocate an order-5 page instead of falling back to the bulk allocator?
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3700
Quoted 6 lines
mm/vmalloc.c +1
-1@@ -3665,7 +3697,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, } } -/* High-order pages or fallback path if "bulk" fails. */ +/* High-order arch pages or fallback path if "bulk" fails. */
I'm not quite clear what this comment change is meant to convey?
Quoted 15 lines · Matthew Wilcox
Quoted 14 lines · Vishal Moola (Oracle)
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259s
Quoted 8 lines
Quoted 5 lines
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sI'd be more confident in the 20kB numbers if you'd done 10x more iterations.
I actually ran my a number of times to mitigate the effects of possibly too small sample sizes, so I do have that number for you too:
[Baseline] [This patch]
real 1m28.119s real 1m32.630s user 0m0.012s user 0m0.011s sys 1m23.270s sys 1m28.529s
Quoted 3 lines
Also, I think 20kB is probably an _interesting_ number, but it's not going to display your change to its best advantage. A 32kB allocation will look much better, for example.
I provided those particular numbers to showcase the beneficial cases as well as the regression case.
I ended up finding that allocating sizes <=20k had noticeable regressions, while [20k, 90k] was approximately the same, and >= 90k had improvements (getting more and more noticeable as size grows in magnitude).
Quoted 5 lines
Also, can you go into more detail of the test? Based on our off-list conversation, we were talking about allocating something like 100MB of memory (in these various sizes) then freeing it, just to be sure that we're measuring the performance of the buddy allocator and not the PCP list.
Yup.
What I did to get the numbers above was: call vmalloc() n number of times on that particular size, then free all those allocations. Then, I did 1000 iterations of that to get a better average.
So none of these allocations were freed until all the allocations were done, every single time.
Quoted 31 lines
Quoted 3 lines
This is an RFC, comments and thoughts are welcomed. There is a clear benefit to be had for large allocations, but there is some regression for smaller allocations.
Also we think that there's probably a later win to be had by not splitting the page we allocated.
At some point, we should also start allocating frozen pages for vmalloc. That's going to be interesting for the users which map vmalloc pages to userspace.
Quoted 20 lines
mm/vmalloc.c +12
-0@@ -3621,6 +3621,38 @@ vm_area_alloc_pages(gfp_t gfp, int nid,unsigned int nr_allocated =0 ;struct page *page;int i; +gfp_t large_gfp = (gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +unsigned int large_order =ilog2 (nr_pages - nr_allocated); + +/* +* Initially, attempt to have the page allocator give us large order +* pages. Do not attempt allocating smaller than order chunks since +* __vmap_pages_range() expects physically contigous pages of exactly +* order long chunks. +*/ +while (large_order > order && nr_allocated < nr_pages) { +/* +* High-order nofail allocations are really expensive and
Quoted 8 lines
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3640
Quoted 5 lines
mm/vmalloc.c +5
-0+* potentially dangerous (pre-mature OOM, disruptive reclaim +* and compaction etc. +*/ +if (gfp & __GFP_NOFAIL) +break ;sure, but we could just clear NOFAIL from the
large_gfpflags instead of giving up on this path so quickly?
Yeah I'll do that.
Quoted 4 lines
Quoted 4 lines
mm/vmalloc.c +4
-0+if (nid ==NUMA_NO_NODE ) + page =alloc_pages_noprof (large_gfp, large_order); +else + page =alloc_pages_node_noprof (nid, large_gfp, large_order);
Quoted 8 lines
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3647
Quoted 3 lines
mm/vmalloc.c +3
-0+ +if (unlikely (!page)) +break ;I'm not entirely convinced here. We might want to fall back to the next larger size. eg if we try to allocate an order-6 page, and there's not one readily available, perhaps we should try to allocate an order-5 page instead of falling back to the bulk allocator?
I'll try that out and see how that affects the numbers.
Quoted 8 lines
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3700
Quoted 6 lines
mm/vmalloc.c +1
-1@@ -3665,7 +3697,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, } } -/* High-order pages or fallback path if "bulk" fails. */ +/* High-order arch pages or fallback path if "bulk" fails. */ I'm not quite clear what this comment change is meant to convey?
Ah that was a comment I had inserted to remind myself that the passed in order is tied to the HAVE_ARCH_HUGE_VMALLOC config. I meant to leave that out.
Uladzislau Rezki
maintainerQuoted 25 lines · Vishal Moola (Oracle)
Quoted 15 lines · Matthew Wilcox
Quoted 14 lines · Vishal Moola (Oracle)
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259sQuoted 8 lines
Quoted 5 lines
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sI'd be more confident in the 20kB numbers if you'd done 10x more iterations.
Quoted 8 lines
I actually ran my a number of times to mitigate the effects of possibly too small sample sizes, so I do have that number for you too:
[Baseline] [This patch]
real 1m28.119s real 1m32.630s user 0m0.012s user 0m0.011s sys 1m23.270s sys 1m28.529s
I have just had a look at performance figures of this patch. The test case is 16K allocation by one single thread, 1 000 000 loops, 10 run:
sudo ./test_vmalloc.sh run_test_mask=1 nr_threads=1 nr_pages=4
BOX: AMD Milan, 256 CPUs, 512GB of memory
# default 16K alloc [ 15.823704] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955334 usec [ 17.751685] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1158739 usec [ 19.443759] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1016522 usec [ 21.035701] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 911381 usec [ 22.727688] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 987286 usec [ 24.199694] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955112 usec [ 25.755675] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 926393 usec [ 27.355670] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 937875 usec [ 28.979671] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1006985 usec [ 30.531674] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 941088 usec
# the patch 16K alloc [ 44.343380] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2296849 usec [ 47.171290] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2014678 usec [ 50.007258] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2094184 usec [ 52.651141] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1953046 usec [ 55.455089] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2209423 usec [ 57.943153] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1941747 usec [ 60.799043] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2038504 usec [ 63.299007] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1788588 usec [ 65.843011] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2137055 usec [ 68.647031] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2193022 usec
2X slower.
perf-cycles, same test but on 64 CPUs:
+ 97.02% 0.13% [test_vmalloc] [k] fix_size_alloc_test
- 82.11% 82.10% [kernel] [k] native_queued_spin_lock_slowpath
26.19% ret_from_fork_asm
ret_from_fork
- kthread
- 25.96% test_func
- fix_size_alloc_test
- 23.49% __vmalloc_node_noprof
- __vmalloc_node_range_noprof
- 54.70% alloc_pages_noprof
alloc_pages_mpol
__alloc_frozen_pages_noprof
get_page_from_freelist
__rmqueue_pcplist
- 5.58% __get_vm_area_node
alloc_vmap_area
- 20.54% vfree.part.0
- 20.43% __free_frozen_pages
free_frozen_page_commit
free_pcppages_bulk
_raw_spin_lock_irqsave
native_queued_spin_lock_slowpath
- 0.77% worker_thread
- process_one_work
- 0.76% vmstat_update
refresh_cpu_vm_stats
decay_pcp_high
free_pcppages_bulk
_raw_spin_lock_irqsave
native_queued_spin_lock_slowpath
+ 76.57% 0.16% [kernel] [k] _raw_spin_lock_irqsave
+ 71.62% 0.00% [kernel] [k] __vmalloc_node_noprof
+ 71.61% 0.58% [kernel] [k] __vmalloc_node_range_noprof
+ 62.35% 0.06% [kernel] [k] alloc_pages_mpol
+ 62.27% 0.17% [kernel] [k] __alloc_frozen_pages_noprof
+ 62.20% 0.02% [kernel] [k] alloc_pages_noprof
+ 62.10% 0.05% [kernel] [k] get_page_from_freelist
+ 55.63% 0.19% [kernel] [k] __rmqueue_pcplist
+ 32.11% 0.00% [kernel] [k] ret_from_fork_asm
+ 32.11% 0.00% [kernel] [k] ret_from_fork
+ 32.11% 0.00% [kernel] [k] kthreadI would say the bottle-neck is a page-allocator. It seems high-order allocations are not good for it.
--
Uladzislau Rezki
Quoted 34 lines · Uladzislau Rezki
Quoted 25 lines · Vishal Moola (Oracle)
Quoted 15 lines · Matthew Wilcox
Quoted 14 lines · Vishal Moola (Oracle)
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259sQuoted 8 lines
Quoted 5 lines
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sI'd be more confident in the 20kB numbers if you'd done 10x more iterations.
Quoted 8 lines
I actually ran my a number of times to mitigate the effects of possibly too small sample sizes, so I do have that number for you too:
[Baseline] [This patch]
real 1m28.119s real 1m32.630s user 0m0.012s user 0m0.011s sys 1m23.270s sys 1m28.529s
Quoted 4 lines
I have just had a look at performance figures of this patch. The test case is 16K allocation by one single thread, 1 000 000 loops, 10 run:
sudo ./test_vmalloc.sh run_test_mask=1 nr_threads=1 nr_pages=4
The reason I didn't use this test module is the same concern Matthew brought up earlier about testing the PCP list rather than buddy allocator. The test module allocates, then frees over and over again, making it incredibly prone to reuse the pages over and over again.
Quoted 77 lines
BOX: AMD Milan, 256 CPUs, 512GB of memory
# default 16K alloc [ 15.823704] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955334 usec [ 17.751685] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1158739 usec [ 19.443759] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1016522 usec [ 21.035701] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 911381 usec [ 22.727688] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 987286 usec [ 24.199694] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955112 usec [ 25.755675] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 926393 usec [ 27.355670] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 937875 usec [ 28.979671] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1006985 usec [ 30.531674] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 941088 usec# the patch 16K alloc [ 44.343380] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2296849 usec [ 47.171290] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2014678 usec [ 50.007258] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2094184 usec [ 52.651141] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1953046 usec [ 55.455089] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2209423 usec [ 57.943153] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1941747 usec [ 60.799043] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2038504 usec [ 63.299007] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1788588 usec [ 65.843011] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2137055 usec [ 68.647031] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2193022 usec2X slower.
perf-cycles, same test but on 64 CPUs:
(quoted diff) +12
-1+ 97.02% 0.13% [test_vmalloc] [k] fix_size_alloc_test - 82.11% 82.10% [kernel] [k] native_queued_spin_lock_slowpath 26.19% ret_from_fork_asm ret_from_fork - kthread - 25.96% test_func - fix_size_alloc_test - 23.49% __vmalloc_node_noprof - __vmalloc_node_range_noprof - 54.70% alloc_pages_noprof alloc_pages_mpol __alloc_frozen_pages_noprof get_page_from_freelist __rmqueue_pcplist - 5.58% __get_vm_area_node alloc_vmap_area - 20.54% vfree.part.0 - 20.43% __free_frozen_pages free_frozen_page_commit free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath - 0.77% worker_thread - process_one_work - 0.76% vmstat_update refresh_cpu_vm_stats decay_pcp_high free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath + 76.57% 0.16% [kernel] [k] _raw_spin_lock_irqsave + 71.62% 0.00% [kernel] [k] __vmalloc_node_noprof + 71.61% 0.58% [kernel] [k] __vmalloc_node_range_noprof + 62.35% 0.06% [kernel] [k] alloc_pages_mpol + 62.27% 0.17% [kernel] [k] __alloc_frozen_pages_noprof + 62.20% 0.02% [kernel] [k] alloc_pages_noprof + 62.10% 0.05% [kernel] [k] get_page_from_freelist + 55.63% 0.19% [kernel] [k] __rmqueue_pcplist + 32.11% 0.00% [kernel] [k] ret_from_fork_asm + 32.11% 0.00% [kernel] [k] ret_from_fork + 32.11% 0.00% [kernel] [k] kthreadI would say the bottle-neck is a page-allocator. It seems high-order allocations are not good for it.
--
Uladzislau Rezki
Quoted 38 lines · Vishal Moola (Oracle)
Quoted 37 lines · Uladzislau Rezki
Quoted 25 lines · Vishal Moola (Oracle)
Quoted 15 lines · Matthew Wilcox
Quoted 14 lines · Vishal Moola (Oracle)
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259sQuoted 8 lines
Quoted 5 lines
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sI'd be more confident in the 20kB numbers if you'd done 10x more iterations.
Quoted 8 lines
I actually ran my a number of times to mitigate the effects of possibly too small sample sizes, so I do have that number for you too:
[Baseline] [This patch]
real 1m28.119s real 1m32.630s user 0m0.012s user 0m0.011s sys 1m23.270s sys 1m28.529sI have just had a look at performance figures of this patch. The test case is 16K allocation by one single thread, 1 000 000 loops, 10 run:
Quoted 81 lines
Quoted 1 line
sudo ./test_vmalloc.sh run_test_mask=1 nr_threads=1 nr_pages=4
The reason I didn't use this test module is the same concern Matthew brought up earlier about testing the PCP list rather than buddy allocator. The test module allocates, then frees over and over again, making it incredibly prone to reuse the pages over and over again.
Quoted 74 lines
BOX: AMD Milan, 256 CPUs, 512GB of memory
# default 16K alloc [ 15.823704] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955334 usec [ 17.751685] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1158739 usec [ 19.443759] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1016522 usec [ 21.035701] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 911381 usec [ 22.727688] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 987286 usec [ 24.199694] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955112 usec [ 25.755675] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 926393 usec [ 27.355670] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 937875 usec [ 28.979671] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1006985 usec [ 30.531674] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 941088 usec# the patch 16K alloc [ 44.343380] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2296849 usec [ 47.171290] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2014678 usec [ 50.007258] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2094184 usec [ 52.651141] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1953046 usec [ 55.455089] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2209423 usec [ 57.943153] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1941747 usec [ 60.799043] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2038504 usec [ 63.299007] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1788588 usec [ 65.843011] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2137055 usec [ 68.647031] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2193022 usec2X slower.
perf-cycles, same test but on 64 CPUs:
(quoted diff) +12
-1+ 97.02% 0.13% [test_vmalloc] [k] fix_size_alloc_test - 82.11% 82.10% [kernel] [k] native_queued_spin_lock_slowpath 26.19% ret_from_fork_asm ret_from_fork - kthread - 25.96% test_func - fix_size_alloc_test - 23.49% __vmalloc_node_noprof - __vmalloc_node_range_noprof - 54.70% alloc_pages_noprof alloc_pages_mpol __alloc_frozen_pages_noprof get_page_from_freelist __rmqueue_pcplist - 5.58% __get_vm_area_node alloc_vmap_area - 20.54% vfree.part.0 - 20.43% __free_frozen_pages free_frozen_page_commit free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath - 0.77% worker_thread - process_one_work - 0.76% vmstat_update refresh_cpu_vm_stats decay_pcp_high free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath + 76.57% 0.16% [kernel] [k] _raw_spin_lock_irqsave + 71.62% 0.00% [kernel] [k] __vmalloc_node_noprof + 71.61% 0.58% [kernel] [k] __vmalloc_node_range_noprof + 62.35% 0.06% [kernel] [k] alloc_pages_mpol + 62.27% 0.17% [kernel] [k] __alloc_frozen_pages_noprof + 62.20% 0.02% [kernel] [k] alloc_pages_noprof + 62.10% 0.05% [kernel] [k] get_page_from_freelist + 55.63% 0.19% [kernel] [k] __rmqueue_pcplist + 32.11% 0.00% [kernel] [k] ret_from_fork_asm + 32.11% 0.00% [kernel] [k] ret_from_fork + 32.11% 0.00% [kernel] [k] kthreadI would say the bottle-neck is a page-allocator. It seems high-order allocations are not good for it.
Ah also just took a closer look at this. I realize that you also did 16k allocations (which is at most order-2), so it may not be a good representation of high-order allocations either.
Plus that falls into the regression range I found that I detailed in response to Matthew elsewhere (I've copy pasted it here for reference)
I ended up finding that allocating sizes <=20k had noticeable regressions, while [20k, 90k] was approximately the same, and >= 90k had improvements (getting more and more noticeable as size grows in magnitude).
Quoted 2 lines
Quoted 2 lines
--
Uladzislau Rezki
Uladzislau Rezki
maintainerQuoted 118 lines · Vishal Moola (Oracle)
Quoted 117 lines · Vishal Moola (Oracle)
Quoted 34 lines · Uladzislau Rezki
Quoted 25 lines · Vishal Moola (Oracle)
Quoted 15 lines · Matthew Wilcox
Quoted 14 lines · Vishal Moola (Oracle)
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259sQuoted 8 lines
Quoted 5 lines
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sI'd be more confident in the 20kB numbers if you'd done 10x more iterations.
Quoted 8 lines
I actually ran my a number of times to mitigate the effects of possibly too small sample sizes, so I do have that number for you too:
[Baseline] [This patch]
real 1m28.119s real 1m32.630s user 0m0.012s user 0m0.011s sys 1m23.270s sys 1m28.529sQuoted 4 lines
I have just had a look at performance figures of this patch. The test case is 16K allocation by one single thread, 1 000 000 loops, 10 run:
sudo ./test_vmalloc.sh run_test_mask=1 nr_threads=1 nr_pages=4
The reason I didn't use this test module is the same concern Matthew brought up earlier about testing the PCP list rather than buddy allocator. The test module allocates, then frees over and over again, making it incredibly prone to reuse the pages over and over again.
Quoted 72 lines
BOX: AMD Milan, 256 CPUs, 512GB of memory
# default 16K alloc [ 15.823704] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955334 usec [ 17.751685] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1158739 usec [ 19.443759] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1016522 usec [ 21.035701] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 911381 usec [ 22.727688] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 987286 usec [ 24.199694] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955112 usec [ 25.755675] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 926393 usec [ 27.355670] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 937875 usec [ 28.979671] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1006985 usec [ 30.531674] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 941088 usec# the patch 16K alloc [ 44.343380] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2296849 usec [ 47.171290] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2014678 usec [ 50.007258] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2094184 usec [ 52.651141] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1953046 usec [ 55.455089] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2209423 usec [ 57.943153] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1941747 usec [ 60.799043] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2038504 usec [ 63.299007] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1788588 usec [ 65.843011] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2137055 usec [ 68.647031] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2193022 usec2X slower.
perf-cycles, same test but on 64 CPUs:
(quoted diff) +12
-1+ 97.02% 0.13% [test_vmalloc] [k] fix_size_alloc_test - 82.11% 82.10% [kernel] [k] native_queued_spin_lock_slowpath 26.19% ret_from_fork_asm ret_from_fork - kthread - 25.96% test_func - fix_size_alloc_test - 23.49% __vmalloc_node_noprof - __vmalloc_node_range_noprof - 54.70% alloc_pages_noprof alloc_pages_mpol __alloc_frozen_pages_noprof get_page_from_freelist __rmqueue_pcplist - 5.58% __get_vm_area_node alloc_vmap_area - 20.54% vfree.part.0 - 20.43% __free_frozen_pages free_frozen_page_commit free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath - 0.77% worker_thread - process_one_work - 0.76% vmstat_update refresh_cpu_vm_stats decay_pcp_high free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath + 76.57% 0.16% [kernel] [k] _raw_spin_lock_irqsave + 71.62% 0.00% [kernel] [k] __vmalloc_node_noprof + 71.61% 0.58% [kernel] [k] __vmalloc_node_range_noprof + 62.35% 0.06% [kernel] [k] alloc_pages_mpol + 62.27% 0.17% [kernel] [k] __alloc_frozen_pages_noprof + 62.20% 0.02% [kernel] [k] alloc_pages_noprof + 62.10% 0.05% [kernel] [k] get_page_from_freelist + 55.63% 0.19% [kernel] [k] __rmqueue_pcplist + 32.11% 0.00% [kernel] [k] ret_from_fork_asm + 32.11% 0.00% [kernel] [k] ret_from_fork + 32.11% 0.00% [kernel] [k] kthread
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
I would say the bottle-neck is a page-allocator. It seems high-order allocations are not good for it.
Ah also just took a closer look at this. I realize that you also did 16k allocations (which is at most order-2), so it may not be a good representation of high-order allocations either.
I agree. But then we should not optimize "small" orders and focus on highest ones. Because of double degrade. I assume stress-ng fork test would alos notice this.
Quoted 8 lines
Plus that falls into the regression range I found that I detailed in response to Matthew elsewhere (I've copy pasted it here for reference)
I ended up finding that allocating sizes <=20k had noticeable regressions, while [20k, 90k] was approximately the same, and >= 90k had improvements (getting more and more noticeable as size grows in magnitude).
Yes, i did 2-order allocations
# default
+ 35.87% 4.24% [kernel] [k] alloc_pages_bulk_noprof
+ 31.94% 0.88% [kernel] [k] vfree.part.0
- 27.38% 27.36% [kernel] [k] clear_page_rep
27.36% ret_from_fork_asm
ret_from_fork
kthread
test_func
fix_size_alloc_test
__vmalloc_node_noprof
__vmalloc_node_range_noprof
alloc_pages_bulk_noprof
clear_page_rep# patch
+ 53.32% 1.12% [kernel] [k] get_page_from_freelist
+ 49.41% 0.71% [kernel] [k] prep_new_page
- 48.70% 48.64% [kernel] [k] clear_page_rep
48.64% ret_from_fork_asm
ret_from_fork
kthread
test_func
fix_size_alloc_test
__vmalloc_node_noprof
__vmalloc_node_range_noprof
alloc_pages_noprof
alloc_pages_mpol
__alloc_frozen_pages_noprof
get_page_from_freelist
prep_new_page
clear_page_repi noticed it is because of clear_page_rep() which with patch consumes double in cycles.
Both versions should mostly go over pcp-cache, as far as i remember order-2 is allowed to be cached.
I wounder why the patch gives x2 of cycles to clear_page_rep()...
--
Uladzislau Rezki
Uladzislau Rezki
maintainerQuoted 172 lines · Uladzislau Rezki
Quoted 118 lines · Vishal Moola (Oracle)
Quoted 117 lines · Vishal Moola (Oracle)
Quoted 34 lines · Uladzislau Rezki
Quoted 25 lines · Vishal Moola (Oracle)
Quoted 15 lines · Matthew Wilcox
Quoted 14 lines · Vishal Moola (Oracle)
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259sQuoted 8 lines
Quoted 5 lines
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sI'd be more confident in the 20kB numbers if you'd done 10x more iterations.
Quoted 8 lines
I actually ran my a number of times to mitigate the effects of possibly too small sample sizes, so I do have that number for you too:
[Baseline] [This patch]
real 1m28.119s real 1m32.630s user 0m0.012s user 0m0.011s sys 1m23.270s sys 1m28.529sQuoted 4 lines
I have just had a look at performance figures of this patch. The test case is 16K allocation by one single thread, 1 000 000 loops, 10 run:
sudo ./test_vmalloc.sh run_test_mask=1 nr_threads=1 nr_pages=4
The reason I didn't use this test module is the same concern Matthew brought up earlier about testing the PCP list rather than buddy allocator. The test module allocates, then frees over and over again, making it incredibly prone to reuse the pages over and over again.
Quoted 72 lines
BOX: AMD Milan, 256 CPUs, 512GB of memory
# default 16K alloc [ 15.823704] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955334 usec [ 17.751685] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1158739 usec [ 19.443759] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1016522 usec [ 21.035701] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 911381 usec [ 22.727688] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 987286 usec [ 24.199694] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955112 usec [ 25.755675] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 926393 usec [ 27.355670] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 937875 usec [ 28.979671] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1006985 usec [ 30.531674] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 941088 usec# the patch 16K alloc [ 44.343380] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2296849 usec [ 47.171290] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2014678 usec [ 50.007258] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2094184 usec [ 52.651141] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1953046 usec [ 55.455089] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2209423 usec [ 57.943153] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1941747 usec [ 60.799043] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2038504 usec [ 63.299007] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1788588 usec [ 65.843011] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2137055 usec [ 68.647031] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2193022 usec2X slower.
perf-cycles, same test but on 64 CPUs:
(quoted diff) +12
-1+ 97.02% 0.13% [test_vmalloc] [k] fix_size_alloc_test - 82.11% 82.10% [kernel] [k] native_queued_spin_lock_slowpath 26.19% ret_from_fork_asm ret_from_fork - kthread - 25.96% test_func - fix_size_alloc_test - 23.49% __vmalloc_node_noprof - __vmalloc_node_range_noprof - 54.70% alloc_pages_noprof alloc_pages_mpol __alloc_frozen_pages_noprof get_page_from_freelist __rmqueue_pcplist - 5.58% __get_vm_area_node alloc_vmap_area - 20.54% vfree.part.0 - 20.43% __free_frozen_pages free_frozen_page_commit free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath - 0.77% worker_thread - process_one_work - 0.76% vmstat_update refresh_cpu_vm_stats decay_pcp_high free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath + 76.57% 0.16% [kernel] [k] _raw_spin_lock_irqsave + 71.62% 0.00% [kernel] [k] __vmalloc_node_noprof + 71.61% 0.58% [kernel] [k] __vmalloc_node_range_noprof + 62.35% 0.06% [kernel] [k] alloc_pages_mpol + 62.27% 0.17% [kernel] [k] __alloc_frozen_pages_noprof + 62.20% 0.02% [kernel] [k] alloc_pages_noprof + 62.10% 0.05% [kernel] [k] get_page_from_freelist + 55.63% 0.19% [kernel] [k] __rmqueue_pcplist + 32.11% 0.00% [kernel] [k] ret_from_fork_asm + 32.11% 0.00% [kernel] [k] ret_from_fork + 32.11% 0.00% [kernel] [k] kthreadQuoted 7 lines
Quoted 2 lines
Quoted 2 lines
I would say the bottle-neck is a page-allocator. It seems high-order allocations are not good for it.
Ah also just took a closer look at this. I realize that you also did 16k allocations (which is at most order-2), so it may not be a good representation of high-order allocations either.
I agree. But then we should not optimize "small" orders and focus on highest ones. Because of double degrade. I assume stress-ng fork test would alos notice this.
Quoted 8 lines
Plus that falls into the regression range I found that I detailed in response to Matthew elsewhere (I've copy pasted it here for reference)
I ended up finding that allocating sizes <=20k had noticeable regressions, while [20k, 90k] was approximately the same, and >= 90k had improvements (getting more and more noticeable as size grows in magnitude).
Yes, i did 2-order allocations
# default(quoted diff) +2
-1+ 35.87% 4.24% [kernel] [k] alloc_pages_bulk_noprof + 31.94% 0.88% [kernel] [k] vfree.part.0 - 27.38% 27.36% [kernel] [k] clear_page_rep 27.36% ret_from_fork_asm ret_from_fork kthread test_func fix_size_alloc_test __vmalloc_node_noprof __vmalloc_node_range_noprof alloc_pages_bulk_noprof clear_page_rep# patch(quoted diff) +2
-1+ 53.32% 1.12% [kernel] [k] get_page_from_freelist + 49.41% 0.71% [kernel] [k] prep_new_page - 48.70% 48.64% [kernel] [k] clear_page_rep 48.64% ret_from_fork_asm ret_from_fork kthread test_func fix_size_alloc_test __vmalloc_node_noprof __vmalloc_node_range_noprof alloc_pages_noprof alloc_pages_mpol __alloc_frozen_pages_noprof get_page_from_freelist prep_new_page clear_page_rep
Quoted 8 lines
i noticed it is because of
clear_page_rep()which with patch consumes double in cycles.Both versions should mostly go over pcp-cache, as far as i remember order-2 is allowed to be cached.
I wounder why the patch gives x2 of cycles to
clear_page_rep()...
And here we go with some results "without" pcp exxecise:
static int fix_size_alloc_test(void)
{
void **ptr;
int i;
if (set_cpus_allowed_ptr(current, cpumask_of(1)) < 0)
pr_err("Failed to set affinity to %d CPU\n", 1);
ptr = vmalloc(sizeof(void *) * test_loop_count);
if (!ptr)
return -1;
for (i = 0; i < test_loop_count; i++)
ptr[i] = vmalloc((nr_pages > 0 ? nr_pages:1) * PAGE_SIZE);
for (i = 0; i < test_loop_count; i++) {
if (ptr[i])
vfree(ptr[i]);
}
return 0;
}time sudo ./test_vmalloc.sh run_test_mask=1 nr_threads=1 nr_pages=nr-pages-in-order
# default order-1 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1423862 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1453518 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1451734 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1455142 usec
# patch order-1 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1431082 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1454855 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1476372 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1433379 usec
# default order-2 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2198130 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2208504 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2219533 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2214151 usec
# patch order-2 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2110344 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2044186 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2083308 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2073572 usec
# default order-3 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3718592 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3740495 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3737213 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3740765 usec
# patch order-3 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3350391 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3374568 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3286374 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3261335 usec
# default order-6(64 pages) Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 23847773 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 24015706 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 24226268 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 24078102 usec
# patch order-6 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 20128225 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 19968964 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 20067469 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 19928870 usec
Now i see that results align with my initial thoughts when i first time saw your patch.
The question which is not clear for me still, why pcp case is doing better even for cached orders.
Do you have any thoughts?
--
Uladzislau Rezki
Quoted 250 lines · Uladzislau Rezki
Quoted 172 lines · Uladzislau Rezki
Quoted 118 lines · Vishal Moola (Oracle)
Quoted 117 lines · Vishal Moola (Oracle)
Quoted 34 lines · Uladzislau Rezki
Quoted 25 lines · Vishal Moola (Oracle)
Quoted 15 lines · Matthew Wilcox
Quoted 14 lines · Vishal Moola (Oracle)
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259sQuoted 8 lines
Quoted 5 lines
10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sI'd be more confident in the 20kB numbers if you'd done 10x more iterations.
Quoted 8 lines
I actually ran my a number of times to mitigate the effects of possibly too small sample sizes, so I do have that number for you too:
[Baseline] [This patch]
real 1m28.119s real 1m32.630s user 0m0.012s user 0m0.011s sys 1m23.270s sys 1m28.529sQuoted 4 lines
I have just had a look at performance figures of this patch. The test case is 16K allocation by one single thread, 1 000 000 loops, 10 run:
sudo ./test_vmalloc.sh run_test_mask=1 nr_threads=1 nr_pages=4
The reason I didn't use this test module is the same concern Matthew brought up earlier about testing the PCP list rather than buddy allocator. The test module allocates, then frees over and over again, making it incredibly prone to reuse the pages over and over again.
Quoted 72 lines
BOX: AMD Milan, 256 CPUs, 512GB of memory
# default 16K alloc [ 15.823704] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955334 usec [ 17.751685] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1158739 usec [ 19.443759] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1016522 usec [ 21.035701] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 911381 usec [ 22.727688] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 987286 usec [ 24.199694] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 955112 usec [ 25.755675] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 926393 usec [ 27.355670] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 937875 usec [ 28.979671] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1006985 usec [ 30.531674] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 941088 usec# the patch 16K alloc [ 44.343380] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2296849 usec [ 47.171290] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2014678 usec [ 50.007258] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2094184 usec [ 52.651141] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1953046 usec [ 55.455089] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2209423 usec [ 57.943153] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1941747 usec [ 60.799043] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2038504 usec [ 63.299007] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1788588 usec [ 65.843011] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2137055 usec [ 68.647031] Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2193022 usec2X slower.
perf-cycles, same test but on 64 CPUs:
(quoted diff) +12
-1+ 97.02% 0.13% [test_vmalloc] [k] fix_size_alloc_test - 82.11% 82.10% [kernel] [k] native_queued_spin_lock_slowpath 26.19% ret_from_fork_asm ret_from_fork - kthread - 25.96% test_func - fix_size_alloc_test - 23.49% __vmalloc_node_noprof - __vmalloc_node_range_noprof - 54.70% alloc_pages_noprof alloc_pages_mpol __alloc_frozen_pages_noprof get_page_from_freelist __rmqueue_pcplist - 5.58% __get_vm_area_node alloc_vmap_area - 20.54% vfree.part.0 - 20.43% __free_frozen_pages free_frozen_page_commit free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath - 0.77% worker_thread - process_one_work - 0.76% vmstat_update refresh_cpu_vm_stats decay_pcp_high free_pcppages_bulk _raw_spin_lock_irqsave native_queued_spin_lock_slowpath + 76.57% 0.16% [kernel] [k] _raw_spin_lock_irqsave + 71.62% 0.00% [kernel] [k] __vmalloc_node_noprof + 71.61% 0.58% [kernel] [k] __vmalloc_node_range_noprof + 62.35% 0.06% [kernel] [k] alloc_pages_mpol + 62.27% 0.17% [kernel] [k] __alloc_frozen_pages_noprof + 62.20% 0.02% [kernel] [k] alloc_pages_noprof + 62.10% 0.05% [kernel] [k] get_page_from_freelist + 55.63% 0.19% [kernel] [k] __rmqueue_pcplist + 32.11% 0.00% [kernel] [k] ret_from_fork_asm + 32.11% 0.00% [kernel] [k] ret_from_fork + 32.11% 0.00% [kernel] [k] kthreadQuoted 7 lines
Quoted 2 lines
Quoted 2 lines
I would say the bottle-neck is a page-allocator. It seems high-order allocations are not good for it.
Ah also just took a closer look at this. I realize that you also did 16k allocations (which is at most order-2), so it may not be a good representation of high-order allocations either.
I agree. But then we should not optimize "small" orders and focus on highest ones. Because of double degrade. I assume stress-ng fork test would alos notice this.
Quoted 8 lines
Plus that falls into the regression range I found that I detailed in response to Matthew elsewhere (I've copy pasted it here for reference)
I ended up finding that allocating sizes <=20k had noticeable regressions, while [20k, 90k] was approximately the same, and >= 90k had improvements (getting more and more noticeable as size grows in magnitude).
Yes, i did 2-order allocations
# default(quoted diff) +2
-1+ 35.87% 4.24% [kernel] [k] alloc_pages_bulk_noprof + 31.94% 0.88% [kernel] [k] vfree.part.0 - 27.38% 27.36% [kernel] [k] clear_page_rep 27.36% ret_from_fork_asm ret_from_fork kthread test_func fix_size_alloc_test __vmalloc_node_noprof __vmalloc_node_range_noprof alloc_pages_bulk_noprof clear_page_rep# patch(quoted diff) +2
-1+ 53.32% 1.12% [kernel] [k] get_page_from_freelist + 49.41% 0.71% [kernel] [k] prep_new_page - 48.70% 48.64% [kernel] [k] clear_page_rep 48.64% ret_from_fork_asm ret_from_fork kthread test_func fix_size_alloc_test __vmalloc_node_noprof __vmalloc_node_range_noprof alloc_pages_noprof alloc_pages_mpol __alloc_frozen_pages_noprof get_page_from_freelist prep_new_page clear_page_repQuoted 8 lines
i noticed it is because of
clear_page_rep()which with patch consumes double in cycles.Both versions should mostly go over pcp-cache, as far as i remember order-2 is allowed to be cached.
I wounder why the patch gives x2 of cycles to
clear_page_rep()...And here we go with some results "without" pcp exxecise:
static int
fix_size_alloc_test(void){ void **ptr; int i; if (set_cpus_allowed_ptr(current, cpumask_of(1)) < 0) pr_err("Failed to set affinity to %d CPU\n", 1); ptr = vmalloc(sizeof(void *) * test_loop_count); if (!ptr) return -1; for (i = 0; i < test_loop_count; i++) ptr[i] = vmalloc((nr_pages > 0 ? nr_pages:1) * PAGE_SIZE); for (i = 0; i < test_loop_count; i++) { if (ptr[i]) vfree(ptr[i]); } return 0; }time sudo ./test_vmalloc.sh run_test_mask=1 nr_threads=1 nr_pages=nr-pages-in-order
# default order-1 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1423862 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1453518 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1451734 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1455142 usec# patch order-1 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1431082 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1454855 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1476372 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 1433379 usec# default order-2 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2198130 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2208504 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2219533 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2214151 usec# patch order-2 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2110344 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2044186 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2083308 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 2073572 usec# default order-3 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3718592 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3740495 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3737213 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3740765 usec# patch order-3 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3350391 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3374568 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3286374 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 3261335 usec# default order-6(64 pages) Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 23847773 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 24015706 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 24226268 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 24078102 usec
Quoted 8 lines
# patch order-6 Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 20128225 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 19968964 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 20067469 usec Summary: fix_size_alloc_test passed: 1 failed: 0 xfailed: 0 repeat: 1 loops: 1000000 avg: 19928870 usecNow i see that results align with my initial thoughts when i first time saw your patch.
Its reassuring that your test results show similar performance even at the order-1 and order-2 cases. This was what I was expecting as well.
I'm assuming this happened because you tested exactly aligned PAGE_SIZE allocations (whereas somehow I hadn't thought to do that).
Quoted 4 lines
The question which is not clear for me still, why pcp case is doing better even for cached orders.
Do you have any thoughts?
I'm not sure either. I'm not familiar with the optimization differences between the standard and bulk allocators :(
When looking at the code, it appears that although the pcp lists can cache up to PAGE_ALLOC_COSTLY_ORDER (3), the bulk allocator doesn't have support for anything outside of order-0. And whenever order-0 pages are available, the bulk allocator appears incredibly efficient at grabbing them.
Uladzislau Rezki
maintainerQuoted 43 lines · Vishal Moola (Oracle)
Sometimes,
vm_area_alloc_pages()will want many pages from the buddy allocator. Rather than making requests to the buddy allocator for at most 100 pages at a time, we can eagerly request large order pages a smaller number of times.We still split the large order pages down to order-0 as the rest of the vmalloc code (and some callers) depend on it. We still defer to the bulk allocator and fallback path in case of order-0 pages or failure.
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259s10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sThis is an RFC, comments and thoughts are welcomed. There is a clear benefit to be had for large allocations, but there is some regression for smaller allocations.
Signed-off-by Vishal Moola (Oracle) <[email protected]>--- mm/vmalloc.c | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)mm/vmalloc.c +0
-0
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3626
Quoted 6 lines
mm/vmalloc.c +2
-0@@ -3621,6 +3621,38 @@ vm_area_alloc_pages(gfp_t gfp, int nid,unsigned int nr_allocated =0 ;struct page *page;int i; +gfp_t large_gfp = (gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +unsigned int large_order =ilog2 (nr_pages - nr_allocated);
If large_order is > MAX_ORDER - 1 then there is no need even try larger_order attempt.
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3627
Quoted 1 line
mm/vmalloc.c +0
-0> unsigned int large_order = ilog2(nr_pages - nr_allocated);
I think, it is better to introduce "remaining" variable which is nr_pages - nr_allocated. And on entry "remaining" can be set to just nr_pages because "nr_allocated" is zero.
Maybe it is worth to drop/warn if __GFP_COMP is set also?
Quoted 21 lines
mm/vmalloc.c +21
-0+ +/* +* Initially, attempt to have the page allocator give us large order +* pages. Do not attempt allocating smaller than order chunks since +* __vmap_pages_range() expects physically contigous pages of exactly +* order long chunks. +*/ +while (large_order > order && nr_allocated < nr_pages) { +/* +* High-order nofail allocations are really expensive and +* potentially dangerous (pre-mature OOM, disruptive reclaim +* and compaction etc. +*/ +if (gfp & __GFP_NOFAIL) +break ; +if (nid ==NUMA_NO_NODE ) + page =alloc_pages_noprof (large_gfp, large_order); +else + page =alloc_pages_node_noprof (nid, large_gfp, large_order); + +if (unlikely (!page))
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3656
Quoted 10 lines
mm/vmalloc.c +9
-0+break ; + +split_page (page, large_order); +for (i =0 ; i < (1U << large_order); i++) + pages[nr_allocated + i] = page + i; + + nr_allocated +=1U << large_order; + large_order =ilog2 (nr_pages - nr_allocated); + }
So this is a third path for page allocation. The question is should we try all orders? Like already noted by Matthew, if there is no 5-order page but there is 4-order page? Try until we check all orders. For example we can get different order pages to fulfill the request.
The concern is then if it is a waste of high-order pages. Because we can easily go with a single page allocator. Whereas someone in a system can not.
Apart of that, maybe we can drop the bulk_path instead of having three paths?
--
Uladzislau Rezki
Quoted 53 lines · Uladzislau Rezki
Quoted 43 lines · Vishal Moola (Oracle)
Sometimes,
vm_area_alloc_pages()will want many pages from the buddy allocator. Rather than making requests to the buddy allocator for at most 100 pages at a time, we can eagerly request large order pages a smaller number of times.We still split the large order pages down to order-0 as the rest of the vmalloc code (and some callers) depend on it. We still defer to the bulk allocator and fallback path in case of order-0 pages or failure.
Running 1000 iterations of allocations on a small 4GB system finds:
1000 2mb allocations:
[Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s10000 200kb allocations:
[Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259s10000 20kb allocations:
[Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671sThis is an RFC, comments and thoughts are welcomed. There is a clear benefit to be had for large allocations, but there is some regression for smaller allocations.
Signed-off-by Vishal Moola (Oracle) <[email protected]>--- mm/vmalloc.c | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)mm/vmalloc.c +0
-0mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3626
Quoted 6 lines
mm/vmalloc.c +2
-0@@ -3621,6 +3621,38 @@ vm_area_alloc_pages(gfp_t gfp, int nid,unsigned int nr_allocated =0 ;struct page *page;int i; +gfp_t large_gfp = (gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +unsigned int large_order =ilog2 (nr_pages - nr_allocated);If
large_orderis >MAX_ORDER- 1 then there is no need even trylarger_orderattempt.
Quoted 4 lines
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3627
Quoted 1 line
mm/vmalloc.c +0
-0> unsigned int large_order = ilog2(nr_pages - nr_allocated);I think, it is better to introduce "remaining" variable which is
nr_pages-nr_allocated. And on entry "remaining" can be set to justnr_pagesbecause "nr_allocated" is zero.
I like the idea too.
Quoted 1 line
Maybe it is worth to drop/warn if
__GFP_COMPis set also?
split_page() has a BUG_ON(PageCompound) within, so we don't need one out here for now.
Quoted 36 lines
Quoted 21 lines
mm/vmalloc.c +21
-0+ +/* +* Initially, attempt to have the page allocator give us large order +* pages. Do not attempt allocating smaller than order chunks since +* __vmap_pages_range() expects physically contigous pages of exactly +* order long chunks. +*/ +while (large_order > order && nr_allocated < nr_pages) { +/* +* High-order nofail allocations are really expensive and +* potentially dangerous (pre-mature OOM, disruptive reclaim +* and compaction etc. +*/ +if (gfp & __GFP_NOFAIL) +break ; +if (nid ==NUMA_NO_NODE ) + page =alloc_pages_noprof (large_gfp, large_order); +else + page =alloc_pages_node_noprof (nid, large_gfp, large_order); + +if (unlikely (!page))mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3656
Quoted 10 lines
mm/vmalloc.c +9
-0+break ; + +split_page (page, large_order); +for (i =0 ; i < (1U << large_order); i++) + pages[nr_allocated + i] = page + i; + + nr_allocated +=1U << large_order; + large_order =ilog2 (nr_pages - nr_allocated); + }So this is a third path for page allocation. The question is should we try all orders? Like already noted by Matthew, if there is no 5-order page but there is 4-order page? Try until we check all orders. For example we can get different order pages to fulfill the request.
Quoted 2 lines
The concern is then if it is a waste of high-order pages. Because we can easily go with a single page allocator. Whereas someone in a system can not.
I feel like if we have high order pages available we'd rather allocate those. Since the buddy allocator just coalesces the pages when they're freed again, as soon as these allocations free up we are much more likely to have large order pages ready to go again.
Quoted 1 line
Apart of that, maybe we can drop the
bulk_pathinstead of having three paths?
Probably. I'd say that just depends on whether we care about maintaining the optimizations for smaller vmallocs() - which I have no strong opinion on.
Quoted 2 lines
--
Uladzislau Rezki
Quoted 7 lines · Vishal Moola (Oracle)
Quoted 6 lines · Uladzislau Rezki
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3625
Quoted 4 lines
mm/vmalloc.c +2
-0int i; +gfp_t large_gfp = (gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +unsigned int large_order =ilog2 (nr_pages - nr_allocated);If
large_orderis >MAX_ORDER- 1 then there is no need even trylarger_orderattempt.
Oh, I meant to mention that too. Yes, this should be min(MAX_ORDER, ilog2()).
Quoted 4 lines
Quoted 1 line
Maybe it is worth to drop/warn if
__GFP_COMPis set also?
split_page()has aBUG_ON(PageCompound)within, so we don't need one out here for now.
I don't think people actually call vmalloc() with __GFP_COMP set, but clearing it would do no harm here.
Quoted 7 lines
Quoted 2 lines
The concern is then if it is a waste of high-order pages. Because we can easily go with a single page allocator. Whereas someone in a system can not.
I feel like if we have high order pages available we'd rather allocate those. Since the buddy allocator just coalesces the pages when they're freed again, as soon as these allocations free up we are much more likely to have large order pages ready to go again.
My PoV is different from either of you -- that we actually want to allocate the high-order pages when we can because it reduces fragmentation. If we allocate five separate pages to satisfy a 20kB allocation, those may come from five different 2MB pages (since they're probably coming from the pcp lists which after a sufficiently long period of running will be a jumble). Whereas if we allocate an order-2 page and an order-0 page, those can come from at most two 2MB pages.
I understand the "allocating order-0 pages helps by using up the remnants of previous allocations" argument. But I think on the whole we need to be doing larger allocations where possible, not smaller ones.
Uladzislau Rezki
maintainerQuoted 11 lines · Matthew Wilcox
Quoted 7 lines · Vishal Moola (Oracle)
Quoted 6 lines · Uladzislau Rezki
mm/vmalloc.c vm_area_alloc_pages(gfp_t gfp, int nid, · line 3625
Quoted 4 lines
mm/vmalloc.c +2
-0int i; +gfp_t large_gfp = (gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +unsigned int large_order =ilog2 (nr_pages - nr_allocated);If
large_orderis >MAX_ORDER- 1 then there is no need even trylarger_orderattempt.Oh, I meant to mention that too. Yes, this should be
min(MAX_ORDER, ilog2()).
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Maybe it is worth to drop/warn if
__GFP_COMPis set also?
split_page()has aBUG_ON(PageCompound)within, so we don't need one out here for now.I don't think people actually call
vmalloc()with__GFP_COMPset, but clearing it would do no harm here.
Agree. We do not want BUG_ON() in split_page(). I think it is better to control this even though nobody invokes vmalloc() with __GFP_COMP.
Quoted 16 lines
Quoted 7 lines
Quoted 2 lines
The concern is then if it is a waste of high-order pages. Because we can easily go with a single page allocator. Whereas someone in a system can not.
I feel like if we have high order pages available we'd rather allocate those. Since the buddy allocator just coalesces the pages when they're freed again, as soon as these allocations free up we are much more likely to have large order pages ready to go again.
My PoV is different from either of you -- that we actually want to allocate the high-order pages when we can because it reduces fragmentation. If we allocate five separate pages to satisfy a 20kB allocation, those may come from five different 2MB pages (since they're probably coming from the pcp lists which after a sufficiently long period of running will be a jumble). Whereas if we allocate an order-2 page and an order-0 page, those can come from at most two 2MB pages.
Quoted 4 lines
I understand the "allocating order-0 pages helps by using up the remnants of previous allocations" argument. But I think on the whole we need to be doing larger allocations where possible, not smaller ones.
OK, i see. Then we can start from highest "fit" order as starting point and switch to lower ones if failing. Fallback to bulk or single page allocator if it is still not enough pages.
Thank you!
--
Uladzislau Rezki
Quoted 2 lines · Uladzislau Rezki
Agree. We do not want
BUG_ON()insplit_page(). I think it is better to control this even though nobody invokesvmalloc()with__GFP_COMP.
Please explicitly warn and reject vmalloc calls with unsupported flag. The fact that many flags get silently ignored or dropped or could lead to behavior in gfp_t based interfaces is a constant source of problems.
Uladzislau Rezki
maintainerQuoted 9 lines · Christoph Hellwig
Quoted 2 lines · Uladzislau Rezki
Agree. We do not want
BUG_ON()insplit_page(). I think it is better to control this even though nobody invokesvmalloc()with__GFP_COMP.Please explicitly warn and reject vmalloc calls with unsupported flag. The fact that many flags get silently ignored or dropped or could lead to behavior in
gfp_tbased interfaces is a constant source of problems.
Thank you for input.
--
Uladzislau Rezki