improve vmap allocation
Subsystems (2)
Uladzislau Rezki (Sony)
maintainerv4 cover improve vmap allocation
Hello.
This is the v4.
Objective
---------
Please have a look for the description at: https://lkml.org/lkml/2018/10/19/786
But let me also summarize it a bit here as well. The current implementation has O(N) complexity. Requests with different permissive parameters can lead to long allocation time. When i say "long" i mean milliseconds.
Description
-----------
This approach organizes the KVA memory layout into free areas of the 1-ULONG_MAX range, i.e. an allocation is done over free areas lookups, instead of finding a hole between two busy blocks. It allows to have lower number of objects which represent the free space, therefore to have less fragmented memory allocator. Because free blocks are always as large as possible.
It uses the augment tree where all free areas are sorted in ascending order of va->va_start address in pair with linked list that provides O(1) access to prev/next elements.
Since the tree is augment, we also maintain the "subtree_max_size" of VA that reflects a maximum available free block in its left or right sub-tree. Knowing that, we can easily traversal toward the lowest(left most path) free area.
Allocation: ~O(log(N)) complexity. It is sequential allocation method therefore tends to maximize locality. The search is done until a first suitable block is large enough to encompass the requested parameters. Bigger areas are split.
I copy paste here the description of how the area is split, since i described it in https://lkml.org/lkml/2018/10/19/786
<snip>
A free block can be split by three different ways. Their names are FL_FIT_TYPE, LE_FIT_TYPE/RE_FIT_TYPE and NE_FIT_TYPE, i.e. they correspond to how requested size and alignment fit to a free block.
FL_FIT_TYPE - in this case a free block is just removed from the free list/tree because it fully fits. Comparing with current design there is an extra work with rb-tree updating.
LE_FIT_TYPE/RE_FIT_TYPE - left/right edges fit. In this case what we do is just cutting a free block. It is as fast as a current design. Most of the vmalloc allocations just end up with this case, because the edge is always aligned to 1.
NE_FIT_TYPE - Is much less common case. Basically it happens when requested size and alignment does not fit left nor right edges, i.e. it is between them. In this case during splitting we have to build a remaining left free area and place it back to the free list/tree.
Comparing with current design there are two extra steps. First one is we have to allocate a new vmap_area structure. Second one we have to insert that remaining free block to the address sorted list/tree.
In order to optimize a first case there is a cache with free_vmap objects. Instead of allocating from slab we just take an object from the cache and reuse it.
Second one is pretty optimized. Since we know a start point in the tree we do not do a search from the top. Instead a traversal begins from a rb-tree node we split. <snip>
De-allocation. ~O(log(N)) complexity. An area is not inserted straight away to the tree/list, instead we identify the spot first, checking if it can be merged around neighbors. The list provides O(1) access to prev/next, so it is pretty fast to check it. Summarizing. If merged then large coalesced areas are created, if not the area is just linked making more fragments.
There is one more thing that i should mention here. After modification of VA node, its subtree_max_size is updated if it was/is the biggest area in its left or right sub-tree. Apart of that it can also be populated back to upper levels to fix the tree. For more details please have a look at the __augment_tree_propagate_from() function and the description.
Tests and stressing
-------------------
I use the "test_vmalloc.sh" test driver available under "tools/testing/selftests/vm/" since 5.1-rc1 kernel. Just trigger "sudo ./test_vmalloc.sh" to find out how to deal with it.
Tested on different platforms including x86_64/i686/ARM64/x86_64_NUMA. Regarding last one, i do not have any physical access to NUMA system, therefore i emulated it. The time of stressing is days.
If you run the test driver in "stress mode", you also need the patch that is in Andrew's tree but not in Linux 5.1-rc1. So, please apply it:
http://git.cmpxchg.org/cgit.cgi/linux-mmotm.git/commit/?id=e0cf7749bade6da318e98e934a24d8b62fab512c
After massive testing, i have not identified any problems like memory leaks, crashes or kernel panics. I find it stable, but more testing would be good.
Performance analysis
--------------------
I have used two systems to test. One is i5-3320M CPU @ 2.60GHz and another is HiKey960(arm64) board. i5-3320M runs on 4.20 kernel, whereas Hikey960 uses 4.15 kernel. I have both system which could run on 5.1-rc1 as well, but the results have not been ready by time i an writing this.
Currently it consist of 8 tests. There are three of them which correspond to different types of splitting(to compare with default). We have 3 ones(see above). Another 5 do allocations in different conditions.
a) sudo ./test_vmalloc.sh performance
When the test driver is run in "performance" mode, it runs all available tests pinned to first online CPU with sequential execution test order. We do it in order to get stable and repeatable results. Take a look at time difference in "long_busy_list_alloc_test". It is not surprising because the worst case is O(N).
# i5-3320M
How many cycles all tests took:
CPU0=646919905370(default) cycles vs CPU0=193290498550(patched) cycles
# See detailed table with results here:
ftp://vps418301.ovh.net/incoming/vmap_test_results_v2/i5-3320M_performance_default.txt ftp://vps418301.ovh.net/incoming/vmap_test_results_v2/i5-3320M_performance_patched.txt
# Hikey960 8x CPUs
How many cycles all tests took:
CPU0=3478683207 cycles vs CPU0=463767978 cycles
# See detailed table with results here:
ftp://vps418301.ovh.net/incoming/vmap_test_results_v2/HiKey960_performance_default.txt ftp://vps418301.ovh.net/incoming/vmap_test_results_v2/HiKey960_performance_patched.txt
b) time sudo ./test_vmalloc.sh test_repeat_count=1
With this configuration, all tests are run on all available online CPUs. Before running each CPU shuffles its tests execution order. It gives random allocation behaviour. So it is rough comparison, but it puts in the picture for sure.
# i5-3320M
<default> vs <patched> real 101m22.813s real 0m56.805s user 0m0.011s user 0m0.015s sys 0m5.076s sys 0m0.023s
# See detailed table with results here:
ftp://vps418301.ovh.net/incoming/vmap_test_results_v2/i5-3320M_test_repeat_count_1_default.txt ftp://vps418301.ovh.net/incoming/vmap_test_results_v2/i5-3320M_test_repeat_count_1_patched.txt
# Hikey960 8x CPUs
<default> vs <patched> real unknown real 4m25.214s user unknown user 0m0.011s sys unknown sys 0m0.670s
I did not manage to complete this test on "default Hikey960" kernel version. After 24 hours it was still running, therefore i had to cancel it. That is why real/user/sys are "unknown".
Changes in v4
-------------
- updated the commit message of [1] patch;
- simplify te compute_subtree_max_size() function by using max3() macro;
- added more explanation to find_va_links() function;
- reworked the function names;
- replace u8 type by using enum fit_type;
- when init the vmap free space, trigger WARN_ON_ONCE() if kmem_cache* fails;
- reworked a bit the pvm_determine_end_from_reverse() function;
- invert "if" condition in __get_va_next_sibling();
- removed intermediate function in [2] patch.
Changes in v3
-------------
- simplify the __get_va_next_sibling() and __find_va_links() functions;
- remove "unlikely". Place the WARN_ON_ONCE directly to the "if" condition;
- replace inline to __always_inline;
- move the debug code to separate patches;
Changes in v2
-------------
- do not distinguish vmalloc and other vmap allocations;
- use kmem_cache for vmap_area objects instead of own implementation;
- remove vmap cache globals;
- fix pcpu allocator on NUMA systems;
- now complexity is ~O(log(N)).
Uladzislau Rezki (Sony) (3):
mm/vmap: keep track of free blocks for vmap allocation mm/vmap: add DEBUG_AUGMENT_PROPAGATE_CHECK macro mm/vmap: add DEBUG_AUGMENT_LOWEST_MATCH_CHECK macro
include/linux/vmalloc.h | 6 +- mm/vmalloc.c | 1095 ++++++++++++++++++++++++++++++++++++----------- 2 files changed, 854 insertions(+), 247 deletions(-)
Signature · 1 line
2.11.0
Uladzislau Rezki (Sony)
maintainerPATCH v4 1/3 mm/vmap: keep track of free blocks for vmap allocation
Currently an allocation of the new vmap area is done over busy list iteration(complexity O(n)) until a suitable hole is found between two busy areas. Therefore each new allocation causes the list being grown. Due to over fragmented list and different permissive parameters an allocation can take a long time. For example on embedded devices it is milliseconds.
This patch organizes the KVA memory layout into free areas of the 1-ULONG_MAX range. It uses an augment red-black tree that keeps blocks sorted by their offsets in pair with linked list keeping the free space in order of increasing addresses.
Nodes are augmented with the size of the maximum available free block in its left or right sub-tree. Thus, that allows to take a decision and traversal toward the block that will fit and will have the lowest start address, i.e. it is sequential allocation.
Allocation: to allocate a new block a search is done over the tree until a suitable lowest(left most) block is large enough to encompass: the requested size, alignment and vstart point. If the block is bigger than requested size - it is split.
De-allocation: when a busy vmap area is freed it can either be merged or inserted to the tree. Red-black tree allows efficiently find a spot whereas a linked list provides a constant-time access to previous and next blocks to check if merging can be done. In case of merging of de-allocated memory chunk a large coalesced area is created.
Complexity: ~O(log(N))
--- include/linux/vmalloc.h | 6 +- mm/vmalloc.c | 1004 +++++++++++++++++++++++++++++++++++------------ 2 files changed, 763 insertions(+), 247 deletions(-)
include/linux/vmalloc.h +5 -1
@@ -45,12 +45,16 @@ struct vm_struct {struct vmap_area {unsigned long va_start;unsigned long va_end; + +/* +* Largest available free size in subtree. +*/ +unsigned long subtree_max_size;unsigned long flags;struct rb_node rb_node;/* address sorted rbtree */ struct list_head list;/* address sorted list */ struct llist_node purge_list;/* "lazy purge" list */ struct vm_struct *vm; -struct rcu_head rcu_head; };/*
mm/vmalloc.c +758 -246
1,192 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [PATCH v4 1/3] mm/vmap: keep track of free blocks for vmap allocation
Quoted 29 lines · Uladzislau Rezki (Sony)
Currently an allocation of the new vmap area is done over busy list iteration(complexity O(n)) until a suitable hole is found between two busy areas. Therefore each new allocation causes the list being grown. Due to over fragmented list and different permissive parameters an allocation can take a long time. For example on embedded devices it is milliseconds.
This patch organizes the KVA memory layout into free areas of the 1-ULONG_MAX range. It uses an augment red-black tree that keeps blocks sorted by their offsets in pair with linked list keeping the free space in order of increasing addresses.
Nodes are augmented with the size of the maximum available free block in its left or right sub-tree. Thus, that allows to take a decision and traversal toward the block that will fit and will have the lowest start address, i.e. it is sequential allocation.
Allocation: to allocate a new block a search is done over the tree until a suitable lowest(left most) block is large enough to encompass: the requested size, alignment and vstart point. If the block is bigger than requested size - it is split.
De-allocation: when a busy vmap area is freed it can either be merged or inserted to the tree. Red-black tree allows efficiently find a spot whereas a linked list provides a constant-time access to previous and next blocks to check if merging can be done. In case of merging of de-allocated memory chunk a large coalesced area is created.
Quoted 3 lines
Complexity: ~O(log(N))
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>
Thanks!
Uladzislau Rezki
maintainerRe: [PATCH v4 1/3] mm/vmap: keep track of free blocks for vmap allocation
Hello, Roman.
Quoted 4 lines
Reviewed-by Roman Gushchin <[email protected]>Thanks!
I appreciate your effort in reviewing to make it better.
Thank you!
Signature · 1 line
Uladzislau Rezki
Andrew Morton
maintainerRe: [PATCH v4 1/3] mm/vmap: keep track of free blocks for vmap allocation
In reply to Uladzislau Rezki (Sony)An earlier version of this patch was accused of crashing the kernel:
https://lists.01.org/pipermail/lkp/2019-April/010004.html
does the v4 series address this?
Uladzislau Rezki
maintainerRe: [PATCH v4 1/3] mm/vmap: keep track of free blocks for vmap allocation
Hello, Andrew.
Quoted 5 lines
An earlier version of this patch was accused of crashing the kernel:
https://lists.01.org/pipermail/lkp/2019-April/010004.html
does the v4 series address this?
I tried before to narrow down that crash but i did not succeed, so i have never seen that before on my test environment as well as during running lkp-tests including trinity test case:
test-url: http://codemonkey.org.uk/projects/trinity/
But after analysis of the Call-trace and slob_alloc():
<snip>
[ 0.395722] Call Trace: [ 0.395722] slob_alloc+0x1c9/0x240 [ 0.395722] kmem_cache_alloc+0x70/0x80 [ 0.395722] acpi_ps_alloc_op+0xc0/0xca [ 0.395722] acpi_ps_get_next_arg+0x3fa/0x6ed
<snip>
<snip>
/* Attempt to alloc */
prev = sp->lru.prev;
b = slob_page_alloc(sp, size, align);
if (!b)
continue;
/* Improve fragment distribution and reduce our average
* search time by starting our next search here. (see
* Knuth vol 1, sec 2.5, pg 449) */
if (prev != slob_list->prev &&
slob_list->next != prev->next)
list_move_tail(slob_list, prev->next); <- Crash is here in __list_add_valid()
break;
}<snip>
i see that it tries to manipulate with "prev" node that may be removed from the list by slob_page_alloc() earlier if whole page is used. I think that crash has to be fixed by the below commit:
https://www.spinics.net/lists/mm-commits/msg137923.html
it was introduced into 5.1-rc3 kernel.
Why ("mm/vmalloc.c: keep track of free blocks for vmap allocation")
was accused is probably because it uses "kmem cache allocations with struct alignment" instead of kmalloc()/kzalloc(). Maybe because of bigger size requests
it became easier to trigger the BUG. But that is theory.
--
Vlad Rezki
Uladzislau Rezki
maintainerRe: [PATCH v4 1/3] mm/vmap: keep track of free blocks for vmap allocation
+Tobin C. Harding <[email protected]>
Quoted 56 lines · Uladzislau Rezki
Hello, Andrew.
Quoted 5 lines
An earlier version of this patch was accused of crashing the kernel:
https://lists.01.org/pipermail/lkp/2019-April/010004.html
does the v4 series address this?
I tried before to narrow down that crash but i did not succeed, so i have never seen that before on my test environment as well as during running lkp-tests including trinity test case:
test-url: http://codemonkey.org.uk/projects/trinity/
But after analysis of the Call-trace and
slob_alloc():<snip>
[ 0.395722] Call Trace: [ 0.395722] slob_alloc+0x1c9/0x240 [ 0.395722] kmem_cache_alloc+0x70/0x80 [ 0.395722] acpi_ps_alloc_op+0xc0/0xca [ 0.395722] acpi_ps_get_next_arg+0x3fa/0x6ed<snip>
<snip>
/* Attempt to alloc */ prev = sp->lru.prev; b = slob_page_alloc(sp, size, align); if (!b) continue; /* Improve fragment distribution and reduce our average * search time by starting our next search here. (see * Knuth vol 1, sec 2.5, pg 449) */ if (prev != slob_list->prev && slob_list->next != prev->next) list_move_tail(slob_list, prev->next); <- Crash is here in __list_add_valid() break; }<snip>
i see that it tries to manipulate with "prev" node that may be removed from the list by
slob_page_alloc()earlier if whole page is used. I think that crash has to be fixed by the below commit:https://www.spinics.net/lists/mm-commits/msg137923.html
it was introduced into 5.1-rc3 kernel.
Why ("
mm/vmalloc.c: keep track of free blocks for vmap allocation")
was accused is probably because it uses "kmem cache allocations with struct alignment" instead of kmalloc()/kzalloc(). Maybe because of bigger size requests
it became easier to trigger the BUG. But that is theory.--
Vlad Rezki
Signature · 1 line
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerPATCH v4 2/3 mm/vmap: add DEBUG_AUGMENT_PROPAGATE_CHECK macro
In reply to Uladzislau Rezki (Sony)This macro adds some debug code to check that the augment tree is maintained correctly, meaning that every node contains valid subtree_max_size value.
By default this option is set to 0 and not active. It requires recompilation of the kernel to activate it. Set to 1, compile the kernel.
--- mm/vmalloc.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)
mm/vmalloc.c +48 -0
@@ -324,6 +324,8 @@ EXPORT_SYMBOL(vmalloc_to_pfn);/*** Global kva allocator ***/ +#define DEBUG_AUGMENT_PROPAGATE_CHECK 0 +#define VM_LAZY_FREE 0x02#define VM_VM_AREA 0x04
@@ -538,6 +540,48 @@ unlink_va(struct vmap_area *va, struct rb_root *root) } } +#if DEBUG_AUGMENT_PROPAGATE_CHECK +static void +augment_tree_propagate_check (struct rb_node *n) +{ +struct vmap_area *va; +struct rb_node *node; +unsigned long size; +bool found = false; + +if (n ==NULL ) +return ; + + va = rb_entry(n, struct vmap_area, rb_node); + size = va->subtree_max_size; + node = n; + +while (node) { + va = rb_entry(node, struct vmap_area, rb_node); + +if (get_subtree_max_size (node->rb_left) == size) { + node = node->rb_left; + }else { +if (va_size (va) == size) { + found = true; +break ; + } + + node = node->rb_right; + } + } + +if (!found) { + va = rb_entry(n, struct vmap_area, rb_node); +pr_emerg ("tree is corrupted: %lu, %lu\n" , +va_size (va), va->subtree_max_size); + } + +augment_tree_propagate_check (n->rb_left); +augment_tree_propagate_check (n->rb_right); +} +#endif +/* * This function populates subtree_max_size from bottom to upper * levels starting from VA point. The propagation must be done
@@ -587,6 +631,10 @@ augment_tree_propagate_from(struct vmap_area *va) va->subtree_max_size = new_va_sub_max_size; node =rb_parent (&va->rb_node); } + +#if DEBUG_AUGMENT_PROPAGATE_CHECK +augment_tree_propagate_check (free_vmap_area_root.rb_node); +#endif } static void
Signature · 1 line
2.11.0
Uladzislau Rezki (Sony)
maintainerPATCH v4 3/3 mm/vmap: add DEBUG_AUGMENT_LOWEST_MATCH_CHECK macro
In reply to Uladzislau Rezki (Sony)This macro adds some debug code to check that vmap allocations are happened in ascending order.
By default this option is set to 0 and not active. It requires recompilation of the kernel to activate it. Set to 1, compile the kernel.
--- mm/vmalloc.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
mm/vmalloc.c +43 -0
@@ -325,6 +325,7 @@ EXPORT_SYMBOL(vmalloc_to_pfn);/*** Global kva allocator ***/ #define DEBUG_AUGMENT_PROPAGATE_CHECK 0 +#define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0#define VM_LAZY_FREE 0x02#define VM_VM_AREA 0x04
@@ -833,6 +834,44 @@ find_vmap_lowest_match(unsigned long size,return NULL ; } +#if DEBUG_AUGMENT_LOWEST_MATCH_CHECK +#include <linux/random.h> + +static struct vmap_area * +find_vmap_lowest_linear_match (unsigned long size, +unsigned long align,unsigned long vstart) +{ +struct vmap_area *va; + +list_for_each_entry (va, &free_vmap_area_list, list) { +if (!is_within_this_va (va, size, align, vstart)) +continue ; + +return va; + } + +return NULL ; +} + +static void +find_vmap_lowest_match_check (unsigned long size) +{ +struct vmap_area *va_1, *va_2; +unsigned long vstart; +unsigned int rnd; + +get_random_bytes (&rnd,sizeof (rnd)); + vstart =VMALLOC_START + rnd; + + va_1 =find_vmap_lowest_match (size,1 , vstart); + va_2 =find_vmap_lowest_linear_match (size,1 , vstart); + +if (va_1 != va_2) +pr_emerg ("not lowest: t: 0x%p, l: 0x%p, v: 0x%lx\n" , + va_1, va_2, vstart); +} +#endif +enum fit_type {NOTHING_FIT =0 ,FL_FIT_TYPE =1 ,/* full fit */
@@ -975,6 +1014,10 @@ __alloc_vmap_area(unsigned long size, unsigned long align,if (ret)return vend; +#if DEBUG_AUGMENT_LOWEST_MATCH_CHECK +find_vmap_lowest_match_check (size); +#endif +return nva_start_addr; }
Signature · 1 line
2.11.0