Mitigate a vmap lock contention v2
Subsystems (9)
Uladzislau Rezki (Sony)
maintainerv2 cover Mitigate a vmap lock contention v2
Hello, folk!
This is the v2, the series which tends to minimize the vmap lock contention. It is based on the tag: v6.5-rc6. Here you can find a documentation about it:
wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf
even though it is a bit outdated(it follows v1), it still gives a good overview on the problem and how it can be solved. On demand and by request i can update it.
The v1 is here: https://lore.kernel.org/linux-mm/ZIAqojPKjChJTssg@pc636/T/
Delta v1 -> v2:
- open coded locking;
- switch to array of nodes instead of per-cpu definition;
- density is 2 cores per one node(not equal to number of CPUs);
- VAs first go back(free path) to an owner node and later to
a global heap if a block is fully freed, nid is saved in va->flags;
- add helpers to drain lazily-freed areas faster, if high pressure;
- picked al Reviewed-by.
Test on AMD Ryzen Threadripper 3970X 32-Core Processor:
sudo ./test_vmalloc.sh run_test_mask=127 nr_threads=64
<v6.5-rc6 perf>
94.17% 0.90% [kernel] [k] _raw_spin_lock 93.27% 93.05% [kernel] [k] native_queued_spin_lock_slowpath 74.69% 0.25% [kernel] [k] __vmalloc_node_range 72.64% 0.01% [kernel] [k] __get_vm_area_node 72.04% 0.89% [kernel] [k] alloc_vmap_area 42.17% 0.00% [kernel] [k] vmalloc 32.53% 0.00% [kernel] [k] __vmalloc_node 24.91% 0.25% [kernel] [k] vfree 24.32% 0.01% [kernel] [k] remove_vm_area 22.63% 0.21% [kernel] [k] find_unlink_vmap_area 15.51% 0.00% [unknown] [k] 0xffffffffc09a74ac 14.35% 0.00% [kernel] [k] ret_from_fork_asm 14.35% 0.00% [kernel] [k] ret_from_fork 14.35% 0.00% [kernel] [k] kthread <v6.5-rc6 perf> vs <v6.5-rc6+v2 perf> 74.32% 2.42% [kernel] [k] __vmalloc_node_range 69.58% 0.01% [kernel] [k] vmalloc 54.21% 1.17% [kernel] [k] __alloc_pages_bulk 48.13% 47.91% [kernel] [k] clear_page_orig 43.60% 0.01% [unknown] [k] 0xffffffffc082f16f 32.06% 0.00% [kernel] [k] ret_from_fork_asm 32.06% 0.00% [kernel] [k] ret_from_fork 32.06% 0.00% [kernel] [k] kthread 31.30% 0.00% [unknown] [k] 0xffffffffc082f889 22.98% 4.16% [kernel] [k] vfree 14.36% 0.28% [kernel] [k] __get_vm_area_node 13.43% 3.35% [kernel] [k] alloc_vmap_area 10.86% 0.04% [kernel] [k] remove_vm_area 8.89% 2.75% [kernel] [k] _raw_spin_lock 7.19% 0.00% [unknown] [k] 0xffffffffc082fba3 6.65% 1.37% [kernel] [k] free_unref_page 6.13% 6.11% [kernel] [k] native_queued_spin_lock_slowpath
<v6.5-rc6+v2 perf>
On smaller systems, for example, 8xCPU Hikey960 board the contention is not that high and is approximately ~16 percent.
Uladzislau Rezki (Sony) (9):
mm: vmalloc: Add va_alloc() helper
mm: vmalloc: Rename adjust_va_to_fit_type() function
mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c
mm: vmalloc: Remove global vmap_area_root rb-tree
mm: vmalloc: Remove global purge_vmap_area_root rb-tree
mm: vmalloc: Offload free_vmap_area_lock lock
mm: vmalloc: Support multiple nodes in vread_iter
mm: vmalloc: Support multiple nodes in vmallocinfo
mm: vmalloc: Set nr_nodes/node_size based on CPU-cores
mm/vmalloc.c | 929 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 683 insertions(+), 246 deletions(-)
Signature · 1 line
2.30.2
Uladzislau Rezki (Sony)
maintainerPATCH v2 1/9 mm: vmalloc: Add va_alloc() helper
Currently __alloc_vmap_area() function contains an open codded logic that finds and adjusts a VA based on allocation request.
Introduce a va_alloc() helper that adjusts found VA only. It will be used later at least in two places.
There is no a functional change as a result of this patch.
--- mm/vmalloc.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)
mm/vmalloc.c +28 -13
@@ -1481,6 +1481,32 @@ adjust_va_to_fit_type(struct rb_root *root, struct list_head *head,return 0 ; } +static unsigned long +va_alloc (struct vmap_area *va, +struct rb_root *root,struct list_head *head, +unsigned long size,unsigned long align, +unsigned long vstart,unsigned long vend) +{ +unsigned long nva_start_addr; +int ret; + +if (va->va_start > vstart) + nva_start_addr =ALIGN (va->va_start, align); +else + nva_start_addr =ALIGN (vstart, align); + +/* Check the "vend" restriction. */ +if (nva_start_addr + size > vend) +return vend; + +/* Update the free vmap_area. */ + ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); +if (WARN_ON_ONCE (ret)) +return vend; + +return nva_start_addr; +} +/* * Returns a start address of the newly allocated area, if success. * Otherwise a vend is returned that indicates failure.
@@ -1493,7 +1519,6 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,bool adjust_search_size = true;unsigned long nva_start_addr;struct vmap_area *va; -int ret;/* * Do not adjust when:
@@ -1511,18 +1536,8 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,if (unlikely (!va))return vend; -if (va->va_start > vstart) - nva_start_addr =ALIGN (va->va_start, align); -else - nva_start_addr =ALIGN (vstart, align); - -/* Check the "vend" restriction. */ -if (nva_start_addr + size > vend) -return vend; - -/* Update the free vmap_area. */ - ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); -if (WARN_ON_ONCE (ret)) + nva_start_addr =va_alloc (va, root, head, size, align, vstart, vend); +if (nva_start_addr == vend)return vend;#if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
Signature · 1 line
2.30.2
Re: [PATCH v2 1/9] mm: vmalloc: Add va_alloc() helper
Quoted 83 lines · Uladzislau Rezki (Sony)
Currently
__alloc_vmap_area()function contains an open codded logic that finds and adjusts a VA based on allocation request.Introduce a
va_alloc()helper that adjusts found VA only. It will be used later at least in two places.There is no a functional change as a result of this patch.
Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)mm/vmalloc.c +28
-13@@ -1481,6 +1481,32 @@ adjust_va_to_fit_type(struct rb_root *root, struct list_head *head,return 0 ; } +static unsigned long +va_alloc (struct vmap_area *va, +struct rb_root *root,struct list_head *head, +unsigned long size,unsigned long align, +unsigned long vstart,unsigned long vend) +{ +unsigned long nva_start_addr; +int ret; + +if (va->va_start > vstart) + nva_start_addr =ALIGN (va->va_start, align); +else + nva_start_addr =ALIGN (vstart, align); + +/* Check the "vend" restriction. */ +if (nva_start_addr + size > vend) +return vend; + +/* Update the free vmap_area. */ + ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); +if (WARN_ON_ONCE (ret)) +return vend; + +return nva_start_addr; +} +/* * Returns a start address of the newly allocated area, if success. * Otherwise a vend is returned that indicates failure. @@ -1493,7 +1519,6 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,bool adjust_search_size = true;unsigned long nva_start_addr;struct vmap_area *va; -int ret;/* * Do not adjust when: @@ -1511,18 +1536,8 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,if (unlikely (!va))return vend; -if (va->va_start > vstart) - nva_start_addr =ALIGN (va->va_start, align); -else - nva_start_addr =ALIGN (vstart, align); - -/* Check the "vend" restriction. */ -if (nva_start_addr + size > vend) -return vend; - -/* Update the free vmap_area. */ - ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); -if (WARN_ON_ONCE (ret)) + nva_start_addr =va_alloc (va, root, head, size, align, vstart, vend); +if (nva_start_addr == vend)return vend;#if DEBUG_AUGMENT_LOWEST_MATCH_CHECK Signature · 1 line
2.30.2
Uladzislau Rezki
maintainerRe: [PATCH v2 1/9] mm: vmalloc: Add va_alloc() helper
Quoted 81 lines · Baoquan He
Quoted 80 lines · Uladzislau Rezki (Sony)
Currently
__alloc_vmap_area()function contains an open codded logic that finds and adjusts a VA based on allocation request.Introduce a
va_alloc()helper that adjusts found VA only. It will be used later at least in two places.There is no a functional change as a result of this patch.
Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-)mm/vmalloc.c +28
-13@@ -1481,6 +1481,32 @@ adjust_va_to_fit_type(struct rb_root *root, struct list_head *head,return 0 ; } +static unsigned long +va_alloc (struct vmap_area *va, +struct rb_root *root,struct list_head *head, +unsigned long size,unsigned long align, +unsigned long vstart,unsigned long vend) +{ +unsigned long nva_start_addr; +int ret; + +if (va->va_start > vstart) + nva_start_addr =ALIGN (va->va_start, align); +else + nva_start_addr =ALIGN (vstart, align); + +/* Check the "vend" restriction. */ +if (nva_start_addr + size > vend) +return vend; + +/* Update the free vmap_area. */ + ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); +if (WARN_ON_ONCE (ret)) +return vend; + +return nva_start_addr; +} +/* * Returns a start address of the newly allocated area, if success. * Otherwise a vend is returned that indicates failure. @@ -1493,7 +1519,6 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,bool adjust_search_size = true;unsigned long nva_start_addr;struct vmap_area *va; -int ret;/* * Do not adjust when: @@ -1511,18 +1536,8 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,if (unlikely (!va))return vend; -if (va->va_start > vstart) - nva_start_addr =ALIGN (va->va_start, align); -else - nva_start_addr =ALIGN (vstart, align); - -/* Check the "vend" restriction. */ -if (nva_start_addr + size > vend) -return vend; - -/* Update the free vmap_area. */ - ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); -if (WARN_ON_ONCE (ret)) + nva_start_addr =va_alloc (va, root, head, size, align, vstart, vend); +if (nva_start_addr == vend)return vend;
Quoted 6 lines
Quoted 3 lines
mm/vmalloc.c +0
-0#if DEBUG_AUGMENT_LOWEST_MATCH_CHECK Signature · 1 line
2.30.2Reviewed-by Baoquan He <[email protected]>
Thanks, i picked it for V3.
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerPATCH v2 2/9 mm: vmalloc: Rename adjust_va_to_fit_type() function
In reply to Uladzislau Rezki (Sony)This patch renames the adjust_va_to_fit_type() function to va_clip() which is shorter and more expressive.
There is no a functional change as a result of this patch.
--- mm/vmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
mm/vmalloc.c +6 -7
@@ -1382,9 +1382,9 @@ classify_va_fit_type(struct vmap_area *va, } static __always_inline int -adjust_va_to_fit_type (struct rb_root *root,struct list_head *head, -struct vmap_area *va,unsigned long nva_start_addr, -unsigned long size) +va_clip (struct rb_root *root,struct list_head *head, +struct vmap_area *va,unsigned long nva_start_addr, +unsigned long size) {struct vmap_area *lva =NULL ;enum fit_type type =classify_va_fit_type (va, nva_start_addr, size);
@@ -1500,7 +1500,7 @@ va_alloc(struct vmap_area *va,return vend;/* Update the free vmap_area. */ - ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); + ret =va_clip (root, head, va, nva_start_addr, size);if (WARN_ON_ONCE (ret))return vend;
@@ -4151,9 +4151,8 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,/* It is a BUG(), but trigger recovery instead. */ goto recovery; - ret =adjust_va_to_fit_type (&free_vmap_area_root, - &free_vmap_area_list, - va, start, size); + ret =va_clip (&free_vmap_area_root, + &free_vmap_area_list, va, start, size);if (WARN_ON_ONCE (unlikely (ret)))/* It is a BUG(), but trigger recovery instead. */ goto recovery;
Signature · 1 line
2.30.2
Re: [PATCH v2 2/9] mm: vmalloc: Rename adjust_va_to_fit_type() function
Quoted 53 lines · Uladzislau Rezki (Sony)
This patch renames the
adjust_va_to_fit_type()function tova_clip()which is shorter and more expressive.There is no a functional change as a result of this patch.
Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)mm/vmalloc.c +6
-7@@ -1382,9 +1382,9 @@ classify_va_fit_type(struct vmap_area *va, } static __always_inline int -adjust_va_to_fit_type (struct rb_root *root,struct list_head *head, -struct vmap_area *va,unsigned long nva_start_addr, -unsigned long size) +va_clip (struct rb_root *root,struct list_head *head, +struct vmap_area *va,unsigned long nva_start_addr, +unsigned long size) {struct vmap_area *lva =NULL ;enum fit_type type =classify_va_fit_type (va, nva_start_addr, size);@@ -1500,7 +1500,7 @@ va_alloc(struct vmap_area *va,return vend;/* Update the free vmap_area. */ - ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); + ret =va_clip (root, head, va, nva_start_addr, size);if (WARN_ON_ONCE (ret))return vend;@@ -4151,9 +4151,8 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,/* It is a BUG(), but trigger recovery instead. */ goto recovery; - ret =adjust_va_to_fit_type (&free_vmap_area_root, - &free_vmap_area_list, - va, start, size); + ret =va_clip (&free_vmap_area_root, + &free_vmap_area_list, va, start, size);if (WARN_ON_ONCE (unlikely (ret)))/* It is a BUG(), but trigger recovery instead. */ goto recovery;Signature · 1 line
2.30.2
Uladzislau Rezki
maintainerRe: [PATCH v2 2/9] mm: vmalloc: Rename adjust_va_to_fit_type() function
Quoted 43 lines · Baoquan He
Quoted 42 lines · Uladzislau Rezki (Sony)
This patch renames the
adjust_va_to_fit_type()function tova_clip()which is shorter and more expressive.There is no a functional change as a result of this patch.
Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)mm/vmalloc.c +4
-4@@ -1382,9 +1382,9 @@ classify_va_fit_type(struct vmap_area *va, } static __always_inline int -adjust_va_to_fit_type (struct rb_root *root,struct list_head *head, -struct vmap_area *va,unsigned long nva_start_addr, -unsigned long size) +va_clip (struct rb_root *root,struct list_head *head, +struct vmap_area *va,unsigned long nva_start_addr, +unsigned long size) {struct vmap_area *lva =NULL ;enum fit_type type =classify_va_fit_type (va, nva_start_addr, size);@@ -1500,7 +1500,7 @@ va_alloc(struct vmap_area *va,return vend;/* Update the free vmap_area. */ - ret =adjust_va_to_fit_type (root, head, va, nva_start_addr, size); + ret =va_clip (root, head, va, nva_start_addr, size);if (WARN_ON_ONCE (ret))return vend;@@ -4151,9 +4151,8 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,/* It is a BUG(), but trigger recovery instead. */ goto recovery;
Quoted 14 lines
Quoted 11 lines
mm/vmalloc.c +2
-3- ret =adjust_va_to_fit_type (&free_vmap_area_root, - &free_vmap_area_list, - va, start, size); + ret =va_clip (&free_vmap_area_root, + &free_vmap_area_list, va, start, size);if (WARN_ON_ONCE (unlikely (ret)))/* It is a BUG(), but trigger recovery instead. */ goto recovery;Signature · 1 line
2.30.2Reviewed-by Baoquan He <[email protected]>
Thank you for the review. Picked it up.
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerPATCH v2 3/9 mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c
In reply to Uladzislau Rezki (Sony)A vmap_init_free_space() is a function that setups a vmap space and is considered as part of initialization phase. Since a main entry which is vmalloc_init(), has been moved down in vmalloc.c it makes sense to follow the pattern.
There is no a functional change as a result of this patch.
--- mm/vmalloc.c | 82 ++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 41 deletions(-)
mm/vmalloc.c +41 -41
@@ -2512,47 +2512,6 @@ void __init vm_area_register_early(struct vm_struct *vm, size_t align)kasan_populate_early_vm_area_shadow (vm->addr, vm->size); } -static void vmap_init_free_space (void ) -{ -unsigned long vmap_start =1 ; -const unsigned long vmap_end =ULONG_MAX ; -struct vmap_area *busy, *free; - -/* -* B F B B B F -* -|-----|.....|-----|-----|-----|.....|- -* | The KVA space | -* |<--------------------------------->| -*/ -list_for_each_entry (busy, &vmap_area_list, list) { -if (busy->va_start - vmap_start >0 ) { - free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); -if (!WARN_ON_ONCE (!free)) { - free->va_start = vmap_start; - free->va_end = busy->va_start; - -insert_vmap_area_augment (free,NULL , - &free_vmap_area_root, - &free_vmap_area_list); - } - } - - vmap_start = busy->va_end; - } - -if (vmap_end - vmap_start >0 ) { - free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); -if (!WARN_ON_ONCE (!free)) { - free->va_start = vmap_start; - free->va_end = vmap_end; - -insert_vmap_area_augment (free,NULL , - &free_vmap_area_root, - &free_vmap_area_list); - } - } -} -static inline void setup_vmalloc_vm_locked (struct vm_struct *vm,struct vmap_area *va,unsigned long flags,const void *caller) {
@@ -4443,6 +4402,47 @@ module_init(proc_vmalloc_init);#endif +static void vmap_init_free_space (void ) +{ +unsigned long vmap_start =1 ; +const unsigned long vmap_end =ULONG_MAX ; +struct vmap_area *busy, *free; + +/* +* B F B B B F +* -|-----|.....|-----|-----|-----|.....|- +* | The KVA space | +* |<--------------------------------->| +*/ +list_for_each_entry (busy, &vmap_area_list, list) { +if (busy->va_start - vmap_start >0 ) { + free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); +if (!WARN_ON_ONCE (!free)) { + free->va_start = vmap_start; + free->va_end = busy->va_start; + +insert_vmap_area_augment (free,NULL , + &free_vmap_area_root, + &free_vmap_area_list); + } + } + + vmap_start = busy->va_end; + } + +if (vmap_end - vmap_start >0 ) { + free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); +if (!WARN_ON_ONCE (!free)) { + free->va_start = vmap_start; + free->va_end = vmap_end; + +insert_vmap_area_augment (free,NULL , + &free_vmap_area_root, + &free_vmap_area_list); + } + } +} +void __init vmalloc_init (void ) {struct vmap_area *va;
Signature · 1 line
2.30.2
Re: [PATCH v2 3/9] mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c
Quoted 117 lines · Uladzislau Rezki (Sony)
A
vmap_init_free_space()is a function that setups a vmap space and is considered as part of initialization phase. Since a main entry which isvmalloc_init(), has been moved down invmalloc.cit makes sense to follow the pattern.There is no a functional change as a result of this patch.
Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 82 ++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 41 deletions(-)mm/vmalloc.c +41
-41@@ -2512,47 +2512,6 @@ void __init vm_area_register_early(struct vm_struct *vm, size_t align)kasan_populate_early_vm_area_shadow (vm->addr, vm->size); } -static void vmap_init_free_space (void ) -{ -unsigned long vmap_start =1 ; -const unsigned long vmap_end =ULONG_MAX ; -struct vmap_area *busy, *free; - -/* -* B F B B B F -* -|-----|.....|-----|-----|-----|.....|- -* | The KVA space | -* |<--------------------------------->| -*/ -list_for_each_entry (busy, &vmap_area_list, list) { -if (busy->va_start - vmap_start >0 ) { - free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); -if (!WARN_ON_ONCE (!free)) { - free->va_start = vmap_start; - free->va_end = busy->va_start; - -insert_vmap_area_augment (free,NULL , - &free_vmap_area_root, - &free_vmap_area_list); - } - } - - vmap_start = busy->va_end; - } - -if (vmap_end - vmap_start >0 ) { - free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); -if (!WARN_ON_ONCE (!free)) { - free->va_start = vmap_start; - free->va_end = vmap_end; - -insert_vmap_area_augment (free,NULL , - &free_vmap_area_root, - &free_vmap_area_list); - } - } -} -static inline void setup_vmalloc_vm_locked (struct vm_struct *vm,struct vmap_area *va,unsigned long flags,const void *caller) {@@ -4443,6 +4402,47 @@ module_init(proc_vmalloc_init);#endif +static void vmap_init_free_space (void ) +{ +unsigned long vmap_start =1 ; +const unsigned long vmap_end =ULONG_MAX ; +struct vmap_area *busy, *free; + +/* +* B F B B B F +* -|-----|.....|-----|-----|-----|.....|- +* | The KVA space | +* |<--------------------------------->| +*/ +list_for_each_entry (busy, &vmap_area_list, list) { +if (busy->va_start - vmap_start >0 ) { + free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); +if (!WARN_ON_ONCE (!free)) { + free->va_start = vmap_start; + free->va_end = busy->va_start; + +insert_vmap_area_augment (free,NULL , + &free_vmap_area_root, + &free_vmap_area_list); + } + } + + vmap_start = busy->va_end; + } + +if (vmap_end - vmap_start >0 ) { + free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); +if (!WARN_ON_ONCE (!free)) { + free->va_start = vmap_start; + free->va_end = vmap_end; + +insert_vmap_area_augment (free,NULL , + &free_vmap_area_root, + &free_vmap_area_list); + } + } +} +void __init vmalloc_init (void ) {struct vmap_area *va;Signature · 1 line
2.30.2
Uladzislau Rezki
maintainerRe: [PATCH v2 3/9] mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c
Quoted 112 lines · Baoquan He
Quoted 111 lines · Uladzislau Rezki (Sony)
A
vmap_init_free_space()is a function that setups a vmap space and is considered as part of initialization phase. Since a main entry which isvmalloc_init(), has been moved down invmalloc.cit makes sense to follow the pattern.There is no a functional change as a result of this patch.
Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Lorenzo Stoakes <[email protected]>Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 82 ++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 41 deletions(-)mm/vmalloc.c +41
-41@@ -2512,47 +2512,6 @@ void __init vm_area_register_early(struct vm_struct *vm, size_t align)kasan_populate_early_vm_area_shadow (vm->addr, vm->size); } -static void vmap_init_free_space (void ) -{ -unsigned long vmap_start =1 ; -const unsigned long vmap_end =ULONG_MAX ; -struct vmap_area *busy, *free; - -/* -* B F B B B F -* -|-----|.....|-----|-----|-----|.....|- -* | The KVA space | -* |<--------------------------------->| -*/ -list_for_each_entry (busy, &vmap_area_list, list) { -if (busy->va_start - vmap_start >0 ) { - free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); -if (!WARN_ON_ONCE (!free)) { - free->va_start = vmap_start; - free->va_end = busy->va_start; - -insert_vmap_area_augment (free,NULL , - &free_vmap_area_root, - &free_vmap_area_list); - } - } - - vmap_start = busy->va_end; - } - -if (vmap_end - vmap_start >0 ) { - free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); -if (!WARN_ON_ONCE (!free)) { - free->va_start = vmap_start; - free->va_end = vmap_end; - -insert_vmap_area_augment (free,NULL , - &free_vmap_area_root, - &free_vmap_area_list); - } - } -} -static inline void setup_vmalloc_vm_locked (struct vm_struct *vm,struct vmap_area *va,unsigned long flags,const void *caller) {@@ -4443,6 +4402,47 @@ module_init(proc_vmalloc_init);#endif +static void vmap_init_free_space (void ) +{ +unsigned long vmap_start =1 ; +const unsigned long vmap_end =ULONG_MAX ; +struct vmap_area *busy, *free; + +/* +* B F B B B F +* -|-----|.....|-----|-----|-----|.....|- +* | The KVA space | +* |<--------------------------------->| +*/ +list_for_each_entry (busy, &vmap_area_list, list) { +if (busy->va_start - vmap_start >0 ) { + free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); +if (!WARN_ON_ONCE (!free)) { + free->va_start = vmap_start; + free->va_end = busy->va_start; + +insert_vmap_area_augment (free,NULL , + &free_vmap_area_root, + &free_vmap_area_list); + } + } + + vmap_start = busy->va_end; + } + +if (vmap_end - vmap_start >0 ) { + free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT ); +if (!WARN_ON_ONCE (!free)) { + free->va_start = vmap_start; + free->va_end = vmap_end; + +insert_vmap_area_augment (free,NULL , + &free_vmap_area_root, + &free_vmap_area_list); + } + } +} +
Quoted 9 lines
Quoted 6 lines
mm/vmalloc.c +0
-0void __init vmalloc_init (void ) {struct vmap_area *va;Signature · 1 line
2.30.2Reviewed-by Baoquan He <[email protected]>
Thanks!
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerPATCH v2 4/9 mm: vmalloc: Remove global vmap_area_root rb-tree
In reply to Uladzislau Rezki (Sony)Store allocated objects in a separate nodes. A va->va_start address is converted into a correct node where it should be placed and resided. An addr_to_node() function is used to do a proper address conversion to determine a node that contains a VA.
Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global vmap_area_lock, vmap_area_root are removed as there is no need in it anymore. The vmap_area_list is still kept and is _empty_. It is exported for a kexec only;
3. The vmallocinfo and vread() have to be reworked to be able to handle multiple nodes.
--- mm/vmalloc.c | 209 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 161 insertions(+), 48 deletions(-)
mm/vmalloc.c +161 -48
@@ -728,11 +728,9 @@ EXPORT_SYMBOL(vmalloc_to_pfn);#define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -static DEFINE_SPINLOCK (vmap_area_lock);static DEFINE_SPINLOCK (free_vmap_area_lock);/* Export for kexec only */ LIST_HEAD (vmap_area_list); -static struct rb_root vmap_area_root =RB_ROOT ;static bool vmap_initialized__read_mostly ;static struct rb_root purge_vmap_area_root =RB_ROOT ;
@@ -772,6 +770,38 @@ static struct rb_root free_vmap_area_root = RB_ROOT;*/ static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); +/* +* An effective vmap-node logic. Users make use of nodes instead +* of a global heap. It allows to balance an access and mitigate +* contention. +*/ +struct rb_list { +struct rb_root root; +struct list_head head; +spinlock_t lock; +}; + +struct vmap_node { +/* Bookkeeping data of this node. */ +struct rb_list busy; +}; + +static struct vmap_node *nodes, snode; +static __read_mostly unsigned int nr_nodes =1 ; +static __read_mostly unsigned int node_size =1 ; + +static inline unsigned int +addr_to_node_id (unsigned long addr) +{ +return (addr / node_size) % nr_nodes; +} + +static inline struct vmap_node * +addr_to_node (unsigned long addr) +{ +return &nodes[addr_to_node_id (addr)]; +} +static __always_inline unsigned long va_size (struct vmap_area *va) {
@@ -803,10 +833,11 @@ unsigned long vmalloc_nr_pages(void) }/* Look up the first VA which satisfies addr < va_end, NULL if none. */ -static struct vmap_area *find_vmap_area_exceed_addr (unsigned long addr) +static struct vmap_area * +find_vmap_area_exceed_addr (unsigned long addr,struct rb_root *root) {struct vmap_area *va =NULL ; -struct rb_node *n = vmap_area_root.rb_node; +struct rb_node *n = root->rb_node; addr = (unsigned long )kasan_reset_tag ((void *)addr);
@@ -1552,12 +1583,14 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,*/ static void free_vmap_area (struct vmap_area *va) { +struct vmap_node *vn =addr_to_node (va->va_start); +/* * Remove from the busy tree/list. */ -spin_lock (&vmap_area_lock); -unlink_va (va, &vmap_area_root); -spin_unlock (&vmap_area_lock); +spin_lock (&vn->busy.lock); +unlink_va (va, &vn->busy.root); +spin_unlock (&vn->busy.lock);/* * Insert/Merge it back to the free tree/list.
@@ -1600,6 +1633,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, int node, gfp_t gfp_mask, unsigned long va_flags) { + struct vmap_node *vn;struct vmap_area *va;unsigned long freed;unsigned long addr;
@@ -1645,9 +1679,11 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->vm =NULL ; va->flags = va_flags; -spin_lock (&vmap_area_lock); -insert_vmap_area (va, &vmap_area_root, &vmap_area_list); -spin_unlock (&vmap_area_lock); + vn =addr_to_node (va->va_start); + +spin_lock (&vn->busy.lock); +insert_vmap_area (va, &vn->busy.root, &vn->busy.head); +spin_unlock (&vn->busy.lock);BUG_ON (!IS_ALIGNED (va->va_start, align));BUG_ON (va->va_start < vstart);
@@ -1871,26 +1907,61 @@ static void free_unmap_vmap_area(struct vmap_area *va)struct vmap_area *find_vmap_area (unsigned long addr) { +struct vmap_node *vn;struct vmap_area *va; +int i, j; + +/* +* An addr_to_node_id(addr) converts an address to a node index +* where a VA is located. If VA spans several zones and passed +* addr is not the same as va->va_start, what is not common, we +* may need to scan an extra nodes. See an example: +* +* <--va--> +* -|-----|-----|-----|-----|- +* 1 2 0 1 +* +* VA resides in node 1 whereas it spans 1 and 2. If passed +* addr is within a second node we should do extra work. We +* should mention that it is rare and is a corner case from +* the other hand it has to be covered. +*/ + i = j =addr_to_node_id (addr); +do { + vn = &nodes[i]; -spin_lock (&vmap_area_lock); - va =__find_vmap_area (addr, &vmap_area_root); -spin_unlock (&vmap_area_lock); +spin_lock (&vn->busy.lock); + va =__find_vmap_area (addr, &vn->busy.root); +spin_unlock (&vn->busy.lock); -return va; +if (va) +return va; + }while ((i = (i +1 ) % nr_nodes) != j); + +return NULL ; }static struct vmap_area *find_unlink_vmap_area (unsigned long addr) { +struct vmap_node *vn;struct vmap_area *va; +int i, j; -spin_lock (&vmap_area_lock); - va =__find_vmap_area (addr, &vmap_area_root); -if (va) -unlink_va (va, &vmap_area_root); -spin_unlock (&vmap_area_lock); + i = j =addr_to_node_id (addr); +do { + vn = &nodes[i]; -return va; +spin_lock (&vn->busy.lock); + va =__find_vmap_area (addr, &vn->busy.root); +if (va) +unlink_va (va, &vn->busy.root); +spin_unlock (&vn->busy.lock); + +if (va) +return va; + }while ((i = (i +1 ) % nr_nodes) != j); + +return NULL ; }/*** Per cpu kva allocator ***/
@@ -2092,6 +2163,7 @@ static void *new_vmap_block(unsigned int order, gfp_t gfp_mask)static void free_vmap_block (struct vmap_block *vb) { +struct vmap_node *vn;struct vmap_block *tmp;struct xarray *xa;
@@ -2099,9 +2171,10 @@ static void free_vmap_block(struct vmap_block *vb) tmp =xa_erase (xa,addr_to_vb_idx (vb->va->va_start));BUG_ON (tmp != vb); -spin_lock (&vmap_area_lock); -unlink_va (vb->va, &vmap_area_root); -spin_unlock (&vmap_area_lock); + vn =addr_to_node (vb->va->va_start); +spin_lock (&vn->busy.lock); +unlink_va (vb->va, &vn->busy.root); +spin_unlock (&vn->busy.lock);free_vmap_area_noflush (vb->va);kfree_rcu (vb, rcu_head);
@@ -2525,9 +2598,11 @@ static inline void setup_vmalloc_vm_locked(struct vm_struct *vm,static void setup_vmalloc_vm (struct vm_struct *vm,struct vmap_area *va,unsigned long flags,const void *caller) { -spin_lock (&vmap_area_lock); +struct vmap_node *vn =addr_to_node (va->va_start); + +spin_lock (&vn->busy.lock);setup_vmalloc_vm_locked (vm, va, flags, caller); -spin_unlock (&vmap_area_lock); +spin_unlock (&vn->busy.lock); }static void clear_vm_uninitialized_flag (struct vm_struct *vm)
@@ -3711,6 +3786,7 @@ static size_t vmap_ram_vread_iter(struct iov_iter *iter, const char *addr,*/ long vread_iter (struct iov_iter *iter,const char *addr,size_t count) { +struct vmap_node *vn;struct vmap_area *va;struct vm_struct *vm;char *vaddr;
@@ -3724,8 +3800,11 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count; -spin_lock (&vmap_area_lock); - va =find_vmap_area_exceed_addr ((unsigned long )addr); +/* Hooked to node_0 so far. */ + vn =addr_to_node (0 ); +spin_lock (&vn->busy.lock); + + va =find_vmap_area_exceed_addr ((unsigned long )addr, &vn->busy.root);if (!va) goto finished_zero;
@@ -3733,7 +3812,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count)if ((unsigned long )addr + remains <= va->va_start) goto finished_zero; -list_for_each_entry_from (va, &vmap_area_list, list) { +list_for_each_entry_from (va, &vn->busy.head, list) {size_t copied;if (remains ==0 )
@@ -3792,12 +3871,12 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) } finished_zero: -spin_unlock (&vmap_area_lock); +spin_unlock (&vn->busy.lock);/* zero-fill memory holes */ return count - remains +zero_iter (iter, remains); finished:/* Nothing remains, or We couldn't copy/zero everything. */ -spin_unlock (&vmap_area_lock); +spin_unlock (&vn->busy.lock);return count - remains; }
@@ -4131,14 +4210,15 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, }/* insert all vm's */ -spin_lock (&vmap_area_lock);for (area =0 ; area < nr_vms; area++) { -insert_vmap_area (vas[area], &vmap_area_root, &vmap_area_list); +struct vmap_node *vn =addr_to_node (vas[area]->va_start); +spin_lock (&vn->busy.lock); +insert_vmap_area (vas[area], &vn->busy.root, &vn->busy.head);setup_vmalloc_vm_locked (vms[area], vas[area],VM_ALLOC , pcpu_get_vm_areas); +spin_unlock (&vn->busy.lock); } -spin_unlock (&vmap_area_lock);/* * Mark allocated areas as accessible. Do it now as a best-effort
@@ -4261,25 +4341,26 @@ bool vmalloc_dump_obj(void *object)#ifdef CONFIG_PROC_FS static void *s_start (struct seq_file *m,loff_t *pos) - __acquires(&vmap_purge_lock) - __acquires(&vmap_area_lock) { +struct vmap_node *vn =addr_to_node (0 ); +mutex_lock (&vmap_purge_lock); -spin_lock (&vmap_area_lock); +spin_lock (&vn->busy.lock); -return seq_list_start (&vmap_area_list, *pos); +return seq_list_start (&vn->busy.head, *pos); }static void *s_next (struct seq_file *m,void *p,loff_t *pos) { -return seq_list_next (p, &vmap_area_list, pos); +struct vmap_node *vn =addr_to_node (0 ); +return seq_list_next (p, &vn->busy.head, pos); }static void s_stop (struct seq_file *m,void *p) - __releases(&vmap_area_lock) - __releases(&vmap_purge_lock) { -spin_unlock (&vmap_area_lock); +struct vmap_node *vn =addr_to_node (0 ); + +spin_unlock (&vn->busy.lock);mutex_unlock (&vmap_purge_lock); }
@@ -4322,9 +4403,11 @@ static void show_purge_info(struct seq_file *m)static int s_show (struct seq_file *m,void *p) { +struct vmap_node *vn;struct vmap_area *va;struct vm_struct *v; + vn =addr_to_node (0 ); va = list_entry(p, struct vmap_area, list);if (!va->vm) {
@@ -4375,7 +4458,7 @@ static int s_show(struct seq_file *m, void *p)* As a final step, dump "unpurged" areas. */ final: -if (list_is_last (&va->list, &vmap_area_list)) +if (list_is_last (&va->list, &vn->busy.head))show_purge_info (m);return 0 ;
@@ -4406,7 +4489,8 @@ static void vmap_init_free_space(void) {unsigned long vmap_start =1 ;const unsigned long vmap_end =ULONG_MAX ; -struct vmap_area *busy, *free; +struct vmap_area *free; +struct vm_struct *busy;/* * B F B B B F
@@ -4414,12 +4498,12 @@ static void vmap_init_free_space(void)* | The KVA space | * |<--------------------------------->| */ -list_for_each_entry (busy, &vmap_area_list, list) { -if (busy->va_start - vmap_start >0 ) { +for (busy = vmlist; busy; busy = busy->next) { +if (busy->addr - vmap_start >0 ) { free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT );if (!WARN_ON_ONCE (!free)) { free->va_start = vmap_start; - free->va_end = busy->va_start; + free->va_end = (unsigned long ) busy->addr; insert_vmap_area_augment(free, NULL, &free_vmap_area_root,
@@ -4427,7 +4511,7 @@ static void vmap_init_free_space(void) } } - vmap_start = busy->va_end; + vmap_start = (unsigned long ) busy->addr + busy->size; }if (vmap_end - vmap_start >0 ) {
@@ -4443,9 +4527,31 @@ static void vmap_init_free_space(void) } } +static void vmap_init_nodes (void ) +{ +struct vmap_node *vn; +int i; + + nodes = &snode; + +if (nr_nodes >1 ) { + vn =kmalloc_array (nr_nodes,sizeof (*vn),GFP_NOWAIT ); +if (vn) + nodes = vn; + } + +for (i =0 ; i < nr_nodes; i++) { + vn = &nodes[i]; + vn->busy.root =RB_ROOT ; +INIT_LIST_HEAD (&vn->busy.head); +spin_lock_init (&vn->busy.lock); + } +} +void __init vmalloc_init (void ) {struct vmap_area *va; +struct vmap_node *vn;struct vm_struct *tmp;int i;
@@ -4467,6 +4573,11 @@ void __init vmalloc_init(void)xa_init (&vbq->vmap_blocks); } +/* +* Setup nodes before importing vmlist. +*/ +vmap_init_nodes (); +/* Import existing vmlist entries. */ for (tmp = vmlist; tmp; tmp = tmp->next) { va =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT );
@@ -4476,7 +4587,9 @@ void __init vmalloc_init(void) va->va_start = (unsigned long )tmp->addr; va->va_end = va->va_start + tmp->size; va->vm = tmp; -insert_vmap_area (va, &vmap_area_root, &vmap_area_list); + + vn =addr_to_node (va->va_start); +insert_vmap_area (va, &vn->busy.root, &vn->busy.head); }/*
Signature · 1 line
2.30.2
kernel test robot
build botRe: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
kernel test robot noticed the following build warnings:
Hi Uladzislau,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on linus/master v6.5]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Uladzislau-Rezki-Sony/mm-vmalloc-Add-va_alloc-helper/20230829-161248 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230829081142.3619-5-urezki%40gmail.com patch subject: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree config: csky-randconfig-r024-20230829 (https://download.01.org/0day-ci/archive/20230829/[email protected]/config) compiler: csky-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20230829/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
mm/vmalloc.c: In function 'vmap_init_free_space':
>> mm/vmalloc.c:4506:45: warning: ordered comparison of pointer with integer zero [-Wextra]
4506 | if (busy->addr - vmap_start > 0) {
| ^vim +4506 mm/vmalloc.c
4491
4492 static void vmap_init_free_space(void)
4493 {
4494 unsigned long vmap_start = 1;
4495 const unsigned long vmap_end = ULONG_MAX;
4496 struct vmap_area *free;
4497 struct vm_struct *busy;
4498
4499 /*
4500 * B F B B B F
4501 * -|-----|.....|-----|-----|-----|.....|-
4502 * | The KVA space |
4503 * |<--------------------------------->|
4504 */
4505 for (busy = vmlist; busy; busy = busy->next) {
> 4506 if (busy->addr - vmap_start > 0) { 4507 free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT);
4508 if (!WARN_ON_ONCE(!free)) {
4509 free->va_start = vmap_start;
4510 free->va_end = (unsigned long) busy->addr;
4511
4512 insert_vmap_area_augment(free, NULL,
4513 &free_vmap_area_root,
4514 &free_vmap_area_list);
4515 }
4516 }
4517
4518 vmap_start = (unsigned long) busy->addr + busy->size;
4519 }
4520
4521 if (vmap_end - vmap_start > 0) {
4522 free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT);
4523 if (!WARN_ON_ONCE(!free)) {
4524 free->va_start = vmap_start;
4525 free->va_end = vmap_end;
4526
4527 insert_vmap_area_augment(free, NULL,
4528 &free_vmap_area_root,
4529 &free_vmap_area_list);
4530 }
4531 }
4532 }
4533
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Uladzislau Rezki
maintainerRe: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 23 lines · kernel test robot
Hi Uladzislau,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on linus/master v6.5]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Uladzislau-Rezki-Sony/mm-vmalloc-Add-va_alloc-helper/20230829-161248 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230829081142.3619-5-urezki%40gmail.com patch subject: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree config: csky-randconfig-r024-20230829 (https://download.01.org/0day-ci/archive/20230829/[email protected]/config) compiler: csky-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20230829/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
Quoted 7 lines
All warnings (new ones prefixed by >>):
mm/vmalloc.c: In function 'vmap_init_free_space':Quoted 1 line
Quoted 1 line
mm/vmalloc.c:4506:45: warning: ordered comparison of pointer with integer zero [-Wextra]4506 | if (busy->addr - vmap_start > 0) { | ^
Right. I will fix it. We should cast the busy->addr to unsigned long.
--
Uladzislau Rezki
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
In reply to Uladzislau Rezki (Sony)Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;
I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via /proc/kcore will get a little surprise.
Empty vmap_area_list will break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys on vmap_area_list to deduce the vmalloc_start value.
Quoted 3 lines
3. The vmallocinfo and
vread()have to be reworked to be able to handle multiple nodes.
Baoquan He
Reviewed-byRe: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 23 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;
Quoted 8 lines
I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.
Except of the empty vmap_area_list, this patch looks good to me.
We may need think of another way to export the vmalloc_start value or deduce it in makedumpfile/Crash utility. And then remove the useless vmap_area_list. I am not sure if we should remove vmap_area_list in this patch because the empty value will cause breakage anyway. Otherwise,
Quoted 5 lines
Quoted 4 lines
3. The vmallocinfo and
vread()have to be reworked to be able to handle multiple nodes.
Uladzislau Rezki
maintainerRe: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 33 lines · Baoquan He
Quoted 23 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;Quoted 8 lines
I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.
Quoted 9 lines
Except of the empty
vmap_area_list, this patch looks good to me.We may need think of another way to export the
vmalloc_startvalue or deduce it in makedumpfile/Crash utility. And then remove the uselessvmap_area_list. I am not sure if we should removevmap_area_listin this patch because the empty value will cause breakage anyway. Otherwise,Reviewed-by Baoquan He <[email protected]>
Thanks for the review!
--
Uladzislau Rezki
Uladzislau Rezki
maintainerRe: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
In reply to Baoquan HeQuoted 29 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.
Quoted 4 lines
Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.
It is left part and i hope i fix it in v3. The problem here is we can not give an opportunity to access to vmap internals from outside. This is just not correct, i.e. you are not allowed to access the list directly.
--
Uladzislau Rezki
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.
Quoted 8 lines
Quoted 4 lines
Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.It is left part and i hope i fix it in v3. The problem here is we can not give an opportunity to access to vmap internals from outside. This is just not correct, i.e. you are not allowed to access the list directly.
Right. Thanks for the fix in v3, that is a relief of makedumpfile and crash.
Hi Kazu,
Meanwhile, I am thinking if we should evaluate the necessity of vmap_area_list in makedumpfile and Crash. In makedumpfile, we just use vmap_area_list to deduce VMALLOC_START. Wondering if we can export VMALLOC_START directly. Surely, the lowest va->va_start in vmap_area_list is a tighter low boundary of vmalloc area and can reduce unnecessary scanning below the lowest va. Not sure if this is the reason people decided to export vmap_area_list.
Thanks
Baoquan
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 45 lines · Baoquan He
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.Quoted 8 lines
Quoted 4 lines
Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.It is left part and i hope i fix it in v3. The problem here is we can not give an opportunity to access to vmap internals from outside. This is just not correct, i.e. you are not allowed to access the list directly.
Right. Thanks for the fix in v3, that is a relief of makedumpfile and crash.
Hi Kazu,
Quoted 7 lines
Meanwhile, I am thinking if we should evaluate the necessity of
vmap_area_listin makedumpfile and Crash. In makedumpfile, we just usevmap_area_listto deduceVMALLOC_START. Wondering if we can exportVMALLOC_STARTdirectly. Surely, the lowestva->va_startinvmap_area_listis a tighter low boundary of vmalloc area and can reduce unnecessary scanning below the lowest va. Not sure if this is the reason people decided to exportvmap_area_list.
The kernel commit acd99dbf5402 introduced the original vmlist entry to vmcoreinfo, but there is no information about why it did not export VMALLOC_START directly.
If VMALLOC_START is exported directly to vmcoreinfo, I think it would be enough for makedumpfile.
Thanks,
Kazu
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 54 lines · HAGIO KAZUHITO(萩尾 一仁)
Quoted 45 lines · Baoquan He
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.Quoted 8 lines
Quoted 4 lines
Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.It is left part and i hope i fix it in v3. The problem here is we can not give an opportunity to access to vmap internals from outside. This is just not correct, i.e. you are not allowed to access the list directly.
Right. Thanks for the fix in v3, that is a relief of makedumpfile and crash.
Hi Kazu,
Quoted 7 lines
Meanwhile, I am thinking if we should evaluate the necessity of
vmap_area_listin makedumpfile and Crash. In makedumpfile, we just usevmap_area_listto deduceVMALLOC_START. Wondering if we can exportVMALLOC_STARTdirectly. Surely, the lowestva->va_startinvmap_area_listis a tighter low boundary of vmalloc area and can reduce unnecessary scanning below the lowest va. Not sure if this is the reason people decided to exportvmap_area_list.
Quoted 6 lines
The kernel commit
acd99dbf5402introduced the original vmlist entry to vmcoreinfo, but there is no information about why it did not exportVMALLOC_STARTdirectly.If
VMALLOC_STARTis exported directly to vmcoreinfo, I think it would be enough for makedumpfile.
Thanks for confirmation, Kazu.
Then, below draft patch should be enough to export VMALLOC_START instead, and remove vmap_area_list. In order to get the base address of vmalloc area, constructing a vmap_area_list from multiple busy-tree seems not worth.
Documentation/admin-guide/kdump/vmcoreinfo.rst +4 -4
@@ -65,11 +65,11 @@ Defines the beginning of the text section. In general, _stext indicates the kernel start address. Used to convert a virtual address from the direct kernel map to a physical address. -vmap_area_list --------------- +VMALLOC_START +------------- -Stores the virtual area list. makedumpfile gets the vmalloc start value -from this variable and its value is necessary for vmalloc translation. +Stores the base address of vmalloc area. makedumpfile gets this value and +its value is necessary for vmalloc translation. mem_map -------
arch/arm64/kernel/crash_core.c +0 -1
@@ -23,7 +23,6 @@ void arch_crash_save_vmcoreinfo(void)/* Please note VMCOREINFO_NUMBER() uses "%d", not "%x" */ vmcoreinfo_append_str ("NUMBER(MODULES_VADDR)=0x%lx\n" ,MODULES_VADDR );vmcoreinfo_append_str ("NUMBER(MODULES_END)=0x%lx\n" ,MODULES_END ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );
arch/riscv/kernel/crash_core.c +0 -1
@@ -9,7 +9,6 @@ void arch_crash_save_vmcoreinfo(void)VMCOREINFO_NUMBER (phys_ram_base);vmcoreinfo_append_str ("NUMBER(PAGE_OFFSET)=0x%lx\n" ,PAGE_OFFSET ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );
include/linux/vmalloc.h +0 -1
@@ -253,7 +253,6 @@ extern long vread_iter(struct iov_iter *iter, const char *addr, size_t count);/* * Internals. Don't use.. */ -extern struct list_head vmap_area_list;extern __init void vm_area_add_early (struct vm_struct *vm);extern __init void vm_area_register_early (struct vm_struct *vm,size_t align);
kernel/crash_core.c +1 -1
@@ -617,7 +617,7 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_SYMBOL_ARRAY (swapper_pg_dir);#endif VMCOREINFO_SYMBOL (_stext); -VMCOREINFO_SYMBOL (vmap_area_list); +vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );#ifndef CONFIG_NUMA VMCOREINFO_SYMBOL (mem_map);
kernel/kallsyms_selftest.c +0 -1
@@ -89,7 +89,6 @@ static struct test_item test_items[] = {ITEM_DATA (kallsyms_test_var_data_static),ITEM_DATA (kallsyms_test_var_bss), ITEM_DATA(kallsyms_test_var_data), - ITEM_DATA(vmap_area_list), #endif };
mm/nommu.c +0 -2
@@ -131,8 +131,6 @@ int follow_pfn(struct vm_area_struct *vma, unsigned long address, }EXPORT_SYMBOL (follow_pfn); -LIST_HEAD (vmap_area_list); -void vfree (const void *addr) {kfree (addr);
mm/vmalloc.c +1 -2
@@ -729,8 +729,7 @@ EXPORT_SYMBOL(vmalloc_to_pfn);static DEFINE_SPINLOCK (free_vmap_area_lock); -/* Export for kexec only */ -LIST_HEAD (vmap_area_list); +static bool vmap_initialized__read_mostly ;/*
Signature · 1 line
2.41.0
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 59 lines · Baoquan He
Quoted 58 lines · HAGIO KAZUHITO(萩尾 一仁)
Quoted 45 lines · Baoquan He
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.Quoted 8 lines
Quoted 4 lines
Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.It is left part and i hope i fix it in v3. The problem here is we can not give an opportunity to access to vmap internals from outside. This is just not correct, i.e. you are not allowed to access the list directly.
Right. Thanks for the fix in v3, that is a relief of makedumpfile and crash.
Hi Kazu,
Quoted 7 lines
Meanwhile, I am thinking if we should evaluate the necessity of
vmap_area_listin makedumpfile and Crash. In makedumpfile, we just usevmap_area_listto deduceVMALLOC_START. Wondering if we can exportVMALLOC_STARTdirectly. Surely, the lowestva->va_startinvmap_area_listis a tighter low boundary of vmalloc area and can reduce unnecessary scanning below the lowest va. Not sure if this is the reason people decided to exportvmap_area_list.The kernel commit
acd99dbf5402introduced the original vmlist entry to vmcoreinfo, but there is no information about why it did not exportVMALLOC_STARTdirectly.
Quoted 7 lines
Quoted 2 lines
If
VMALLOC_STARTis exported directly to vmcoreinfo, I think it would be enough for makedumpfile.Thanks for confirmation, Kazu.
Then, below draft patch should be enough to export
VMALLOC_STARTinstead, and removevmap_area_list.
also the following entries can be removed.
VMCOREINFO_OFFSET(vmap_area, va_start);
VMCOREINFO_OFFSET(vmap_area, list);Thanks,
Kazu
In order to get the base address of
Quoted 111 lines
vmalloc area, constructing a
vmap_area_listfrom multiple busy-tree seems not worth.Documentation/admin-guide/kdump/vmcoreinfo.rst +4
-4@@ -65,11 +65,11 @@ Defines the beginning of the text section. In general, _stext indicates the kernel start address. Used to convert a virtual address from the direct kernel map to a physical address. -vmap_area_list --------------- +VMALLOC_START +------------- -Stores the virtual area list. makedumpfile gets the vmalloc start value -from this variable and its value is necessary for vmalloc translation. +Stores the base address of vmalloc area. makedumpfile gets this value and +its value is necessary for vmalloc translation. mem_map -------arch/arm64/kernel/crash_core.c +0
-1@@ -23,7 +23,6 @@ void arch_crash_save_vmcoreinfo(void)/* Please note VMCOREINFO_NUMBER() uses "%d", not "%x" */ vmcoreinfo_append_str ("NUMBER(MODULES_VADDR)=0x%lx\n" ,MODULES_VADDR );vmcoreinfo_append_str ("NUMBER(MODULES_END)=0x%lx\n" ,MODULES_END ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );arch/riscv/kernel/crash_core.c +0
-1@@ -9,7 +9,6 @@ void arch_crash_save_vmcoreinfo(void)VMCOREINFO_NUMBER (phys_ram_base);vmcoreinfo_append_str ("NUMBER(PAGE_OFFSET)=0x%lx\n" ,PAGE_OFFSET ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );include/linux/vmalloc.h +0
-1@@ -253,7 +253,6 @@ extern long vread_iter(struct iov_iter *iter, const char *addr, size_t count);/* * Internals. Don't use.. */ -extern struct list_head vmap_area_list;extern __init void vm_area_add_early (struct vm_struct *vm);extern __init void vm_area_register_early (struct vm_struct *vm,size_t align);kernel/crash_core.c +1
-1@@ -617,7 +617,7 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_SYMBOL_ARRAY (swapper_pg_dir);#endif VMCOREINFO_SYMBOL (_stext); -VMCOREINFO_SYMBOL (vmap_area_list); +vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );#ifndef CONFIG_NUMA VMCOREINFO_SYMBOL (mem_map);kernel/kallsyms_selftest.c +0
-1@@ -89,7 +89,6 @@ static struct test_item test_items[] = {ITEM_DATA (kallsyms_test_var_data_static),ITEM_DATA (kallsyms_test_var_bss), ITEM_DATA(kallsyms_test_var_data), - ITEM_DATA(vmap_area_list), #endif };mm/nommu.c +0
-2@@ -131,8 +131,6 @@ int follow_pfn(struct vm_area_struct *vma, unsigned long address, }EXPORT_SYMBOL (follow_pfn); -LIST_HEAD (vmap_area_list); -void vfree (const void *addr) {kfree (addr);mm/vmalloc.c +1
-2@@ -729,8 +729,7 @@ EXPORT_SYMBOL(vmalloc_to_pfn);static DEFINE_SPINLOCK (free_vmap_area_lock); -/* Export for kexec only */ -LIST_HEAD (vmap_area_list); +static bool vmap_initialized__read_mostly ;/*
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 65 lines · HAGIO KAZUHITO(萩尾 一仁)
Quoted 64 lines · Baoquan He
Quoted 54 lines · HAGIO KAZUHITO(萩尾 一仁)
Quoted 45 lines · Baoquan He
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.Quoted 8 lines
Quoted 4 lines
Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.It is left part and i hope i fix it in v3. The problem here is we can not give an opportunity to access to vmap internals from outside. This is just not correct, i.e. you are not allowed to access the list directly.
Right. Thanks for the fix in v3, that is a relief of makedumpfile and crash.
Hi Kazu,
Quoted 7 lines
Meanwhile, I am thinking if we should evaluate the necessity of
vmap_area_listin makedumpfile and Crash. In makedumpfile, we just usevmap_area_listto deduceVMALLOC_START. Wondering if we can exportVMALLOC_STARTdirectly. Surely, the lowestva->va_startinvmap_area_listis a tighter low boundary of vmalloc area and can reduce unnecessary scanning below the lowest va. Not sure if this is the reason people decided to exportvmap_area_list.Quoted 6 lines
The kernel commit
acd99dbf5402introduced the original vmlist entry to vmcoreinfo, but there is no information about why it did not exportVMALLOC_STARTdirectly.If
VMALLOC_STARTis exported directly to vmcoreinfo, I think it would be enough for makedumpfile.Thanks for confirmation, Kazu.
Quoted 7 lines
Quoted 2 lines
Then, below draft patch should be enough to export
VMALLOC_STARTinstead, and removevmap_area_list.also the following entries can be removed.
VMCOREINFO_OFFSET(vmap_area, va_start); VMCOREINFO_OFFSET(vmap_area, list);
Right, they are useless now. I updated to remove them in below patch.
From a867fada34fd9e96528fcc5e72ae50b3b5685015 Mon Sep 17 00:00:00 2001
From: Baoquan He <[email protected]>
Date: Fri, 8 Sep 2023 11:53:22 +0800
Subject: [PATCH] mm/vmalloc: remove vmap_area_list
Content-type: text/plain
Earlier, vmap_area_list is exported to vmcoreinfo so that makedumpfile get the base address of vmalloc area. Now, vmap_area_list is empty, so export VMALLOC_START to vmcoreinfo instead, and remove vmap_area_list.
--- Documentation/admin-guide/kdump/vmcoreinfo.rst | 8 ++++---- arch/arm64/kernel/crash_core.c | 1 - arch/riscv/kernel/crash_core.c | 1 - include/linux/vmalloc.h | 1 - kernel/crash_core.c | 4 +--- kernel/kallsyms_selftest.c | 1 - mm/nommu.c | 2 -- mm/vmalloc.c | 3 +-- 8 files changed, 6 insertions(+), 15 deletions(-)
Documentation/admin-guide/kdump/vmcoreinfo.rst +4 -4
@@ -65,11 +65,11 @@ Defines the beginning of the text section. In general, _stext indicates the kernel start address. Used to convert a virtual address from the direct kernel map to a physical address. -vmap_area_list --------------- +VMALLOC_START +------------- -Stores the virtual area list. makedumpfile gets the vmalloc start value -from this variable and its value is necessary for vmalloc translation. +Stores the base address of vmalloc area. makedumpfile gets this value +since is necessary for vmalloc translation. mem_map -------
arch/arm64/kernel/crash_core.c +0 -1
@@ -23,7 +23,6 @@ void arch_crash_save_vmcoreinfo(void)/* Please note VMCOREINFO_NUMBER() uses "%d", not "%x" */ vmcoreinfo_append_str ("NUMBER(MODULES_VADDR)=0x%lx\n" ,MODULES_VADDR );vmcoreinfo_append_str ("NUMBER(MODULES_END)=0x%lx\n" ,MODULES_END ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );
arch/riscv/kernel/crash_core.c +0 -1
@@ -9,7 +9,6 @@ void arch_crash_save_vmcoreinfo(void)VMCOREINFO_NUMBER (phys_ram_base);vmcoreinfo_append_str ("NUMBER(PAGE_OFFSET)=0x%lx\n" ,PAGE_OFFSET ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );
include/linux/vmalloc.h +0 -1
@@ -253,7 +253,6 @@ extern long vread_iter(struct iov_iter *iter, const char *addr, size_t count);/* * Internals. Don't use.. */ -extern struct list_head vmap_area_list;extern __init void vm_area_add_early (struct vm_struct *vm);extern __init void vm_area_register_early (struct vm_struct *vm,size_t align);
kernel/crash_core.c +1 -3
@@ -617,7 +617,7 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_SYMBOL_ARRAY (swapper_pg_dir);#endif VMCOREINFO_SYMBOL (_stext); -VMCOREINFO_SYMBOL (vmap_area_list); +vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );#ifndef CONFIG_NUMA VMCOREINFO_SYMBOL (mem_map);
@@ -658,8 +658,6 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_OFFSET (free_area, free_list);VMCOREINFO_OFFSET (list_head, next);VMCOREINFO_OFFSET (list_head, prev); -VMCOREINFO_OFFSET (vmap_area, va_start); -VMCOREINFO_OFFSET (vmap_area, list);VMCOREINFO_LENGTH (zone.free_area,MAX_ORDER +1 );log_buf_vmcoreinfo_setup ();VMCOREINFO_LENGTH (free_area.free_list,MIGRATE_TYPES );
kernel/kallsyms_selftest.c +0 -1
@@ -89,7 +89,6 @@ static struct test_item test_items[] = {ITEM_DATA (kallsyms_test_var_data_static),ITEM_DATA (kallsyms_test_var_bss), ITEM_DATA(kallsyms_test_var_data), - ITEM_DATA(vmap_area_list), #endif };
mm/nommu.c +0 -2
@@ -131,8 +131,6 @@ int follow_pfn(struct vm_area_struct *vma, unsigned long address, }EXPORT_SYMBOL (follow_pfn); -LIST_HEAD (vmap_area_list); -void vfree (const void *addr) {kfree (addr);
mm/vmalloc.c +1 -2
@@ -729,8 +729,7 @@ EXPORT_SYMBOL(vmalloc_to_pfn);static DEFINE_SPINLOCK (free_vmap_area_lock); -/* Export for kexec only */ -LIST_HEAD (vmap_area_list); +static bool vmap_initialized__read_mostly ;/*
Signature · 1 line
2.41.0
Uladzislau Rezki
maintainerRe: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 218 lines · Baoquan He
Quoted 65 lines · HAGIO KAZUHITO(萩尾 一仁)
Quoted 64 lines · Baoquan He
Quoted 54 lines · HAGIO KAZUHITO(萩尾 一仁)
Quoted 45 lines · Baoquan He
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.Quoted 8 lines
Quoted 4 lines
Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.It is left part and i hope i fix it in v3. The problem here is we can not give an opportunity to access to vmap internals from outside. This is just not correct, i.e. you are not allowed to access the list directly.
Right. Thanks for the fix in v3, that is a relief of makedumpfile and crash.
Hi Kazu,
Quoted 7 lines
Meanwhile, I am thinking if we should evaluate the necessity of
vmap_area_listin makedumpfile and Crash. In makedumpfile, we just usevmap_area_listto deduceVMALLOC_START. Wondering if we can exportVMALLOC_STARTdirectly. Surely, the lowestva->va_startinvmap_area_listis a tighter low boundary of vmalloc area and can reduce unnecessary scanning below the lowest va. Not sure if this is the reason people decided to exportvmap_area_list.Quoted 6 lines
The kernel commit
acd99dbf5402introduced the original vmlist entry to vmcoreinfo, but there is no information about why it did not exportVMALLOC_STARTdirectly.If
VMALLOC_STARTis exported directly to vmcoreinfo, I think it would be enough for makedumpfile.Thanks for confirmation, Kazu.
Quoted 7 lines
Quoted 2 lines
Then, below draft patch should be enough to export
VMALLOC_STARTinstead, and removevmap_area_list.also the following entries can be removed.
VMCOREINFO_OFFSET(vmap_area, va_start); VMCOREINFO_OFFSET(vmap_area, list);Right, they are useless now. I updated to remove them in below patch.
From
a867fada34fd9e96528fcc5e72ae50b3b5685015Mon Sep 17 00:00:00 2001
From: Baoquan He <[email protected]>
Date: Fri, 8 Sep 2023 11:53:22 +0800
Subject: [PATCH] mm/vmalloc: removevmap_area_list
Content-type: text/plainEarlier,
vmap_area_listis exported to vmcoreinfo so that makedumpfile get the base address of vmalloc area. Now,vmap_area_listis empty, so exportVMALLOC_STARTto vmcoreinfo instead, and removevmap_area_list.Signed-off-by Baoquan He <[email protected]>--- Documentation/admin-guide/kdump/vmcoreinfo.rst | 8 ++++---- arch/arm64/kernel/crash_core.c | 1 - arch/riscv/kernel/crash_core.c | 1 - include/linux/vmalloc.h | 1 - kernel/crash_core.c | 4 +--- kernel/kallsyms_selftest.c | 1 - mm/nommu.c | 2 -- mm/vmalloc.c | 3 +-- 8 files changed, 6 insertions(+), 15 deletions(-)Documentation/admin-guide/kdump/vmcoreinfo.rst +4
-4@@ -65,11 +65,11 @@ Defines the beginning of the text section. In general, _stext indicates the kernel start address. Used to convert a virtual address from the direct kernel map to a physical address. -vmap_area_list --------------- +VMALLOC_START +------------- -Stores the virtual area list. makedumpfile gets the vmalloc start value -from this variable and its value is necessary for vmalloc translation. +Stores the base address of vmalloc area. makedumpfile gets this value +since is necessary for vmalloc translation. mem_map -------arch/arm64/kernel/crash_core.c +0
-1@@ -23,7 +23,6 @@ void arch_crash_save_vmcoreinfo(void)/* Please note VMCOREINFO_NUMBER() uses "%d", not "%x" */ vmcoreinfo_append_str ("NUMBER(MODULES_VADDR)=0x%lx\n" ,MODULES_VADDR );vmcoreinfo_append_str ("NUMBER(MODULES_END)=0x%lx\n" ,MODULES_END ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );arch/riscv/kernel/crash_core.c +0
-1@@ -9,7 +9,6 @@ void arch_crash_save_vmcoreinfo(void)VMCOREINFO_NUMBER (phys_ram_base);vmcoreinfo_append_str ("NUMBER(PAGE_OFFSET)=0x%lx\n" ,PAGE_OFFSET ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );include/linux/vmalloc.h +0
-1@@ -253,7 +253,6 @@ extern long vread_iter(struct iov_iter *iter, const char *addr, size_t count);/* * Internals. Don't use.. */ -extern struct list_head vmap_area_list;extern __init void vm_area_add_early (struct vm_struct *vm);extern __init void vm_area_register_early (struct vm_struct *vm,size_t align);kernel/crash_core.c +1
-3@@ -617,7 +617,7 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_SYMBOL_ARRAY (swapper_pg_dir);#endif VMCOREINFO_SYMBOL (_stext); -VMCOREINFO_SYMBOL (vmap_area_list); +vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );#ifndef CONFIG_NUMA VMCOREINFO_SYMBOL (mem_map);@@ -658,8 +658,6 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_OFFSET (free_area, free_list);VMCOREINFO_OFFSET (list_head, next);VMCOREINFO_OFFSET (list_head, prev); -VMCOREINFO_OFFSET (vmap_area, va_start); -VMCOREINFO_OFFSET (vmap_area, list);VMCOREINFO_LENGTH (zone.free_area,MAX_ORDER +1 );log_buf_vmcoreinfo_setup ();VMCOREINFO_LENGTH (free_area.free_list,MIGRATE_TYPES );kernel/kallsyms_selftest.c +0
-1@@ -89,7 +89,6 @@ static struct test_item test_items[] = {ITEM_DATA (kallsyms_test_var_data_static),ITEM_DATA (kallsyms_test_var_bss), ITEM_DATA(kallsyms_test_var_data), - ITEM_DATA(vmap_area_list), #endif };mm/nommu.c +0
-2@@ -131,8 +131,6 @@ int follow_pfn(struct vm_area_struct *vma, unsigned long address, }EXPORT_SYMBOL (follow_pfn); -LIST_HEAD (vmap_area_list); -void vfree (const void *addr) {kfree (addr);mm/vmalloc.c +1
-2@@ -729,8 +729,7 @@ EXPORT_SYMBOL(vmalloc_to_pfn);static DEFINE_SPINLOCK (free_vmap_area_lock); -/* Export for kexec only */ -LIST_HEAD (vmap_area_list); +static bool vmap_initialized__read_mostly ;/* Signature · 1 line
2.41.0
Appreciate for your great input. This patch can go as standalone with slight commit message updating or i can take it and send it out as part of v3.
Either way i am totally fine. What do you prefer?
Thank you!
--
Uladzislau Rezki
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 223 lines · Uladzislau Rezki
Quoted 218 lines · Baoquan He
Quoted 65 lines · HAGIO KAZUHITO(萩尾 一仁)
Quoted 64 lines · Baoquan He
Quoted 54 lines · HAGIO KAZUHITO(萩尾 一仁)
Quoted 45 lines · Baoquan He
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Baoquan He
Add Kazu and Lianbo to CC, and kexec mailing list
Quoted 12 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
Quoted 7 lines
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;I haven't taken a test, while accessing all nodes' busy tree to get va of the lowest address could severely impact kcore reading efficiency on system with many vmap nodes. People doing live debugging via
/proc/kcorewill get a little surprise.Quoted 8 lines
Quoted 4 lines
Empty
vmap_area_listwill break makedumpfile utility, Crash utility could be impactd too. I checked makedumpfile code, it relys onvmap_area_listto deduce thevmalloc_startvalue.It is left part and i hope i fix it in v3. The problem here is we can not give an opportunity to access to vmap internals from outside. This is just not correct, i.e. you are not allowed to access the list directly.
Right. Thanks for the fix in v3, that is a relief of makedumpfile and crash.
Hi Kazu,
Quoted 7 lines
Meanwhile, I am thinking if we should evaluate the necessity of
vmap_area_listin makedumpfile and Crash. In makedumpfile, we just usevmap_area_listto deduceVMALLOC_START. Wondering if we can exportVMALLOC_STARTdirectly. Surely, the lowestva->va_startinvmap_area_listis a tighter low boundary of vmalloc area and can reduce unnecessary scanning below the lowest va. Not sure if this is the reason people decided to exportvmap_area_list.Quoted 6 lines
The kernel commit
acd99dbf5402introduced the original vmlist entry to vmcoreinfo, but there is no information about why it did not exportVMALLOC_STARTdirectly.If
VMALLOC_STARTis exported directly to vmcoreinfo, I think it would be enough for makedumpfile.Thanks for confirmation, Kazu.
Quoted 7 lines
Quoted 2 lines
Then, below draft patch should be enough to export
VMALLOC_STARTinstead, and removevmap_area_list.also the following entries can be removed.
VMCOREINFO_OFFSET(vmap_area, va_start); VMCOREINFO_OFFSET(vmap_area, list);Right, they are useless now. I updated to remove them in below patch.
From
a867fada34fd9e96528fcc5e72ae50b3b5685015Mon Sep 17 00:00:00 2001
From: Baoquan He <[email protected]>
Date: Fri, 8 Sep 2023 11:53:22 +0800
Subject: [PATCH] mm/vmalloc: removevmap_area_list
Content-type: text/plainEarlier,
vmap_area_listis exported to vmcoreinfo so that makedumpfile get the base address of vmalloc area. Now,vmap_area_listis empty, so exportVMALLOC_STARTto vmcoreinfo instead, and removevmap_area_list.Signed-off-by Baoquan He <[email protected]>--- Documentation/admin-guide/kdump/vmcoreinfo.rst | 8 ++++---- arch/arm64/kernel/crash_core.c | 1 - arch/riscv/kernel/crash_core.c | 1 - include/linux/vmalloc.h | 1 - kernel/crash_core.c | 4 +--- kernel/kallsyms_selftest.c | 1 - mm/nommu.c | 2 -- mm/vmalloc.c | 3 +-- 8 files changed, 6 insertions(+), 15 deletions(-)Documentation/admin-guide/kdump/vmcoreinfo.rst +4
-4@@ -65,11 +65,11 @@ Defines the beginning of the text section. In general, _stext indicates the kernel start address. Used to convert a virtual address from the direct kernel map to a physical address. -vmap_area_list --------------- +VMALLOC_START +------------- -Stores the virtual area list. makedumpfile gets the vmalloc start value -from this variable and its value is necessary for vmalloc translation. +Stores the base address of vmalloc area. makedumpfile gets this value +since is necessary for vmalloc translation. mem_map -------arch/arm64/kernel/crash_core.c +0
-1@@ -23,7 +23,6 @@ void arch_crash_save_vmcoreinfo(void)/* Please note VMCOREINFO_NUMBER() uses "%d", not "%x" */ vmcoreinfo_append_str ("NUMBER(MODULES_VADDR)=0x%lx\n" ,MODULES_VADDR );vmcoreinfo_append_str ("NUMBER(MODULES_END)=0x%lx\n" ,MODULES_END ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );arch/riscv/kernel/crash_core.c +0
-1@@ -9,7 +9,6 @@ void arch_crash_save_vmcoreinfo(void)VMCOREINFO_NUMBER (phys_ram_base);vmcoreinfo_append_str ("NUMBER(PAGE_OFFSET)=0x%lx\n" ,PAGE_OFFSET ); -vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );vmcoreinfo_append_str ("NUMBER(VMALLOC_END)=0x%lx\n" ,VMALLOC_END );vmcoreinfo_append_str ("NUMBER(VMEMMAP_START)=0x%lx\n" ,VMEMMAP_START );vmcoreinfo_append_str ("NUMBER(VMEMMAP_END)=0x%lx\n" ,VMEMMAP_END );include/linux/vmalloc.h +0
-1@@ -253,7 +253,6 @@ extern long vread_iter(struct iov_iter *iter, const char *addr, size_t count);/* * Internals. Don't use.. */ -extern struct list_head vmap_area_list;extern __init void vm_area_add_early (struct vm_struct *vm);extern __init void vm_area_register_early (struct vm_struct *vm,size_t align);kernel/crash_core.c +1
-3@@ -617,7 +617,7 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_SYMBOL_ARRAY (swapper_pg_dir);#endif VMCOREINFO_SYMBOL (_stext); -VMCOREINFO_SYMBOL (vmap_area_list); +vmcoreinfo_append_str ("NUMBER(VMALLOC_START)=0x%lx\n" ,VMALLOC_START );#ifndef CONFIG_NUMA VMCOREINFO_SYMBOL (mem_map);@@ -658,8 +658,6 @@ static int __init crash_save_vmcoreinfo_init(void)VMCOREINFO_OFFSET (free_area, free_list);VMCOREINFO_OFFSET (list_head, next);VMCOREINFO_OFFSET (list_head, prev); -VMCOREINFO_OFFSET (vmap_area, va_start); -VMCOREINFO_OFFSET (vmap_area, list);VMCOREINFO_LENGTH (zone.free_area,MAX_ORDER +1 );log_buf_vmcoreinfo_setup ();VMCOREINFO_LENGTH (free_area.free_list,MIGRATE_TYPES );kernel/kallsyms_selftest.c +0
-1@@ -89,7 +89,6 @@ static struct test_item test_items[] = {ITEM_DATA (kallsyms_test_var_data_static),ITEM_DATA (kallsyms_test_var_bss), ITEM_DATA(kallsyms_test_var_data), - ITEM_DATA(vmap_area_list), #endif };mm/nommu.c +0
-2@@ -131,8 +131,6 @@ int follow_pfn(struct vm_area_struct *vma, unsigned long address, }EXPORT_SYMBOL (follow_pfn); -LIST_HEAD (vmap_area_list); -void vfree (const void *addr) {kfree (addr);mm/vmalloc.c +1
-2@@ -729,8 +729,7 @@ EXPORT_SYMBOL(vmalloc_to_pfn);static DEFINE_SPINLOCK (free_vmap_area_lock); -/* Export for kexec only */ -LIST_HEAD (vmap_area_list); +static bool vmap_initialized__read_mostly ;/* Signature · 1 line
2.41.0Appreciate for your great input. This patch can go as standalone with slight commit message updating or i can take it and send it out as part of v3.
Quoted 1 line
Either way i am totally fine. What do you prefer?
Maybe take it together with this patchset in v3. Thanks.
Uladzislau Rezki
maintainerRe: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
Appreciate for your great input. This patch can go as standalone with slight commit message updating or i can take it and send it out as part of v3.
Either way i am totally fine. What do you prefer?
Maybe take it together with this patchset in v3. Thanks.
OK, i will deliver it with v3. Again, thank you very much.
--
Uladzislau Rezki
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
In reply to Uladzislau Rezki (Sony)Quoted 57 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;3. The vmallocinfo and
vread()have to be reworked to be able to handle multiple nodes.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 209 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 161 insertions(+), 48 deletions(-)mm/vmalloc.c +9
-2@@ -728,11 +728,9 @@ EXPORT_SYMBOL(vmalloc_to_pfn);#define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -static DEFINE_SPINLOCK (vmap_area_lock);static DEFINE_SPINLOCK (free_vmap_area_lock);/* Export for kexec only */ LIST_HEAD (vmap_area_list); -static struct rb_root vmap_area_root =RB_ROOT ;static bool vmap_initialized__read_mostly ;static struct rb_root purge_vmap_area_root =RB_ROOT ;@@ -772,6 +770,38 @@ static struct rb_root free_vmap_area_root = RB_ROOT;*/ static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); +/* +* An effective vmap-node logic. Users make use of nodes instead +* of a global heap. It allows to balance an access and mitigate +* contention. +*/ +struct rb_list { +struct rb_root root; +struct list_head head; +spinlock_t lock;
mm/vmalloc.c static struct rb_root free_vmap_area_root = RB_ROOT; · line 791
Quoted 10 lines
mm/vmalloc.c +10
-0+}; + +struct vmap_node { +/* Bookkeeping data of this node. */ +struct rb_list busy; +}; + +static struct vmap_node *nodes, snode; +static __read_mostly unsigned int nr_nodes =1 ; +static __read_mostly unsigned int node_size =1 ;
It could be better if calling these global variables a meaningful name, e.g vmap_nodes, static_vmap_nodes, nr_vmap_nodes. When I use vim+cscope to reference them, it gives me a super long list. Aside from that, a simple name often makes me mistake it as a local virable. A weak opinion.
Quoted 16 lines
mm/vmalloc.c +13
-0+ +static inline unsigned int +addr_to_node_id (unsigned long addr) +{ +return (addr / node_size) % nr_nodes; +} + +static inline struct vmap_node * +addr_to_node (unsigned long addr) +{ +return &nodes[addr_to_node_id (addr)]; +} +static __always_inline unsigned long va_size (struct vmap_area *va) {
Uladzislau Rezki
maintainerRe: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 66 lines · Baoquan He
Quoted 65 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;3. The vmallocinfo and
vread()have to be reworked to be able to handle multiple nodes.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 209 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 161 insertions(+), 48 deletions(-)mm/vmalloc.c +17
-2@@ -728,11 +728,9 @@ EXPORT_SYMBOL(vmalloc_to_pfn);#define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -static DEFINE_SPINLOCK (vmap_area_lock);static DEFINE_SPINLOCK (free_vmap_area_lock);/* Export for kexec only */ LIST_HEAD (vmap_area_list); -static struct rb_root vmap_area_root =RB_ROOT ;static bool vmap_initialized__read_mostly ;static struct rb_root purge_vmap_area_root =RB_ROOT ;@@ -772,6 +770,38 @@ static struct rb_root free_vmap_area_root = RB_ROOT;*/ static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); +/* +* An effective vmap-node logic. Users make use of nodes instead +* of a global heap. It allows to balance an access and mitigate +* contention. +*/ +struct rb_list { +struct rb_root root; +struct list_head head; +spinlock_t lock; +}; + +struct vmap_node { +/* Bookkeeping data of this node. */ +struct rb_list busy; +}; + +static struct vmap_node *nodes, snode;
Quoted 9 lines
mm/vmalloc.c static struct rb_root free_vmap_area_root = RB_ROOT; · line 791
Quoted 2 lines
mm/vmalloc.c +2
-0+static __read_mostly unsigned int nr_nodes =1 ; +static __read_mostly unsigned int node_size =1 ;It could be better if calling these global variables a meaningful name, e.g
vmap_nodes,static_vmap_nodes,nr_vmap_nodes. When I use vim+cscope to reference them, it gives me a super long list. Aside from that, a simple name often makes me mistake it as a local virable. A weak opinion.
I am OK to add "vmap_" prefix:
vmap_nodes;vmap_nr_nodes;vmap_node_size;
..
If you are not OK with that, feel free to propose other variants.
Thank you!
--
Uladzislau Rezki
Re: [PATCH v2 4/9] mm: vmalloc: Remove global vmap_area_root rb-tree
Quoted 76 lines · Uladzislau Rezki
Quoted 66 lines · Baoquan He
Quoted 65 lines · Uladzislau Rezki (Sony)
Store allocated objects in a separate nodes. A
va->va_startaddress is converted into a correct node where it should be placed and resided. Anaddr_to_node()function is used to do a proper address conversion to determine a node that contains a VA.Such approach balances VAs across nodes as a result an access becomes scalable. Number of nodes in a system depends on number of CPUs divided by two. The density factor in this case is 1/2.
Please note:
1. As of now allocated VAs are bound to a node-0. It means the patch does not give any difference comparing with a current behavior;
2. The global
vmap_area_lock,vmap_area_rootare removed as there is no need in it anymore. Thevmap_area_listis still kept and is _empty_. It is exported for a kexec only;3. The vmallocinfo and
vread()have to be reworked to be able to handle multiple nodes.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 209 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 161 insertions(+), 48 deletions(-)mm/vmalloc.c +17
-2@@ -728,11 +728,9 @@ EXPORT_SYMBOL(vmalloc_to_pfn);#define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -static DEFINE_SPINLOCK (vmap_area_lock);static DEFINE_SPINLOCK (free_vmap_area_lock);/* Export for kexec only */ LIST_HEAD (vmap_area_list); -static struct rb_root vmap_area_root =RB_ROOT ;static bool vmap_initialized__read_mostly ;static struct rb_root purge_vmap_area_root =RB_ROOT ;@@ -772,6 +770,38 @@ static struct rb_root free_vmap_area_root = RB_ROOT;*/ static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); +/* +* An effective vmap-node logic. Users make use of nodes instead +* of a global heap. It allows to balance an access and mitigate +* contention. +*/ +struct rb_list { +struct rb_root root; +struct list_head head; +spinlock_t lock; +}; + +struct vmap_node { +/* Bookkeeping data of this node. */ +struct rb_list busy; +}; + +static struct vmap_node *nodes, snode;Quoted 9 lines
mm/vmalloc.c static struct rb_root free_vmap_area_root = RB_ROOT; · line 791
Quoted 2 lines
mm/vmalloc.c +2
-0+static __read_mostly unsigned int nr_nodes =1 ; +static __read_mostly unsigned int node_size =1 ;It could be better if calling these global variables a meaningful name, e.g
vmap_nodes,static_vmap_nodes,nr_vmap_nodes. When I use vim+cscope to reference them, it gives me a super long list. Aside from that, a simple name often makes me mistake it as a local virable. A weak opinion.
Quoted 4 lines
I am OK to add "
vmap_" prefix:
vmap_nodes;vmap_nr_nodes;
~ nr_vmap_nodes?
Quoted 4 lines
vmap_node_size;
..If you are not OK with that, feel free to propose other variants.
Other than the nr_nodes one, others look good to me, thanks a lot.
Uladzislau Rezki (Sony)
maintainerPATCH v2 5/9 mm: vmalloc: Remove global purge_vmap_area_root rb-tree
In reply to Uladzislau Rezki (Sony)Similar to busy VA, lazily-freed area is stored to a node it belongs to. Such approach does not require any global locking primitive, instead an access becomes scalable what mitigates a contention.
This patch removes a global purge-lock, global purge-tree and global purge list.
--- mm/vmalloc.c | 135 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 82 insertions(+), 53 deletions(-)
mm/vmalloc.c +82 -53
@@ -733,10 +733,6 @@ static DEFINE_SPINLOCK(free_vmap_area_lock);LIST_HEAD (vmap_area_list);static bool vmap_initialized__read_mostly ; -static struct rb_root purge_vmap_area_root =RB_ROOT ; -static LIST_HEAD (purge_vmap_area_list); -static DEFINE_SPINLOCK (purge_vmap_area_lock); -/* * This kmem_cache is used for vmap_area objects. Instead of * allocating from slab we reuse an object from this cache to
@@ -784,6 +780,12 @@ struct rb_list {struct vmap_node {/* Bookkeeping data of this node. */ struct rb_list busy; +struct rb_list lazy; + +/* +* Ready-to-free areas. +*/ +struct list_head purge_list; };static struct vmap_node *nodes, snode;
@@ -1768,40 +1770,22 @@ static DEFINE_MUTEX(vmap_purge_lock);/* for per-CPU blocks */ static void purge_fragmented_blocks_allcpus (void ); +static cpumask_t purge_nodes;/* * Purges all lazily-freed vmap areas. */ -static bool __purge_vmap_area_lazy (unsigned long start,unsigned long end) +static unsigned long +purge_vmap_node (struct vmap_node *vn) { -unsigned long resched_threshold; -unsigned int num_purged_areas =0 ; -struct list_head local_purge_list; +unsigned long num_purged_areas =0 ;struct vmap_area *va, *n_va; -lockdep_assert_held (&vmap_purge_lock); - -spin_lock (&purge_vmap_area_lock); - purge_vmap_area_root =RB_ROOT ; -list_replace_init (&purge_vmap_area_list, &local_purge_list); -spin_unlock (&purge_vmap_area_lock); - -if (unlikely (list_empty (&local_purge_list))) - goto out; - - start =min (start, -list_first_entry (&local_purge_list, - struct vmap_area, list)->va_start); - - end =max (end, -list_last_entry (&local_purge_list, - struct vmap_area, list)->va_end); - -flush_tlb_kernel_range (start, end); - resched_threshold =lazy_max_pages () <<1 ; +if (list_empty (&vn->purge_list)) +return 0 ;spin_lock (&free_vmap_area_lock); -list_for_each_entry_safe (va, n_va, &local_purge_list, list) { +list_for_each_entry_safe (va, n_va, &vn->purge_list, list) {unsigned long nr = (va->va_end - va->va_start) >>PAGE_SHIFT ;unsigned long orig_start = va->va_start;unsigned long orig_end = va->va_end;
@@ -1823,13 +1807,55 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)atomic_long_sub (nr, &vmap_lazy_nr); num_purged_areas++; - -if (atomic_long_read (&vmap_lazy_nr) < resched_threshold) -cond_resched_lock (&free_vmap_area_lock); }spin_unlock (&free_vmap_area_lock); -out: +return num_purged_areas; +} + +/* +* Purges all lazily-freed vmap areas. +*/ +static bool __purge_vmap_area_lazy (unsigned long start,unsigned long end) +{ +unsigned long num_purged_areas =0 ; +struct vmap_node *vn; +int i; + +lockdep_assert_held (&vmap_purge_lock); + purge_nodes =CPU_MASK_NONE ; + +for (i =0 ; i < nr_nodes; i++) { + vn = &nodes[i]; + +INIT_LIST_HEAD (&vn->purge_list); + +if (RB_EMPTY_ROOT (&vn->lazy.root)) +continue ; + +spin_lock (&vn->lazy.lock); +WRITE_ONCE (vn->lazy.root.rb_node,NULL ); +list_replace_init (&vn->lazy.head, &vn->purge_list); +spin_unlock (&vn->lazy.lock); + + start =min (start,list_first_entry (&vn->purge_list, + struct vmap_area, list)->va_start); + + end =max (end,list_last_entry (&vn->purge_list, + struct vmap_area, list)->va_end); + +cpumask_set_cpu (i, &purge_nodes); + } + +if (cpumask_weight (&purge_nodes) >0 ) { +flush_tlb_kernel_range (start, end); + +for_each_cpu (i, &purge_nodes) { + vn = &nodes[i]; + num_purged_areas +=purge_vmap_node (vn); + } + } +trace_purge_vmap_area_lazy (start, end, num_purged_areas);return num_purged_areas >0 ; }
@@ -1848,16 +1874,9 @@ static void reclaim_and_purge_vmap_areas(void)static void drain_vmap_area_work (struct work_struct *work) { -unsigned long nr_lazy; - -do { -mutex_lock (&vmap_purge_lock); -__purge_vmap_area_lazy (ULONG_MAX ,0 ); -mutex_unlock (&vmap_purge_lock); - -/* Recheck if further work is required. */ - nr_lazy =atomic_long_read (&vmap_lazy_nr); - }while (nr_lazy >lazy_max_pages ()); +mutex_lock (&vmap_purge_lock); +__purge_vmap_area_lazy (ULONG_MAX ,0 ); +mutex_unlock (&vmap_purge_lock); }/*
@@ -1867,6 +1886,7 @@ static void drain_vmap_area_work(struct work_struct *work)*/ static void free_vmap_area_noflush (struct vmap_area *va) { +struct vmap_node *vn =addr_to_node (va->va_start);unsigned long nr_lazy_max =lazy_max_pages ();unsigned long va_start = va->va_start;unsigned long nr_lazy;
@@ -1880,10 +1900,9 @@ static void free_vmap_area_noflush(struct vmap_area *va)/* * Merge or place it to the purge tree/list. */ -spin_lock (&purge_vmap_area_lock); -merge_or_add_vmap_area (va, - &purge_vmap_area_root, &purge_vmap_area_list); -spin_unlock (&purge_vmap_area_lock); +spin_lock (&vn->lazy.lock); +merge_or_add_vmap_area (va, &vn->lazy.root, &vn->lazy.head); +spin_unlock (&vn->lazy.lock);trace_free_vmap_area_noflush (va_start, nr_lazy, nr_lazy_max);
@@ -4390,15 +4409,21 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v)static void show_purge_info (struct seq_file *m) { +struct vmap_node *vn;struct vmap_area *va; +int i; -spin_lock (&purge_vmap_area_lock); -list_for_each_entry (va, &purge_vmap_area_list, list) { -seq_printf (m,"0x%pK-0x%pK %7ld unpurged vm_area\n" , - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); +for (i =0 ; i < nr_nodes; i++) { + vn = &nodes[i]; + +spin_lock (&vn->lazy.lock); +list_for_each_entry (va, &vn->lazy.head, list) { +seq_printf (m,"0x%pK-0x%pK %7ld unpurged vm_area\n" , + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } +spin_unlock (&vn->lazy.lock); } -spin_unlock (&purge_vmap_area_lock); }static int s_show (struct seq_file *m,void *p)
@@ -4545,6 +4570,10 @@ static void vmap_init_nodes(void) vn->busy.root =RB_ROOT ;INIT_LIST_HEAD (&vn->busy.head);spin_lock_init (&vn->busy.lock); + + vn->lazy.root =RB_ROOT ; +INIT_LIST_HEAD (&vn->lazy.head); +spin_lock_init (&vn->lazy.lock); } }
Signature · 1 line
2.30.2
Re: [PATCH v2 5/9] mm: vmalloc: Remove global purge_vmap_area_root rb-tree
Quoted 5 lines · Uladzislau Rezki (Sony)
Similar to busy VA, lazily-freed area is stored to a node it belongs to. Such approach does not require any global locking primitive, instead an access becomes scalable what mitigates a contention.
Quoted 7 lines
This patch removes a global purge-lock, global purge-tree and global purge list.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 135 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 82 insertions(+), 53 deletions(-)
LGTM,
Quoted 221 lines
mm/vmalloc.c +82
-53@@ -733,10 +733,6 @@ static DEFINE_SPINLOCK(free_vmap_area_lock);LIST_HEAD (vmap_area_list);static bool vmap_initialized__read_mostly ; -static struct rb_root purge_vmap_area_root =RB_ROOT ; -static LIST_HEAD (purge_vmap_area_list); -static DEFINE_SPINLOCK (purge_vmap_area_lock); -/* * This kmem_cache is used for vmap_area objects. Instead of * allocating from slab we reuse an object from this cache to @@ -784,6 +780,12 @@ struct rb_list {struct vmap_node {/* Bookkeeping data of this node. */ struct rb_list busy; +struct rb_list lazy; + +/* +* Ready-to-free areas. +*/ +struct list_head purge_list; };static struct vmap_node *nodes, snode;@@ -1768,40 +1770,22 @@ static DEFINE_MUTEX(vmap_purge_lock);/* for per-CPU blocks */ static void purge_fragmented_blocks_allcpus (void ); +static cpumask_t purge_nodes;/* * Purges all lazily-freed vmap areas. */ -static bool __purge_vmap_area_lazy (unsigned long start,unsigned long end) +static unsigned long +purge_vmap_node (struct vmap_node *vn) { -unsigned long resched_threshold; -unsigned int num_purged_areas =0 ; -struct list_head local_purge_list; +unsigned long num_purged_areas =0 ;struct vmap_area *va, *n_va; -lockdep_assert_held (&vmap_purge_lock); - -spin_lock (&purge_vmap_area_lock); - purge_vmap_area_root =RB_ROOT ; -list_replace_init (&purge_vmap_area_list, &local_purge_list); -spin_unlock (&purge_vmap_area_lock); - -if (unlikely (list_empty (&local_purge_list))) - goto out; - - start =min (start, -list_first_entry (&local_purge_list, - struct vmap_area, list)->va_start); - - end =max (end, -list_last_entry (&local_purge_list, - struct vmap_area, list)->va_end); - -flush_tlb_kernel_range (start, end); - resched_threshold =lazy_max_pages () <<1 ; +if (list_empty (&vn->purge_list)) +return 0 ;spin_lock (&free_vmap_area_lock); -list_for_each_entry_safe (va, n_va, &local_purge_list, list) { +list_for_each_entry_safe (va, n_va, &vn->purge_list, list) {unsigned long nr = (va->va_end - va->va_start) >>PAGE_SHIFT ;unsigned long orig_start = va->va_start;unsigned long orig_end = va->va_end;@@ -1823,13 +1807,55 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)atomic_long_sub (nr, &vmap_lazy_nr); num_purged_areas++; - -if (atomic_long_read (&vmap_lazy_nr) < resched_threshold) -cond_resched_lock (&free_vmap_area_lock); }spin_unlock (&free_vmap_area_lock); -out: +return num_purged_areas; +} + +/* +* Purges all lazily-freed vmap areas. +*/ +static bool __purge_vmap_area_lazy (unsigned long start,unsigned long end) +{ +unsigned long num_purged_areas =0 ; +struct vmap_node *vn; +int i; + +lockdep_assert_held (&vmap_purge_lock); + purge_nodes =CPU_MASK_NONE ; + +for (i =0 ; i < nr_nodes; i++) { + vn = &nodes[i]; + +INIT_LIST_HEAD (&vn->purge_list); + +if (RB_EMPTY_ROOT (&vn->lazy.root)) +continue ; + +spin_lock (&vn->lazy.lock); +WRITE_ONCE (vn->lazy.root.rb_node,NULL ); +list_replace_init (&vn->lazy.head, &vn->purge_list); +spin_unlock (&vn->lazy.lock); + + start =min (start,list_first_entry (&vn->purge_list, + struct vmap_area, list)->va_start); + + end =max (end,list_last_entry (&vn->purge_list, + struct vmap_area, list)->va_end); + +cpumask_set_cpu (i, &purge_nodes); + } + +if (cpumask_weight (&purge_nodes) >0 ) { +flush_tlb_kernel_range (start, end); + +for_each_cpu (i, &purge_nodes) { + vn = &nodes[i]; + num_purged_areas +=purge_vmap_node (vn); + } + } +trace_purge_vmap_area_lazy (start, end, num_purged_areas);return num_purged_areas >0 ; }@@ -1848,16 +1874,9 @@ static void reclaim_and_purge_vmap_areas(void)static void drain_vmap_area_work (struct work_struct *work) { -unsigned long nr_lazy; - -do { -mutex_lock (&vmap_purge_lock); -__purge_vmap_area_lazy (ULONG_MAX ,0 ); -mutex_unlock (&vmap_purge_lock); - -/* Recheck if further work is required. */ - nr_lazy =atomic_long_read (&vmap_lazy_nr); - }while (nr_lazy >lazy_max_pages ()); +mutex_lock (&vmap_purge_lock); +__purge_vmap_area_lazy (ULONG_MAX ,0 ); +mutex_unlock (&vmap_purge_lock); }/* @@ -1867,6 +1886,7 @@ static void drain_vmap_area_work(struct work_struct *work)*/ static void free_vmap_area_noflush (struct vmap_area *va) { +struct vmap_node *vn =addr_to_node (va->va_start);unsigned long nr_lazy_max =lazy_max_pages ();unsigned long va_start = va->va_start;unsigned long nr_lazy;@@ -1880,10 +1900,9 @@ static void free_vmap_area_noflush(struct vmap_area *va)/* * Merge or place it to the purge tree/list. */ -spin_lock (&purge_vmap_area_lock); -merge_or_add_vmap_area (va, - &purge_vmap_area_root, &purge_vmap_area_list); -spin_unlock (&purge_vmap_area_lock); +spin_lock (&vn->lazy.lock); +merge_or_add_vmap_area (va, &vn->lazy.root, &vn->lazy.head); +spin_unlock (&vn->lazy.lock);trace_free_vmap_area_noflush (va_start, nr_lazy, nr_lazy_max);@@ -4390,15 +4409,21 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v)static void show_purge_info (struct seq_file *m) { +struct vmap_node *vn;struct vmap_area *va; +int i; -spin_lock (&purge_vmap_area_lock); -list_for_each_entry (va, &purge_vmap_area_list, list) { -seq_printf (m,"0x%pK-0x%pK %7ld unpurged vm_area\n" , - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); +for (i =0 ; i < nr_nodes; i++) { + vn = &nodes[i]; + +spin_lock (&vn->lazy.lock); +list_for_each_entry (va, &vn->lazy.head, list) { +seq_printf (m,"0x%pK-0x%pK %7ld unpurged vm_area\n" , + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } +spin_unlock (&vn->lazy.lock); } -spin_unlock (&purge_vmap_area_lock); }static int s_show (struct seq_file *m,void *p)@@ -4545,6 +4570,10 @@ static void vmap_init_nodes(void) vn->busy.root =RB_ROOT ;INIT_LIST_HEAD (&vn->busy.head);spin_lock_init (&vn->busy.lock); + + vn->lazy.root =RB_ROOT ; +INIT_LIST_HEAD (&vn->lazy.head); +spin_lock_init (&vn->lazy.lock); } }Signature · 1 line
2.30.2
Re: [PATCH v2 5/9] mm: vmalloc: Remove global purge_vmap_area_root rb-tree
Quoted 9 lines · Uladzislau Rezki (Sony)
Quoted 8 lines
Similar to busy VA, lazily-freed area is stored to a node it belongs to. Such approach does not require any global locking primitive, instead an access becomes scalable what mitigates a contention.
This patch removes a global purge-lock, global purge-tree and global purge list.
Quoted 9 lines
Quoted 4 lines
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 135 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 82 insertions(+), 53 deletions(-)LGTM,
Reviewed-by Baoquan He <[email protected]>
Applied.
Thank you for review!
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerPATCH v2 6/9 mm: vmalloc: Offload free_vmap_area_lock lock
In reply to Uladzislau Rezki (Sony)Concurrent access to a global vmap space is a bottle-neck. We can simulate a high contention by running a vmalloc test suite.
To address it, introduce an effective vmap node logic. Each node behaves as independent entity. When a node is accessed it serves a request directly(if possible) also it can fetch a new block from a global heap to its internals if no space or low capacity is left.
This technique reduces a pressure on the global vmap lock.
--- mm/vmalloc.c | 316 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 279 insertions(+), 37 deletions(-)
mm/vmalloc.c +279 -37
@@ -779,6 +779,7 @@ struct rb_list {struct vmap_node {/* Bookkeeping data of this node. */ +struct rb_list free;struct rb_list busy;struct rb_list lazy;
@@ -786,6 +787,13 @@ struct vmap_node {* Ready-to-free areas. */ struct list_head purge_list; +struct work_struct purge_work; +unsigned long nr_purged; + +/* +* Control that only one user can pre-fetch this node. +*/ +atomic_t fill_in_progress; };static struct vmap_node *nodes, snode;
@@ -804,6 +812,32 @@ addr_to_node(unsigned long addr)return &nodes[addr_to_node_id (addr)]; } +static inline struct vmap_node * +id_to_node (int id) +{ +return &nodes[id % nr_nodes]; +} + +static inline int +this_node_id (void ) +{ +return raw_smp_processor_id () % nr_nodes; +} + +static inline unsigned long +encode_vn_id (int node_id) +{ +/* Can store U8_MAX [0:254] nodes. */ +return (node_id +1 ) <<BITS_PER_BYTE ; +} + +static inline int +decode_vn_id (unsigned long val) +{ +/* Can store U8_MAX [0:254] nodes. */ +return (val >>BITS_PER_BYTE ) -1 ; +} +static __always_inline unsigned long va_size (struct vmap_area *va) {
@@ -1586,6 +1620,7 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,static void free_vmap_area (struct vmap_area *va) {struct vmap_node *vn =addr_to_node (va->va_start); +int vn_id =decode_vn_id (va->flags);/* * Remove from the busy tree/list.
@@ -1594,12 +1629,19 @@ static void free_vmap_area(struct vmap_area *va)unlink_va (va, &vn->busy.root);spin_unlock (&vn->busy.lock); -/* -* Insert/Merge it back to the free tree/list. -*/ -spin_lock (&free_vmap_area_lock); -merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); -spin_unlock (&free_vmap_area_lock); +if (vn_id >=0 ) { + vn =id_to_node (vn_id); + +/* Belongs to this node. */ +spin_lock (&vn->free.lock); +merge_or_add_vmap_area_augment (va, &vn->free.root, &vn->free.head); +spin_unlock (&vn->free.lock); + }else { +/* Goes to global. */ +spin_lock (&free_vmap_area_lock); +merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); +spin_unlock (&free_vmap_area_lock); + } } static inline void
@@ -1625,6 +1667,134 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node)kmem_cache_free (vmap_area_cachep, va); } +static unsigned long +node_alloc_fill (struct vmap_node *vn, +unsigned long size,unsigned long align, +gfp_t gfp_mask,int node) +{ +struct vmap_area *va; +unsigned long addr; + + va =kmem_cache_alloc_node (vmap_area_cachep, gfp_mask, node); +if (unlikely (!va)) +return VMALLOC_END ; + +/* +* Please note, an allocated block is not aligned to its size. +* Therefore it can span several zones what means addr_to_node() +* can point to two different nodes: +* <-----> +* -|-----|-----|-----|-----|- +* 1 2 0 1 +* +* an alignment would just increase fragmentation thus more heap +* consumption what we would like to avoid. +*/ +spin_lock (&free_vmap_area_lock); + addr =__alloc_vmap_area (&free_vmap_area_root, &free_vmap_area_list, + node_size,1 ,VMALLOC_START ,VMALLOC_END ); +spin_unlock (&free_vmap_area_lock); + +if (addr ==VMALLOC_END ) { +kmem_cache_free (vmap_area_cachep, va); +return VMALLOC_END ; + } + +/* +* Statement and condition of the problem: +* +* a) where to free allocated areas from a node: +* - directly to a global heap; +* - to a node that we got a VA from; +* - what is a condition to return allocated areas +* to a global heap then; +* b) how to properly handle left small free fragments +* of a node in order to mitigate a fragmentation. +* +* How to address described points: +* When a new block is allocated(from a global heap) we shrink +* it deliberately by one page from both sides and place it to +* this node to serve a request. +* +* Why we shrink. We would like to distinguish VAs which were +* obtained from a node and a global heap. This is for a free +* path. A va->flags contains a node-id it belongs to. No VAs +* merging is possible between each other unless they are part +* of same block. +* +* A free-path in its turn can detect a correct node where a +* VA has to be returned. Thus as a block is freed entirely, +* its size becomes(merging): node_size - (2 * PAGE_SIZE) it +* recovers its edges, thus is released to a global heap for +* reuse elsewhere. In partly freed case, VAs go back to the +* node not bothering a global vmap space. +* +* 1 2 3 +* |<------------>|<------------>|<------------>| +* |..<-------->..|..<-------->..|..<-------->..| +*/ + va->va_start = addr +PAGE_SIZE ; + va->va_end = (addr + node_size) -PAGE_SIZE ; + +spin_lock (&vn->free.lock); +/* Never merges. See explanation above. */ +insert_vmap_area_augment (va,NULL , &vn->free.root, &vn->free.head); + addr =va_alloc (va, &vn->free.root, &vn->free.head, + size, align,VMALLOC_START ,VMALLOC_END ); +spin_unlock (&vn->free.lock); + +return addr; +} + +static unsigned long +node_alloc (int vn_id,unsigned long size,unsigned long align, +unsigned long vstart,unsigned long vend, +gfp_t gfp_mask,int node) +{ +struct vmap_node *vn =id_to_node (vn_id); +unsigned long extra = align >PAGE_SIZE ? align :0 ; +bool do_alloc_fill = false; +unsigned long addr; + +/* +* Fallback to a global heap if not vmalloc. +*/ +if (vstart !=VMALLOC_START || vend !=VMALLOC_END ) +return vend; + +/* +* A maximum allocation limit is 1/4 of capacity. This +* is done in order to prevent a fast depleting of zone +* by few requests. +*/ +if (size + extra > (node_size >>2 )) +return vend; + +spin_lock (&vn->free.lock); + addr =__alloc_vmap_area (&vn->free.root, &vn->free.head, + size, align, vstart, vend); + +if (addr == vend) { +/* +* Set the fetch flag under the critical section. +* This guarantees that only one user is eligible +* to perform a pre-fetch. A reset operation can +* be concurrent. +*/ +if (!atomic_xchg (&vn->fill_in_progress,1 )) + do_alloc_fill = true; + } +spin_unlock (&vn->free.lock); + +/* Only if fails a previous attempt. */ +if (do_alloc_fill) { + addr =node_alloc_fill (vn, size, align, gfp_mask, node); +atomic_set (&vn->fill_in_progress,0 ); + } + +return addr; +} +/* * Allocate a region of KVA of the specified size and alignment, within the * vstart and vend.
@@ -1640,7 +1810,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,unsigned long freed;unsigned long addr;int purged =0 ; -int ret; +int ret, vn_id;if (unlikely (!size ||offset_in_page (size) || !is_power_of_2 (align)))return ERR_PTR (-EINVAL );
@@ -1661,11 +1831,17 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,*/ kmemleak_scan_area (&va->rb_node,SIZE_MAX , gfp_mask); + vn_id =this_node_id (); + addr =node_alloc (vn_id, size, align, vstart, vend, gfp_mask, node); + va->flags = (addr != vend) ?encode_vn_id (vn_id) :0 ; + retry: -preload_this_cpu_lock (&free_vmap_area_lock, gfp_mask, node); - addr =__alloc_vmap_area (&free_vmap_area_root, &free_vmap_area_list, - size, align, vstart, vend); -spin_unlock (&free_vmap_area_lock); +if (addr == vend) { +preload_this_cpu_lock (&free_vmap_area_lock, gfp_mask, node); + addr =__alloc_vmap_area (&free_vmap_area_root, &free_vmap_area_list, + size, align, vstart, vend); +spin_unlock (&free_vmap_area_lock); + }trace_alloc_vmap_area (addr, size, align, vstart, vend, addr == vend);
@@ -1679,7 +1855,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->va_start = addr; va->va_end = addr + size; va->vm =NULL ; - va->flags = va_flags; + va->flags |= va_flags; vn =addr_to_node (va->va_start);
@@ -1772,31 +1948,58 @@ static DEFINE_MUTEX(vmap_purge_lock);static void purge_fragmented_blocks_allcpus (void );static cpumask_t purge_nodes; -/* -* Purges all lazily-freed vmap areas. -*/ -static unsigned long -purge_vmap_node (struct vmap_node *vn) +static void +reclaim_list_global (struct list_head *head) +{ +struct vmap_area *va, *n; + +if (list_empty (head)) +return ; + +spin_lock (&free_vmap_area_lock); +list_for_each_entry_safe (va, n, head, list) +merge_or_add_vmap_area_augment (va, + &free_vmap_area_root, &free_vmap_area_list); +spin_unlock (&free_vmap_area_lock); +} + +static void purge_vmap_node (struct work_struct *work) { -unsigned long num_purged_areas =0 ; +struct vmap_node *vn =container_of (work, + struct vmap_node, purge_work);struct vmap_area *va, *n_va; +LIST_HEAD (global); + + vn->nr_purged =0 ;if (list_empty (&vn->purge_list)) -return 0 ; +return ; -spin_lock (&free_vmap_area_lock); +spin_lock (&vn->free.lock);list_for_each_entry_safe (va, n_va, &vn->purge_list, list) {unsigned long nr = (va->va_end - va->va_start) >>PAGE_SHIFT ;unsigned long orig_start = va->va_start;unsigned long orig_end = va->va_end; +int vn_id =decode_vn_id (va->flags); -/* -* Finally insert or merge lazily-freed area. It is -* detached and there is no need to "unlink" it from -* anything. -*/ - va =merge_or_add_vmap_area_augment (va, &free_vmap_area_root, - &free_vmap_area_list); +list_del_init (&va->list); + +if (vn_id >=0 ) { +if (va_size (va) != node_size - (2 *PAGE_SIZE )) + va =merge_or_add_vmap_area_augment (va, &vn->free.root, &vn->free.head); + +if (va_size (va) == node_size - (2 *PAGE_SIZE )) { +if (!list_empty (&va->list)) +unlink_va_augment (va, &vn->free.root); + +/* Restore the block size. */ + va->va_start -=PAGE_SIZE ; + va->va_end +=PAGE_SIZE ; +list_add (&va->list, &global); + } + }else { +list_add (&va->list, &global); + }if (!va)continue ;
@@ -1806,11 +2009,10 @@ purge_vmap_node(struct vmap_node *vn) va->va_start, va->va_end);atomic_long_sub (nr, &vmap_lazy_nr); - num_purged_areas++; + vn->nr_purged++; } -spin_unlock (&free_vmap_area_lock); - -return num_purged_areas; +spin_unlock (&vn->free.lock); +reclaim_list_global (&global); }/*
@@ -1818,11 +2020,17 @@ purge_vmap_node(struct vmap_node *vn)*/ static bool __purge_vmap_area_lazy (unsigned long start,unsigned long end) { -unsigned long num_purged_areas =0 ; +unsigned long nr_purged_areas =0 ; +unsigned int nr_purge_helpers; +unsigned int nr_purge_nodes;struct vmap_node *vn;int i;lockdep_assert_held (&vmap_purge_lock); + +/* +* Use cpumask to mark which node has to be processed. +*/ purge_nodes =CPU_MASK_NONE ;for (i =0 ; i < nr_nodes; i++) {
@@ -1847,17 +2055,45 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)cpumask_set_cpu (i, &purge_nodes); } -if (cpumask_weight (&purge_nodes) >0 ) { + nr_purge_nodes =cpumask_weight (&purge_nodes); +if (nr_purge_nodes >0 ) {flush_tlb_kernel_range (start, end); +/* One extra worker is per a lazy_max_pages() full set minus one. */ + nr_purge_helpers =atomic_long_read (&vmap_lazy_nr) /lazy_max_pages (); + nr_purge_helpers =clamp (nr_purge_helpers,1U , nr_purge_nodes) -1 ; + +for_each_cpu (i, &purge_nodes) { + vn = &nodes[i]; + +if (nr_purge_helpers >0 ) { +INIT_WORK (&vn->purge_work, purge_vmap_node); + +if (cpumask_test_cpu (i, cpu_online_mask)) +schedule_work_on (i, &vn->purge_work); +else +schedule_work (&vn->purge_work); + + nr_purge_helpers--; + }else { + vn->purge_work.func =NULL ; +purge_vmap_node (&vn->purge_work); + nr_purged_areas += vn->nr_purged; + } + } +for_each_cpu (i, &purge_nodes) { vn = &nodes[i]; - num_purged_areas +=purge_vmap_node (vn); + +if (vn->purge_work.func) { +flush_work (&vn->purge_work); + nr_purged_areas += vn->nr_purged; + } } } -trace_purge_vmap_area_lazy (start, end, num_purged_areas); -return num_purged_areas >0 ; +trace_purge_vmap_area_lazy (start, end, nr_purged_areas); +return nr_purged_areas >0 ; }/*
@@ -1886,9 +2122,11 @@ static void drain_vmap_area_work(struct work_struct *work)*/ static void free_vmap_area_noflush (struct vmap_area *va) { -struct vmap_node *vn =addr_to_node (va->va_start);unsigned long nr_lazy_max =lazy_max_pages ();unsigned long va_start = va->va_start; +int vn_id =decode_vn_id (va->flags); +struct vmap_node *vn = vn_id >=0 ?id_to_node (vn_id): +addr_to_node (va->va_start);;unsigned long nr_lazy;if (WARN_ON_ONCE (!list_empty (&va->list)))
@@ -4574,6 +4812,10 @@ static void vmap_init_nodes(void) vn->lazy.root =RB_ROOT ;INIT_LIST_HEAD (&vn->lazy.head);spin_lock_init (&vn->lazy.lock); + + vn->free.root =RB_ROOT ; +INIT_LIST_HEAD (&vn->free.head); +spin_lock_init (&vn->free.lock); } }
Signature · 1 line
2.30.2
Re: [PATCH v2 6/9] mm: vmalloc: Offload free_vmap_area_lock lock
Quoted 85 lines · Uladzislau Rezki (Sony)
Concurrent access to a global vmap space is a bottle-neck. We can simulate a high contention by running a vmalloc test suite.
To address it, introduce an effective vmap node logic. Each node behaves as independent entity. When a node is accessed it serves a request directly(if possible) also it can fetch a new block from a global heap to its internals if no space or low capacity is left.
This technique reduces a pressure on the global vmap lock.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 316 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 279 insertions(+), 37 deletions(-)mm/vmalloc.c +35
-0@@ -779,6 +779,7 @@ struct rb_list {struct vmap_node {/* Bookkeeping data of this node. */ +struct rb_list free;struct rb_list busy;struct rb_list lazy;@@ -786,6 +787,13 @@ struct vmap_node {* Ready-to-free areas. */ struct list_head purge_list; +struct work_struct purge_work; +unsigned long nr_purged; + +/* +* Control that only one user can pre-fetch this node. +*/ +atomic_t fill_in_progress; };static struct vmap_node *nodes, snode;@@ -804,6 +812,32 @@ addr_to_node(unsigned long addr)return &nodes[addr_to_node_id (addr)]; } +static inline struct vmap_node * +id_to_node (int id) +{ +return &nodes[id % nr_nodes]; +} + +static inline int +this_node_id (void ) +{ +return raw_smp_processor_id () % nr_nodes; +} + +static inline unsigned long +encode_vn_id (int node_id) +{ +/* Can store U8_MAX [0:254] nodes. */ +return (node_id +1 ) <<BITS_PER_BYTE ; +} + +static inline int +decode_vn_id (unsigned long val) +{ +/* Can store U8_MAX [0:254] nodes. */ +return (val >>BITS_PER_BYTE ) -1 ; +} +static __always_inline unsigned long va_size (struct vmap_area *va) {@@ -1586,6 +1620,7 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,static void free_vmap_area (struct vmap_area *va) {struct vmap_node *vn =addr_to_node (va->va_start); +int vn_id =decode_vn_id (va->flags);/* * Remove from the busy tree/list.
mm/vmalloc.c static void free_vmap_area(struct vmap_area *va) · line 1632
Quoted 10 lines
mm/vmalloc.c +1
-6@@ -1594,12 +1629,19 @@ static void free_vmap_area(struct vmap_area *va)unlink_va (va, &vn->busy.root);spin_unlock (&vn->busy.lock); -/* -* Insert/Merge it back to the free tree/list. -*/ -spin_lock (&free_vmap_area_lock); -merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); -spin_unlock (&free_vmap_area_lock); +if (vn_id >=0 ) {
In alloc_vmap_area(), the vn_id is encoded into va->flags. When allocation failed, the vn_id = 0. Here should we change to check 'if (vn_id > 0)' becasue the vn_id == 0 means no available vn_id encoded into. And I do not get how we treat the case vn_id truly is 0.
va->flags = (addr != vend) ? encode_vn_id(vn_id) : 0;
Quoted 15 lines
mm/vmalloc.c +12
-0+ vn =id_to_node (vn_id); + +/* Belongs to this node. */ +spin_lock (&vn->free.lock); +merge_or_add_vmap_area_augment (va, &vn->free.root, &vn->free.head); +spin_unlock (&vn->free.lock); + }else { +/* Goes to global. */ +spin_lock (&free_vmap_area_lock); +merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); +spin_unlock (&free_vmap_area_lock); + } } static inline void
......
Quoted 225 lines
mm/vmalloc.c +103
-31@@ -1640,7 +1810,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,unsigned long freed;unsigned long addr;int purged =0 ; -int ret; +int ret, vn_id;if (unlikely (!size ||offset_in_page (size) || !is_power_of_2 (align)))return ERR_PTR (-EINVAL );@@ -1661,11 +1831,17 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,*/ kmemleak_scan_area (&va->rb_node,SIZE_MAX , gfp_mask); + vn_id =this_node_id (); + addr =node_alloc (vn_id, size, align, vstart, vend, gfp_mask, node); + va->flags = (addr != vend) ?encode_vn_id (vn_id) :0 ; + retry: -preload_this_cpu_lock (&free_vmap_area_lock, gfp_mask, node); - addr =__alloc_vmap_area (&free_vmap_area_root, &free_vmap_area_list, - size, align, vstart, vend); -spin_unlock (&free_vmap_area_lock); +if (addr == vend) { +preload_this_cpu_lock (&free_vmap_area_lock, gfp_mask, node); + addr =__alloc_vmap_area (&free_vmap_area_root, &free_vmap_area_list, + size, align, vstart, vend); +spin_unlock (&free_vmap_area_lock); + }trace_alloc_vmap_area (addr, size, align, vstart, vend, addr == vend);@@ -1679,7 +1855,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->va_start = addr; va->va_end = addr + size; va->vm =NULL ; - va->flags = va_flags; + va->flags |= va_flags; vn =addr_to_node (va->va_start);@@ -1772,31 +1948,58 @@ static DEFINE_MUTEX(vmap_purge_lock);static void purge_fragmented_blocks_allcpus (void );static cpumask_t purge_nodes; -/* -* Purges all lazily-freed vmap areas. -*/ -static unsigned long -purge_vmap_node (struct vmap_node *vn) +static void +reclaim_list_global (struct list_head *head) +{ +struct vmap_area *va, *n; + +if (list_empty (head)) +return ; + +spin_lock (&free_vmap_area_lock); +list_for_each_entry_safe (va, n, head, list) +merge_or_add_vmap_area_augment (va, + &free_vmap_area_root, &free_vmap_area_list); +spin_unlock (&free_vmap_area_lock); +} + +static void purge_vmap_node (struct work_struct *work) { -unsigned long num_purged_areas =0 ; +struct vmap_node *vn =container_of (work, + struct vmap_node, purge_work);struct vmap_area *va, *n_va; +LIST_HEAD (global); + + vn->nr_purged =0 ;if (list_empty (&vn->purge_list)) -return 0 ; +return ; -spin_lock (&free_vmap_area_lock); +spin_lock (&vn->free.lock);list_for_each_entry_safe (va, n_va, &vn->purge_list, list) {unsigned long nr = (va->va_end - va->va_start) >>PAGE_SHIFT ;unsigned long orig_start = va->va_start;unsigned long orig_end = va->va_end; +int vn_id =decode_vn_id (va->flags); -/* -* Finally insert or merge lazily-freed area. It is -* detached and there is no need to "unlink" it from -* anything. -*/ - va =merge_or_add_vmap_area_augment (va, &free_vmap_area_root, - &free_vmap_area_list); +list_del_init (&va->list); + +if (vn_id >=0 ) { +if (va_size (va) != node_size - (2 *PAGE_SIZE )) + va =merge_or_add_vmap_area_augment (va, &vn->free.root, &vn->free.head); + +if (va_size (va) == node_size - (2 *PAGE_SIZE )) { +if (!list_empty (&va->list)) +unlink_va_augment (va, &vn->free.root); + +/* Restore the block size. */ + va->va_start -=PAGE_SIZE ; + va->va_end +=PAGE_SIZE ; +list_add (&va->list, &global); + } + }else { +list_add (&va->list, &global); + }if (!va)continue ;@@ -1806,11 +2009,10 @@ purge_vmap_node(struct vmap_node *vn) va->va_start, va->va_end);atomic_long_sub (nr, &vmap_lazy_nr); - num_purged_areas++; + vn->nr_purged++; } -spin_unlock (&free_vmap_area_lock); - -return num_purged_areas; +spin_unlock (&vn->free.lock); +reclaim_list_global (&global); }/* @@ -1818,11 +2020,17 @@ purge_vmap_node(struct vmap_node *vn)*/ static bool __purge_vmap_area_lazy (unsigned long start,unsigned long end) { -unsigned long num_purged_areas =0 ; +unsigned long nr_purged_areas =0 ; +unsigned int nr_purge_helpers; +unsigned int nr_purge_nodes;struct vmap_node *vn;int i;lockdep_assert_held (&vmap_purge_lock); + +/* +* Use cpumask to mark which node has to be processed. +*/ purge_nodes =CPU_MASK_NONE ;for (i =0 ; i < nr_nodes; i++) {@@ -1847,17 +2055,45 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)cpumask_set_cpu (i, &purge_nodes); } -if (cpumask_weight (&purge_nodes) >0 ) { + nr_purge_nodes =cpumask_weight (&purge_nodes); +if (nr_purge_nodes >0 ) {flush_tlb_kernel_range (start, end); +/* One extra worker is per a lazy_max_pages() full set minus one. */ + nr_purge_helpers =atomic_long_read (&vmap_lazy_nr) /lazy_max_pages (); + nr_purge_helpers =clamp (nr_purge_helpers,1U , nr_purge_nodes) -1 ; + +for_each_cpu (i, &purge_nodes) { + vn = &nodes[i]; + +if (nr_purge_helpers >0 ) { +INIT_WORK (&vn->purge_work, purge_vmap_node); + +if (cpumask_test_cpu (i, cpu_online_mask)) +schedule_work_on (i, &vn->purge_work); +else +schedule_work (&vn->purge_work); + + nr_purge_helpers--; + }else { + vn->purge_work.func =NULL ; +purge_vmap_node (&vn->purge_work); + nr_purged_areas += vn->nr_purged; + } + } +for_each_cpu (i, &purge_nodes) { vn = &nodes[i]; - num_purged_areas +=purge_vmap_node (vn); + +if (vn->purge_work.func) { +flush_work (&vn->purge_work); + nr_purged_areas += vn->nr_purged; + } } } -trace_purge_vmap_area_lazy (start, end, num_purged_areas); -return num_purged_areas >0 ; +trace_purge_vmap_area_lazy (start, end, nr_purged_areas); +return nr_purged_areas >0 ; }/* @@ -1886,9 +2122,11 @@ static void drain_vmap_area_work(struct work_struct *work)*/ static void free_vmap_area_noflush (struct vmap_area *va) { -struct vmap_node *vn =addr_to_node (va->va_start);unsigned long nr_lazy_max =lazy_max_pages ();unsigned long va_start = va->va_start; +int vn_id =decode_vn_id (va->flags); +struct vmap_node *vn = vn_id >=0 ?id_to_node (vn_id): +addr_to_node (va->va_start);;unsigned long nr_lazy;if (WARN_ON_ONCE (!list_empty (&va->list)))@@ -4574,6 +4812,10 @@ static void vmap_init_nodes(void) vn->lazy.root =RB_ROOT ;INIT_LIST_HEAD (&vn->lazy.head);spin_lock_init (&vn->lazy.lock); + + vn->free.root =RB_ROOT ; +INIT_LIST_HEAD (&vn->free.head); +spin_lock_init (&vn->free.lock); } }Signature · 1 line
2.30.2
Uladzislau Rezki
maintainerRe: [PATCH v2 6/9] mm: vmalloc: Offload free_vmap_area_lock lock
Quoted 24 lines
Quoted 8 lines
static void
free_vmap_area(struct vmap_area *va)
{mm/vmalloc.c +1
-0struct vmap_node *vn =addr_to_node (va->va_start); +int vn_id =decode_vn_id (va->flags);/* * Remove from the busy tree/list. mm/vmalloc.c static void free_vmap_area(struct vmap_area *va) · line 1632
Quoted 10 lines
mm/vmalloc.c +1
-6@@ -1594,12 +1629,19 @@ static void free_vmap_area(struct vmap_area *va)unlink_va (va, &vn->busy.root);spin_unlock (&vn->busy.lock); -/* -* Insert/Merge it back to the free tree/list. -*/ -spin_lock (&free_vmap_area_lock); -merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); -spin_unlock (&free_vmap_area_lock); +if (vn_id >=0 ) {In
alloc_vmap_area(), thevn_idis encoded intova->flags. When allocation failed, thevn_id= 0. Here should we change to check 'if (vn_id> 0)' becasue thevn_id== 0 means no availablevn_idencoded into. And I do not get how we treat the casevn_idtruly is 0.
Quoted 2 lines
va->flags = (addr != vend) ? encode_vn_id(vn_id) : 0;
Yes, vn_id always >= 0, so it is positive since it is an index. We encode a vn_id as vn_id + 1. For example if it is zero we write 1.
If not node allocation path or an error zero is written. Decoding is done as: zero - 1 = -1, so it is negative value, i.e. decode_vn_id() function returns -1.
--
Uladzislau Rezki
Re: [PATCH v2 6/9] mm: vmalloc: Offload free_vmap_area_lock lock
Quoted 24 lines · Uladzislau Rezki
Quoted 24 lines
Quoted 8 lines
static void
free_vmap_area(struct vmap_area *va)
{mm/vmalloc.c +1
-0struct vmap_node *vn =addr_to_node (va->va_start); +int vn_id =decode_vn_id (va->flags);/* * Remove from the busy tree/list. mm/vmalloc.c static void free_vmap_area(struct vmap_area *va) · line 1632
Quoted 10 lines
mm/vmalloc.c +1
-6@@ -1594,12 +1629,19 @@ static void free_vmap_area(struct vmap_area *va)unlink_va (va, &vn->busy.root);spin_unlock (&vn->busy.lock); -/* -* Insert/Merge it back to the free tree/list. -*/ -spin_lock (&free_vmap_area_lock); -merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); -spin_unlock (&free_vmap_area_lock); +if (vn_id >=0 ) {In
alloc_vmap_area(), thevn_idis encoded intova->flags. When allocation failed, thevn_id= 0. Here should we change to check 'if (vn_id> 0)' becasue thevn_id== 0 means no availablevn_idencoded into. And I do not get how we treat the casevn_idtruly is 0.
Quoted 8 lines
Quoted 2 lines
va->flags = (addr != vend) ? encode_vn_id(vn_id) : 0;Yes,
vn_idalways >= 0, so it is positive since it is an index. We encode avn_idasvn_id+ 1. For example if it is zero we write 1.If not node allocation path or an error zero is written. Decoding is done as: zero - 1 = -1, so it is negative value, i.e.
decode_vn_id()function returns -1.
Ah, I see it now, thanks. It would be helpful to add some explanation above decode_vn_id() lest people misunderstand this like me?
Uladzislau Rezki
maintainerRe: [PATCH v2 6/9] mm: vmalloc: Offload free_vmap_area_lock lock
Quoted 30 lines · Baoquan He
Quoted 29 lines · Uladzislau Rezki
Quoted 24 lines
Quoted 8 lines
static void
free_vmap_area(struct vmap_area *va)
{mm/vmalloc.c +1
-0struct vmap_node *vn =addr_to_node (va->va_start); +int vn_id =decode_vn_id (va->flags);/* * Remove from the busy tree/list. mm/vmalloc.c static void free_vmap_area(struct vmap_area *va) · line 1632
Quoted 10 lines
mm/vmalloc.c +1
-6@@ -1594,12 +1629,19 @@ static void free_vmap_area(struct vmap_area *va)unlink_va (va, &vn->busy.root);spin_unlock (&vn->busy.lock); -/* -* Insert/Merge it back to the free tree/list. -*/ -spin_lock (&free_vmap_area_lock); -merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); -spin_unlock (&free_vmap_area_lock); +if (vn_id >=0 ) {In
alloc_vmap_area(), thevn_idis encoded intova->flags. When allocation failed, thevn_id= 0. Here should we change to check 'if (vn_id> 0)' becasue thevn_id== 0 means no availablevn_idencoded into. And I do not get how we treat the casevn_idtruly is 0.Quoted 2 lines
va->flags = (addr != vend) ? encode_vn_id(vn_id) : 0;Yes,
vn_idalways >= 0, so it is positive since it is an index. We encode avn_idasvn_id+ 1. For example if it is zero we write 1.
Quoted 7 lines
Quoted 3 lines
If not node allocation path or an error zero is written. Decoding is done as: zero - 1 = -1, so it is negative value, i.e.
decode_vn_id()function returns -1.Ah, I see it now, thanks. It would be helpful to add some explanation above
decode_vn_id()lest people misunderstand this like me?
I got that feeling also. This makes sense, so i will comment it!
--
Uladzislau Rezki
Re: [PATCH v2 6/9] mm: vmalloc: Offload free_vmap_area_lock lock
In reply to Uladzislau Rezki (Sony)Quoted 62 lines · Uladzislau Rezki (Sony)
Concurrent access to a global vmap space is a bottle-neck. We can simulate a high contention by running a vmalloc test suite.
To address it, introduce an effective vmap node logic. Each node behaves as independent entity. When a node is accessed it serves a request directly(if possible) also it can fetch a new block from a global heap to its internals if no space or low capacity is left.
This technique reduces a pressure on the global vmap lock.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 316 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 279 insertions(+), 37 deletions(-)mm/vmalloc.c +23
-0@@ -779,6 +779,7 @@ struct rb_list {struct vmap_node {/* Bookkeeping data of this node. */ +struct rb_list free;struct rb_list busy;struct rb_list lazy;@@ -786,6 +787,13 @@ struct vmap_node {* Ready-to-free areas. */ struct list_head purge_list; +struct work_struct purge_work; +unsigned long nr_purged; + +/* +* Control that only one user can pre-fetch this node. +*/ +atomic_t fill_in_progress; };static struct vmap_node *nodes, snode;@@ -804,6 +812,32 @@ addr_to_node(unsigned long addr)return &nodes[addr_to_node_id (addr)]; } +static inline struct vmap_node * +id_to_node (int id) +{ +return &nodes[id % nr_nodes]; +} + +static inline int +this_node_id (void ) +{ +return raw_smp_processor_id () % nr_nodes; +} + +static inline unsigned long +encode_vn_id (int node_id) +{
mm/vmalloc.c addr_to_node(unsigned long addr) · line 839
Quoted 10 lines
mm/vmalloc.c +10
-0+/* Can store U8_MAX [0:254] nodes. */ +return (node_id +1 ) <<BITS_PER_BYTE ; +} + +static inline int +decode_vn_id (unsigned long val) +{ +/* Can store U8_MAX [0:254] nodes. */ +return (val >>BITS_PER_BYTE ) -1 ; +}
This patch looks good to me. However, should we split out the encoding vn_id into va->flags optimization into another patch? It looks like an independent optimization which can be described better with specific log. At least, in the pdf file pasted or patch log, it's not obvious that:
1) node's free tree could contains any address range;
2) nodes' busy tree only contains address range belonging to this node;
- could contain crossing node range, corner case.
3) nodes' purge tree could contain any address range;
- decided by encoded vn_id in va->flags.
- decided by address via addr_to_node(va->va_start).
Personal opinion, feel it will make reviewing easier.
Quoted 398 lines
mm/vmalloc.c +246
-37+static __always_inline unsigned long va_size (struct vmap_area *va) {@@ -1586,6 +1620,7 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,static void free_vmap_area (struct vmap_area *va) {struct vmap_node *vn =addr_to_node (va->va_start); +int vn_id =decode_vn_id (va->flags);/* * Remove from the busy tree/list. @@ -1594,12 +1629,19 @@ static void free_vmap_area(struct vmap_area *va)unlink_va (va, &vn->busy.root);spin_unlock (&vn->busy.lock); -/* -* Insert/Merge it back to the free tree/list. -*/ -spin_lock (&free_vmap_area_lock); -merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); -spin_unlock (&free_vmap_area_lock); +if (vn_id >=0 ) { + vn =id_to_node (vn_id); + +/* Belongs to this node. */ +spin_lock (&vn->free.lock); +merge_or_add_vmap_area_augment (va, &vn->free.root, &vn->free.head); +spin_unlock (&vn->free.lock); + }else { +/* Goes to global. */ +spin_lock (&free_vmap_area_lock); +merge_or_add_vmap_area_augment (va, &free_vmap_area_root, &free_vmap_area_list); +spin_unlock (&free_vmap_area_lock); + } } static inline void@@ -1625,6 +1667,134 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node)kmem_cache_free (vmap_area_cachep, va); } +static unsigned long +node_alloc_fill (struct vmap_node *vn, +unsigned long size,unsigned long align, +gfp_t gfp_mask,int node) +{ +struct vmap_area *va; +unsigned long addr; + + va =kmem_cache_alloc_node (vmap_area_cachep, gfp_mask, node); +if (unlikely (!va)) +return VMALLOC_END ; + +/* +* Please note, an allocated block is not aligned to its size. +* Therefore it can span several zones what means addr_to_node() +* can point to two different nodes: +* <-----> +* -|-----|-----|-----|-----|- +* 1 2 0 1 +* +* an alignment would just increase fragmentation thus more heap +* consumption what we would like to avoid. +*/ +spin_lock (&free_vmap_area_lock); + addr =__alloc_vmap_area (&free_vmap_area_root, &free_vmap_area_list, + node_size,1 ,VMALLOC_START ,VMALLOC_END ); +spin_unlock (&free_vmap_area_lock); + +if (addr ==VMALLOC_END ) { +kmem_cache_free (vmap_area_cachep, va); +return VMALLOC_END ; + } + +/* +* Statement and condition of the problem: +* +* a) where to free allocated areas from a node: +* - directly to a global heap; +* - to a node that we got a VA from; +* - what is a condition to return allocated areas +* to a global heap then; +* b) how to properly handle left small free fragments +* of a node in order to mitigate a fragmentation. +* +* How to address described points: +* When a new block is allocated(from a global heap) we shrink +* it deliberately by one page from both sides and place it to +* this node to serve a request. +* +* Why we shrink. We would like to distinguish VAs which were +* obtained from a node and a global heap. This is for a free +* path. A va->flags contains a node-id it belongs to. No VAs +* merging is possible between each other unless they are part +* of same block. +* +* A free-path in its turn can detect a correct node where a +* VA has to be returned. Thus as a block is freed entirely, +* its size becomes(merging): node_size - (2 * PAGE_SIZE) it +* recovers its edges, thus is released to a global heap for +* reuse elsewhere. In partly freed case, VAs go back to the +* node not bothering a global vmap space. +* +* 1 2 3 +* |<------------>|<------------>|<------------>| +* |..<-------->..|..<-------->..|..<-------->..| +*/ + va->va_start = addr +PAGE_SIZE ; + va->va_end = (addr + node_size) -PAGE_SIZE ; + +spin_lock (&vn->free.lock); +/* Never merges. See explanation above. */ +insert_vmap_area_augment (va,NULL , &vn->free.root, &vn->free.head); + addr =va_alloc (va, &vn->free.root, &vn->free.head, + size, align,VMALLOC_START ,VMALLOC_END ); +spin_unlock (&vn->free.lock); + +return addr; +} + +static unsigned long +node_alloc (int vn_id,unsigned long size,unsigned long align, +unsigned long vstart,unsigned long vend, +gfp_t gfp_mask,int node) +{ +struct vmap_node *vn =id_to_node (vn_id); +unsigned long extra = align >PAGE_SIZE ? align :0 ; +bool do_alloc_fill = false; +unsigned long addr; + +/* +* Fallback to a global heap if not vmalloc. +*/ +if (vstart !=VMALLOC_START || vend !=VMALLOC_END ) +return vend; + +/* +* A maximum allocation limit is 1/4 of capacity. This +* is done in order to prevent a fast depleting of zone +* by few requests. +*/ +if (size + extra > (node_size >>2 )) +return vend; + +spin_lock (&vn->free.lock); + addr =__alloc_vmap_area (&vn->free.root, &vn->free.head, + size, align, vstart, vend); + +if (addr == vend) { +/* +* Set the fetch flag under the critical section. +* This guarantees that only one user is eligible +* to perform a pre-fetch. A reset operation can +* be concurrent. +*/ +if (!atomic_xchg (&vn->fill_in_progress,1 )) + do_alloc_fill = true; + } +spin_unlock (&vn->free.lock); + +/* Only if fails a previous attempt. */ +if (do_alloc_fill) { + addr =node_alloc_fill (vn, size, align, gfp_mask, node); +atomic_set (&vn->fill_in_progress,0 ); + } + +return addr; +} +/* * Allocate a region of KVA of the specified size and alignment, within the * vstart and vend. @@ -1640,7 +1810,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,unsigned long freed;unsigned long addr;int purged =0 ; -int ret; +int ret, vn_id;if (unlikely (!size ||offset_in_page (size) || !is_power_of_2 (align)))return ERR_PTR (-EINVAL );@@ -1661,11 +1831,17 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,*/ kmemleak_scan_area (&va->rb_node,SIZE_MAX , gfp_mask); + vn_id =this_node_id (); + addr =node_alloc (vn_id, size, align, vstart, vend, gfp_mask, node); + va->flags = (addr != vend) ?encode_vn_id (vn_id) :0 ; + retry: -preload_this_cpu_lock (&free_vmap_area_lock, gfp_mask, node); - addr =__alloc_vmap_area (&free_vmap_area_root, &free_vmap_area_list, - size, align, vstart, vend); -spin_unlock (&free_vmap_area_lock); +if (addr == vend) { +preload_this_cpu_lock (&free_vmap_area_lock, gfp_mask, node); + addr =__alloc_vmap_area (&free_vmap_area_root, &free_vmap_area_list, + size, align, vstart, vend); +spin_unlock (&free_vmap_area_lock); + }trace_alloc_vmap_area (addr, size, align, vstart, vend, addr == vend);@@ -1679,7 +1855,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->va_start = addr; va->va_end = addr + size; va->vm =NULL ; - va->flags = va_flags; + va->flags |= va_flags; vn =addr_to_node (va->va_start);@@ -1772,31 +1948,58 @@ static DEFINE_MUTEX(vmap_purge_lock);static void purge_fragmented_blocks_allcpus (void );static cpumask_t purge_nodes; -/* -* Purges all lazily-freed vmap areas. -*/ -static unsigned long -purge_vmap_node (struct vmap_node *vn) +static void +reclaim_list_global (struct list_head *head) +{ +struct vmap_area *va, *n; + +if (list_empty (head)) +return ; + +spin_lock (&free_vmap_area_lock); +list_for_each_entry_safe (va, n, head, list) +merge_or_add_vmap_area_augment (va, + &free_vmap_area_root, &free_vmap_area_list); +spin_unlock (&free_vmap_area_lock); +} + +static void purge_vmap_node (struct work_struct *work) { -unsigned long num_purged_areas =0 ; +struct vmap_node *vn =container_of (work, + struct vmap_node, purge_work);struct vmap_area *va, *n_va; +LIST_HEAD (global); + + vn->nr_purged =0 ;if (list_empty (&vn->purge_list)) -return 0 ; +return ; -spin_lock (&free_vmap_area_lock); +spin_lock (&vn->free.lock);list_for_each_entry_safe (va, n_va, &vn->purge_list, list) {unsigned long nr = (va->va_end - va->va_start) >>PAGE_SHIFT ;unsigned long orig_start = va->va_start;unsigned long orig_end = va->va_end; +int vn_id =decode_vn_id (va->flags); -/* -* Finally insert or merge lazily-freed area. It is -* detached and there is no need to "unlink" it from -* anything. -*/ - va =merge_or_add_vmap_area_augment (va, &free_vmap_area_root, - &free_vmap_area_list); +list_del_init (&va->list); + +if (vn_id >=0 ) { +if (va_size (va) != node_size - (2 *PAGE_SIZE )) + va =merge_or_add_vmap_area_augment (va, &vn->free.root, &vn->free.head); + +if (va_size (va) == node_size - (2 *PAGE_SIZE )) { +if (!list_empty (&va->list)) +unlink_va_augment (va, &vn->free.root); + +/* Restore the block size. */ + va->va_start -=PAGE_SIZE ; + va->va_end +=PAGE_SIZE ; +list_add (&va->list, &global); + } + }else { +list_add (&va->list, &global); + }if (!va)continue ;@@ -1806,11 +2009,10 @@ purge_vmap_node(struct vmap_node *vn) va->va_start, va->va_end);atomic_long_sub (nr, &vmap_lazy_nr); - num_purged_areas++; + vn->nr_purged++; } -spin_unlock (&free_vmap_area_lock); - -return num_purged_areas; +spin_unlock (&vn->free.lock); +reclaim_list_global (&global); }/* @@ -1818,11 +2020,17 @@ purge_vmap_node(struct vmap_node *vn)*/ static bool __purge_vmap_area_lazy (unsigned long start,unsigned long end) { -unsigned long num_purged_areas =0 ; +unsigned long nr_purged_areas =0 ; +unsigned int nr_purge_helpers; +unsigned int nr_purge_nodes;struct vmap_node *vn;int i;lockdep_assert_held (&vmap_purge_lock); + +/* +* Use cpumask to mark which node has to be processed. +*/ purge_nodes =CPU_MASK_NONE ;for (i =0 ; i < nr_nodes; i++) {@@ -1847,17 +2055,45 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)cpumask_set_cpu (i, &purge_nodes); } -if (cpumask_weight (&purge_nodes) >0 ) { + nr_purge_nodes =cpumask_weight (&purge_nodes); +if (nr_purge_nodes >0 ) {flush_tlb_kernel_range (start, end); +/* One extra worker is per a lazy_max_pages() full set minus one. */ + nr_purge_helpers =atomic_long_read (&vmap_lazy_nr) /lazy_max_pages (); + nr_purge_helpers =clamp (nr_purge_helpers,1U , nr_purge_nodes) -1 ; + +for_each_cpu (i, &purge_nodes) { + vn = &nodes[i]; + +if (nr_purge_helpers >0 ) { +INIT_WORK (&vn->purge_work, purge_vmap_node); + +if (cpumask_test_cpu (i, cpu_online_mask)) +schedule_work_on (i, &vn->purge_work); +else +schedule_work (&vn->purge_work); + + nr_purge_helpers--; + }else { + vn->purge_work.func =NULL ; +purge_vmap_node (&vn->purge_work); + nr_purged_areas += vn->nr_purged; + } + } +for_each_cpu (i, &purge_nodes) { vn = &nodes[i]; - num_purged_areas +=purge_vmap_node (vn); + +if (vn->purge_work.func) { +flush_work (&vn->purge_work); + nr_purged_areas += vn->nr_purged; + } } } -trace_purge_vmap_area_lazy (start, end, num_purged_areas); -return num_purged_areas >0 ; +trace_purge_vmap_area_lazy (start, end, nr_purged_areas); +return nr_purged_areas >0 ; }/* @@ -1886,9 +2122,11 @@ static void drain_vmap_area_work(struct work_struct *work)*/ static void free_vmap_area_noflush (struct vmap_area *va) { -struct vmap_node *vn =addr_to_node (va->va_start);unsigned long nr_lazy_max =lazy_max_pages ();unsigned long va_start = va->va_start; +int vn_id =decode_vn_id (va->flags); +struct vmap_node *vn = vn_id >=0 ?id_to_node (vn_id): +addr_to_node (va->va_start);;unsigned long nr_lazy;if (WARN_ON_ONCE (!list_empty (&va->list)))@@ -4574,6 +4812,10 @@ static void vmap_init_nodes(void) vn->lazy.root =RB_ROOT ;INIT_LIST_HEAD (&vn->lazy.head);spin_lock_init (&vn->lazy.lock); + + vn->free.root =RB_ROOT ; +INIT_LIST_HEAD (&vn->free.head); +spin_lock_init (&vn->free.lock); } }Signature · 1 line
2.30.2
Uladzislau Rezki
maintainerRe: [PATCH v2 6/9] mm: vmalloc: Offload free_vmap_area_lock lock
Quoted 86 lines · Baoquan He
Quoted 62 lines · Uladzislau Rezki (Sony)
Concurrent access to a global vmap space is a bottle-neck. We can simulate a high contention by running a vmalloc test suite.
To address it, introduce an effective vmap node logic. Each node behaves as independent entity. When a node is accessed it serves a request directly(if possible) also it can fetch a new block from a global heap to its internals if no space or low capacity is left.
This technique reduces a pressure on the global vmap lock.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 316 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 279 insertions(+), 37 deletions(-)mm/vmalloc.c +23
-0@@ -779,6 +779,7 @@ struct rb_list {struct vmap_node {/* Bookkeeping data of this node. */ +struct rb_list free;struct rb_list busy;struct rb_list lazy;@@ -786,6 +787,13 @@ struct vmap_node {* Ready-to-free areas. */ struct list_head purge_list; +struct work_struct purge_work; +unsigned long nr_purged; + +/* +* Control that only one user can pre-fetch this node. +*/ +atomic_t fill_in_progress; };static struct vmap_node *nodes, snode;@@ -804,6 +812,32 @@ addr_to_node(unsigned long addr)return &nodes[addr_to_node_id (addr)]; } +static inline struct vmap_node * +id_to_node (int id) +{ +return &nodes[id % nr_nodes]; +} + +static inline int +this_node_id (void ) +{ +return raw_smp_processor_id () % nr_nodes; +} + +static inline unsigned long +encode_vn_id (int node_id) +{mm/vmalloc.c addr_to_node(unsigned long addr) · line 839
Quoted 10 lines
mm/vmalloc.c +10
-0+/* Can store U8_MAX [0:254] nodes. */ +return (node_id +1 ) <<BITS_PER_BYTE ; +} + +static inline int +decode_vn_id (unsigned long val) +{ +/* Can store U8_MAX [0:254] nodes. */ +return (val >>BITS_PER_BYTE ) -1 ; +}This patch looks good to me. However, should we split out the encoding
vn_idintova->flagsoptimization into another patch? It looks like an independent optimization which can be described better with specific log. At least, in the pdf file pasted or patch log, it's not obvious that:
1) node's free tree could contains any address range;
2) nodes' busy tree only contains address range belonging to this node;
- could contain crossing node range, corner case.
3) nodes' purge tree could contain any address range;
- decided by encodedvn_idinva->flags.
- decided by address viaaddr_to_node(va->va_start).
Quoted 2 lines
Personal opinion, feel it will make reviewing easier.
Sure, if it is easier to review, then i will split these two parts. All three statements are correct and valid. The pdf file only covers v1, so it is not up to date.
Anyway i will update a cover letter in v3 with more details.
--
Uladzislau Rezki
Re: [PATCH v2 6/9] mm: vmalloc: Offload free_vmap_area_lock lock
Quoted 87 lines · Uladzislau Rezki
Quoted 86 lines · Baoquan He
Quoted 62 lines · Uladzislau Rezki (Sony)
Concurrent access to a global vmap space is a bottle-neck. We can simulate a high contention by running a vmalloc test suite.
To address it, introduce an effective vmap node logic. Each node behaves as independent entity. When a node is accessed it serves a request directly(if possible) also it can fetch a new block from a global heap to its internals if no space or low capacity is left.
This technique reduces a pressure on the global vmap lock.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 316 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 279 insertions(+), 37 deletions(-)mm/vmalloc.c +23
-0@@ -779,6 +779,7 @@ struct rb_list {struct vmap_node {/* Bookkeeping data of this node. */ +struct rb_list free;struct rb_list busy;struct rb_list lazy;@@ -786,6 +787,13 @@ struct vmap_node {* Ready-to-free areas. */ struct list_head purge_list; +struct work_struct purge_work; +unsigned long nr_purged; + +/* +* Control that only one user can pre-fetch this node. +*/ +atomic_t fill_in_progress; };static struct vmap_node *nodes, snode;@@ -804,6 +812,32 @@ addr_to_node(unsigned long addr)return &nodes[addr_to_node_id (addr)]; } +static inline struct vmap_node * +id_to_node (int id) +{ +return &nodes[id % nr_nodes]; +} + +static inline int +this_node_id (void ) +{ +return raw_smp_processor_id () % nr_nodes; +} + +static inline unsigned long +encode_vn_id (int node_id) +{mm/vmalloc.c addr_to_node(unsigned long addr) · line 839
Quoted 10 lines
mm/vmalloc.c +10
-0+/* Can store U8_MAX [0:254] nodes. */ +return (node_id +1 ) <<BITS_PER_BYTE ; +} + +static inline int +decode_vn_id (unsigned long val) +{ +/* Can store U8_MAX [0:254] nodes. */ +return (val >>BITS_PER_BYTE ) -1 ; +}This patch looks good to me. However, should we split out the encoding
vn_idintova->flagsoptimization into another patch? It looks like an independent optimization which can be described better with specific log. At least, in the pdf file pasted or patch log, it's not obvious that:
1) node's free tree could contains any address range;
2) nodes' busy tree only contains address range belonging to this node;
- could contain crossing node range, corner case.
3) nodes' purge tree could contain any address range;
- decided by encodedvn_idinva->flags.
- decided by address viaaddr_to_node(va->va_start).
Quoted 7 lines
Quoted 2 lines
Personal opinion, feel it will make reviewing easier.
Sure, if it is easier to review, then i will split these two parts. All three statements are correct and valid. The pdf file only covers v1, so it is not up to date.
Anyway i will update a cover letter in v3 with more details.
Maybe providing these details in patch log or cover letter is enough. Leave it to you to decide if splitting patch is still needed. Thanks.
Uladzislau Rezki (Sony)
maintainerPATCH v2 7/9 mm: vmalloc: Support multiple nodes in vread_iter
In reply to Uladzislau Rezki (Sony)Extend the vread_iter() to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the /dev/kmem correctly reflects a vmalloc memory layout.
--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)
mm/vmalloc.c +53 -14
@@ -870,7 +870,7 @@ unsigned long vmalloc_nr_pages(void)/* Look up the first VA which satisfies addr < va_end, NULL if none. */ static struct vmap_area * -find_vmap_area_exceed_addr (unsigned long addr,struct rb_root *root) +__find_vmap_area_exceed_addr (unsigned long addr,struct rb_root *root) {struct vmap_area *va =NULL ;struct rb_node *n = root->rb_node;
@@ -894,6 +894,41 @@ find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root)return va; } +/* +* Returns a node where a first VA, that satisfies addr < va_end, resides. +* If success, a node is locked. A user is responsible to unlock it when a +* VA is no longer needed to be accessed. +* +* Returns NULL if nothing found. +*/ +static struct vmap_node * +find_vmap_area_exceed_addr_lock (unsigned long addr,struct vmap_area **va) +{ +struct vmap_node *vn, *va_node =NULL ; +struct vmap_area *va_lowest; +int i; + +for (i =0 ; i < nr_nodes; i++) { + vn = &nodes[i]; + +spin_lock (&vn->busy.lock); + va_lowest =__find_vmap_area_exceed_addr (addr, &vn->busy.root); +if (va_lowest) { +if (!va_node || va_lowest->va_start < (*va)->va_start) { +if (va_node) +spin_unlock (&va_node->busy.lock); + + *va = va_lowest; + va_node = vn; +continue ; + } + } +spin_unlock (&vn->busy.lock); + } + +return va_node; +} +static struct vmap_area *__find_vmap_area (unsigned long addr,struct rb_root *root) {struct rb_node *n = root->rb_node;
@@ -4048,6 +4083,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count)struct vm_struct *vm;char *vaddr;size_t n, size, flags, remains; +unsigned long next; addr =kasan_reset_tag (addr);
@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count; -/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock); - - va =find_vmap_area_exceed_addr ((unsigned long )addr, &vn->busy.root); -if (!va) + vn =find_vmap_area_exceed_addr_lock ((unsigned long ) addr, &va); +if (!vn) goto finished_zero;/* no intersects with alive vmap_area */ if ((unsigned long )addr + remains <= va->va_start) goto finished_zero; -list_for_each_entry_from (va, &vn->busy.head, list) { + do { size_t copied; if (remains == 0)
@@ -4084,10 +4116,10 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count)WARN_ON (flags ==VMAP_BLOCK );if (!vm && !flags) -continue ; + goto next_va;if (vm && (vm->flags &VM_UNINITIALIZED )) -continue ; + goto next_va;/* Pair with smp_wmb() in clear_vm_uninitialized_flag() */ smp_rmb ();
@@ -4096,7 +4128,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) size = vm ?get_vm_area_size (vm) :va_size (va);if (addr >= vaddr + size) -continue ; + goto next_va;if (addr < vaddr) {size_t to_zero =min_t (size_t, vaddr - addr, remains);
@@ -4125,15 +4157,22 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count)if (copied != n) goto finished; - } + + next_va: + next = va->va_end; +spin_unlock (&vn->busy.lock); + }while ((vn =find_vmap_area_exceed_addr_lock (next, &va))); finished_zero: -spin_unlock (&vn->busy.lock); +if (vn) +spin_unlock (&vn->busy.lock); +/* zero-fill memory holes */ return count - remains +zero_iter (iter, remains); finished:/* Nothing remains, or We couldn't copy/zero everything. */ -spin_unlock (&vn->busy.lock); +if (vn) +spin_unlock (&vn->busy.lock);return count - remains; }
Signature · 1 line
2.30.2
Re: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0
......
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 7 lines
mm/vmalloc.c +0
-3@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count; -/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);
This could change the vread behaviour a little bit. Before, once we take vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access /proc/kcore, e.g dynamic debugging.
And, the reading will be a little slower because each va finding need iterate all vmap_nodes[].
Otherwise, the patch itself looks good to me.
Quoted 67 lines
mm/vmalloc.c +16
-10- - va =find_vmap_area_exceed_addr ((unsigned long )addr, &vn->busy.root); -if (!va) + vn =find_vmap_area_exceed_addr_lock ((unsigned long ) addr, &va); +if (!vn) goto finished_zero;/* no intersects with alive vmap_area */ if ((unsigned long )addr + remains <= va->va_start) goto finished_zero; -list_for_each_entry_from (va, &vn->busy.head, list) { + do { size_t copied; if (remains == 0)@@ -4084,10 +4116,10 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count)WARN_ON (flags ==VMAP_BLOCK );if (!vm && !flags) -continue ; + goto next_va;if (vm && (vm->flags &VM_UNINITIALIZED )) -continue ; + goto next_va;/* Pair with smp_wmb() in clear_vm_uninitialized_flag() */ smp_rmb ();@@ -4096,7 +4128,7 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) size = vm ?get_vm_area_size (vm) :va_size (va);if (addr >= vaddr + size) -continue ; + goto next_va;if (addr < vaddr) {size_t to_zero =min_t (size_t, vaddr - addr, remains);@@ -4125,15 +4157,22 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count)if (copied != n) goto finished; - } + + next_va: + next = va->va_end; +spin_unlock (&vn->busy.lock); + }while ((vn =find_vmap_area_exceed_addr_lock (next, &va))); finished_zero: -spin_unlock (&vn->busy.lock); +if (vn) +spin_unlock (&vn->busy.lock); +/* zero-fill memory holes */ return count - remains +zero_iter (iter, remains); finished:/* Nothing remains, or We couldn't copy/zero everything. */ -spin_unlock (&vn->busy.lock); +if (vn) +spin_unlock (&vn->busy.lock);return count - remains; }Signature · 1 line
2.30.2
Re: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
Quoted 20 lines · Baoquan He
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0......
Quoted 4 lines
mm/vmalloc.c +0
-0@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count;
Quoted 10 lines
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 3 lines
mm/vmalloc.c +0
-3-/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);This could change the vread behaviour a little bit. Before, once we take
vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access/proc/kcore, e.g dynamic debugging.
With one big tree you anyway drop the lock after one cycle of reading. As far as i see, kcore.c's read granularity is a PAGE_SIZE.
Please correct me if i am wrong.
Quoted 4 lines
And, the reading will be a little slower because each va finding need iterate all vmap_nodes[].
Right. It is a bit tough here, because we have multiple nodes which represent zones(address space), i.e. there is an offset between them, it means that, reading fully one tree, will not provide a sequential reading.
Quoted 5 lines
Otherwise, the patch itself looks good to me.
Reviewed-by Baoquan He <[email protected]>
Applied.
Thank you for looking at it!
--
Uladzislau Rezki
Re: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
Quoted 31 lines · Uladzislau Rezki
Quoted 20 lines · Baoquan He
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0......
Quoted 4 lines
mm/vmalloc.c +0
-0@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count;Quoted 10 lines
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 3 lines
mm/vmalloc.c +0
-3-/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);This could change the vread behaviour a little bit. Before, once we take
vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access/proc/kcore, e.g dynamic debugging.
Quoted 2 lines
With one big tree you anyway drop the lock after one cycle of reading. As far as i see, kcore.c's read granularity is a
PAGE_SIZE.
With my understanding, kcore reading on vmalloc does read page by page, it will continue after one page reading if the required size is bigger than one page. Please see aligned_vread_iter() code. During the complete process, vmap_area_lock is held before this patch.
Quoted 9 lines
Quoted 5 lines
And, the reading will be a little slower because each va finding need iterate all vmap_nodes[].
Right. It is a bit tough here, because we have multiple nodes which represent zones(address space), i.e. there is an offset between them, it means that, reading fully one tree, will not provide a sequential reading.
Understood. Suppose the kcore reading on vmalloc is not critical. If I get chance to test on a machine with 256 cpu, I will report here.
Uladzislau Rezki
maintainerRe: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
Quoted 45 lines · Baoquan He
Quoted 31 lines · Uladzislau Rezki
Quoted 20 lines · Baoquan He
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0......
Quoted 4 lines
mm/vmalloc.c +0
-0@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count;Quoted 10 lines
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 3 lines
mm/vmalloc.c +0
-3-/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);This could change the vread behaviour a little bit. Before, once we take
vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access/proc/kcore, e.g dynamic debugging.Quoted 2 lines
With one big tree you anyway drop the lock after one cycle of reading. As far as i see, kcore.c's read granularity is a
PAGE_SIZE.With my understanding, kcore reading on vmalloc does read page by page, it will continue after one page reading if the required size is bigger than one page. Please see
aligned_vread_iter()code. During the complete process,vmap_area_lockis held before this patch.Quoted 5 lines
Quoted 5 lines
And, the reading will be a little slower because each va finding need iterate all vmap_nodes[].
Quoted 8 lines
Quoted 4 lines
Right. It is a bit tough here, because we have multiple nodes which represent zones(address space), i.e. there is an offset between them, it means that, reading fully one tree, will not provide a sequential reading.
Understood. Suppose the kcore reading on vmalloc is not critical. If I get chance to test on a machine with 256 cpu, I will report here.
It would be great! Unfortunately i do not have an access to such big systems. What i have is 64 CPUs max system. If you, by chance can test on bigger systems or can provide a temporary ssh access that would be awesome.
--
Uladzislau Rezki
Re: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
Quoted 51 lines · Uladzislau Rezki
Quoted 50 lines · Baoquan He
Quoted 31 lines · Uladzislau Rezki
Quoted 20 lines · Baoquan He
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0......
Quoted 4 lines
mm/vmalloc.c +0
-0@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count;Quoted 10 lines
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 3 lines
mm/vmalloc.c +0
-3-/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);This could change the vread behaviour a little bit. Before, once we take
vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access/proc/kcore, e.g dynamic debugging.Quoted 2 lines
With one big tree you anyway drop the lock after one cycle of reading. As far as i see, kcore.c's read granularity is a
PAGE_SIZE.With my understanding, kcore reading on vmalloc does read page by page, it will continue after one page reading if the required size is bigger than one page. Please see
aligned_vread_iter()code. During the complete process,vmap_area_lockis held before this patch.Quoted 9 lines
Quoted 5 lines
And, the reading will be a little slower because each va finding need iterate all vmap_nodes[].
Right. It is a bit tough here, because we have multiple nodes which represent zones(address space), i.e. there is an offset between them, it means that, reading fully one tree, will not provide a sequential reading.
Quoted 7 lines
Quoted 3 lines
Understood. Suppose the kcore reading on vmalloc is not critical. If I get chance to test on a machine with 256 cpu, I will report here.
It would be great! Unfortunately i do not have an access to such big systems. What i have is 64 CPUs max system. If you, by chance can test on bigger systems or can provide a temporary ssh access that would be awesome.
I got one with 288 cpus, have sent you ip address in private mail.
Re: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
In reply to Uladzislau RezkiQuoted 51 lines · Uladzislau Rezki
Quoted 50 lines · Baoquan He
Quoted 31 lines · Uladzislau Rezki
Quoted 20 lines · Baoquan He
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0......
Quoted 4 lines
mm/vmalloc.c +0
-0@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count;Quoted 10 lines
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 3 lines
mm/vmalloc.c +0
-3-/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);This could change the vread behaviour a little bit. Before, once we take
vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access/proc/kcore, e.g dynamic debugging.Quoted 2 lines
With one big tree you anyway drop the lock after one cycle of reading. As far as i see, kcore.c's read granularity is a
PAGE_SIZE.With my understanding, kcore reading on vmalloc does read page by page, it will continue after one page reading if the required size is bigger than one page. Please see
aligned_vread_iter()code. During the complete process,vmap_area_lockis held before this patch.Quoted 9 lines
Quoted 5 lines
And, the reading will be a little slower because each va finding need iterate all vmap_nodes[].
Right. It is a bit tough here, because we have multiple nodes which represent zones(address space), i.e. there is an offset between them, it means that, reading fully one tree, will not provide a sequential reading.
Quoted 7 lines
Quoted 3 lines
Understood. Suppose the kcore reading on vmalloc is not critical. If I get chance to test on a machine with 256 cpu, I will report here.
It would be great! Unfortunately i do not have an access to such big systems. What i have is 64 CPUs max system. If you, by chance can test on bigger systems or can provide a temporary ssh access that would be awesome.
10.16.216.205
user:root
password:redhat
This is a testing server in our lab, we apply for usage each time and it will reinstall OS, root user should be OK. I will take it for two days.
If accessing is not available, I can do some testing if you want me to run some commands.
Re: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
Quoted 60 lines · Baoquan He
Quoted 51 lines · Uladzislau Rezki
Quoted 50 lines · Baoquan He
Quoted 31 lines · Uladzislau Rezki
Quoted 20 lines · Baoquan He
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0......
Quoted 4 lines
mm/vmalloc.c +0
-0@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count;Quoted 10 lines
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 3 lines
mm/vmalloc.c +0
-3-/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);This could change the vread behaviour a little bit. Before, once we take
vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access/proc/kcore, e.g dynamic debugging.Quoted 2 lines
With one big tree you anyway drop the lock after one cycle of reading. As far as i see, kcore.c's read granularity is a
PAGE_SIZE.With my understanding, kcore reading on vmalloc does read page by page, it will continue after one page reading if the required size is bigger than one page. Please see
aligned_vread_iter()code. During the complete process,vmap_area_lockis held before this patch.Quoted 9 lines
Quoted 5 lines
And, the reading will be a little slower because each va finding need iterate all vmap_nodes[].
Right. It is a bit tough here, because we have multiple nodes which represent zones(address space), i.e. there is an offset between them, it means that, reading fully one tree, will not provide a sequential reading.
Quoted 7 lines
Quoted 3 lines
Understood. Suppose the kcore reading on vmalloc is not critical. If I get chance to test on a machine with 256 cpu, I will report here.
It would be great! Unfortunately i do not have an access to such big systems. What i have is 64 CPUs max system. If you, by chance can test on bigger systems or can provide a temporary ssh access that would be awesome.
Quoted 6 lines
10.16.216.205
user:root
password:redhatThis is a testing server in our lab, we apply for usage each time and it will reinstall OS, root user should be OK. I will take it for two days.
Oops, I sent it out publicly.
Quoted 4 lines
If accessing is not available, I can do some testing if you want me to run some commands.
Re: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
In reply to Uladzislau RezkiQuoted 31 lines · Uladzislau Rezki
Quoted 20 lines · Baoquan He
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0......
Quoted 4 lines
mm/vmalloc.c +0
-0@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count;Quoted 10 lines
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 3 lines
mm/vmalloc.c +0
-3-/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);This could change the vread behaviour a little bit. Before, once we take
vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access/proc/kcore, e.g dynamic debugging.
Quoted 2 lines
With one big tree you anyway drop the lock after one cycle of reading. As far as i see, kcore.c's read granularity is a
PAGE_SIZE.
You are right, kcore.c's reading granularity is truly PAGE_SIZE. I don't know procfs well, still need to study the code. Then it doesn't matter much with the multiple nodes in vread_iter(). Sorry for the noise.
static ssize_t read_kcore_iter(struct kiocb *iocb, struct iov_iter *iter)
{
......
start = kc_offset_to_vaddr(*fpos - data_offset);
if ((tsz = (PAGE_SIZE - (start & ~PAGE_MASK))) > buflen)
tsz = buflen;
m = NULL;
while (buflen) {
}
...
}Uladzislau Rezki
maintainerRe: [PATCH v2 7/9] mm: vmalloc: Support multiple nodes in vread_iter
Quoted 40 lines · Baoquan He
Quoted 31 lines · Uladzislau Rezki
Quoted 20 lines · Baoquan He
Quoted 14 lines · Uladzislau Rezki (Sony)
Extend the
vread_iter()to be able to perform a sequential reading of VAs which are spread among multiple nodes. So a data read over the/dev/kmemcorrectly reflects a vmalloc memory layout.Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 67 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 14 deletions(-)mm/vmalloc.c +0
-0......
Quoted 4 lines
mm/vmalloc.c +0
-0@@ -4057,19 +4093,15 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) remains = count;Quoted 10 lines
mm/vmalloc.c long vread_iter(struct iov_iter *iter, const char *addr, size_t count) · line 4062
Quoted 3 lines
mm/vmalloc.c +0
-3-/* Hooked to node_0 so far. */ - vn =addr_to_node (0 ); -spin_lock (&vn->busy.lock);This could change the vread behaviour a little bit. Before, once we take
vmap_area_lock, the vread will read out the content of snapshot at the moment. Now, reading out in one node's tree won't disrupt other nodes' tree accessing. Not sure if this matters when people need access/proc/kcore, e.g dynamic debugging.Quoted 2 lines
With one big tree you anyway drop the lock after one cycle of reading. As far as i see, kcore.c's read granularity is a
PAGE_SIZE.You are right, kcore.c's reading granularity is truly
PAGE_SIZE. I don't know procfs well, still need to study the code. Then it doesn't matter much with the multiple nodes invread_iter(). Sorry for the noise.static
ssize_tread_kcore_iter(struct kiocb *iocb, struct iov_iter *iter)
Quoted 12 lines
{
......
start = kc_offset_to_vaddr(*fpos - data_offset);
if ((tsz = (PAGE_SIZE - (start & ~PAGE_MASK))) > buflen)
tsz = buflen;
m = NULL;
while (buflen) {
}
...
}Good. Then we are on the same page :)
Thank you!
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerPATCH v2 8/9 mm: vmalloc: Support multiple nodes in vmallocinfo
In reply to Uladzislau Rezki (Sony)Allocated areas are spread among nodes, it implies that the scanning has to be performed individually of each node in order to dump all existing VAs.
--- mm/vmalloc.c | 120 ++++++++++++++++++++------------------------------- 1 file changed, 47 insertions(+), 73 deletions(-)
mm/vmalloc.c +47 -73
@@ -4636,30 +4636,6 @@ bool vmalloc_dump_obj(void *object)#endif #ifdef CONFIG_PROC_FS -static void *s_start (struct seq_file *m,loff_t *pos) -{ -struct vmap_node *vn =addr_to_node (0 ); - -mutex_lock (&vmap_purge_lock); -spin_lock (&vn->busy.lock); - -return seq_list_start (&vn->busy.head, *pos); -} - -static void *s_next (struct seq_file *m,void *p,loff_t *pos) -{ -struct vmap_node *vn =addr_to_node (0 ); -return seq_list_next (p, &vn->busy.head, pos); -} - -static void s_stop (struct seq_file *m,void *p) -{ -struct vmap_node *vn =addr_to_node (0 ); - -spin_unlock (&vn->busy.lock); -mutex_unlock (&vmap_purge_lock); -} -static void show_numa_info (struct seq_file *m,struct vm_struct *v) {if (IS_ENABLED (CONFIG_NUMA )) {
@@ -4703,84 +4679,82 @@ static void show_purge_info(struct seq_file *m) } } -static int s_show (struct seq_file *m,void *p) +static int vmalloc_info_show (struct seq_file *m,void *p) {struct vmap_node *vn;struct vmap_area *va;struct vm_struct *v; +int i; - vn =addr_to_node (0 ); - va = list_entry(p, struct vmap_area, list); +for (i =0 ; i < nr_nodes; i++) { + vn = &nodes[i]; -if (!va->vm) { -if (va->flags &VMAP_RAM ) -seq_printf (m,"0x%pK-0x%pK %7ld vm_map_ram\n" , - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); +spin_lock (&vn->busy.lock); +list_for_each_entry (va, &vn->busy.head, list) { +if (!va->vm) { +if (va->flags &VMAP_RAM ) +seq_printf (m,"0x%pK-0x%pK %7ld vm_map_ram\n" , + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); - goto final; - } +continue ; + } - v = va->vm; + v = va->vm; -seq_printf (m,"0x%pK-0x%pK %7ld" , - v->addr, v->addr + v->size, v->size); +seq_printf (m,"0x%pK-0x%pK %7ld" , + v->addr, v->addr + v->size, v->size); -if (v->caller) -seq_printf (m," %pS" , v->caller); +if (v->caller) +seq_printf (m," %pS" , v->caller); -if (v->nr_pages) -seq_printf (m," pages=%d" , v->nr_pages); +if (v->nr_pages) +seq_printf (m," pages=%d" , v->nr_pages); -if (v->phys_addr) -seq_printf (m," phys=%pa" , &v->phys_addr); +if (v->phys_addr) +seq_printf (m," phys=%pa" , &v->phys_addr); -if (v->flags &VM_IOREMAP ) -seq_puts (m," ioremap" ); +if (v->flags &VM_IOREMAP ) +seq_puts (m," ioremap" ); -if (v->flags &VM_ALLOC ) -seq_puts (m," vmalloc" ); +if (v->flags &VM_ALLOC ) +seq_puts (m," vmalloc" ); -if (v->flags &VM_MAP ) -seq_puts (m," vmap" ); +if (v->flags &VM_MAP ) +seq_puts (m," vmap" ); -if (v->flags &VM_USERMAP ) -seq_puts (m," user" ); +if (v->flags &VM_USERMAP ) +seq_puts (m," user" ); -if (v->flags &VM_DMA_COHERENT ) -seq_puts (m," dma-coherent" ); +if (v->flags &VM_DMA_COHERENT ) +seq_puts (m," dma-coherent" ); -if (is_vmalloc_addr (v->pages)) -seq_puts (m," vpages" ); +if (is_vmalloc_addr (v->pages)) +seq_puts (m," vpages" ); -show_numa_info (m, v); -seq_putc (m,'\n' ); +show_numa_info (m, v); +seq_putc (m,'\n' ); + } +spin_unlock (&vn->busy.lock); + }/* * As a final step, dump "unpurged" areas. */ -final: -if (list_is_last (&va->list, &vn->busy.head)) -show_purge_info (m); - +show_purge_info (m);return 0 ; } -static const struct seq_operations vmalloc_op = { - .start = s_start, - .next = s_next, - .stop = s_stop, - .show = s_show, -}; -static int __init proc_vmalloc_init (void ) { +void *priv_data =NULL ; +if (IS_ENABLED (CONFIG_NUMA )) -proc_create_seq_private ("vmallocinfo" ,0400 ,NULL , - &vmalloc_op, - nr_node_ids *sizeof (unsigned int ),NULL ); -else -proc_create_seq ("vmallocinfo" ,0400 ,NULL , &vmalloc_op); + priv_data =kmalloc (nr_node_ids *sizeof (unsigned int ),GFP_KERNEL ); + +proc_create_single_data ("vmallocinfo" , +0400 ,NULL , vmalloc_info_show, priv_data); +return 0 ; }module_init (proc_vmalloc_init);
Signature · 1 line
2.30.2
Re: [PATCH v2 8/9] mm: vmalloc: Support multiple nodes in vmallocinfo
Quoted 8 lines · Uladzislau Rezki (Sony)
Allocated areas are spread among nodes, it implies that the scanning has to be performed individually of each node in order to dump all existing VAs.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 120 ++++++++++++++++++++------------------------------- 1 file changed, 47 insertions(+), 73 deletions(-)
LGTM,
Quoted 171 lines
mm/vmalloc.c +47
-73@@ -4636,30 +4636,6 @@ bool vmalloc_dump_obj(void *object)#endif #ifdef CONFIG_PROC_FS -static void *s_start (struct seq_file *m,loff_t *pos) -{ -struct vmap_node *vn =addr_to_node (0 ); - -mutex_lock (&vmap_purge_lock); -spin_lock (&vn->busy.lock); - -return seq_list_start (&vn->busy.head, *pos); -} - -static void *s_next (struct seq_file *m,void *p,loff_t *pos) -{ -struct vmap_node *vn =addr_to_node (0 ); -return seq_list_next (p, &vn->busy.head, pos); -} - -static void s_stop (struct seq_file *m,void *p) -{ -struct vmap_node *vn =addr_to_node (0 ); - -spin_unlock (&vn->busy.lock); -mutex_unlock (&vmap_purge_lock); -} -static void show_numa_info (struct seq_file *m,struct vm_struct *v) {if (IS_ENABLED (CONFIG_NUMA )) {@@ -4703,84 +4679,82 @@ static void show_purge_info(struct seq_file *m) } } -static int s_show (struct seq_file *m,void *p) +static int vmalloc_info_show (struct seq_file *m,void *p) {struct vmap_node *vn;struct vmap_area *va;struct vm_struct *v; +int i; - vn =addr_to_node (0 ); - va = list_entry(p, struct vmap_area, list); +for (i =0 ; i < nr_nodes; i++) { + vn = &nodes[i]; -if (!va->vm) { -if (va->flags &VMAP_RAM ) -seq_printf (m,"0x%pK-0x%pK %7ld vm_map_ram\n" , - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); +spin_lock (&vn->busy.lock); +list_for_each_entry (va, &vn->busy.head, list) { +if (!va->vm) { +if (va->flags &VMAP_RAM ) +seq_printf (m,"0x%pK-0x%pK %7ld vm_map_ram\n" , + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); - goto final; - } +continue ; + } - v = va->vm; + v = va->vm; -seq_printf (m,"0x%pK-0x%pK %7ld" , - v->addr, v->addr + v->size, v->size); +seq_printf (m,"0x%pK-0x%pK %7ld" , + v->addr, v->addr + v->size, v->size); -if (v->caller) -seq_printf (m," %pS" , v->caller); +if (v->caller) +seq_printf (m," %pS" , v->caller); -if (v->nr_pages) -seq_printf (m," pages=%d" , v->nr_pages); +if (v->nr_pages) +seq_printf (m," pages=%d" , v->nr_pages); -if (v->phys_addr) -seq_printf (m," phys=%pa" , &v->phys_addr); +if (v->phys_addr) +seq_printf (m," phys=%pa" , &v->phys_addr); -if (v->flags &VM_IOREMAP ) -seq_puts (m," ioremap" ); +if (v->flags &VM_IOREMAP ) +seq_puts (m," ioremap" ); -if (v->flags &VM_ALLOC ) -seq_puts (m," vmalloc" ); +if (v->flags &VM_ALLOC ) +seq_puts (m," vmalloc" ); -if (v->flags &VM_MAP ) -seq_puts (m," vmap" ); +if (v->flags &VM_MAP ) +seq_puts (m," vmap" ); -if (v->flags &VM_USERMAP ) -seq_puts (m," user" ); +if (v->flags &VM_USERMAP ) +seq_puts (m," user" ); -if (v->flags &VM_DMA_COHERENT ) -seq_puts (m," dma-coherent" ); +if (v->flags &VM_DMA_COHERENT ) +seq_puts (m," dma-coherent" ); -if (is_vmalloc_addr (v->pages)) -seq_puts (m," vpages" ); +if (is_vmalloc_addr (v->pages)) +seq_puts (m," vpages" ); -show_numa_info (m, v); -seq_putc (m,'\n' ); +show_numa_info (m, v); +seq_putc (m,'\n' ); + } +spin_unlock (&vn->busy.lock); + }/* * As a final step, dump "unpurged" areas. */ -final: -if (list_is_last (&va->list, &vn->busy.head)) -show_purge_info (m); - +show_purge_info (m);return 0 ; } -static const struct seq_operations vmalloc_op = { - .start = s_start, - .next = s_next, - .stop = s_stop, - .show = s_show, -}; -static int __init proc_vmalloc_init (void ) { +void *priv_data =NULL ; +if (IS_ENABLED (CONFIG_NUMA )) -proc_create_seq_private ("vmallocinfo" ,0400 ,NULL , - &vmalloc_op, - nr_node_ids *sizeof (unsigned int ),NULL ); -else -proc_create_seq ("vmallocinfo" ,0400 ,NULL , &vmalloc_op); + priv_data =kmalloc (nr_node_ids *sizeof (unsigned int ),GFP_KERNEL ); + +proc_create_single_data ("vmallocinfo" , +0400 ,NULL , vmalloc_info_show, priv_data); +return 0 ; }module_init (proc_vmalloc_init);Signature · 1 line
2.30.2
Uladzislau Rezki
maintainerRe: [PATCH v2 8/9] mm: vmalloc: Support multiple nodes in vmallocinfo
Quoted 5 lines · Baoquan He
Quoted 4 lines · Uladzislau Rezki (Sony)
Allocated areas are spread among nodes, it implies that the scanning has to be performed individually of each node in order to dump all existing VAs.
Quoted 9 lines
Quoted 4 lines
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 120 ++++++++++++++++++++------------------------------- 1 file changed, 47 insertions(+), 73 deletions(-)LGTM,
Reviewed-by Baoquan He <[email protected]>
Thank you for review, applied for v3.
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerPATCH v2 9/9 mm: vmalloc: Set nr_nodes/node_size based on CPU-cores
In reply to Uladzislau Rezki (Sony)The density ratio is set to 2, i.e. two users per one node. For example if there are 6 cores in a system the "nr_nodes" is 3.
The "node_size" also depends on number of physical cores. A high-threshold limit is hard-coded and set to SZ_4M.
For 32-bit, single/dual core systems an access to a global vmap heap is not balanced. Such small systems do not suffer from lock contentions due to limitation of CPU-cores.
Test on AMD Ryzen Threadripper 3970X 32-Core Processor:
sudo ./test_vmalloc.sh run_test_mask=127 nr_threads=64
<default perf>
94.17% 0.90% [kernel] [k] _raw_spin_lock 93.27% 93.05% [kernel] [k] native_queued_spin_lock_slowpath 74.69% 0.25% [kernel] [k] __vmalloc_node_range 72.64% 0.01% [kernel] [k] __get_vm_area_node 72.04% 0.89% [kernel] [k] alloc_vmap_area 42.17% 0.00% [kernel] [k] vmalloc 32.53% 0.00% [kernel] [k] __vmalloc_node 24.91% 0.25% [kernel] [k] vfree 24.32% 0.01% [kernel] [k] remove_vm_area 22.63% 0.21% [kernel] [k] find_unlink_vmap_area 15.51% 0.00% [unknown] [k] 0xffffffffc09a74ac 14.35% 0.00% [kernel] [k] ret_from_fork_asm 14.35% 0.00% [kernel] [k] ret_from_fork 14.35% 0.00% [kernel] [k] kthread <default perf> vs <patch-series perf> 74.32% 2.42% [kernel] [k] __vmalloc_node_range 69.58% 0.01% [kernel] [k] vmalloc 54.21% 1.17% [kernel] [k] __alloc_pages_bulk 48.13% 47.91% [kernel] [k] clear_page_orig 43.60% 0.01% [unknown] [k] 0xffffffffc082f16f 32.06% 0.00% [kernel] [k] ret_from_fork_asm 32.06% 0.00% [kernel] [k] ret_from_fork 32.06% 0.00% [kernel] [k] kthread 31.30% 0.00% [unknown] [k] 0xffffffffc082f889 22.98% 4.16% [kernel] [k] vfree 14.36% 0.28% [kernel] [k] __get_vm_area_node 13.43% 3.35% [kernel] [k] alloc_vmap_area 10.86% 0.04% [kernel] [k] remove_vm_area 8.89% 2.75% [kernel] [k] _raw_spin_lock 7.19% 0.00% [unknown] [k] 0xffffffffc082fba3 6.65% 1.37% [kernel] [k] free_unref_page 6.13% 6.11% [kernel] [k] native_queued_spin_lock_slowpath
<patch-series perf>
confirms that a native_queued_spin_lock_slowpath bottle-neck can be considered as negligible for the patch-series version.
The throughput is ~15x higher:
urezki@pc638:~$ time sudo ./test_vmalloc.sh run_test_mask=127 nr_threads=64 Run the test with following parameters: run_test_mask=127 nr_threads=64 Done. Check the kernel ring buffer to see the summary.
real 24m3.305s user 0m0.361s sys 0m0.013s
urezki@pc638:~$
urezki@pc638:~$ time sudo ./test_vmalloc.sh run_test_mask=127 nr_threads=64 Run the test with following parameters: run_test_mask=127 nr_threads=64 Done. Check the kernel ring buffer to see the summary.
real 1m28.382s user 0m0.014s sys 0m0.026s
urezki@pc638:~$
--- mm/vmalloc.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
mm/vmalloc.c +26 -0
@@ -796,6 +796,9 @@ struct vmap_node {atomic_t fill_in_progress; }; +#define MAX_NODES U8_MAX +#define MAX_NODE_SIZE SZ_4M +static struct vmap_node *nodes, snode;static __read_mostly unsigned int nr_nodes =1 ;static __read_mostly unsigned int node_size =1 ;
@@ -4803,11 +4806,24 @@ static void vmap_init_free_space(void) } } +static unsigned int calculate_nr_nodes (void ) +{ +unsigned int nr_cpus; + + nr_cpus =num_present_cpus (); +if (nr_cpus <=1 ) + nr_cpus =num_possible_cpus (); + +/* Density factor. Two users per a node. */ + return clamp_t(unsigned int, nr_cpus >> 1, 1, MAX_NODES); +} +static void vmap_init_nodes (void ) {struct vmap_node *vn;int i; + nr_nodes =calculate_nr_nodes (); nodes = &snode;if (nr_nodes >1 ) {
@@ -4830,6 +4846,16 @@ static void vmap_init_nodes(void)INIT_LIST_HEAD (&vn->free.head);spin_lock_init (&vn->free.lock); } + +/* +* Scale a node size to number of CPUs. Each power of two +* value doubles a node size. A high-threshold limit is set +* to 4M. +*/ +#if BITS_PER_LONG ==64 +if (nr_nodes >1 ) + node_size =min (SZ_64K <<fls (num_possible_cpus ()),SZ_4M ); +#endif }void __init vmalloc_init (void )
Signature · 1 line
2.30.2
Re: [PATCH v2 9/9] mm: vmalloc: Set nr_nodes/node_size based on CPU-cores
On 08/29/23 at 10:11am, Uladzislau Rezki (Sony) wrote:
......
Quoted 9 lines
real 1m28.382s user 0m0.014s sys 0m0.026surezki@pc638:~$
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
LGTM,
Quoted 60 lines
mm/vmalloc.c +26
-0@@ -796,6 +796,9 @@ struct vmap_node {atomic_t fill_in_progress; }; +#define MAX_NODES U8_MAX +#define MAX_NODE_SIZE SZ_4M +static struct vmap_node *nodes, snode;static __read_mostly unsigned int nr_nodes =1 ;static __read_mostly unsigned int node_size =1 ;@@ -4803,11 +4806,24 @@ static void vmap_init_free_space(void) } } +static unsigned int calculate_nr_nodes (void ) +{ +unsigned int nr_cpus; + + nr_cpus =num_present_cpus (); +if (nr_cpus <=1 ) + nr_cpus =num_possible_cpus (); + +/* Density factor. Two users per a node. */ + return clamp_t(unsigned int, nr_cpus >> 1, 1, MAX_NODES); +} +static void vmap_init_nodes (void ) {struct vmap_node *vn;int i; + nr_nodes =calculate_nr_nodes (); nodes = &snode;if (nr_nodes >1 ) {@@ -4830,6 +4846,16 @@ static void vmap_init_nodes(void)INIT_LIST_HEAD (&vn->free.head);spin_lock_init (&vn->free.lock); } + +/* +* Scale a node size to number of CPUs. Each power of two +* value doubles a node size. A high-threshold limit is set +* to 4M. +*/ +#if BITS_PER_LONG ==64 +if (nr_nodes >1 ) + node_size =min (SZ_64K <<fls (num_possible_cpus ()),SZ_4M ); +#endif }void __init vmalloc_init (void )Signature · 1 line
2.30.2
Re: [PATCH v2 9/9] mm: vmalloc: Set nr_nodes/node_size based on CPU-cores
Quoted 7 lines · Baoquan He
On 08/29/23 at 10:11am, Uladzislau Rezki (Sony) wrote:
......Quoted 5 lines
real 1m28.382s user 0m0.014s sys 0m0.026surezki@pc638:~$
Quoted 9 lines
Quoted 4 lines
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)LGTM,
Reviewed-by Baoquan He <[email protected]>
Applied. Thank you!
--
Uladzislau Rezki
Quoted 7 lines · Uladzislau Rezki (Sony)
Hello, folk!
This is the v2, the series which tends to minimize the vmap lock contention. It is based on the tag: v6.5-rc6. Here you can find a documentation about it:
wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf
Seems the wget command doesn't work for me. Not sure if other people can retrieve it successfully.
--2023-08-30 21:14:20-- ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf
=> ‘Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf’
Resolving vps418301.ovh.net (vps418301.ovh.net)... 37.187.244.100
Connecting to vps418301.ovh.net (vps418301.ovh.net)|37.187.244.100|:21... connected. Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /incoming ... done.
==> SIZE Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf ... done.
==> PASV ... done. ==> RETR Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf ...
No such file ‘Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf’.
Uladzislau Rezki
maintainerQuoted 21 lines · Baoquan He
Quoted 7 lines · Uladzislau Rezki (Sony)
Hello, folk!
This is the v2, the series which tends to minimize the vmap lock contention. It is based on the tag: v6.5-rc6. Here you can find a documentation about it:
wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf
Seems the wget command doesn't work for me. Not sure if other people can retrieve it successfully.
--2023-08-30 21:14:20-- ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf
=> ‘Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf’
Resolving vps418301.ovh.net (vps418301.ovh.net)... 37.187.244.100
Connecting to vps418301.ovh.net (vps418301.ovh.net)|37.187.244.100|:21... connected. Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /incoming ... done.
==> SIZE Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf ... done.
Quoted 3 lines
==> PASV ... done. ==> RETR Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf ...
No such file ‘Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf’.
Right. Same issue as a last time. I renamed the file name but pointed to the old name. Here we go:
wget ftp://vps418301.ovh.net/incoming/Mitigate_a_vmalloc_lock_contention_in_SMP_env_v2.pdf
--
Uladzislau Rezki
Uladzislau Rezki
maintainerHello, Andrew!
Quoted 77 lines
Hello, folk!
This is the v2, the series which tends to minimize the vmap lock contention. It is based on the tag: v6.5-rc6. Here you can find a documentation about it:
wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf
even though it is a bit outdated(it follows v1), it still gives a good overview on the problem and how it can be solved. On demand and by request i can update it.
The v1 is here: https://lore.kernel.org/linux-mm/ZIAqojPKjChJTssg@pc636/T/
Delta v1 -> v2:
- open coded locking;
- switch to array of nodes instead of per-cpu definition;
- density is 2 cores per one node(not equal to number of CPUs);
- VAs first go back(free path) to an owner node and later toa global heap if a block is fully freed, nid is saved in va->flags;- add helpers to drain lazily-freed areas faster, if high pressure;
- picked al Reviewed-by.Test on AMD Ryzen Threadripper 3970X 32-Core Processor:
sudo ./test_vmalloc.sh run_test_mask=127 nr_threads=64<v6.5-rc6 perf>
94.17% 0.90% [kernel] [k] _raw_spin_lock 93.27% 93.05% [kernel] [k] native_queued_spin_lock_slowpath 74.69% 0.25% [kernel] [k] __vmalloc_node_range 72.64% 0.01% [kernel] [k] __get_vm_area_node 72.04% 0.89% [kernel] [k] alloc_vmap_area 42.17% 0.00% [kernel] [k] vmalloc 32.53% 0.00% [kernel] [k] __vmalloc_node 24.91% 0.25% [kernel] [k] vfree 24.32% 0.01% [kernel] [k] remove_vm_area 22.63% 0.21% [kernel] [k] find_unlink_vmap_area 15.51% 0.00% [unknown] [k] 0xffffffffc09a74ac 14.35% 0.00% [kernel] [k] ret_from_fork_asm 14.35% 0.00% [kernel] [k] ret_from_fork 14.35% 0.00% [kernel] [k] kthread <v6.5-rc6 perf> vs <v6.5-rc6+v2 perf> 74.32% 2.42% [kernel] [k] __vmalloc_node_range 69.58% 0.01% [kernel] [k] vmalloc 54.21% 1.17% [kernel] [k] __alloc_pages_bulk 48.13% 47.91% [kernel] [k] clear_page_orig 43.60% 0.01% [unknown] [k] 0xffffffffc082f16f 32.06% 0.00% [kernel] [k] ret_from_fork_asm 32.06% 0.00% [kernel] [k] ret_from_fork 32.06% 0.00% [kernel] [k] kthread 31.30% 0.00% [unknown] [k] 0xffffffffc082f889 22.98% 4.16% [kernel] [k] vfree 14.36% 0.28% [kernel] [k] __get_vm_area_node 13.43% 3.35% [kernel] [k] alloc_vmap_area 10.86% 0.04% [kernel] [k] remove_vm_area 8.89% 2.75% [kernel] [k] _raw_spin_lock 7.19% 0.00% [unknown] [k] 0xffffffffc082fba3 6.65% 1.37% [kernel] [k] free_unref_page 6.13% 6.11% [kernel] [k] native_queued_spin_lock_slowpath<v6.5-rc6+v2 perf>
On smaller systems, for example, 8xCPU Hikey960 board the contention is not that high and is approximately ~16 percent.
Uladzislau Rezki (Sony) (9):
mm: vmalloc: Addva_alloc()helper
mm: vmalloc: Renameadjust_va_to_fit_type()function
mm: vmalloc: Movevmap_init_free_space()down invmalloc.c
mm: vmalloc: Remove globalvmap_area_rootrb-tree
mm: vmalloc: Remove globalpurge_vmap_area_rootrb-tree
mm: vmalloc: Offloadfree_vmap_area_locklock
mm: vmalloc: Support multiple nodes invread_iter
mm: vmalloc: Support multiple nodes in vmallocinfo
mm: vmalloc: Set nr_nodes/node_size based on CPU-cores
Quoted 6 lines
mm/vmalloc.c | 929 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 683 insertions(+), 246 deletions(-)Signature · 1 line
2.30.2
It would be good if this series somehow could be tested having some runtime from the people. So far there was a warning from the test robot:
https://lore.kernel.org/lkml/[email protected]/T/#m397b3834cb3b7a0a53b8dffb3624384c8e278007
<snip>
urezki@pc638:~/data/raid0/coding/linux.git$ git diff
mm/vmalloc.c +1 -1
@@ -4778,7 +4778,7 @@ static void vmap_init_free_space(void)* |<--------------------------------->| */ for (busy = vmlist; busy; busy = busy->next) { -if (busy->addr - vmap_start >0 ) { +if ((unsigned long ) busy->addr - vmap_start >0 ) { free =kmem_cache_zalloc (vmap_area_cachep,GFP_NOWAIT );if (!WARN_ON_ONCE (!free)) { free->va_start = vmap_start;
urezki@pc638:~/data/raid0/coding/linux.git$
<snip>
This extra patch has to be applied to fix the warning.
From my side i have tested it as much as i can. Can it be plugged into linux-next to get some runtime? Or is there any other way you prefer to go?
Thank you in advance!
--
Uladzislau Rezki
Andrew Morton
maintainerQuoted 2 lines · On Mon, 4 Sep 2023 16:55:38 +0200 Uladzislau Rezki <[email protected]> wrote:
It would be good if this series somehow could be tested having some runtime from the people.
I grabbed it. We're supposed to avoid adding new material to -next until after -rc1 is released, but I've cheated before ;)
That (inaccessible) pdf file is awkward. Could you please send out a suitable [0/N] cover letter for this series, which can be incorporated into the git record?
Uladzislau Rezki
maintainerQuoted 5 lines · Andrew Morton
Quoted 2 lines · On Mon, 4 Sep 2023 16:55:38 +0200 Uladzislau Rezki <[email protected]> wrote:
It would be good if this series somehow could be tested having some runtime from the people.
Quoted 7 lines
I grabbed it. We're supposed to avoid adding new material to -next until after -rc1 is released, but I've cheated before ;)
That (inaccessible) pdf file is awkward. Could you please send out a suitable [0/N] cover letter for this series, which can be incorporated into the git record?
There will be a v3 anyway where i update the cover latter. The v2 is not adapted to recently introduced Joel's patch, which is not in linux-next but will land soon:
<snip>
From: "Joel Fernandes (Google)" <[email protected]>
Subject: mm/vmalloc: add a safer version of find_vm_area() for debug
Date: Mon, 4 Sep 2023 18:08:04 +0000
It is unsafe to dump vmalloc area information when trying to do so from some contexts. Add a safer trylock version of the same function to do a best-effort VMA finding and use it from vmalloc_dump_obj(). <snip>
Also it might come some extra reviews and comments for v2.
Thanks!
--
Uladzislau Rezki
Quoted 5 lines · Uladzislau Rezki (Sony)
Hello, folk!
This is the v2, the series which tends to minimize the vmap lock contention. It is based on the tag: v6.5-rc6. Here you can find a documentation about it:
Will take a look when I get a chance at v3 as I gather you're spinning another version :)
Cheers!
Quoted 78 lines
wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf
even though it is a bit outdated(it follows v1), it still gives a good overview on the problem and how it can be solved. On demand and by request i can update it.
The v1 is here: https://lore.kernel.org/linux-mm/ZIAqojPKjChJTssg@pc636/T/
Delta v1 -> v2:
- open coded locking;
- switch to array of nodes instead of per-cpu definition;
- density is 2 cores per one node(not equal to number of CPUs);
- VAs first go back(free path) to an owner node and later toa global heap if a block is fully freed, nid is saved in va->flags;- add helpers to drain lazily-freed areas faster, if high pressure;
- picked al Reviewed-by.Test on AMD Ryzen Threadripper 3970X 32-Core Processor:
sudo ./test_vmalloc.sh run_test_mask=127 nr_threads=64<v6.5-rc6 perf>
94.17% 0.90% [kernel] [k] _raw_spin_lock 93.27% 93.05% [kernel] [k] native_queued_spin_lock_slowpath 74.69% 0.25% [kernel] [k] __vmalloc_node_range 72.64% 0.01% [kernel] [k] __get_vm_area_node 72.04% 0.89% [kernel] [k] alloc_vmap_area 42.17% 0.00% [kernel] [k] vmalloc 32.53% 0.00% [kernel] [k] __vmalloc_node 24.91% 0.25% [kernel] [k] vfree 24.32% 0.01% [kernel] [k] remove_vm_area 22.63% 0.21% [kernel] [k] find_unlink_vmap_area 15.51% 0.00% [unknown] [k] 0xffffffffc09a74ac 14.35% 0.00% [kernel] [k] ret_from_fork_asm 14.35% 0.00% [kernel] [k] ret_from_fork 14.35% 0.00% [kernel] [k] kthread <v6.5-rc6 perf> vs <v6.5-rc6+v2 perf> 74.32% 2.42% [kernel] [k] __vmalloc_node_range 69.58% 0.01% [kernel] [k] vmalloc 54.21% 1.17% [kernel] [k] __alloc_pages_bulk 48.13% 47.91% [kernel] [k] clear_page_orig 43.60% 0.01% [unknown] [k] 0xffffffffc082f16f 32.06% 0.00% [kernel] [k] ret_from_fork_asm 32.06% 0.00% [kernel] [k] ret_from_fork 32.06% 0.00% [kernel] [k] kthread 31.30% 0.00% [unknown] [k] 0xffffffffc082f889 22.98% 4.16% [kernel] [k] vfree 14.36% 0.28% [kernel] [k] __get_vm_area_node 13.43% 3.35% [kernel] [k] alloc_vmap_area 10.86% 0.04% [kernel] [k] remove_vm_area 8.89% 2.75% [kernel] [k] _raw_spin_lock 7.19% 0.00% [unknown] [k] 0xffffffffc082fba3 6.65% 1.37% [kernel] [k] free_unref_page 6.13% 6.11% [kernel] [k] native_queued_spin_lock_slowpath<v6.5-rc6+v2 perf>
On smaller systems, for example, 8xCPU Hikey960 board the contention is not that high and is approximately ~16 percent.
Uladzislau Rezki (Sony) (9):
mm: vmalloc: Addva_alloc()helper
mm: vmalloc: Renameadjust_va_to_fit_type()function
mm: vmalloc: Movevmap_init_free_space()down invmalloc.c
mm: vmalloc: Remove globalvmap_area_rootrb-tree
mm: vmalloc: Remove globalpurge_vmap_area_rootrb-tree
mm: vmalloc: Offloadfree_vmap_area_locklock
mm: vmalloc: Support multiple nodes invread_iter
mm: vmalloc: Support multiple nodes in vmallocinfo
mm: vmalloc: Set nr_nodes/node_size based on CPU-coresmm/vmalloc.c | 929 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 683 insertions(+), 246 deletions(-)--
2.30.2
Uladzislau Rezki
maintainerQuoted 10 lines · Lorenzo Stoakes
Quoted 5 lines · Uladzislau Rezki (Sony)
Hello, folk!
This is the v2, the series which tends to minimize the vmap lock contention. It is based on the tag: v6.5-rc6. Here you can find a documentation about it:
Will take a look when I get a chance at v3 as I gather you're spinning another version :)
Correct. I will do that :)
--
Uladzislau Rezki