vmalloc: Use atomic_long_add_return_relaxed()
Subsystems (2)
Uladzislau Rezki (Sony)
maintainerPATCH 1/1 vmalloc: Use atomic_long_add_return_relaxed()
Switch from the atomic_long_add_return() to its relaxed version.
We do not need a full memory barrier or any memory ordering during increasing the "vmap_lazy_nr" variable. What we only need is to do it atomically. This is what atomic_long_add_return_relaxed() guarantees.
AARCH64:
<snip>
Default:
40ec: d34cfe94 lsr x20, x20, #12
40f0: 14000044 b 4200 <free_vmap_area_noflush+0x19c>
40f4: 94000000 bl 0 <__sanitizer_cov_trace_pc>
40f8: 90000000 adrp x0, 0 <__traceiter_alloc_vmap_area>
40fc: 91000000 add x0, x0, #0x0
4100: f8f40016 ldaddal x20, x22, [x0]
4104: 8b160296 add x22, x20, x22Relaxed:
40ec: d34cfe94 lsr x20, x20, #12
40f0: 14000044 b 4200 <free_vmap_area_noflush+0x19c>
40f4: 94000000 bl 0 <__sanitizer_cov_trace_pc>
40f8: 90000000 adrp x0, 0 <__traceiter_alloc_vmap_area>
40fc: 91000000 add x0, x0, #0x0
4100: f8340016 ldadd x20, x22, [x0]
4104: 8b160296 add x22, x20, x22<snip>
--- mm/vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmalloc.c +1 -1
@@ -2370,7 +2370,7 @@ static void free_vmap_area_noflush(struct vmap_area *va)if (WARN_ON_ONCE (!list_empty (&va->list)))return ; - nr_lazy =atomic_long_add_return (va_size (va) >>PAGE_SHIFT , + nr_lazy =atomic_long_add_return_relaxed (va_size (va) >>PAGE_SHIFT , &vmap_lazy_nr);/*
Signature · 1 line
2.39.5
Quoted 38 lines · Uladzislau Rezki (Sony)
Switch from the
atomic_long_add_return()to its relaxed version.We do not need a full memory barrier or any memory ordering during increasing the "
vmap_lazy_nr" variable. What we only need is to do it atomically. This is whatatomic_long_add_return_relaxed()guarantees.AARCH64:
<snip>
Default:40ec: d34cfe94 lsr x20, x20, #12 40f0: 14000044 b 4200 <free_vmap_area_noflush+0x19c> 40f4: 94000000 bl 0 <__sanitizer_cov_trace_pc> 40f8: 90000000 adrp x0, 0 <__traceiter_alloc_vmap_area> 40fc: 91000000 add x0, x0, #0x0 4100: f8f40016 ldaddal x20, x22, [x0] 4104: 8b160296 add x22, x20, x22Relaxed:
40ec: d34cfe94 lsr x20, x20, #12 40f0: 14000044 b 4200 <free_vmap_area_noflush+0x19c> 40f4: 94000000 bl 0 <__sanitizer_cov_trace_pc> 40f8: 90000000 adrp x0, 0 <__traceiter_alloc_vmap_area> 40fc: 91000000 add x0, x0, #0x0 4100: f8340016 ldadd x20, x22, [x0] 4104: 8b160296 add x22, x20, x22<snip>
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/vmalloc.c +0
-0
mm/vmalloc.c static void free_vmap_area_noflush(struct vmap_area *va) · line 2374
Quoted 6 lines
mm/vmalloc.c +1
-1@@ -2370,7 +2370,7 @@ static void free_vmap_area_noflush(struct vmap_area *va)if (WARN_ON_ONCE (!list_empty (&va->list)))return ; - nr_lazy =atomic_long_add_return (va_size (va) >>PAGE_SHIFT , + nr_lazy =atomic_long_add_return_relaxed (va_size (va) >>PAGE_SHIFT , &vmap_lazy_nr);
LGTM,
Quoted 38 lines · Uladzislau Rezki (Sony)
Switch from the
atomic_long_add_return()to its relaxed version.We do not need a full memory barrier or any memory ordering during increasing the "
vmap_lazy_nr" variable. What we only need is to do it atomically. This is whatatomic_long_add_return_relaxed()guarantees.AARCH64:
<snip>
Default:40ec: d34cfe94 lsr x20, x20, #12 40f0: 14000044 b 4200 <free_vmap_area_noflush+0x19c> 40f4: 94000000 bl 0 <__sanitizer_cov_trace_pc> 40f8: 90000000 adrp x0, 0 <__traceiter_alloc_vmap_area> 40fc: 91000000 add x0, x0, #0x0 4100: f8f40016 ldaddal x20, x22, [x0] 4104: 8b160296 add x22, x20, x22Relaxed:
40ec: d34cfe94 lsr x20, x20, #12 40f0: 14000044 b 4200 <free_vmap_area_noflush+0x19c> 40f4: 94000000 bl 0 <__sanitizer_cov_trace_pc> 40f8: 90000000 adrp x0, 0 <__traceiter_alloc_vmap_area> 40fc: 91000000 add x0, x0, #0x0 4100: f8340016 ldadd x20, x22, [x0] 4104: 8b160296 add x22, x20, x22<snip>
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/vmalloc.c +0
-0
mm/vmalloc.c static void free_vmap_area_noflush(struct vmap_area *va) · line 2376
Quoted 8 lines
mm/vmalloc.c +1
-1@@ -2370,7 +2370,7 @@ static void free_vmap_area_noflush(struct vmap_area *va)if (WARN_ON_ONCE (!list_empty (&va->list)))return ; - nr_lazy =atomic_long_add_return (va_size (va) >>PAGE_SHIFT , + nr_lazy =atomic_long_add_return_relaxed (va_size (va) >>PAGE_SHIFT , &vmap_lazy_nr);/*
If touching this, maybe a step further -- I see false sharing with:
nm vmlinux | sort -k 1
[snip]ffffffff845cb240 b vmap_lazy_nrffffffff845cb248 b nr_vmalloc_pagesffffffff845cb260 b single
[/snip]
If this is used enough to warrant messing with eliding a fence, then it probably also wants to elide some false sharing.
So at least the first two should be annotated with __cacheline_aligned_in_smp?
I am not going to push though, feel free to ignore.
Uladzislau Rezki
maintainerQuoted 57 lines · Mateusz Guzik
Quoted 38 lines · Uladzislau Rezki (Sony)
Switch from the
atomic_long_add_return()to its relaxed version.We do not need a full memory barrier or any memory ordering during increasing the "
vmap_lazy_nr" variable. What we only need is to do it atomically. This is whatatomic_long_add_return_relaxed()guarantees.AARCH64:
<snip>
Default:40ec: d34cfe94 lsr x20, x20, #12 40f0: 14000044 b 4200 <free_vmap_area_noflush+0x19c> 40f4: 94000000 bl 0 <__sanitizer_cov_trace_pc> 40f8: 90000000 adrp x0, 0 <__traceiter_alloc_vmap_area> 40fc: 91000000 add x0, x0, #0x0 4100: f8f40016 ldaddal x20, x22, [x0] 4104: 8b160296 add x22, x20, x22Relaxed:
40ec: d34cfe94 lsr x20, x20, #12 40f0: 14000044 b 4200 <free_vmap_area_noflush+0x19c> 40f4: 94000000 bl 0 <__sanitizer_cov_trace_pc> 40f8: 90000000 adrp x0, 0 <__traceiter_alloc_vmap_area> 40fc: 91000000 add x0, x0, #0x0 4100: f8340016 ldadd x20, x22, [x0] 4104: 8b160296 add x22, x20, x22<snip>
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- mm/vmalloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/vmalloc.c +0
-0mm/vmalloc.c static void free_vmap_area_noflush(struct vmap_area *va) · line 2376
Quoted 8 lines
mm/vmalloc.c +1
-1@@ -2370,7 +2370,7 @@ static void free_vmap_area_noflush(struct vmap_area *va)if (WARN_ON_ONCE (!list_empty (&va->list)))return ; - nr_lazy =atomic_long_add_return (va_size (va) >>PAGE_SHIFT , + nr_lazy =atomic_long_add_return_relaxed (va_size (va) >>PAGE_SHIFT , &vmap_lazy_nr);/* If touching this, maybe a step further -- I see false sharing with:
nm vmlinux | sort -k 1[snip]
ffffffff845cb240bvmap_lazy_nrffffffff845cb248bnr_vmalloc_pagesffffffff845cb260b single
[/snip]
Quoted 7 lines
If this is used enough to warrant messing with eliding a fence, then it probably also wants to elide some false sharing.
So at least the first two should be annotated with
__cacheline_aligned_in_smp?I am not going to push though, feel free to ignore.
OK, i see your point. I will check closer. We can place both into two different cache-lines to prevent interference between each other.
But that will be a different patch.
Thank you for the nit!
--
Uladzislau Rezki