mm/vmalloc: Assorted fixes and improvements
Subsystems (2)
cover mm/vmalloc: Assorted fixes and improvements
Following up to the discussion about excessive TLB flushes
https://lore.kernel.org/all/87a5y5a6kj.ffs@tglx
this series addresses the following issues:
1) Prevent the stale TLB problem related to fully utilized vmap blocks
2) Avoid the double per CPU list walk in _vm_unmap_aliases()
3) Avoid flushing dirty space over and over
4) Add a lockless quickcheck in vb_alloc() and add missing READ/WRITE_ONCE() annotations
5) Prevent overeager purging of usable vmap_blocks if not under memory/address space pressure.
Thanks,
tglx
PATCH 1/6 mm/vmalloc: Prevent stale TLBs in fully utilized blocks
_vm_unmap_aliases() is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.
This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
So the per CPU list iteration does not find the block and if the page was mapped via such a block and the TLB has not yet been flushed, the guarantee of _vm_unmap_aliases() that there are no stale TLBs after returning is broken:
x = vb_alloc() // Removes vmap_block from free list because vb->free became 0vb_free(x) // Unmaps page and marks in dirty_min/max range
// Page is reusedvm_unmap_aliases() // Can't find vmap block with the dirty space -> FAIL
So instead of walking the per CPU free lists, walk the per CPU xarrays which hold pointers to _all_ active blocks in the system including those removed from the free lists.
--- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/vmalloc.c +2 -1
@@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned lfor_each_possible_cpu (cpu) {struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);struct vmap_block *vb; +unsigned long idx;rcu_read_lock (); -list_for_each_entry_rcu (vb, &vbq->free, free_list) { +xa_for_each (&vbq->vmap_blocks, idx, vb) {spin_lock (&vb->lock);if (vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Looks good:
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 3 lines · Christoph Hellwig
Looks good:
Reviewed-by Christoph Hellwig <[email protected]>
Thanks!
This probably wants a Fixes tag and if I'm not completely off then this problem exists since 2.6.28: db64fe02258f ("mm: rewrite vmap layer")
Uladzislau Rezki
maintainerRe: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 10 lines · Thomas Gleixner
Quoted 3 lines · Christoph Hellwig
Looks good:
Reviewed-by Christoph Hellwig <[email protected]>Thanks!
This probably wants a Fixes tag and if I'm not completely off then this problem exists since 2.6.28:
db64fe02258f("mm: rewrite vmap layer")
Yes. This code is quite old and the commit, you pointed looks correct.
--
Uladzislau Rezki
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
In reply to Thomas GleixnerQuoted 7 lines · Thomas Gleixner
_vm_unmap_aliases()is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
Oh Lord.
Quoted 27 lines
So the per CPU list iteration does not find the block and if the page was mapped via such a block and the TLB has not yet been flushed, the guarantee of
_vm_unmap_aliases()that there are no stale TLBs after returning is broken:x =
vb_alloc()// Removesvmap_blockfrom free list becausevb->freebecame 0vb_free(x)// Unmaps page and marks in dirty_min/max range// Page is reused
vm_unmap_aliases()// Can't find vmap block with the dirty space -> FAILSo instead of walking the per CPU free lists, walk the per CPU xarrays which hold pointers to _all_ active blocks in the system including those removed from the free lists.
Signed-off-by Thomas Gleixner <[email protected]>--- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/vmalloc.c +0
-0@@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned lfor_each_possible_cpu (cpu) {
mm/vmalloc.c static void _vm_unmap_aliases(unsigned l · line 2245
Quoted 10 lines
mm/vmalloc.c +2
-1struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);struct vmap_block *vb; +unsigned long idx;rcu_read_lock (); -list_for_each_entry_rcu (vb, &vbq->free, free_list) { +xa_for_each (&vbq->vmap_blocks, idx, vb) {spin_lock (&vb->lock);if (vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;
LGTM.
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 51 lines · Lorenzo Stoakes
Quoted 7 lines · Thomas Gleixner
_vm_unmap_aliases()is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
Oh Lord.
Quoted 27 lines
So the per CPU list iteration does not find the block and if the page was mapped via such a block and the TLB has not yet been flushed, the guarantee of
_vm_unmap_aliases()that there are no stale TLBs after returning is broken:x =
vb_alloc()// Removesvmap_blockfrom free list becausevb->freebecame 0vb_free(x)// Unmaps page and marks in dirty_min/max range// Page is reused
vm_unmap_aliases()// Can't find vmap block with the dirty space -> FAILSo instead of walking the per CPU free lists, walk the per CPU xarrays which hold pointers to _all_ active blocks in the system including those removed from the free lists.
Signed-off-by Thomas Gleixner <[email protected]>--- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/vmalloc.c +0
-0@@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned lfor_each_possible_cpu (cpu) {mm/vmalloc.c static void _vm_unmap_aliases(unsigned l · line 2245
Quoted 10 lines
mm/vmalloc.c +2
-1struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);struct vmap_block *vb; +unsigned long idx;rcu_read_lock (); -list_for_each_entry_rcu (vb, &vbq->free, free_list) { +xa_for_each (&vbq->vmap_blocks, idx, vb) {spin_lock (&vb->lock);if (vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;LGTM.
Quoted 1 line
Reviewed-by Lorenzo Stoakes <[email protected]>
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
In reply to Thomas GleixnerQuoted 7 lines · Thomas Gleixner
_vm_unmap_aliases()is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
The problem description is not accurate. This is tried to achieve for va associated with vmap_block by walking the per CPU free lists, those fully utilized vmap blocks can still be flushed in __purge_vmap_area_lazy() by calculating the [min:max] of purge_vmap_area_list, because va of vmap_blocks will be added to purge_vmap_area_list too via vb_free().
If we finally exclude the vmap_block va from purge list in __purge_vmap_area_lazy(), then we can say that stale TLBs is missed to flush. No?
IMHO, this is a preparation work for the vmap_block va excluding from purge list flushing. Please correct me if I am wrong.
Quoted 36 lines
So the per CPU list iteration does not find the block and if the page was mapped via such a block and the TLB has not yet been flushed, the guarantee of
_vm_unmap_aliases()that there are no stale TLBs after returning is broken:x =
vb_alloc()// Removesvmap_blockfrom free list becausevb->freebecame 0vb_free(x)// Unmaps page and marks in dirty_min/max range// Page is reused
vm_unmap_aliases()// Can't find vmap block with the dirty space -> FAILSo instead of walking the per CPU free lists, walk the per CPU xarrays which hold pointers to _all_ active blocks in the system including those removed from the free lists.
Signed-off-by Thomas Gleixner <[email protected]>--- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/vmalloc.c +2
-1@@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned lfor_each_possible_cpu (cpu) {struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);struct vmap_block *vb; +unsigned long idx;rcu_read_lock (); -list_for_each_entry_rcu (vb, &vbq->free, free_list) { +xa_for_each (&vbq->vmap_blocks, idx, vb) {spin_lock (&vb->lock);if (vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 9 lines · Baoquan He
Quoted 7 lines · Thomas Gleixner
_vm_unmap_aliases()is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
Quoted 5 lines
The problem description is not accurate. This is tried to achieve for va associated with
vmap_blockby walking the per CPU free lists, those fully utilized vmap blocks can still be flushed in__purge_vmap_area_lazy()by calculating the [min:max] ofpurge_vmap_area_list, because va ofvmap_blockswill be added topurge_vmap_area_listtoo viavb_free().
No. The fully utilized block cannot be purged when there are still active mappings on it. Again:
X = vb_alloc()
...
Y = vb_alloc()
vb->free -= order;
if (!vb->vb_free)
list_del(vb->free_list);
...
vb_free(Y)
vb->dirty += order;
if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_
free_block(); So because $X is not yet unmapped the block is neither on the free list nor on purge_vmap_area_list.
Thanks,
tglx
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 22 lines · Thomas Gleixner
Quoted 9 lines · Baoquan He
Quoted 7 lines · Thomas Gleixner
_vm_unmap_aliases()is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
Quoted 5 lines
The problem description is not accurate. This is tried to achieve for va associated with
vmap_blockby walking the per CPU free lists, those fully utilized vmap blocks can still be flushed in__purge_vmap_area_lazy()by calculating the [min:max] ofpurge_vmap_area_list, because va ofvmap_blockswill be added topurge_vmap_area_listtoo viavb_free().No. The fully utilized block cannot be purged when there are still active mappings on it. Again:
X =
vb_alloc()
...
Y =vb_alloc()
Quoted 8 lines
vb->free -= order;
if (!vb->vb_free)
list_del(vb->free_list);
...
vb_free(Y)
vb->dirty += order;
if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_
free_block(); vb_free(Y)
vb->dirty += order;
if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_
free_vmap_block();
-->free_vmap_area_noflush()
-->merge_or_add_vmap_area(va,
&purge_vmap_area_root, &purge_vmap_area_list);The last mapped region will be freed and added to purge list via vb_free(), it will be flushed with other va in purge list. When it's mapped via vb_alloc(), it's detached from vbq->free list. When it's freed via vb_alloc(), it's added to purge list, and flushed, the thing is duplicated flushing, no missing flush seen here?
Quoted 3 lines
So because $X is not yet unmapped the block is neither on the free list nor on
purge_vmap_area_list.
Yeah, because $X is not yet unmapped, the block could have unmapped part flushed, or unflushed. For unflushed part, it's got flushed with $X altogether in the purge list.
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 41 lines · Baoquan He
Quoted 22 lines · Thomas Gleixner
Quoted 9 lines · Baoquan He
Quoted 7 lines · Thomas Gleixner
_vm_unmap_aliases()is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
Quoted 5 lines
The problem description is not accurate. This is tried to achieve for va associated with
vmap_blockby walking the per CPU free lists, those fully utilized vmap blocks can still be flushed in__purge_vmap_area_lazy()by calculating the [min:max] ofpurge_vmap_area_list, because va ofvmap_blockswill be added topurge_vmap_area_listtoo viavb_free().No. The fully utilized block cannot be purged when there are still active mappings on it. Again:
X =
vb_alloc()
...
Y =vb_alloc()Quoted 8 lines
vb->free -= order; if (!vb->vb_free) list_del(vb->free_list); ... vb_free(Y) vb->dirty += order; if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_ free_block();vb_free(Y) vb->dirty += order; if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_ free_vmap_block(); -->free_vmap_area_noflush() -->merge_or_add_vmap_area(va, &purge_vmap_area_root, &purge_vmap_area_list);
Quoted 4 lines
The last mapped region will be freed and added to purge list via
vb_free(), it will be flushed with other va in purge list. When it's mapped viavb_alloc(), it's detached fromvbq->freelist. When it's freed viavb_alloc(), it's added to purge list, and flushed, the thing
~~~~~~~~vb_free(), typo,
Quoted 10 lines
is duplicated flushing, no missing flush seen here?
Quoted 3 lines
So because $X is not yet unmapped the block is neither on the free list nor on
purge_vmap_area_list.Yeah, because $X is not yet unmapped, the block could have unmapped part flushed, or unflushed. For unflushed part, it's got flushed with $X altogether in the purge list.
Uladzislau Rezki
maintainerRe: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
In reply to Baoquan HeQuoted 48 lines · Baoquan He
Quoted 22 lines · Thomas Gleixner
Quoted 9 lines · Baoquan He
Quoted 7 lines · Thomas Gleixner
_vm_unmap_aliases()is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
Quoted 5 lines
The problem description is not accurate. This is tried to achieve for va associated with
vmap_blockby walking the per CPU free lists, those fully utilized vmap blocks can still be flushed in__purge_vmap_area_lazy()by calculating the [min:max] ofpurge_vmap_area_list, because va ofvmap_blockswill be added topurge_vmap_area_listtoo viavb_free().No. The fully utilized block cannot be purged when there are still active mappings on it. Again:
X =
vb_alloc()
...
Y =vb_alloc()Quoted 8 lines
vb->free -= order; if (!vb->vb_free) list_del(vb->free_list); ... vb_free(Y) vb->dirty += order; if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_ free_block();vb_free(Y) vb->dirty += order; if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_ free_vmap_block(); -->free_vmap_area_noflush() -->merge_or_add_vmap_area(va, &purge_vmap_area_root, &purge_vmap_area_list);The last mapped region will be freed and added to purge list via
vb_free(), it will be flushed with other va in purge list. When it's mapped viavb_alloc(), it's detached fromvbq->freelist. When it's freed viavb_alloc(), it's added to purge list, and flushed, the thing is duplicated flushing, no missing flush seen here?Quoted 1 line
Quoted 7 lines
Quoted 2 lines
So because $X is not yet unmapped the block is neither on the free list nor on
purge_vmap_area_list.Yeah, because $X is not yet unmapped, the block could have unmapped part flushed, or unflushed. For unflushed part, it's got flushed with $X altogether in the purge list.
If we gurantee that vb is fully flushed, then just do not add it it purge list:
@@ -2086,12 +2090,7 @@ static void free_vmap_block(struct vmap_block *vb)
BUG_ON(tmp != vb);
z = addr_to_cvz(vb->va->va_start);
-
- spin_lock(&z->busy.lock);
- unlink_va(vb->va, &z->busy.root);
- spin_unlock(&z->busy.lock);
-
- free_vmap_area_noflush(vb->va);
+ free_vmap_area(vb->va);
kfree_rcu(vb, rcu_head);}
and directly return back into global vmap heap.
--
Uladzislau Rezki
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 7 lines · Uladzislau Rezki
Quoted 4 lines · Baoquan He
Yeah, because $X is not yet unmapped, the block could have unmapped part flushed, or unflushed. For unflushed part, it's got flushed with $X altogether in the purge list.
If we gurantee that vb is fully flushed, then just do not add it it purge list:
There is no such guarantee today and there wont be one even with patch 3/6 applied because:
unmap_aliases()
flush(dirty_min/max)
reset(dirty_min/max)
vb_free(lastmapping)
vunmap_range_noflush();
vb->dirty += order;
if (vb->dirty == VMAP_BBMAP_BITS)
free_block(vb); <-- the TLBs related to @lastmapping are
_NOT_ yet flushed.
Thanks,
tglx
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
In reply to Baoquan HeQuoted 8 lines · Baoquan He
On 05/24/23 at 11:51am, Thomas Gleixner wrote:
vb_free(Y) vb->dirty += order; if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_ free_vmap_block(); -->free_vmap_area_noflush() -->merge_or_add_vmap_area(va, &purge_vmap_area_root, &purge_vmap_area_list);
This is irrelevant. The path is _NOT_ taken. You even copied the comment:
if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_
Did you actually read what I wrote?
Again: It _CANNOT_ be on the purge list because it has active mappings:
1 X = vb_alloc()
...
Y = vb_alloc()
vb->free -= order; // Free space goes to 0
if (!vb->vb_free)
2 list_del(vb->free_list); // Block is removed from free list
...
vb_free(Y)
vb->dirty += order;
3 if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_
// because #1 $X is still mapped
// so block is _NOT_ freed and
// _NOT_ put on the purge list4 unmap_aliases()
walk_free_list() // Does not find it because of #2
walk_purge_list() // Does not find it because of #3
If the resulting flush range is not covering the $Y TLBs then stale TLBs stay around.
The xarray walk finds it and guarantees that the TLBs are gone when unmap_aliases() returns, which is the whole purpose of that function.
Thanks,
tglx
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 10 lines · Thomas Gleixner
Quoted 8 lines · Baoquan He
On 05/24/23 at 11:51am, Thomas Gleixner wrote:
vb_free(Y) vb->dirty += order; if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_ free_vmap_block(); -->free_vmap_area_noflush() -->merge_or_add_vmap_area(va, &purge_vmap_area_root, &purge_vmap_area_list);
Quoted 2 lines
This is irrelevant. The path is _NOT_ taken. You even copied the comment:
Ah, just copied the whole, didn't notice that. I am not a scrupulous person.
Quoted 10 lines
if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_Did you actually read what I wrote?
Again: It _CANNOT_ be on the purge list because it has active mappings:
1 X =
vb_alloc()
...
Y =vb_alloc()
Quoted 10 lines
vb->free -= order; // Free space goes to 0
if (!vb->vb_free)
2 list_del(vb->free_list); // Block is removed from free list
...
vb_free(Y)
vb->dirty += order;
3 if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_
// because #1 $X is still mapped
// so block is _NOT_ freed and
// _NOT_ put on the purge listSo what if $X is unmapped via vb_free($X)? Does the condition satisfied and can the vb put into purge list?
In your above example, $Y's flush is deferred, but not missed?
Quoted 7 lines
4
unmap_aliases()
walk_free_list()// Does not find it because of #2
walk_purge_list()// Does not find it because of #3If the resulting flush range is not covering the $Y TLBs then stale TLBs stay around.
OK, your mean the TLB of $Y will stay around after vb_free() until the whole vb becomes dirty, and fix that in this patch, you are right. vm_unmap_aliases() may need try to flush all unmapped ranges in this case but failed on $Y, while the page which is being reused has the old alias of $Y.
My thought was attracted to the repeated flush of vmap_block va on purge list.
By the way, you don't fix issue that in vm_reset_perms(), the direct map range will be accumulated with vb va and purge va and could produce flushing range including huge gap, do you still plan to fix that? I remember you said you will use array to gather ranges and flush them one by one.
Quoted 8 lines
The xarray walk finds it and guarantees that the TLBs are gone when
unmap_aliases()returns, which is the whole purpose of that function.Thanks,
tglx
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 18 lines · Baoquan He
Quoted 6 lines · Thomas Gleixner
On Wed, May 24 2023 at 19:24, Baoquan He wrote:
Again: It _CANNOT_ be on the purge list because it has active mappings:1 X =
vb_alloc()
...
Y =vb_alloc()Quoted 10 lines
vb->free -= order; // Free space goes to 0 if (!vb->vb_free) 2 list_del(vb->free_list); // Block is removed from free list ... vb_free(Y) vb->dirty += order; 3 if (vb->dirty == VMAP_BBMAP_BITS) // Condition is _false_ // because #1 $X is still mapped // so block is _NOT_ freed and // _NOT_ put on the purge list
Quoted 2 lines
So what if $X is unmapped via
vb_free($X)? Does the condition satisfied and can the vb put into purge list?
Yes, but it is _irrelevant_ for the problem at hand.
Quoted 1 line
In your above example, $Y's flush is deferred, but not missed?
Yes, but that violates the guarantee of vm_unmap_aliases():
* The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily
* to amortize TLB flushing overheads. What this means is that any page you
* have now, may, in a former life, have been mapped into kernel virtual
* address by the vmap layer and so there might be some CPUs with TLB entries
* still referencing that page (additional to the regular 1:1 kernel mapping). *
* vm_unmap_aliases flushes all such lazy mappings. After it returns, we can
* be sure that none of the pages we have control over will have any aliases
* from the vmap layer.
Quoted 4 lines
Quoted 4 lines
4
unmap_aliases()
walk_free_list()// Does not find it because of #2
walk_purge_list()// Does not find it because of #3
Quoted 8 lines
Quoted 2 lines
If the resulting flush range is not covering the $Y TLBs then stale TLBs stay around.
OK, your mean the TLB of $Y will stay around after
vb_free()until the whole vb becomes dirty, and fix that in this patch, you are right.vm_unmap_aliases()may need try to flush all unmapped ranges in this case but failed on $Y, while the page which is being reused has the old alias of $Y.
vm_unmap_aliases() _must_ guarantee that the old TLBs for $Y are gone.
Quoted 8 lines
My thought was attracted to the repeated flush of
vmap_blockva on purge list.By the way, you don't fix issue that in
vm_reset_perms(), the direct map range will be accumulated with vb va and purge va and could produce flushing range including huge gap, do you still plan to fix that? I remember you said you will use array to gather ranges and flush them one by one.
One thing at a time. This series is a prerequisite.
Thanks,
tglx
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
In reply to Thomas GleixnerQuoted 31 lines · Thomas Gleixner
_vm_unmap_aliases()is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc.This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero.
So the per CPU list iteration does not find the block and if the page was mapped via such a block and the TLB has not yet been flushed, the guarantee of
_vm_unmap_aliases()that there are no stale TLBs after returning is broken:x =
vb_alloc()// Removesvmap_blockfrom free list becausevb->freebecame 0vb_free(x)// Unmaps page and marks in dirty_min/max range// Page is reused
vm_unmap_aliases()// Can't find vmap block with the dirty space -> FAILSo instead of walking the per CPU free lists, walk the per CPU xarrays which hold pointers to _all_ active blocks in the system including those removed from the free lists.
Signed-off-by Thomas Gleixner <[email protected]>--- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/vmalloc.c static void _vm_unmap_aliases(unsigned l · line 2241
Quoted 6 lines
mm/vmalloc.c +1
-0@@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned lfor_each_possible_cpu (cpu) {struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);struct vmap_block *vb; +unsigned long idx;rcu_read_lock ();
Do we need to remove this rcu_read_xx() pair since it marks the RCU read-side critical section on vbq-free list?
Quoted 6 lines
mm/vmalloc.c +1
-1-list_for_each_entry_rcu (vb, &vbq->free, free_list) { +xa_for_each (&vbq->vmap_blocks, idx, vb) {spin_lock (&vb->lock);if (vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 11 lines · Baoquan He
mm/vmalloc.c static void _vm_unmap_aliases(unsigned l · line 2241
Quoted 7 lines · Thomas Gleixner
mm/vmalloc.c +1
-0@@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned lfor_each_possible_cpu (cpu) {struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);struct vmap_block *vb; +unsigned long idx;rcu_read_lock ();Do we need to remove this
rcu_read_xx()pair since it marks the RCU read-side critical section on vbq-free list?
And what protects the xarray lookup?
Quoted 6 lines
Quoted 6 lines
mm/vmalloc.c +1
-1-list_for_each_entry_rcu (vb, &vbq->free, free_list) { +xa_for_each (&vbq->vmap_blocks, idx, vb) {spin_lock (&vb->lock);if (vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Baoquan He
Quoted 6 lines · Thomas Gleixner
mm/vmalloc.c +1
-0@@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned lfor_each_possible_cpu (cpu) {struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);struct vmap_block *vb; +unsigned long idx;
Quoted 6 lines
Quoted 4 lines
mm/vmalloc.c static void _vm_unmap_aliases(unsigned l · line 2241
Quoted 1 line
mm/vmalloc.c +0
-0rcu_read_lock ();Do we need to remove this
rcu_read_xx()pair since it marks the RCU read-side critical section on vbq-free list?And what protects the xarray lookup?
We have put rcu_read_lock() pair around the xas_find(). And it will find into xarray for each iteration item. We won't lose the connection to the next element like list adding or deleting? not very sure, I could be wrong.
xa_for_each()
-->xa_for_each_start()
-->xa_for_each_range()
-->xa_find()
void *xa_find(struct xarray *xa, unsigned long *indexp,
unsigned long max, xa_mark_t filter)
{
......
rcu_read_lock();
do {
if ((__force unsigned int)filter < XA_MAX_MARKS)
entry = xas_find_marked(&xas, max, filter);
else
entry = xas_find(&xas, max);
} while (xas_retry(&xas, entry));
rcu_read_unlock();
if (entry)
*indexp = xas.xa_index;
return entry;
}Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
mm/vmalloc.c +1
-1-list_for_each_entry_rcu (vb, &vbq->free, free_list) { +xa_for_each (&vbq->vmap_blocks, idx, vb) {spin_lock (&vb->lock);if (vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;
Re: [patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks
Quoted 21 lines · Baoquan He
Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Baoquan He
Quoted 6 lines · Thomas Gleixner
mm/vmalloc.c +1
-0@@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned lfor_each_possible_cpu (cpu) {struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);struct vmap_block *vb; +unsigned long idx;Quoted 6 lines
Quoted 4 lines
mm/vmalloc.c static void _vm_unmap_aliases(unsigned l · line 2241
Quoted 1 line
mm/vmalloc.c +0
-0rcu_read_lock ();Do we need to remove this
rcu_read_xx()pair since it marks the RCU read-side critical section on vbq-free list?And what protects the xarray lookup?
We have put
rcu_read_lock()pair around thexas_find(). And it will find into xarray for each iteration item. We won't lose the connection to the next element like list adding or deleting? not very sure, I could be wrong.
Quoted 4 lines
xa_for_each()
-->xa_for_each_start()
-->xa_for_each_range()-->xa_find()
I know how xarray works. No need to copy the code.
rcu_read_lock() inside xa_find() protects the search, but it does not protect the returned entry, which might go away right after xa_find() does rcu_read_unlock().
Thanks,
tglx
PATCH 2/6 mm/vmalloc: Avoid iterating over per CPU vmap blocks twice
In reply to Thomas Gleixner_vunmap_aliases() walks the per CPU xarrays to find partially unmapped blocks and then walks the per cpu free lists to purge fragmented blocks.
Arguably that's waste of CPU cycles and cache lines as the full xarray walk already touches every block.
Avoid this double iteration:
- Split out the code to purge one block and the code to free the local purge list into helper functions.
- Try to purge the fragmented blocks in the xarray walk before looking at their dirty space.
--- mm/vmalloc.c | 66 ++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 43 insertions(+), 23 deletions(-)
mm/vmalloc.c +43 -23
@@ -2086,39 +2086,52 @@ static void free_vmap_block(struct vmap_kfree_rcu (vb, rcu_head); } +static bool purge_fragmented_block (struct vmap_block *vb,struct vmap_block_queue *vbq, +struct list_head *purge_list) +{ +if (!(vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS )) +return false; + +/* prevent further allocs after releasing lock */ + vb->free =0 ; +/* prevent purging it again */ + vb->dirty =VMAP_BBMAP_BITS ; + vb->dirty_min =0 ; + vb->dirty_max =VMAP_BBMAP_BITS ; +spin_lock (&vbq->lock); +list_del_rcu (&vb->free_list); +spin_unlock (&vbq->lock); +list_add_tail (&vb->purge, purge_list); +return true; +} + +static void free_purged_blocks (struct list_head *purge_list) +{ +struct vmap_block *vb, *n_vb; + +list_for_each_entry_safe (vb, n_vb, purge_list, purge) { +list_del (&vb->purge); +free_vmap_block (vb); + } +} +static void purge_fragmented_blocks (int cpu) {LIST_HEAD (purge);struct vmap_block *vb; -struct vmap_block *n_vb;struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);rcu_read_lock ();list_for_each_entry_rcu (vb, &vbq->free, free_list) { -if (!(vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS ))continue ;spin_lock (&vb->lock); -if (vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS ) { - vb->free =0 ;/* prevent further allocs after releasing lock */ - vb->dirty =VMAP_BBMAP_BITS ;/* prevent purging it again */ - vb->dirty_min =0 ; - vb->dirty_max =VMAP_BBMAP_BITS ; -spin_lock (&vbq->lock); -list_del_rcu (&vb->free_list); -spin_unlock (&vbq->lock); -spin_unlock (&vb->lock); -list_add_tail (&vb->purge, &purge); - }else -spin_unlock (&vb->lock); +purge_fragmented_block (vb, vbq, &purge); +spin_unlock (&vb->lock); }rcu_read_unlock (); - -list_for_each_entry_safe (vb, n_vb, &purge, purge) { -list_del (&vb->purge); -free_vmap_block (vb); - } +free_purged_blocks (&purge); }static void purge_fragmented_blocks_allcpus (void )
@@ -2226,12 +2239,13 @@ static void vb_free(unsigned long addr,static void _vm_unmap_aliases (unsigned long start,unsigned long end,int flush) { +LIST_HEAD (purge_list);int cpu;if (unlikely (!vmap_initialized))return ; -might_sleep (); +mutex_lock (&vmap_purge_lock);for_each_possible_cpu (cpu) {struct vmap_block_queue *vbq = &per_cpu (vmap_block_queue, cpu);
@@ -2241,7 +2255,14 @@ static void _vm_unmap_aliases(unsigned lrcu_read_lock ();xa_for_each (&vbq->vmap_blocks, idx, vb) {spin_lock (&vb->lock); -if (vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) { + +/* +* Try to purge a fragmented block first. If it's +* not purgeable, check whether there is dirty +* space to be flushed. +*/ +if (!purge_fragmented_block (vb, vbq, &purge_list) && + vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;unsigned long s, e;
@@ -2257,9 +2278,8 @@ static void _vm_unmap_aliases(unsigned l }rcu_read_unlock (); } +free_purged_blocks (&purge_list); -mutex_lock (&vmap_purge_lock); -purge_fragmented_blocks_allcpus ();if (!__purge_vmap_area_lazy (start, end) && flush)flush_tlb_kernel_range (start, end);mutex_unlock (&vmap_purge_lock);
Re: [patch 2/6] mm/vmalloc: Avoid iterating over per CPU vmap blocks twice
Quoted 22 lines · Thomas Gleixner
_vunmap_aliases()walks the per CPU xarrays to find partially unmapped blocks and then walks the per cpu free lists to purge fragmented blocks.Arguably that's waste of CPU cycles and cache lines as the full xarray walk already touches every block.
Avoid this double iteration:
- Split out the code to purge one block and the code to free the local purge list into helper functions.
- Try to purge the fragmented blocks in the xarray walk before looking at their dirty space.
Signed-off-by Thomas Gleixner <[email protected]>--- mm/vmalloc.c | 66 ++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 43 insertions(+), 23 deletions(-)
mm/vmalloc.c static void free_vmap_block(struct vmap_ · line 2090
Quoted 5 lines
mm/vmalloc.c +2
-0@@ -2086,39 +2086,52 @@ static void free_vmap_block(struct vmap_kfree_rcu (vb, rcu_head); } +static bool purge_fragmented_block (struct vmap_block *vb,struct vmap_block_queue *vbq, +struct list_head *purge_list)
Please stick to 80 character lines for the vmalloc code. And while it's personal preference, two tab indents for the continuation make prototypes like this a lot more readable.
a lot easier if you
mm/vmalloc.c static void free_vmap_block(struct vmap_ · line 2093
Quoted 2 lines
mm/vmalloc.c +2
-0+if (!(vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS )) +return false;
This comes from the old code, but it does looks almost intentionally obsfucated..
if (vb->free + vb->dirty != VMAP_BBMAP_BITS || vb->dirty == VMAP_BBMAP_BITS)
actually gets the logic across much better.
mm/vmalloc.c static void free_vmap_block(struct vmap_ · line 2097
Quoted 3 lines
mm/vmalloc.c +3
-0+/* prevent further allocs after releasing lock */ + vb->free =0 ; +/* prevent purging it again */
extra spaces before the comments.
Otherwise the refactoring looks nice, thanks.
PATCH 3/6 mm/vmalloc: Prevent flushing dirty space over and over
In reply to Thomas Gleixnervmap blocks which have active mappings cannot be purged. Allocations which have been freed are accounted for in vmap_block::dirty_min/max, so that they can be detected in _vm_unmap_aliases() as potentially stale TLBs.
If there are several invocations of _vm_unmap_aliases() then each of them will flush the dirty range. That's pointless and just increases the probability of full TLB flushes.
Avoid that by resetting the flush range after accounting for it. That's safe versus other invocations of _vm_unmap_aliases() because this is all serialized with vmap_purge_lock.
--- mm/vmalloc.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/vmalloc.c +6 -2
@@ -2224,7 +2224,7 @@ static void vb_free(unsigned long addr,spin_lock (&vb->lock); -/* Expand dirty range */ +/* Expand the not yet TLB flushed dirty range */ vb->dirty_min =min (vb->dirty_min, offset); vb->dirty_max =max (vb->dirty_max, offset + (1UL << order));
@@ -2262,7 +2262,7 @@ static void _vm_unmap_aliases(unsigned l* space to be flushed. */ if (!purge_fragmented_block (vb, vbq, &purge_list) && - vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) { + vb->dirty_max && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;unsigned long s, e;
@@ -2272,6 +2272,10 @@ static void _vm_unmap_aliases(unsigned l start =min (s, start); end =max (e, end); +/* Prevent that this is flushed more than once */ + vb->dirty_min =VMAP_BBMAP_BITS ; + vb->dirty_max =0 ; + flush =1 ; }spin_unlock (&vb->lock);
Re: [patch 3/6] mm/vmalloc: Prevent flushing dirty space over and over
Quoted 11 lines · Thomas Gleixner
vmap blocks which have active mappings cannot be purged. Allocations which have been freed are accounted for in vmap_block::dirty_min/max, so that they can be detected in
_vm_unmap_aliases()as potentially stale TLBs.If there are several invocations of
_vm_unmap_aliases()then each of them will flush the dirty range. That's pointless and just increases the probability of full TLB flushes.Avoid that by resetting the flush range after accounting for it. That's safe versus other invocations of
_vm_unmap_aliases()because this is all serialized withvmap_purge_lock.
Just nitpicking, but isn't vb->lock the actually relevant lock here? vmap_purge_lock is only taken after the loop.
Re: [patch 3/6] mm/vmalloc: Prevent flushing dirty space over and over
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Thomas Gleixner
vmap blocks which have active mappings cannot be purged. Allocations which have been freed are accounted for in vmap_block::dirty_min/max, so that they can be detected in
_vm_unmap_aliases()as potentially stale TLBs.If there are several invocations of
_vm_unmap_aliases()then each of them will flush the dirty range. That's pointless and just increases the probability of full TLB flushes.
Quoted 6 lines
Quoted 3 lines
Avoid that by resetting the flush range after accounting for it. That's safe versus other invocations of
_vm_unmap_aliases()because this is all serialized withvmap_purge_lock.Just nitpicking, but isn't
vb->lockthe actually relevant lock here?vmap_purge_lockis only taken after the loop.
No. The avoid double list iteration change moves the purge lock up before the loop as it needs to protect against a concurrent purge attempt.
Thanks,
tglx
Re: [patch 3/6] mm/vmalloc: Prevent flushing dirty space over and over
In reply to Thomas GleixnerQuoted 38 lines · Thomas Gleixner
vmap blocks which have active mappings cannot be purged. Allocations which have been freed are accounted for in vmap_block::dirty_min/max, so that they can be detected in
_vm_unmap_aliases()as potentially stale TLBs.If there are several invocations of
_vm_unmap_aliases()then each of them will flush the dirty range. That's pointless and just increases the probability of full TLB flushes.Avoid that by resetting the flush range after accounting for it. That's safe versus other invocations of
_vm_unmap_aliases()because this is all serialized withvmap_purge_lock.Signed-off-by Thomas Gleixner <[email protected]>--- mm/vmalloc.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)mm/vmalloc.c +2
-2@@ -2224,7 +2224,7 @@ static void vb_free(unsigned long addr,spin_lock (&vb->lock); -/* Expand dirty range */ +/* Expand the not yet TLB flushed dirty range */ vb->dirty_min =min (vb->dirty_min, offset); vb->dirty_max =max (vb->dirty_max, offset + (1UL << order));@@ -2262,7 +2262,7 @@ static void _vm_unmap_aliases(unsigned l* space to be flushed. */ if (!purge_fragmented_block (vb, vbq, &purge_list) && - vb->dirty && vb->dirty !=VMAP_BBMAP_BITS ) { + vb->dirty_max && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;unsigned long s, e;
mm/vmalloc.c static void _vm_unmap_aliases(unsigned l · line 2278
Quoted 7 lines
mm/vmalloc.c +4
-0@@ -2272,6 +2272,10 @@ static void _vm_unmap_aliases(unsigned l start =min (s, start); end =max (e, end); +/* Prevent that this is flushed more than once */ + vb->dirty_min =VMAP_BBMAP_BITS ; + vb->dirty_max =0 ; +
This is really a great catch and improvement.
Quoted 4 lines
mm/vmalloc.c +0
-0flush =1 ; }spin_unlock (&vb->lock);
PATCH 4/6 mm/vmalloc: Check free space in vmap_block lockless
In reply to Thomas Gleixnervb_alloc() unconditionally locks a vmap_block on the free list to check the free space.
This can be done locklessly because vmap_block::free never increases, it's only decreased on allocations.
Check the free space lockless and only if that succeeds, recheck under the lock.
--- mm/vmalloc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/vmalloc.c +4 -1
@@ -2166,6 +2166,9 @@ static void *vb_alloc(unsigned long sizelist_for_each_entry_rcu (vb, &vbq->free, free_list) {unsigned long pages_off; +if (READ_ONCE (vb->free) < (1UL << order)) +continue ; +spin_lock (&vb->lock);if (vb->free < (1UL << order)) {spin_unlock (&vb->lock);
@@ -2174,7 +2177,7 @@ static void *vb_alloc(unsigned long size pages_off =VMAP_BBMAP_BITS - vb->free; vaddr =vmap_block_vaddr (vb->va->va_start, pages_off); - vb->free -=1UL << order; +WRITE_ONCE (vb->free, vb->free - (1UL << order));bitmap_set (vb->used_map, pages_off, (1UL << order));if (vb->free ==0 ) {spin_lock (&vbq->lock);
Re: [patch 4/6] mm/vmalloc: Check free space in vmap_block lockless
mm/vmalloc.c static void *vb_alloc(unsigned long size · line 2180
Quoted 12 lines · Thomas Gleixner
mm/vmalloc.c +4
-1+if (READ_ONCE (vb->free) < (1UL << order)) +continue ; +spin_lock (&vb->lock);if (vb->free < (1UL << order)) {spin_unlock (&vb->lock);@@ -2174,7 +2177,7 @@ static void *vb_alloc(unsigned long size pages_off =VMAP_BBMAP_BITS - vb->free; vaddr =vmap_block_vaddr (vb->va->va_start, pages_off); - vb->free -=1UL << order; +WRITE_ONCE (vb->free, vb->free - (1UL << order));
Maybe just a matter of preference, but wouldn't an atomic_t be better here? We'd have another locked instruction in the alloc path, but I always find the READ_ONCE/WRITE_ONCE usage a bit fragile that I'd rather reserve them to well documented hot path code.
Re: [patch 4/6] mm/vmalloc: Check free space in vmap_block lockless
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Thomas Gleixner
mm/vmalloc.c +3
-0+if (READ_ONCE (vb->free) < (1UL << order)) +continue ; +spin_lock (&vb->lock);if (vb->free < (1UL << order)) {spin_unlock (&vb->lock);@@ -2174,7 +2177,7 @@ static void *vb_alloc(unsigned long size
Quoted 10 lines
mm/vmalloc.c static void *vb_alloc(unsigned long size · line 2180
Quoted 4 lines
mm/vmalloc.c +1
-1pages_off =VMAP_BBMAP_BITS - vb->free; vaddr =vmap_block_vaddr (vb->va->va_start, pages_off); - vb->free -=1UL << order; +WRITE_ONCE (vb->free, vb->free - (1UL << order));Maybe just a matter of preference, but wouldn't an
atomic_tbe better here? We'd have another locked instruction in the alloc path, but I always find the READ_ONCE/WRITE_ONCE usage a bit fragile that I'd rather reserve them to well documented hot path code.
I don't see a problem with these lockless quickchecks, especially not in this particular case, but no strong opinion either.
Thanks
tglx
Re: [patch 4/6] mm/vmalloc: Check free space in vmap_block lockless
Quoted 21 lines · Thomas Gleixner
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Thomas Gleixner
mm/vmalloc.c +3
-0+if (READ_ONCE (vb->free) < (1UL << order)) +continue ; +spin_lock (&vb->lock);if (vb->free < (1UL << order)) {spin_unlock (&vb->lock);@@ -2174,7 +2177,7 @@ static void *vb_alloc(unsigned long sizeQuoted 10 lines
mm/vmalloc.c static void *vb_alloc(unsigned long size · line 2180
Quoted 4 lines
mm/vmalloc.c +1
-1pages_off =VMAP_BBMAP_BITS - vb->free; vaddr =vmap_block_vaddr (vb->va->va_start, pages_off); - vb->free -=1UL << order; +WRITE_ONCE (vb->free, vb->free - (1UL << order));Maybe just a matter of preference, but wouldn't an
atomic_tbe better here? We'd have another locked instruction in the alloc path, but I always find the READ_ONCE/WRITE_ONCE usage a bit fragile that I'd rather reserve them to well documented hot path code.
Quoted 3 lines
I don't see a problem with these lockless quickchecks, especially not in this particular case, but no strong opinion either.
PATCH 5/6 mm/vmalloc: Add missing READ/WRITE_ONCE() annotations
In reply to Thomas Gleixnerpurge_fragmented_blocks() accesses vmap_block::free and vmap_block::dirty lockless for a quick check.
Add the missing READ/WRITE_ONCE() annotations.
--- mm/vmalloc.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
mm/vmalloc.c +7 -4
@@ -2093,9 +2093,9 @@ static bool purge_fragmented_block(strucreturn false;/* prevent further allocs after releasing lock */ - vb->free =0 ; +WRITE_ONCE (vb->free,0 );/* prevent purging it again */ - vb->dirty =VMAP_BBMAP_BITS ; +WRITE_ONCE (vb->dirty,VMAP_BBMAP_BITS ); vb->dirty_min =0 ; vb->dirty_max =VMAP_BBMAP_BITS ;spin_lock (&vbq->lock);
@@ -2123,7 +2123,10 @@ static void purge_fragmented_blocks(intrcu_read_lock ();list_for_each_entry_rcu (vb, &vbq->free, free_list) { -if (!(vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS )) +unsigned long free =READ_ONCE (vb->free); +unsigned long dirty =READ_ONCE (vb->dirty); + +if (!(free + dirty ==VMAP_BBMAP_BITS && dirty !=VMAP_BBMAP_BITS ))continue ;spin_lock (&vb->lock);
@@ -2231,7 +2234,7 @@ static void vb_free(unsigned long addr, vb->dirty_min =min (vb->dirty_min, offset); vb->dirty_max =max (vb->dirty_max, offset + (1UL << order)); - vb->dirty +=1UL << order; +WRITE_ONCE (vb->dirty, vb->dirty + (1UL << order));if (vb->dirty ==VMAP_BBMAP_BITS ) {BUG_ON (vb->free);spin_unlock (&vb->lock);
Re: [patch 5/6] mm/vmalloc: Add missing READ/WRITE_ONCE() annotations
Quoted 38 lines · Thomas Gleixner
purge_fragmented_blocks()accessesvmap_block::freeandvmap_block::dirtylockless for a quick check.Add the missing READ/WRITE_ONCE() annotations.
Signed-off-by Thomas Gleixner <[email protected]>--- mm/vmalloc.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)mm/vmalloc.c +6
-3@@ -2093,9 +2093,9 @@ static bool purge_fragmented_block(strucreturn false;/* prevent further allocs after releasing lock */ - vb->free =0 ; +WRITE_ONCE (vb->free,0 );/* prevent purging it again */ - vb->dirty =VMAP_BBMAP_BITS ; +WRITE_ONCE (vb->dirty,VMAP_BBMAP_BITS ); vb->dirty_min =0 ; vb->dirty_max =VMAP_BBMAP_BITS ;spin_lock (&vbq->lock);@@ -2123,7 +2123,10 @@ static void purge_fragmented_blocks(intrcu_read_lock ();list_for_each_entry_rcu (vb, &vbq->free, free_list) { -if (!(vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS )) +unsigned long free =READ_ONCE (vb->free); +unsigned long dirty =READ_ONCE (vb->dirty); + +if (!(free + dirty ==VMAP_BBMAP_BITS && dirty !=VMAP_BBMAP_BITS ))continue ;spin_lock (&vb->lock);
mm/vmalloc.c static void vb_free(unsigned long addr, · line 2240
Quoted 8 lines
mm/vmalloc.c +1
-1@@ -2231,7 +2234,7 @@ static void vb_free(unsigned long addr, vb->dirty_min =min (vb->dirty_min, offset); vb->dirty_max =max (vb->dirty_max, offset + (1UL << order)); - vb->dirty +=1UL << order; +WRITE_ONCE (vb->dirty, vb->dirty + (1UL << order));if (vb->dirty ==VMAP_BBMAP_BITS ) {BUG_ON (vb->free);spin_unlock (&vb->lock);
--
Uladzislau Rezki
PATCH 6/6 mm/vmalloc: Dont purge usable blocks unnecessarily
In reply to Thomas GleixnerPurging fragmented blocks is done unconditionally in several contexts:
1) From drain_vmap_area_work(), when the number of lazy to be freed vmap_areas reached the threshold
2) Reclaiming vmalloc address space from pcpu_get_vm_areas()
3) _unmap_aliases()
#1 There is no reason to zap fragmented vmap blocks unconditionally, simply
because reclaiming all lazy areas drains at least
32MB * fls(num_online_cpus())
per invocation which is plenty.
#2 Reclaiming when running out of space or due to memory pressure makes a
lot of sense
#3 _unmap_aliases() requires to touch everything because the caller has no
clue which vmap_area used a particular page last and the vmap_area lost that information too.
Except for the vfree + VM_FLUSH_RESET_PERMS case, which removes the vmap area first and then cares about the flush. That in turn requires a full walk of _all_ vmap areas including the one which was just added to the purge list.
But as this has to be flushed anyway this is an opportunity to combine outstanding TLB flushes and do the housekeeping of purging freed areas, but like #1 there is no real good reason to zap usable vmap blocks unconditionally.
Add a @force_purge argument to the relevant functions and if not true only purge fragmented blocks which have less than 1/4 of their capacity left.
--- mm/vmalloc.c | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-)
mm/vmalloc.c +22 -12
@@ -791,7 +791,7 @@ get_subtree_max_size(struct rb_node *nodRB_DECLARE_CALLBACKS_MAX (static, free_vmap_area_rb_augment_cb, struct vmap_area, rb_node, unsigned long, subtree_max_size, va_size) -static void purge_vmap_area_lazy (void ); +static void purge_vmap_area_lazy (bool force_purge);static BLOCKING_NOTIFIER_HEAD (vmap_notify_list);static void drain_vmap_area_work (struct work_struct *work); static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work);
@@ -1649,7 +1649,7 @@ static struct vmap_area *alloc_vmap_area overflow:if (!purged) { -purge_vmap_area_lazy (); +purge_vmap_area_lazy (true); purged =1 ; goto retry; }
@@ -1717,7 +1717,7 @@ static atomic_long_t vmap_lazy_nr = ATOMstatic DEFINE_MUTEX (vmap_purge_lock);/* for per-CPU blocks */ -static void purge_fragmented_blocks_allcpus (void ); +static void purge_fragmented_blocks_allcpus (bool force_purge);/* * Purges all lazily-freed vmap areas.
@@ -1787,10 +1787,10 @@ static bool __purge_vmap_area_lazy(unsig/* * Kick off a purge of the outstanding lazy areas. */ -static void purge_vmap_area_lazy (void ) +static void purge_vmap_area_lazy (bool force_purge) {mutex_lock (&vmap_purge_lock); -purge_fragmented_blocks_allcpus (); +purge_fragmented_blocks_allcpus (force_purge);__purge_vmap_area_lazy (ULONG_MAX ,0 );mutex_unlock (&vmap_purge_lock); }
@@ -1908,6 +1908,12 @@ static struct vmap_area *find_unlink_vma#define VMAP_BLOCK_SIZE (VMAP_BBMAP_BITS * PAGE_SIZE) +/* +* Purge threshold to prevent overeager purging of fragmented blocks for +* regular operations: Purge if vb->free is less than 1/4 of the capacity. +*/ +#define VMAP_PURGE_THRESHOLD (VMAP_BBMAP_BITS / 4) +#define VMAP_RAM 0x1/* indicates vm_map_ram area*/ #define VMAP_BLOCK 0x2/* mark out the vmap_block sub-type*/ #define VMAP_FLAGS_MASK 0x3
@@ -2087,12 +2093,16 @@ static void free_vmap_block(struct vmap_ }static bool purge_fragmented_block (struct vmap_block *vb,struct vmap_block_queue *vbq, -struct list_head *purge_list) +struct list_head *purge_list,bool force_purge) {if (!(vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS ))return false; -/* prevent further allocs after releasing lock */ +/* Don't overeagerly purge usable blocks unless requested */ +if (!force_purge && vb->free <VMAP_PURGE_THRESHOLD ) +return false; + +/* prevent further allocs after releasing lock */ WRITE_ONCE (vb->free,0 );/* prevent purging it again */ WRITE_ONCE (vb->dirty,VMAP_BBMAP_BITS );
@@ -2115,7 +2125,7 @@ static void free_purged_blocks(struct li } } -static void purge_fragmented_blocks (int cpu) +static void purge_fragmented_blocks (int cpu,bool force_purge) {LIST_HEAD (purge);struct vmap_block *vb;
@@ -2130,19 +2140,19 @@ static void purge_fragmented_blocks(intcontinue ;spin_lock (&vb->lock); -purge_fragmented_block (vb, vbq, &purge); +purge_fragmented_block (vb, vbq, &purge, force_purge);spin_unlock (&vb->lock); }rcu_read_unlock ();free_purged_blocks (&purge); } -static void purge_fragmented_blocks_allcpus (void ) +static void purge_fragmented_blocks_allcpus (bool force_purge) {int cpu;for_each_possible_cpu (cpu) -purge_fragmented_blocks (cpu); +purge_fragmented_blocks (cpu, force_purge); }static void *vb_alloc (unsigned long size,gfp_t gfp_mask)
@@ -4173,7 +4183,7 @@ struct vm_struct **pcpu_get_vm_areas(con overflow:spin_unlock (&free_vmap_area_lock);if (!purged) { -purge_vmap_area_lazy (); +purge_vmap_area_lazy (true); purged = true;/* Before "retry", check if we recover. */
Re: [patch 6/6] mm/vmalloc: Dont purge usable blocks unnecessarily
Looks good:
Re: [patch 6/6] mm/vmalloc: Dont purge usable blocks unnecessarily
In reply to Thomas GleixnerQuoted 148 lines · Thomas Gleixner
Purging fragmented blocks is done unconditionally in several contexts:
1) From
drain_vmap_area_work(), when the number of lazy to be freedvmap_areasreached the threshold2) Reclaiming vmalloc address space from
pcpu_get_vm_areas()3)
_unmap_aliases()#1 There is no reason to zap fragmented vmap blocks unconditionally, simply
because reclaiming all lazy areas drains at least32MB * fls(num_online_cpus())per invocation which is plenty.
#2 Reclaiming when running out of space or due to memory pressure makes a
lot of sense#3
_unmap_aliases()requires to touch everything because the caller has no
clue whichvmap_areaused a particular page last and thevmap_arealost that information too.Except for the vfree +
VM_FLUSH_RESET_PERMScase, which removes the vmap area first and then cares about the flush. That in turn requires a full walk of _all_ vmap areas including the one which was just added to the purge list.But as this has to be flushed anyway this is an opportunity to combine outstanding TLB flushes and do the housekeeping of purging freed areas, but like #1 there is no real good reason to zap usable vmap blocks unconditionally.
Add a @force_purge argument to the relevant functions and if not true only purge fragmented blocks which have less than 1/4 of their capacity left.
Signed-off-by Thomas Gleixner <[email protected]>--- mm/vmalloc.c | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-)mm/vmalloc.c +21
-11@@ -791,7 +791,7 @@ get_subtree_max_size(struct rb_node *nodRB_DECLARE_CALLBACKS_MAX (static, free_vmap_area_rb_augment_cb, struct vmap_area, rb_node, unsigned long, subtree_max_size, va_size) -static void purge_vmap_area_lazy (void ); +static void purge_vmap_area_lazy (bool force_purge);static BLOCKING_NOTIFIER_HEAD (vmap_notify_list);static void drain_vmap_area_work (struct work_struct *work); static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work);@@ -1649,7 +1649,7 @@ static struct vmap_area *alloc_vmap_area overflow:if (!purged) { -purge_vmap_area_lazy (); +purge_vmap_area_lazy (true); purged =1 ; goto retry; }@@ -1717,7 +1717,7 @@ static atomic_long_t vmap_lazy_nr = ATOMstatic DEFINE_MUTEX (vmap_purge_lock);/* for per-CPU blocks */ -static void purge_fragmented_blocks_allcpus (void ); +static void purge_fragmented_blocks_allcpus (bool force_purge);/* * Purges all lazily-freed vmap areas. @@ -1787,10 +1787,10 @@ static bool __purge_vmap_area_lazy(unsig/* * Kick off a purge of the outstanding lazy areas. */ -static void purge_vmap_area_lazy (void ) +static void purge_vmap_area_lazy (bool force_purge) {mutex_lock (&vmap_purge_lock); -purge_fragmented_blocks_allcpus (); +purge_fragmented_blocks_allcpus (force_purge);__purge_vmap_area_lazy (ULONG_MAX ,0 );mutex_unlock (&vmap_purge_lock); }@@ -1908,6 +1908,12 @@ static struct vmap_area *find_unlink_vma#define VMAP_BLOCK_SIZE (VMAP_BBMAP_BITS * PAGE_SIZE) +/* +* Purge threshold to prevent overeager purging of fragmented blocks for +* regular operations: Purge if vb->free is less than 1/4 of the capacity. +*/ +#define VMAP_PURGE_THRESHOLD (VMAP_BBMAP_BITS / 4) +#define VMAP_RAM 0x1/* indicates vm_map_ram area*/ #define VMAP_BLOCK 0x2/* mark out the vmap_block sub-type*/ #define VMAP_FLAGS_MASK 0x3@@ -2087,12 +2093,16 @@ static void free_vmap_block(struct vmap_ }static bool purge_fragmented_block (struct vmap_block *vb,struct vmap_block_queue *vbq, -struct list_head *purge_list) +struct list_head *purge_list,bool force_purge) {if (!(vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS ))return false; -/* prevent further allocs after releasing lock */ +/* Don't overeagerly purge usable blocks unless requested */ +if (!force_purge && vb->free <VMAP_PURGE_THRESHOLD ) +return false; + +/* prevent further allocs after releasing lock */ WRITE_ONCE (vb->free,0 );/* prevent purging it again */ WRITE_ONCE (vb->dirty,VMAP_BBMAP_BITS );@@ -2115,7 +2125,7 @@ static void free_purged_blocks(struct li } } -static void purge_fragmented_blocks (int cpu) +static void purge_fragmented_blocks (int cpu,bool force_purge) {LIST_HEAD (purge);struct vmap_block *vb;@@ -2130,19 +2140,19 @@ static void purge_fragmented_blocks(intcontinue ;spin_lock (&vb->lock); -purge_fragmented_block (vb, vbq, &purge); +purge_fragmented_block (vb, vbq, &purge, force_purge);spin_unlock (&vb->lock); }rcu_read_unlock ();free_purged_blocks (&purge); } -static void purge_fragmented_blocks_allcpus (void ) +static void purge_fragmented_blocks_allcpus (bool force_purge) {int cpu;for_each_possible_cpu (cpu) -purge_fragmented_blocks (cpu); +purge_fragmented_blocks (cpu, force_purge); }static void *vb_alloc (unsigned long size,gfp_t gfp_mask)
mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(con · line 4189
Quoted 8 lines
mm/vmalloc.c +1
-1@@ -4173,7 +4183,7 @@ struct vm_struct **pcpu_get_vm_areas(con overflow:spin_unlock (&free_vmap_area_lock);if (!purged) { -purge_vmap_area_lazy (); +purge_vmap_area_lazy (true); purged = true;/* Before "retry", check if we recover. */
Wondering why bothering to add 'force_purge' to purge_vmap_area_lazy(), purge_fragmented_blocks_allcpus() if they are all true. Can't we set 'force_purge' as true for purge_fragmented_block() in purge_fragmented_blocks()?
alloc_vmap_area()pcpu_get_vm_areas() -->purge_vmap_area_lazy(true)
-->purge_fragmented_blocks_allcpus(force_purge=true)
-->purge_fragmented_block(force_purge=true)
mm/vmalloc.c +1 -1
@@ -2140,7 +2140,7 @@ static void purge_fragmented_blocks(int cpu, bool force_purge)continue ;spin_lock (&vb->lock); -purge_fragmented_block (vb, vbq, &purge, force_purge); +purge_fragmented_block (vb, vbq, &purge, true);spin_unlock (&vb->lock); }rcu_read_unlock ();
And one place of change is missing, it will fail building.
mm/vmalloc.c +1 -1
@@ -2277,7 +2277,7 @@ static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush)* not purgeable, check whether there is dirty * space to be flushed. */ -if (!purge_fragmented_block (vb, vbq, &purge_list) && +if (!purge_fragmented_block (vb, vbq, &purge_list, false) && vb->dirty_max && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;unsigned long s, e;
Quoted 1 line
Re: [patch 6/6] mm/vmalloc: Dont purge usable blocks unnecessarily
Quoted 1 line · Baoquan He
On 05/23/23 at 04:02pm, Thomas Gleixner wrote:
Can you please trim your replies? It's a pain to find them.
Quoted 16 lines
Review struct vm_struct **pcpu_get_vm_areas(con · line 4189
Quoted 10 lines
static void *vb_alloc(unsigned long size,
gfp_tgfp_mask)(quoted diff) +1
-1@@ -4173,7 +4183,7 @@ struct vm_struct **pcpu_get_vm_areas(con overflow: spin_unlock(&free_vmap_area_lock); if (!purged) { - purge_vmap_area_lazy(); + purge_vmap_area_lazy(true); purged = true; /* Before "retry", check if we recover. */Wondering why bothering to add '
force_purge' topurge_vmap_area_lazy(),purge_fragmented_blocks_allcpus()if they are all true. Can't we set 'force_purge' as true forpurge_fragmented_block()inpurge_fragmented_blocks()?
Quoted 5 lines
alloc_vmap_area()pcpu_get_vm_areas()-->purge_vmap_area_lazy(true)
-->purge_fragmented_blocks_allcpus(force_purge=true)-->purge_fragmented_block(force_purge=true)
Yes. purge_fragmented_blocks_allcpus() has only the purge_vmap_area_lazy() caller left.
Quoted 1 line
And one place of change is missing, it will fail building.
Already reported by Uladzislau and I sent a fix in reply.
Uladzislau Rezki
maintainerQuoted 19 lines · Thomas Gleixner
Following up to the discussion about excessive TLB flushes
https://lore.kernel.org/all/87a5y5a6kj.ffs@tglxthis series addresses the following issues:
1) Prevent the stale TLB problem related to fully utilized vmap blocks
2) Avoid the double per CPU list walk in
_vm_unmap_aliases()3) Avoid flushing dirty space over and over
4) Add a lockless quickcheck in
vb_alloc()and add missing READ/WRITE_ONCE() annotations
Quoted 6 lines
5) Prevent overeager purging of usable
vmap_blocksif not under memory/address space pressure.Thanks,
tglx
Thank you for fixing it!
<snip>
urezki@pc638:~/data/raid0/coding/linux.git$ make -j64 bzImage
DESCEND objtool
INSTALL libsubcmd_headers
CALL scripts/checksyscalls.sh
CC mm/vmalloc.o
In file included from ./include/linux/list_lru.h:14,
from ./include/linux/fs.h:13,
from ./include/linux/huge_mm.h:8,
from ./include/linux/mm.h:855,
from mm/vmalloc.c:12:
mm/vmalloc.c: In function ‘_vm_unmap_aliases’:
mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 | xa_for_each(&vbq->vmap_blocks, idx, vb) {
| ^~
./include/linux/xarray.h:449:23: note: in definition of macro ‘xa_for_each_range’
449 | entry = xa_find(xa, &index, last, XA_PRESENT); \
| ^~
./include/linux/xarray.h:501:2: note: in expansion of macro ‘xa_for_each_start’
501 | xa_for_each_start(xa, index, entry, 0)
| ^~~~~~~~~~~~~~~~~
mm/vmalloc.c:2220:3: note: in expansion of macro ‘xa_for_each’
2220 | xa_for_each(&vbq->vmap_blocks, idx, vb) {
| ^~~~~~~~~~~
mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 | xa_for_each(&vbq->vmap_blocks, idx, vb) {
| ^~
./include/linux/xarray.h:451:29: note: in definition of macro ‘xa_for_each_range’
451 | entry = xa_find_after(xa, &index, last, XA_PRESENT))
| ^~
./include/linux/xarray.h:501:2: note: in expansion of macro ‘xa_for_each_start’
501 | xa_for_each_start(xa, index, entry, 0)
| ^~~~~~~~~~~~~~~~~
mm/vmalloc.c:2220:3: note: in expansion of macro ‘xa_for_each’
2220 | xa_for_each(&vbq->vmap_blocks, idx, vb) {
| ^~~~~~~~~~~
mm/vmalloc.c:2228:9: error: too few arguments to function ‘purge_fragmented_block’
2228 | if (!purge_fragmented_block(vb, vbq, &purge_list) &&
| ^~~~~~~~~~~~~~~~~~~~~~
mm/vmalloc.c:2047:13: note: declared here
2047 | static bool purge_fragmented_block(struct vmap_block *vb, struct vmap_block_queue *vbq,
| ^~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [scripts/Makefile.build:252: mm/vmalloc.o] Error 1
make[1]: *** [scripts/Makefile.build:494: mm] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:2025: .] Error 2
urezki@pc638:~/data/raid0/coding/linux.git$ ^C
<snip>Could please fix it? :)
--
Uladzislau Rezki
Quoted 5 lines · Uladzislau Rezki
On Tue, May 23, 2023 at 04:02:09PM +0200, Thomas Gleixner wrote:
mm/vmalloc.c: In function ‘_vm_unmap_aliases’:mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){
| ^~
Duh. I surely had that compile fail fixed before I boot tested that pile. And then I did something stupid obviously.
Quoted 9 lines · Thomas Gleixner
Quoted 5 lines · Uladzislau Rezki
On Tue, May 23, 2023 at 04:02:09PM +0200, Thomas Gleixner wrote:
mm/vmalloc.c: In function ‘_vm_unmap_aliases’:mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){
| ^~Duh. I surely had that compile fail fixed before I boot tested that pile. And then I did something stupid obviously.
No. This one not. I only had the one in the last patch (missing force_purge argument)
And this one makes me scratch my head:
struct vmap_block_queue {
spinlock_t lock;
struct list_head free;
/*
* An xarray requires an extra memory dynamically to
* be allocated. If it is an issue, we can use rb-tree
* instead.
*/
struct xarray vmap_blocks;
};So how can your compiler complain?
Quoted 2 lines
Quoted 2 lines
mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){
Mine does not, but I might be missing something.
Thanks,
tglx
Uladzislau Rezki
maintainerQuoted 30 lines · Thomas Gleixner
Quoted 9 lines · Thomas Gleixner
Quoted 5 lines · Uladzislau Rezki
On Tue, May 23, 2023 at 04:02:09PM +0200, Thomas Gleixner wrote:
mm/vmalloc.c: In function ‘_vm_unmap_aliases’:mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){
| ^~Duh. I surely had that compile fail fixed before I boot tested that pile. And then I did something stupid obviously.
No. This one not. I only had the one in the last patch (missing
force_purgeargument)And this one makes me scratch my head:
struct vmap_block_queue { spinlock_t lock; struct list_head free; /* * An xarray requires an extra memory dynamically to * be allocated. If it is an issue, we can use rb-tree * instead. */ struct xarray vmap_blocks; };
Quoted 7 lines
So how can your compiler complain?
Quoted 2 lines
Quoted 2 lines
mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){Mine does not, but I might be missing something.
Right. I have applied your patches on the v6.3 what is not correct. I thought it should be fine, because that part was not touched quite a lot of time. Apparently, me, Lorenzo and Baoquan placed the vmap_blocks under the vmap_block_queue structure.
So, v6.3 does not contain that patch. I have to use the next instead.
--
Uladzislau Rezki
Uladzislau Rezki
maintainerQuoted 36 lines · Uladzislau Rezki
Quoted 35 lines · Thomas Gleixner
Quoted 9 lines · Thomas Gleixner
Quoted 5 lines · Uladzislau Rezki
On Tue, May 23, 2023 at 04:02:09PM +0200, Thomas Gleixner wrote:
mm/vmalloc.c: In function ‘_vm_unmap_aliases’:mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){
| ^~Duh. I surely had that compile fail fixed before I boot tested that pile. And then I did something stupid obviously.
No. This one not. I only had the one in the last patch (missing
force_purgeargument)And this one makes me scratch my head:
struct vmap_block_queue { spinlock_t lock; struct list_head free; /* * An xarray requires an extra memory dynamically to * be allocated. If it is an issue, we can use rb-tree * instead. */ struct xarray vmap_blocks; };So how can your compiler complain?
Quoted 2 lines
Quoted 2 lines
mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){
Quoted 9 lines
Quoted 2 lines
Mine does not, but I might be missing something.
Right. I have applied your patches on the v6.3 what is not correct. I thought it should be fine, because that part was not touched quite a lot of time. Apparently, me, Lorenzo and Baoquan placed the
vmap_blocksunder thevmap_block_queuestructure.So, v6.3 does not contain that patch. I have to use the next instead.
My fault :)
--
Uladzislau Rezki
Uladzislau Rezki
maintainerQuoted 36 lines · Uladzislau Rezki
Quoted 35 lines · Thomas Gleixner
Quoted 9 lines · Thomas Gleixner
Quoted 5 lines · Uladzislau Rezki
On Tue, May 23, 2023 at 04:02:09PM +0200, Thomas Gleixner wrote:
mm/vmalloc.c: In function ‘_vm_unmap_aliases’:mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){
| ^~Duh. I surely had that compile fail fixed before I boot tested that pile. And then I did something stupid obviously.
No. This one not. I only had the one in the last patch (missing
force_purgeargument)And this one makes me scratch my head:
struct vmap_block_queue { spinlock_t lock; struct list_head free; /* * An xarray requires an extra memory dynamically to * be allocated. If it is an issue, we can use rb-tree * instead. */ struct xarray vmap_blocks; };So how can your compiler complain?
Quoted 2 lines
Quoted 2 lines
mm/vmalloc.c:2220:19: error: ‘struct vmap_block_queue’ has no member named ‘vmap_blocks’
2220 |xa_for_each(&vbq->vmap_blocks, idx, vb){
Quoted 9 lines
Quoted 2 lines
Mine does not, but I might be missing something.
Right. I have applied your patches on the v6.3 what is not correct. I thought it should be fine, because that part was not touched quite a lot of time. Apparently, me, Lorenzo and Baoquan placed the
vmap_blocksunder thevmap_block_queuestructure.So, v6.3 does not contain that patch. I have to use the next instead.
next-20230523:
mm/vmalloc.c: In function ‘_vm_unmap_aliases’:mm/vmalloc.c:2280:9: error: too few arguments to function ‘purge_fragmented_block’
2280 | if (!purge_fragmented_block(vb, vbq, &purge_list) &&
| ^~~~~~~~~~~~~~~~~~~~~~
mm/vmalloc.c:2095:13: note: declared here
2095 | static bool purge_fragmented_block(struct vmap_block *vb, struct vmap_block_queue *vbq,
| ^~~~~~~~~~~~~~~~~~~~~~ CC drivers/acpi/pmic/intel_pmic_bxtwc.o
there is only one complain in fact.
--
Uladzislau Rezki
Quoted 4 lines · Uladzislau Rezki
Quoted 2 lines · Uladzislau Rezki
So, v6.3 does not contain that patch. I have to use the next instead.
next-20230523:
It works against linus tree just fine.
Quoted 11 lines
mm/vmalloc.c: In function ‘_vm_unmap_aliases’:mm/vmalloc.c:2280:9: error: too few arguments to function ‘purge_fragmented_block’
2280 | if (!purge_fragmented_block(vb, vbq,&purge_list) &&| ^~~~~~~~~~~~~~~~~~~~~~ mm/vmalloc.c:2095:13: note: declared here 2095 | static bool purge_fragmented_block(struct vmap_block *vb, struct vmap_block_queue *vbq, | ^~~~~~~~~~~~~~~~~~~~~~CC
drivers/acpi/pmic/intel_pmic_bxtwc.othere is only one complain in fact.
That's the one where I failed to pull the refreshed patch back from the test machine.
---
Subject: mm/vmalloc: Don't purge usable blocks unnecessarily
From: Thomas Gleixner <[email protected]>
Date: Mon, 22 May 2023 19:38:32 +0200
Purging fragmented blocks is done unconditionally in several contexts:
1) From drain_vmap_area_work(), when the number of lazy to be freed vmap_areas reached the threshold
2) Reclaiming vmalloc address space from pcpu_get_vm_areas()
3) _unmap_aliases()
#1 There is no reason to zap fragmented vmap blocks unconditionally, simply
because reclaiming all lazy areas drains at least
32MB * fls(num_online_cpus())
per invocation which is plenty.
#2 Reclaiming when running out of space or due to memory pressure makes a
lot of sense
#3 _unmap_aliases() requires to touch everything because the caller has no
clue which vmap_area used a particular page last and the vmap_area lost that information too.
Except for the vfree + VM_FLUSH_RESET_PERMS case, which removes the vmap area first and then cares about the flush. That in turn requires a full walk of _all_ vmap areas including the one which was just added to the purge list.
But as this has to be flushed anyway this is an opportunity to combine outstanding TLB flushes and do the housekeeping of purging freed areas, but like #1 there is no real good reason to zap usable vmap blocks unconditionally.
Add a @force_purge argument to the relevant functions and if not true only purge fragmented blocks which have less than 1/4 of their capacity left.
--- mm/vmalloc.c | 36 +++++++++++++++++++++++------------- 1 file changed, 23 insertions(+), 13 deletions(-)
mm/vmalloc.c +23 -13
@@ -791,7 +791,7 @@ get_subtree_max_size(struct rb_node *nodRB_DECLARE_CALLBACKS_MAX (static, free_vmap_area_rb_augment_cb, struct vmap_area, rb_node, unsigned long, subtree_max_size, va_size) -static void purge_vmap_area_lazy (void ); +static void purge_vmap_area_lazy (bool force_purge);static BLOCKING_NOTIFIER_HEAD (vmap_notify_list);static void drain_vmap_area_work (struct work_struct *work); static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work);
@@ -1649,7 +1649,7 @@ static struct vmap_area *alloc_vmap_area overflow:if (!purged) { -purge_vmap_area_lazy (); +purge_vmap_area_lazy (true); purged =1 ; goto retry; }
@@ -1717,7 +1717,7 @@ static atomic_long_t vmap_lazy_nr = ATOMstatic DEFINE_MUTEX (vmap_purge_lock);/* for per-CPU blocks */ -static void purge_fragmented_blocks_allcpus (void ); +static void purge_fragmented_blocks_allcpus (bool force_purge);/* * Purges all lazily-freed vmap areas.
@@ -1787,10 +1787,10 @@ static bool __purge_vmap_area_lazy(unsig/* * Kick off a purge of the outstanding lazy areas. */ -static void purge_vmap_area_lazy (void ) +static void purge_vmap_area_lazy (bool force_purge) {mutex_lock (&vmap_purge_lock); -purge_fragmented_blocks_allcpus (); +purge_fragmented_blocks_allcpus (force_purge);__purge_vmap_area_lazy (ULONG_MAX ,0 );mutex_unlock (&vmap_purge_lock); }
@@ -1908,6 +1908,12 @@ static struct vmap_area *find_unlink_vma#define VMAP_BLOCK_SIZE (VMAP_BBMAP_BITS * PAGE_SIZE) +/* +* Purge threshold to prevent overeager purging of fragmented blocks for +* regular operations: Purge if vb->free is less than 1/4 of the capacity. +*/ +#define VMAP_PURGE_THRESHOLD (VMAP_BBMAP_BITS / 4) +#define VMAP_RAM 0x1/* indicates vm_map_ram area*/ #define VMAP_BLOCK 0x2/* mark out the vmap_block sub-type*/ #define VMAP_FLAGS_MASK 0x3
@@ -2087,12 +2093,16 @@ static void free_vmap_block(struct vmap_ }static bool purge_fragmented_block (struct vmap_block *vb,struct vmap_block_queue *vbq, -struct list_head *purge_list) +struct list_head *purge_list,bool force_purge) {if (!(vb->free + vb->dirty ==VMAP_BBMAP_BITS && vb->dirty !=VMAP_BBMAP_BITS ))return false; -/* prevent further allocs after releasing lock */ +/* Don't overeagerly purge usable blocks unless requested */ +if (!force_purge && vb->free <VMAP_PURGE_THRESHOLD ) +return false; + +/* prevent further allocs after releasing lock */ WRITE_ONCE (vb->free,0 );/* prevent purging it again */ WRITE_ONCE (vb->dirty,VMAP_BBMAP_BITS );
@@ -2115,7 +2125,7 @@ static void free_purged_blocks(struct li } } -static void purge_fragmented_blocks (int cpu) +static void purge_fragmented_blocks (int cpu,bool force_purge) {LIST_HEAD (purge);struct vmap_block *vb;
@@ -2130,19 +2140,19 @@ static void purge_fragmented_blocks(intcontinue ;spin_lock (&vb->lock); -purge_fragmented_block (vb, vbq, &purge); +purge_fragmented_block (vb, vbq, &purge, force_purge);spin_unlock (&vb->lock); }rcu_read_unlock ();free_purged_blocks (&purge); } -static void purge_fragmented_blocks_allcpus (void ) +static void purge_fragmented_blocks_allcpus (bool force_purge) {int cpu;for_each_possible_cpu (cpu) -purge_fragmented_blocks (cpu); +purge_fragmented_blocks (cpu, force_purge); }static void *vb_alloc (unsigned long size,gfp_t gfp_mask)
@@ -2267,7 +2277,7 @@ static void _vm_unmap_aliases(unsigned l* not purgeable, check whether there is dirty * space to be flushed. */ -if (!purge_fragmented_block (vb, vbq, &purge_list) && +if (!purge_fragmented_block (vb, vbq, &purge_list, false) && vb->dirty_max && vb->dirty !=VMAP_BBMAP_BITS ) {unsigned long va_start = vb->va->va_start;unsigned long s, e;
@@ -4173,7 +4183,7 @@ struct vm_struct **pcpu_get_vm_areas(con overflow:spin_unlock (&free_vmap_area_lock);if (!purged) { -purge_vmap_area_lazy (); +purge_vmap_area_lazy (true); purged = true;/* Before "retry", check if we recover. */