Skip to contentSkip to conversation

mm/vmalloc.c: code cleanup and improvements

PATCH Superseded Baoquan He 4 participants 32 messages Apr 15, 2025 – Apr 17, 2025

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Baoquan He

author
#0 lore raw

cover mm/vmalloc.c: code cleanup and improvements

These were made from code inspection in mm/vmalloc.c.

Baoquan He (5):
mm/vmalloc.c: change purge_ndoes as local static variable mm/vmalloc.c: find the vmap of vmap_nodes in reverse order mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit mm/vmalloc: optimize function vm_unmap_aliases() mm/vmalloc.c: return explicit error value in alloc_vmap_area()

 mm/vmalloc.c | 68 +++++++++++++++++++++++++---------------------------
 1 file changed, 32 insertions(+), 36 deletions(-)
Signature · 1 line
2.41.0
author
#1 lore raw

PATCH 1/5 mm/vmalloc.c: change purge_ndoes as local static variable

Static variable 'purge_ndoes' is defined in global scope, while it's only used in function __purge_vmap_area_lazy(). It mainly serves to avoid memory allocation repeatedly, especially when NR_CPUS is big.

While a local static variable can also satisfy the demand, and can improve code readibility. Hence move its definition into __purge_vmap_area_lazy().

---
 mm/vmalloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmalloc.c +1 -1
index 6ee7fc2ec986..aca1905d3397 100644
@@ -2128,7 +2128,6 @@ static DEFINE_MUTEX(vmap_purge_lock);
 
 /* for per-CPU blocks */
 static void purge_fragmented_blocks_allcpus(void);
-static cpumask_t purge_nodes;
 
 static void
 reclaim_list_global(struct list_head *head)
@@ -2261,6 +2260,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
 {
 	unsigned long nr_purged_areas = 0;
 	unsigned int nr_purge_helpers;
+	static cpumask_t purge_nodes;
 	unsigned int nr_purge_nodes;
 	struct vmap_node *vn;
 	int i;
Signature · 1 line
2.41.0
maintainer Reviewed-by on v1 1/5
#2 lore raw

Re: [PATCH 1/5] mm/vmalloc.c: change purge_ndoes as local static variable

Quoted 36 lines · Baoquan He

Static variable 'purge_ndoes' is defined in global scope, while it's only used in function __purge_vmap_area_lazy(). It mainly serves to avoid memory allocation repeatedly, especially when NR_CPUS is big.

While a local static variable can also satisfy the demand, and can improve code readibility. Hence move its definition into __purge_vmap_area_lazy().

---
 mm/vmalloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmalloc.c +1 -1
index 6ee7fc2ec986..aca1905d3397 100644
@@ -2128,7 +2128,6 @@ static DEFINE_MUTEX(vmap_purge_lock);
 
 /* for per-CPU blocks */
 static void purge_fragmented_blocks_allcpus(void);
-static cpumask_t purge_nodes;
 
 static void
 reclaim_list_global(struct list_head *head)
@@ -2261,6 +2260,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
 {
 	unsigned long nr_purged_areas = 0;
 	unsigned int nr_purge_helpers;
+	static cpumask_t purge_nodes;
 	unsigned int nr_purge_nodes;
 	struct vmap_node *vn;
 	int i;
Signature · 1 line
2.41.0

Well. I do not have a strong opinion here. But right, it is only used inside the function.

--
Uladzislau Rezki

Reviewed-by Tested-by on v1 1/5
#3 lore raw

Re: [PATCH 1/5] mm/vmalloc.c: change purge_ndoes as local static variable

In reply to Baoquan He
Quoted 26 lines · Baoquan He

Static variable 'purge_ndoes' is defined in global scope, while it's only used in function __purge_vmap_area_lazy(). It mainly serves to avoid memory allocation repeatedly, especially when NR_CPUS is big.

While a local static variable can also satisfy the demand, and can improve code readibility. Hence move its definition into __purge_vmap_area_lazy().

---
 mm/vmalloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/vmalloc.c +0 -1
index 6ee7fc2ec986..aca1905d3397 100644
@@ -2128,7 +2128,6 @@ static DEFINE_MUTEX(vmap_purge_lock);
 
 /* for per-CPU blocks */
 static void purge_fragmented_blocks_allcpus(void);
-static cpumask_t purge_nodes;
 
 static void
 reclaim_list_global(struct list_head *head)

mm/vmalloc.c static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end, · line 2266

Quoted 7 lines
mm/vmalloc.c +1 -0
@@ -2261,6 +2260,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
 {
 	unsigned long nr_purged_areas = 0;
 	unsigned int nr_purge_helpers;
+	static cpumask_t purge_nodes;
 	unsigned int nr_purge_nodes;
 	struct vmap_node *vn;
 	int i;
Reviewed-by on v1 1/5
#4 lore raw

Re: [PATCH 1/5] mm/vmalloc.c: change purge_ndoes as local static variable

In reply to Baoquan He
Quoted 9 lines · Baoquan He

Static variable 'purge_ndoes' is defined in global scope, while it's only used in function __purge_vmap_area_lazy(). It mainly serves to avoid memory allocation repeatedly, especially when NR_CPUS is big.

While a local static variable can also satisfy the demand, and can improve code readibility. Hence move its definition into __purge_vmap_area_lazy().

author
#5 lore raw

PATCH 2/5 mm/vmalloc.c: find the vmap of vmap_nodes in reverse order

In reply to Baoquan He

When finding VA in vn->busy, if VA spans several zones and the passed addr is not the same as va->va_start, we should scan the vn in reverse odrdr because the starting address of VA must be smaller than the passed addr if it really resides in the VA.

E.g on a system nr_vmap_nodes=100,

     <----va---->
 -|-----|-----|-----|-----|-----|-----|-----|-----|-----|-
    ...   n-1   n    n+1   n+2   ...   100     0     1

VA resides in node 'n' whereas it spans 'n', 'n+1' and 'n+2'. If passed addr is within 'n+2', we should try nodes backwards on 'n+1' and 'n', then succeed very soon.

Meanwhile we still need loop around because VA could spans node from 'n' to node 100, node 0, node 1.

Anyway, changing to find in reverse order can improve efficiency on many CPUs system.

---
 mm/vmalloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/vmalloc.c +2 -2
index aca1905d3397..488d69b56765 100644
@@ -2436,7 +2436,7 @@ struct vmap_area *find_vmap_area(unsigned long addr)
 
 		if (va)
 			return va;
-	} while ((i = (i + 1) % nr_vmap_nodes) != j);
+	} while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j);
 
 	return NULL;
 }
@@ -2462,7 +2462,7 @@ static struct vmap_area *find_unlink_vmap_area(unsigned long addr)
 
 		if (va)
 			return va;
-	} while ((i = (i + 1) % nr_vmap_nodes) != j);
+	} while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j);
 
 	return NULL;
 }
Signature · 1 line
2.41.0
maintainer Reviewed-by on v1 2/5
#6 lore raw

Re: [PATCH 2/5] mm/vmalloc.c: find the vmap of vmap_nodes in reverse order

Quoted 51 lines · Baoquan He

When finding VA in vn->busy, if VA spans several zones and the passed addr is not the same as va->va_start, we should scan the vn in reverse odrdr because the starting address of VA must be smaller than the passed addr if it really resides in the VA.

E.g on a system nr_vmap_nodes=100,

     <----va---->
 -|-----|-----|-----|-----|-----|-----|-----|-----|-----|-
    ...   n-1   n    n+1   n+2   ...   100     0     1

VA resides in node 'n' whereas it spans 'n', 'n+1' and 'n+2'. If passed addr is within 'n+2', we should try nodes backwards on 'n+1' and 'n', then succeed very soon.

Meanwhile we still need loop around because VA could spans node from 'n' to node 100, node 0, node 1.

Anyway, changing to find in reverse order can improve efficiency on many CPUs system.

---
 mm/vmalloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/vmalloc.c +2 -2
index aca1905d3397..488d69b56765 100644
@@ -2436,7 +2436,7 @@ struct vmap_area *find_vmap_area(unsigned long addr)
 
 		if (va)
 			return va;
-	} while ((i = (i + 1) % nr_vmap_nodes) != j);
+	} while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j);
 
 	return NULL;
 }
@@ -2462,7 +2462,7 @@ static struct vmap_area *find_unlink_vmap_area(unsigned long addr)
 
 		if (va)
 			return va;
-	} while ((i = (i + 1) % nr_vmap_nodes) != j);
+	} while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j);
 
 	return NULL;
 }
Signature · 1 line
2.41.0

It depends. Consider a below situation:

             addr
              |
        VA    V

<------------>
<---|---|---|---|---|---|---|--->
0 1 2 3 0 1 2 3

basically it matters how big VA and how many nodes it spans. But i agree that an assumption to reverse back is more convinced in most cases.

--
Uladzislau Rezki

author
#7 lore raw

Re: [PATCH 2/5] mm/vmalloc.c: find the vmap of vmap_nodes in reverse order

Quoted 57 lines · Uladzislau Rezki
Quoted 51 lines · Baoquan He

When finding VA in vn->busy, if VA spans several zones and the passed addr is not the same as va->va_start, we should scan the vn in reverse odrdr because the starting address of VA must be smaller than the passed addr if it really resides in the VA.

E.g on a system nr_vmap_nodes=100,

     <----va---->
 -|-----|-----|-----|-----|-----|-----|-----|-----|-----|-
    ...   n-1   n    n+1   n+2   ...   100     0     1

VA resides in node 'n' whereas it spans 'n', 'n+1' and 'n+2'. If passed addr is within 'n+2', we should try nodes backwards on 'n+1' and 'n', then succeed very soon.

Meanwhile we still need loop around because VA could spans node from 'n' to node 100, node 0, node 1.

Anyway, changing to find in reverse order can improve efficiency on many CPUs system.

---
 mm/vmalloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/vmalloc.c +2 -2
index aca1905d3397..488d69b56765 100644
@@ -2436,7 +2436,7 @@ struct vmap_area *find_vmap_area(unsigned long addr)
 
 		if (va)
 			return va;
-	} while ((i = (i + 1) % nr_vmap_nodes) != j);
+	} while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j);
 
 	return NULL;
 }
@@ -2462,7 +2462,7 @@ static struct vmap_area *find_unlink_vmap_area(unsigned long addr)
 
 		if (va)
 			return va;
-	} while ((i = (i + 1) % nr_vmap_nodes) != j);
+	} while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j);
 
 	return NULL;
 }
Signature · 1 line
2.41.0

It depends. Consider a below situation:

             addr
              |
        VA    V
Quoted 7 lines

<------------>
<---|---|---|---|---|---|---|--->
0 1 2 3 0 1 2 3

basically it matters how big VA and how many nodes it spans. But i agree that an assumption to reverse back is more convinced in most cases.

Agree, on small system with few CPUs and big VA case, the advantage is not apparent.

Quoted 2 lines

Thanks.

Reviewed-by Tested-by on v1 2/5
#8 lore raw

Re: [PATCH 2/5] mm/vmalloc.c: find the vmap of vmap_nodes in reverse order

In reply to Baoquan He
Quoted 40 lines · Baoquan He

When finding VA in vn->busy, if VA spans several zones and the passed addr is not the same as va->va_start, we should scan the vn in reverse odrdr because the starting address of VA must be smaller than the passed addr if it really resides in the VA.

E.g on a system nr_vmap_nodes=100,

     <----va---->
 -|-----|-----|-----|-----|-----|-----|-----|-----|-----|-
    ...   n-1   n    n+1   n+2   ...   100     0     1

VA resides in node 'n' whereas it spans 'n', 'n+1' and 'n+2'. If passed addr is within 'n+2', we should try nodes backwards on 'n+1' and 'n', then succeed very soon.

Meanwhile we still need loop around because VA could spans node from 'n' to node 100, node 0, node 1.

Anyway, changing to find in reverse order can improve efficiency on many CPUs system.

---
 mm/vmalloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/vmalloc.c +1 -1
index aca1905d3397..488d69b56765 100644
@@ -2436,7 +2436,7 @@ struct vmap_area *find_vmap_area(unsigned long addr)
 
 		if (va)
 			return va;
-	} while ((i = (i + 1) % nr_vmap_nodes) != j);
+	} while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j);
 
 	return NULL;
 }

mm/vmalloc.c static struct vmap_area *find_unlink_vmap_area(unsigned long addr) · line 2468

Quoted 8 lines
mm/vmalloc.c +1 -1
@@ -2462,7 +2462,7 @@ static struct vmap_area *find_unlink_vmap_area(unsigned long addr)
 
 		if (va)
 			return va;
-	} while ((i = (i + 1) % nr_vmap_nodes) != j);
+	} while ((i = (i + nr_vmap_nodes - 1) % nr_vmap_nodes) != j);
 
 	return NULL;
 }

Thanks,
Shivank

author
#9 lore raw

PATCH 3/5 mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit

In reply to Baoquan He

When purge lazily freed vmap areas, VA stored in vn->pool[] will also be taken away into free vmap tree partially or completely accordingly, that is done in decay_va_pool_node(). When doing that, for each pool of node, the whole list is detached from the pool for handling. At this time, that pool is empty. It's not necessary to update the pool size each time when one VA is removed and addded into free vmap tree.

Here change code to update the pool size when attaching the pool back.

---
 mm/vmalloc.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/vmalloc.c +11 -12
index 488d69b56765..bf735c890878 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay;
+	unsigned long n_decay, len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,22 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		if (full_decay)
-			WRITE_ONCE(vn->pool[i].len, 0);
+		len = n_decay = vn->pool[i].len;
+		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
-		n_decay = vn->pool[i].len >> 2;
+		if (!full_decay)
+			n_decay >>= 2;
+		len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
+			if (!n_decay)
+				break;
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-
-			if (!full_decay) {
-				WRITE_ONCE(vn->pool[i].len, vn->pool[i].len - 1);
-
-				if (!--n_decay)
-					break;
-			}
+			n_decay--;
 		}
 
 		/*
@@ -2188,9 +2186,10 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		 * can populate the pool therefore a simple list replace
 		 * operation takes place here.
 		 */
-		if (!full_decay && !list_empty(&tmp_list)) {
+		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
+			vn->pool[i].len = len;
 			spin_unlock(&vn->pool_lock);
 		}
 	}
Signature · 1 line
2.41.0
#10 lore raw

Re: [PATCH 3/5] mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit

Quoted 59 lines · Baoquan He

When purge lazily freed vmap areas, VA stored in vn->pool[] will also be taken away into free vmap tree partially or completely accordingly, that is done in decay_va_pool_node(). When doing that, for each pool of node, the whole list is detached from the pool for handling. At this time, that pool is empty. It's not necessary to update the pool size each time when one VA is removed and addded into free vmap tree.

Here change code to update the pool size when attaching the pool back.

---
 mm/vmalloc.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/vmalloc.c +9 -11
index 488d69b56765..bf735c890878 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay;
+	unsigned long n_decay, len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,22 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		if (full_decay)
-			WRITE_ONCE(vn->pool[i].len, 0);
+		len = n_decay = vn->pool[i].len;
+		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
-		n_decay = vn->pool[i].len >> 2;
+		if (!full_decay)
+			n_decay >>= 2;
+		len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
+			if (!n_decay)
+				break;
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-
-			if (!full_decay) {
-				WRITE_ONCE(vn->pool[i].len, vn->pool[i].len - 1);
-
-				if (!--n_decay)
-					break;
-			}
+			n_decay--;
 		}
 
 		/*

mm/vmalloc.c decay_va_pool_node(struct vmap_node *vn, bool full_decay) · line 2192

Quoted 8 lines
mm/vmalloc.c +2 -1
@@ -2188,9 +2186,10 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		 * can populate the pool therefore a simple list replace
 		 * operation takes place here.
 		 */
-		if (!full_decay && !list_empty(&tmp_list)) {
+		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
+			vn->pool[i].len = len;

Current logic uses WRITE_ONCE() to update vn->pool[i].len. Could this lead to consistency issues?

Quoted 3 lines
mm/vmalloc.c +0 -0
 			spin_unlock(&vn->pool_lock);
 		}
 	}
author
#11 lore raw

Re: [PATCH 3/5] mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit

Quoted 58 lines · Shivank Garg
Quoted 57 lines · Baoquan He

When purge lazily freed vmap areas, VA stored in vn->pool[] will also be taken away into free vmap tree partially or completely accordingly, that is done in decay_va_pool_node(). When doing that, for each pool of node, the whole list is detached from the pool for handling. At this time, that pool is empty. It's not necessary to update the pool size each time when one VA is removed and addded into free vmap tree.

Here change code to update the pool size when attaching the pool back.

---
 mm/vmalloc.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/vmalloc.c +9 -11
index 488d69b56765..bf735c890878 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay;
+	unsigned long n_decay, len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,22 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		if (full_decay)
-			WRITE_ONCE(vn->pool[i].len, 0);
+		len = n_decay = vn->pool[i].len;
+		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
-		n_decay = vn->pool[i].len >> 2;
+		if (!full_decay)
+			n_decay >>= 2;
+		len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
+			if (!n_decay)
+				break;
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-
-			if (!full_decay) {
-				WRITE_ONCE(vn->pool[i].len, vn->pool[i].len - 1);
-
-				if (!--n_decay)
-					break;
-			}
+			n_decay--;
 		}
 
Quoted 13 lines

mm/vmalloc.c decay_va_pool_node(struct vmap_node *vn, bool full_decay) · line 2192

Quoted 10 lines
mm/vmalloc.c +2 -1
 		/*
@@ -2188,9 +2186,10 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		 * can populate the pool therefore a simple list replace
 		 * operation takes place here.
 		 */
-		if (!full_decay && !list_empty(&tmp_list)) {
+		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
+			vn->pool[i].len = len;

Current logic uses WRITE_ONCE() to update vn->pool[i].len. Could this lead to consistency issues?

Seems no necessary to use WRITE_ONCE(). I can change back to use WRITE_ONCE() just in case. Currently, it's only updated in node_alloc(), decay_va_pool_node(), purge_vmap_node(). And the latter two are inside vmap_purge_lock area.

Quoted 5 lines
Quoted 4 lines
mm/vmalloc.c +0 -0
 			spin_unlock(&vn->pool_lock);
 		}
 	}
Reviewed-by Tested-by
#12 lore raw

Re: [PATCH 3/5] mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit

Quoted 73 lines · Baoquan He
Quoted 58 lines · Shivank Garg
Quoted 57 lines · Baoquan He

When purge lazily freed vmap areas, VA stored in vn->pool[] will also be taken away into free vmap tree partially or completely accordingly, that is done in decay_va_pool_node(). When doing that, for each pool of node, the whole list is detached from the pool for handling. At this time, that pool is empty. It's not necessary to update the pool size each time when one VA is removed and addded into free vmap tree.

Here change code to update the pool size when attaching the pool back.

---
 mm/vmalloc.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/vmalloc.c +9 -11
index 488d69b56765..bf735c890878 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay;
+	unsigned long n_decay, len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,22 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		if (full_decay)
-			WRITE_ONCE(vn->pool[i].len, 0);
+		len = n_decay = vn->pool[i].len;
+		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
-		n_decay = vn->pool[i].len >> 2;
+		if (!full_decay)
+			n_decay >>= 2;
+		len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
+			if (!n_decay)
+				break;
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-
-			if (!full_decay) {
-				WRITE_ONCE(vn->pool[i].len, vn->pool[i].len - 1);
-
-				if (!--n_decay)
-					break;
-			}
+			n_decay--;
 		}
 
Quoted 13 lines

mm/vmalloc.c decay_va_pool_node(struct vmap_node *vn, bool full_decay) · line 2192

Quoted 10 lines
mm/vmalloc.c +2 -1
 		/*
@@ -2188,9 +2186,10 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		 * can populate the pool therefore a simple list replace
 		 * operation takes place here.
 		 */
-		if (!full_decay && !list_empty(&tmp_list)) {
+		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
+			vn->pool[i].len = len;

Current logic uses WRITE_ONCE() to update vn->pool[i].len. Could this lead to consistency issues?

Quoted 5 lines

Seems no necessary to use WRITE_ONCE(). I can change back to use WRITE_ONCE() just in case. Currently, it's only updated in node_alloc(), decay_va_pool_node(), purge_vmap_node(). And the latter two are inside vmap_purge_lock area.

Okay.

Best,
Shivank

Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
mm/vmalloc.c +0 -0
 			spin_unlock(&vn->pool_lock);
 		}
 	}
maintainer
#13 lore raw

Re: [PATCH 3/5] mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit

In reply to Baoquan He
Quoted 73 lines · Baoquan He

When purge lazily freed vmap areas, VA stored in vn->pool[] will also be taken away into free vmap tree partially or completely accordingly, that is done in decay_va_pool_node(). When doing that, for each pool of node, the whole list is detached from the pool for handling. At this time, that pool is empty. It's not necessary to update the pool size each time when one VA is removed and addded into free vmap tree.

Here change code to update the pool size when attaching the pool back.

---
 mm/vmalloc.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/vmalloc.c +11 -12
index 488d69b56765..bf735c890878 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay;
+	unsigned long n_decay, len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,22 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		if (full_decay)
-			WRITE_ONCE(vn->pool[i].len, 0);
+		len = n_decay = vn->pool[i].len;
+		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
-		n_decay = vn->pool[i].len >> 2;
+		if (!full_decay)
+			n_decay >>= 2;
+		len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
+			if (!n_decay)
+				break;
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-
-			if (!full_decay) {
-				WRITE_ONCE(vn->pool[i].len, vn->pool[i].len - 1);
-
-				if (!--n_decay)
-					break;
-			}
+			n_decay--;
 		}
 
 		/*
@@ -2188,9 +2186,10 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		 * can populate the pool therefore a simple list replace
 		 * operation takes place here.
 		 */
-		if (!full_decay && !list_empty(&tmp_list)) {
+		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
+			vn->pool[i].len = len;
 			spin_unlock(&vn->pool_lock);
 		}
 	}
Signature · 1 line
2.41.0

Which Linux version this patch is based on? I can not apply it.

Small nits:

<snip>

mm/vmalloc.c +5 -6
index c909b8fea6eb..0ae53c997219 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay, len;
+	unsigned long n_decay, pool_len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,21 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		len = n_decay = vn->pool[i].len;
+		pool_len = n_decay = vn->pool[i].len;
 		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
 		if (!full_decay)
 			n_decay >>= 2;
-		len -= n_decay;
+		pool_len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
-			if (!n_decay)
+			if (!n_decay--)
 				break;
 
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-			n_decay--;
 		}
 
 		/*
@@ -2190,7 +2189,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
-			vn->pool[i].len = len;
+			vn->pool[i].len = pool_len;
 			spin_unlock(&vn->pool_lock);
 		}
 	}

<snip>

on top of this?

a) decay in "if" statement, no need extra line;
b) rename len to something obvious - pool_len.

--
Uladzislau Rezki

author
#14 lore raw

Re: [PATCH 3/5] mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit

Quoted 67 lines · Uladzislau Rezki
Quoted 66 lines · Baoquan He

When purge lazily freed vmap areas, VA stored in vn->pool[] will also be taken away into free vmap tree partially or completely accordingly, that is done in decay_va_pool_node(). When doing that, for each pool of node, the whole list is detached from the pool for handling. At this time, that pool is empty. It's not necessary to update the pool size each time when one VA is removed and addded into free vmap tree.

Here change code to update the pool size when attaching the pool back.

---
 mm/vmalloc.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/vmalloc.c +10 -12
index 488d69b56765..bf735c890878 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay;
+	unsigned long n_decay, len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,22 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		if (full_decay)
-			WRITE_ONCE(vn->pool[i].len, 0);
+		len = n_decay = vn->pool[i].len;
+		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
-		n_decay = vn->pool[i].len >> 2;
+		if (!full_decay)
+			n_decay >>= 2;
+		len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
+			if (!n_decay)
+				break;
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-
-			if (!full_decay) {
-				WRITE_ONCE(vn->pool[i].len, vn->pool[i].len - 1);
-
-				if (!--n_decay)
-					break;
-			}
+			n_decay--;
 		}
 
 		/*
@@ -2188,9 +2186,10 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		 * can populate the pool therefore a simple list replace
 		 * operation takes place here.
 		 */
-		if (!full_decay && !list_empty(&tmp_list)) {
+		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
Quoted 8 lines
Quoted 7 lines
mm/vmalloc.c +1 -0
+			vn->pool[i].len = len;
 			spin_unlock(&vn->pool_lock);
 		}
 	}
Signature · 1 line
2.41.0

Which Linux version this patch is based on? I can not apply it.

I can apply them on the latest mainline kernel, next/master and mm-new branch of akpm/mm.git. I checked just now.

Quoted 53 lines

Small nits:

<snip>

mm/vmalloc.c +5 -6
index c909b8fea6eb..0ae53c997219 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay, len;
+	unsigned long n_decay, pool_len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,21 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		len = n_decay = vn->pool[i].len;
+		pool_len = n_decay = vn->pool[i].len;
 		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
 		if (!full_decay)
 			n_decay >>= 2;
-		len -= n_decay;
+		pool_len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
-			if (!n_decay)
+			if (!n_decay--)
 				break;
 
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-			n_decay--;
 		}
 
 		/*
@@ -2190,7 +2189,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
-			vn->pool[i].len = len;
+			vn->pool[i].len = pool_len;
 			spin_unlock(&vn->pool_lock);
 		}
 	}
<snip>
Quoted 4 lines

on top of this?

a) decay in "if" statement, no need extra line;
b) rename len to something obvious - pool_len.

All look good, will add them to this patch 3 of v2. Thanks a lot.

maintainer
#15 lore raw

Re: [PATCH 3/5] mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit

Quoted 77 lines · Baoquan He
Quoted 67 lines · Uladzislau Rezki
Quoted 66 lines · Baoquan He

When purge lazily freed vmap areas, VA stored in vn->pool[] will also be taken away into free vmap tree partially or completely accordingly, that is done in decay_va_pool_node(). When doing that, for each pool of node, the whole list is detached from the pool for handling. At this time, that pool is empty. It's not necessary to update the pool size each time when one VA is removed and addded into free vmap tree.

Here change code to update the pool size when attaching the pool back.

---
 mm/vmalloc.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
mm/vmalloc.c +10 -12
index 488d69b56765..bf735c890878 100644
@@ -2150,7 +2150,7 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 	LIST_HEAD(decay_list);
 	struct rb_root decay_root = RB_ROOT;
 	struct vmap_area *va, *nva;
-	unsigned long n_decay;
+	unsigned long n_decay, len;
 	int i;
 
 	for (i = 0; i < MAX_VA_SIZE_PAGES; i++) {
@@ -2164,22 +2164,20 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		list_replace_init(&vn->pool[i].head, &tmp_list);
 		spin_unlock(&vn->pool_lock);
 
-		if (full_decay)
-			WRITE_ONCE(vn->pool[i].len, 0);
+		len = n_decay = vn->pool[i].len;
+		WRITE_ONCE(vn->pool[i].len, 0);
 
 		/* Decay a pool by ~25% out of left objects. */
-		n_decay = vn->pool[i].len >> 2;
+		if (!full_decay)
+			n_decay >>= 2;
+		len -= n_decay;
 
 		list_for_each_entry_safe(va, nva, &tmp_list, list) {
+			if (!n_decay)
+				break;
 			list_del_init(&va->list);
 			merge_or_add_vmap_area(va, &decay_root, &decay_list);
-
-			if (!full_decay) {
-				WRITE_ONCE(vn->pool[i].len, vn->pool[i].len - 1);
-
-				if (!--n_decay)
-					break;
-			}
+			n_decay--;
 		}
 
 		/*
@@ -2188,9 +2186,10 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay)
 		 * can populate the pool therefore a simple list replace
 		 * operation takes place here.
 		 */
-		if (!full_decay && !list_empty(&tmp_list)) {
+		if (!list_empty(&tmp_list)) {
 			spin_lock(&vn->pool_lock);
 			list_replace_init(&tmp_list, &vn->pool[i].head);
Quoted 8 lines
Quoted 7 lines
mm/vmalloc.c +1 -0
+			vn->pool[i].len = len;
 			spin_unlock(&vn->pool_lock);
 		}
 	}
Signature · 1 line
2.41.0

Which Linux version this patch is based on? I can not apply it.

Quoted 3 lines

I can apply them on the latest mainline kernel, next/master and mm-new branch of akpm/mm.git. I checked just now.

Sounds good.

--
Uladzislau Rezki

author
#16 lore raw

PATCH 4/5 mm/vmalloc: optimize function vm_unmap_aliases()

In reply to Baoquan He

Remove unneeded local variables and replace them with values.

---
 mm/vmalloc.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
mm/vmalloc.c +1 -4
index bf735c890878..3f38a232663b 100644
@@ -2930,10 +2930,7 @@ static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush)
  */
 void vm_unmap_aliases(void)
 {
-	unsigned long start = ULONG_MAX, end = 0;
-	int flush = 0;
-
-	_vm_unmap_aliases(start, end, flush);
+	_vm_unmap_aliases(ULONG_MAX, 0, 0);
 }
 EXPORT_SYMBOL_GPL(vm_unmap_aliases);
 
Signature · 1 line
2.41.0
maintainer Reviewed-by on v1 4/5
#17 lore raw

Re: [PATCH 4/5] mm/vmalloc: optimize function vm_unmap_aliases()

Quoted 26 lines · Baoquan He

Remove unneeded local variables and replace them with values.

---
 mm/vmalloc.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
mm/vmalloc.c +1 -4
index bf735c890878..3f38a232663b 100644
@@ -2930,10 +2930,7 @@ static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush)
  */
 void vm_unmap_aliases(void)
 {
-	unsigned long start = ULONG_MAX, end = 0;
-	int flush = 0;
-
-	_vm_unmap_aliases(start, end, flush);
+	_vm_unmap_aliases(ULONG_MAX, 0, 0);
 }
 EXPORT_SYMBOL_GPL(vm_unmap_aliases);
 
Signature · 1 line
2.41.0

Makes sense!

--
Uladzislau Rezki

Reviewed-by Tested-by on v1 4/5
#18 lore raw

Re: [PATCH 4/5] mm/vmalloc: optimize function vm_unmap_aliases()

In reply to Baoquan He
Quoted 13 lines · Baoquan He

Remove unneeded local variables and replace them with values.

---
 mm/vmalloc.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
mm/vmalloc.c +0 -0
index bf735c890878..3f38a232663b 100644
@@ -2930,10 +2930,7 @@ static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush)
  */

mm/vmalloc.c static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush) · line 2936

Quoted 10 lines
mm/vmalloc.c +1 -4
 void vm_unmap_aliases(void)
 {
-	unsigned long start = ULONG_MAX, end = 0;
-	int flush = 0;
-
-	_vm_unmap_aliases(start, end, flush);
+	_vm_unmap_aliases(ULONG_MAX, 0, 0);
 }
 EXPORT_SYMBOL_GPL(vm_unmap_aliases);
 
author
#20 lore raw

PATCH 5/5 mm/vmalloc.c: return explicit error value in alloc_vmap_area()

In reply to Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +17 -17
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
 		/*
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}
 
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 
 	/* Check the "vend" restriction. */
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
+	if (ret)
+		return ret;
 
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long
 __alloc_vmap_area(struct rb_root *root, struct list_head *head,
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
 	va->va_start = addr;
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */
+		if ((unlikely(ret))) {
+			WARN_ONCE(1, "%s error: errno (%d)\n", __func__, ret);
 			goto recovery;
+		}
 
 		/* Allocated area. */
 		va = vas[area];
Signature · 1 line
2.41.0
author
#21 lore raw

Re: [PATCH 5/5] mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 8 lines · Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

^^^^^^^^^
Sorry, above line was added mistakenly in log draft, should be removed.

Quoted 120 lines
---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +17 -17
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
 		/*
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}
 
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 
 	/* Check the "vend" restriction. */
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
+	if (ret)
+		return ret;
 
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long
 __alloc_vmap_area(struct rb_root *root, struct list_head *head,
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
 	va->va_start = addr;
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */
+		if ((unlikely(ret))) {
+			WARN_ONCE(1, "%s error: errno (%d)\n", __func__, ret);
 			goto recovery;
+		}
 
 		/* Allocated area. */
 		va = vas[area];
Signature · 1 line
2.41.0
#22 lore raw

Re: [PATCH 5/5] mm/vmalloc.c: return explicit error value in alloc_vmap_area()

In reply to Baoquan He
Quoted 27 lines · Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +1 -1
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
 		/*

mm/vmalloc.c va_clip(struct rb_root *root, struct list_head *head, · line 1733

Quoted 6 lines
mm/vmalloc.c +1 -1
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}

Braces around return -EINVAL seem unnecessary.
They can be dropped.

Quoted 5 lines
mm/vmalloc.c +0 -0
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 
 	/* Check the "vend" restriction. */

mm/vmalloc.c va_alloc(struct vmap_area *va, · line 1766

Quoted 10 lines
mm/vmalloc.c +3 -3
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
+	if (ret)
+		return ret;

Is it safe to remove the warning, or was it critical for debugging?

Quoted 64 lines
mm/vmalloc.c +9 -10
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long
 __alloc_vmap_area(struct rb_root *root, struct list_head *head,
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
 	va->va_start = addr;

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4755

Quoted 6 lines
mm/vmalloc.c +1 -2
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */
+		if ((unlikely(ret))) {
		    ^^		   ^^

The extra parentheses are redundant and can be removed for clarity.

Quoted 6 lines
mm/vmalloc.c +2 -0
+			WARN_ONCE(1, "%s error: errno (%d)\n", __func__, ret);
 			goto recovery;
+		}
 
 		/* Allocated area. */
 		va = vas[area];
author
#23 lore raw

Re: [PATCH 5/5] mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 26 lines · Shivank Garg
Quoted 25 lines · Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +1 -1
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
Quoted 8 lines

mm/vmalloc.c va_clip(struct rb_root *root, struct list_head *head, · line 1733

Quoted 8 lines
mm/vmalloc.c +1 -1
 		/*
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}

Thanks for reviewing.

Quoted 3 lines

Braces around return -EINVAL seem unnecessary.
They can be dropped.

This complys with the codeing style required in 3) Placing Braces and Spaces of Documentation/process/coding-style.rst because other branches are multiple statements.

Quoted 10 lines
Quoted 10 lines
mm/vmalloc.c +1 -1
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 
 	/* Check the "vend" restriction. */
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
Quoted 8 lines

mm/vmalloc.c va_alloc(struct vmap_area *va, · line 1766

Quoted 6 lines
mm/vmalloc.c +2 -2
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
+	if (ret)
+		return ret;

Is it safe to remove the warning, or was it critical for debugging?

This comes from a reported concern because va_clip() could be failed by NOTHING_FIT or kmem_cache_alloc failure. The warning here could cause confusion misleading people to think vmap area management is failed.

Quoted 66 lines
Quoted 66 lines
mm/vmalloc.c +9 -10
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long
 __alloc_vmap_area(struct rb_root *root, struct list_head *head,
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
 	va->va_start = addr;
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
Quoted 7 lines

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4755

Quoted 5 lines
mm/vmalloc.c +1 -2
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */
+		if ((unlikely(ret))) {
		    ^^		   ^^

The extra parentheses are redundant and can be removed for clarity.

You are right, I will remove it. Thanks.

Quoted 8 lines
Quoted 7 lines
mm/vmalloc.c +2 -0
+			WARN_ONCE(1, "%s error: errno (%d)\n", __func__, ret);
 			goto recovery;
+		}
 
 		/* Allocated area. */
 		va = vas[area];
Reviewed-by Tested-by
#24 lore raw

Re: [PATCH 5/5] mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 143 lines · Baoquan He
Quoted 26 lines · Shivank Garg
Quoted 25 lines · Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +1 -1
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
Quoted 8 lines

mm/vmalloc.c va_clip(struct rb_root *root, struct list_head *head, · line 1733

Quoted 8 lines
mm/vmalloc.c +1 -1
 		/*
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}

Thanks for reviewing.

Quoted 3 lines

Braces around return -EINVAL seem unnecessary.
They can be dropped.

This complys with the codeing style required in 3) Placing Braces and Spaces of Documentation/process/coding-style.rst because other branches are multiple statements.

Quoted 10 lines
Quoted 10 lines
mm/vmalloc.c +1 -1
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 
 	/* Check the "vend" restriction. */
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
Quoted 8 lines

mm/vmalloc.c va_alloc(struct vmap_area *va, · line 1766

Quoted 6 lines
mm/vmalloc.c +2 -2
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
+	if (ret)
+		return ret;

Is it safe to remove the warning, or was it critical for debugging?

This comes from a reported concern because va_clip() could be failed by NOTHING_FIT or kmem_cache_alloc failure. The warning here could cause confusion misleading people to think vmap area management is failed.

Quoted 66 lines
Quoted 66 lines
mm/vmalloc.c +9 -10
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long
 __alloc_vmap_area(struct rb_root *root, struct list_head *head,
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
 	va->va_start = addr;
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
Quoted 7 lines

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4755

Quoted 5 lines
mm/vmalloc.c +1 -2
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */
+		if ((unlikely(ret))) {
		    ^^		   ^^

The extra parentheses are redundant and can be removed for clarity.

Quoted 2 lines

You are right, I will remove it. Thanks.

Please feel free to add following in next version.

Thanks,
Shivank

Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
mm/vmalloc.c +2 -0
+			WARN_ONCE(1, "%s error: errno (%d)\n", __func__, ret);
 			goto recovery;
+		}
 
 		/* Allocated area. */
 		va = vas[area];
author
#25 lore raw

Re: [PATCH 5/5] mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 146 lines · Shivank Garg
Quoted 143 lines · Baoquan He
Quoted 26 lines · Shivank Garg
Quoted 25 lines · Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +1 -1
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
Quoted 8 lines

mm/vmalloc.c va_clip(struct rb_root *root, struct list_head *head, · line 1733

Quoted 8 lines
mm/vmalloc.c +1 -1
 		/*
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}

Thanks for reviewing.

Quoted 3 lines

Braces around return -EINVAL seem unnecessary.
They can be dropped.

This complys with the codeing style required in 3) Placing Braces and Spaces of Documentation/process/coding-style.rst because other branches are multiple statements.

Quoted 10 lines
Quoted 10 lines
mm/vmalloc.c +1 -1
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 
 	/* Check the "vend" restriction. */
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
Quoted 8 lines

mm/vmalloc.c va_alloc(struct vmap_area *va, · line 1766

Quoted 6 lines
mm/vmalloc.c +2 -2
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
+	if (ret)
+		return ret;

Is it safe to remove the warning, or was it critical for debugging?

This comes from a reported concern because va_clip() could be failed by NOTHING_FIT or kmem_cache_alloc failure. The warning here could cause confusion misleading people to think vmap area management is failed.

Quoted 73 lines
Quoted 65 lines
mm/vmalloc.c +9 -10
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long
 __alloc_vmap_area(struct rb_root *root, struct list_head *head,
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
 	va->va_start = addr;

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4755

Quoted 6 lines
mm/vmalloc.c +1 -2
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */
+		if ((unlikely(ret))) {
		    ^^		   ^^

The extra parentheses are redundant and can be removed for clarity.

Quoted 7 lines
Quoted 2 lines

You are right, I will remove it. Thanks.

Please feel free to add following in next version.

Thanks a lot for your careful reviewing and testing.

Quoted 11 lines
Quoted 10 lines
Quoted 9 lines
Quoted 8 lines
mm/vmalloc.c +2 -0
+			WARN_ONCE(1, "%s error: errno (%d)\n", __func__, ret);
 			goto recovery;
+		}
 
 		/* Allocated area. */
 		va = vas[area];
maintainer
#26 lore raw

Re: [PATCH 5/5] mm/vmalloc.c: return explicit error value in alloc_vmap_area()

In reply to Baoquan He
Quoted 37 lines · Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +2 -2
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
 		/*
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}
 
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 

mm/vmalloc.c va_alloc(struct vmap_area *va, · line 1765

Quoted 10 lines
mm/vmalloc.c +1 -3
 	/* Check the "vend" restriction. */
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
 

Not clear why you remove this WARN_ON by this patch. It should be a separate patch or just keep it as is. The warning here can mean that something is really wrong, especially if NOTHING_FIT. So we definitely want the warning.

Quoted 18 lines
mm/vmalloc.c +3 -1
+	if (ret)
+		return ret;
 
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long

__alloc_vmap_area(struct rb_root *root, struct list_head *head,

mm/vmalloc.c +0 -1
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;

mm/vmalloc.c __alloc_vmap_area(struct rb_root *root, struct list_head *head, · line 1804

Quoted 10 lines
mm/vmalloc.c +2 -3
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
 

Just keep it as it was. No need to check if addr is valid or not.

Quoted 39 lines
mm/vmalloc.c +1 -0
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
mm/vmalloc.c +5 -5
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
 	va->va_start = addr;

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4757

Quoted 5 lines
mm/vmalloc.c +0 -2
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */

Keep the comment.

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4759

Quoted 5 lines
mm/vmalloc.c +3 -0
+		if ((unlikely(ret))) {
+			WARN_ONCE(1, "%s error: errno (%d)\n", __func__, ret);
 			goto recovery;
+		}
 

--
Uladzislau Rezki

author
#27 lore raw

Re: [PATCH 5/5] mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 43 lines · Uladzislau Rezki
Quoted 42 lines · Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +3 -3
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
 		/*
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}
 
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 
 	/* Check the "vend" restriction. */
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
Quoted 9 lines

mm/vmalloc.c va_alloc(struct vmap_area *va, · line 1767

Quoted 5 lines
mm/vmalloc.c +0 -2
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
 

Not clear why you remove this WARN_ON by this patch. It should be a separate patch or just keep it as is. The warning here can mean that something is really wrong, especially if NOTHING_FIT. So we definitely want the warning.

I remember one time someone reported that the slab allocation failure triggered this warning which is confusing to them. But yes, it should be discussed in a separate post or thread, not appropriate to remove it silently. I will add it back in v2.

Quoted 25 lines
Quoted 25 lines
mm/vmalloc.c +3 -1
+	if (ret)
+		return ret;
 
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long

__alloc_vmap_area(struct rb_root *root, struct list_head *head,

mm/vmalloc.c +1 -3
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
Quoted 5 lines

mm/vmalloc.c __alloc_vmap_area(struct rb_root *root, struct list_head *head, · line 1806

Quoted 4 lines
mm/vmalloc.c +1 -1
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
 

Just keep it as it was. No need to check if addr is valid or not.

This is to keep consistent with the old code. Before this patch, if va_alloc() return vend, it returns directly, no find_vmap_lowest_match_check() invocation is done. I tried to keep the behaviour unchanged. That code is for debugging, both is fine to me.

Quoted 38 lines
Quoted 38 lines
mm/vmalloc.c +1 -0
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
mm/vmalloc.c +5 -5
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
Quoted 8 lines

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4757

Quoted 7 lines
mm/vmalloc.c +0 -2
 	va->va_start = addr;
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */

Keep the comment.

OK, will add it back.

Quoted 10 lines

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4759

Quoted 6 lines
mm/vmalloc.c +3 -0
+		if ((unlikely(ret))) {
+			WARN_ONCE(1, "%s error: errno (%d)\n", __func__, ret);
 			goto recovery;
+		}
 

--
Uladzislau Rezki

maintainer
#28 lore raw

Re: [PATCH 5/5] mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 54 lines · Baoquan He
Quoted 43 lines · Uladzislau Rezki
Quoted 42 lines · Baoquan He

In codes of alloc_vmap_area(), it returns the upper bound 'vend' to indicate if the allocation is successful or failed. That is not very clear.

Here change to return explicit error values and check them to judge if allocation is successful.

IS_ERR_VALUE already uses unlikely() internally

---
 mm/vmalloc.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
mm/vmalloc.c +3 -3
index 3f38a232663b..5b21cd09b2b4 100644
@@ -1715,7 +1715,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 			 */
 			lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
 			if (!lva)
-				return -1;
+				return -ENOMEM;
 		}
 
 		/*
@@ -1729,7 +1729,7 @@ va_clip(struct rb_root *root, struct list_head *head,
 		 */
 		va->va_start = nva_start_addr + size;
 	} else {
-		return -1;
+		return -EINVAL;
 	}
 
 	if (type != FL_FIT_TYPE) {
@@ -1758,19 +1758,19 @@ va_alloc(struct vmap_area *va,
 
 	/* Check the "vend" restriction. */
 	if (nva_start_addr + size > vend)
-		return vend;
+		return -ERANGE;
 
Quoted 9 lines

mm/vmalloc.c va_alloc(struct vmap_area *va, · line 1767

Quoted 5 lines
mm/vmalloc.c +0 -2
 	/* Update the free vmap_area. */
 	ret = va_clip(root, head, va, nva_start_addr, size);
-	if (WARN_ON_ONCE(ret))
-		return vend;
 

Not clear why you remove this WARN_ON by this patch. It should be a separate patch or just keep it as is. The warning here can mean that something is really wrong, especially if NOTHING_FIT. So we definitely want the warning.

Quoted 5 lines

I remember one time someone reported that the slab allocation failure triggered this warning which is confusing to them. But yes, it should be discussed in a separate post or thread, not appropriate to remove it silently. I will add it back in v2.

Thanks!

Quoted 31 lines
Quoted 25 lines
Quoted 25 lines
mm/vmalloc.c +3 -1
+	if (ret)
+		return ret;
 
 	return nva_start_addr;
 }
 
 /*
  * Returns a start address of the newly allocated area, if success.
- * Otherwise a vend is returned that indicates failure.
+ * Otherwise an error value is returned that indicates failure.
  */
 static __always_inline unsigned long

__alloc_vmap_area(struct rb_root *root, struct list_head *head,

mm/vmalloc.c +1 -3
@@ -1795,14 +1795,13 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head,
 
 	va = find_vmap_lowest_match(root, size, align, vstart, adjust_search_size);
 	if (unlikely(!va))
-		return vend;
+		return -ENOENT;
 
 	nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
-	if (nva_start_addr == vend)
-		return vend;
 
Quoted 5 lines

mm/vmalloc.c __alloc_vmap_area(struct rb_root *root, struct list_head *head, · line 1806

Quoted 4 lines
mm/vmalloc.c +1 -1
 #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
-	find_vmap_lowest_match_check(root, head, size, align);
+	if (!IS_ERR_VALUE(nva_start_addr))
 

Just keep it as it was. No need to check if addr is valid or not.

Quoted 5 lines

This is to keep consistent with the old code. Before this patch, if va_alloc() return vend, it returns directly, no find_vmap_lowest_match_check() invocation is done. I tried to keep the behaviour unchanged. That code is for debugging, both is fine to me.

Ack. Makes sense to keep same behaviour as it was/is.

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
mm/vmalloc.c +1 -0
+		find_vmap_lowest_match_check(root, head, size, align);
 #endif
 
mm/vmalloc.c +5 -5
 	return nva_start_addr;
@@ -1932,7 +1931,7 @@ node_alloc(unsigned long size, unsigned long align,
 	struct vmap_area *va;
 
 	*vn_id = 0;
-	*addr = vend;
+	*addr = -EINVAL;
 
 	/*
 	 * Fallback to a global heap if not vmalloc or there
@@ -2012,20 +2011,20 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
 	}
 
 retry:
-	if (addr == vend) {
+	if (IS_ERR_VALUE(addr)) {
 		preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node);
 		addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list,
 			size, align, vstart, vend);
 		spin_unlock(&free_vmap_area_lock);
 	}
 
-	trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend);
+	trace_alloc_vmap_area(addr, size, align, vstart, vend, IS_ERR_VALUE(addr));
 
 	/*
-	 * If an allocation fails, the "vend" address is
+	 * If an allocation fails, the error value is
 	 * returned. Therefore trigger the overflow path.
 	 */
-	if (unlikely(addr == vend))
+	if (IS_ERR_VALUE(addr))
 		goto overflow;
 
 	va->va_start = addr;
@@ -4753,9 +4752,10 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 
Quoted 8 lines
Quoted 5 lines

mm/vmalloc.c struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, · line 4757

Quoted 4 lines
mm/vmalloc.c +0 -2
 		ret = va_clip(&free_vmap_area_root,
 			&free_vmap_area_list, va, start, size);
-		if (WARN_ON_ONCE(unlikely(ret)))
-			/* It is a BUG(), but trigger recovery instead. */

Keep the comment.

OK, will add it back.

Thank you.

--
Uladzislau Rezki

maintainer
#29 lore raw
In reply to Baoquan He
Quoted 9 lines · Baoquan He

These were made from code inspection in mm/vmalloc.c.

Baoquan He (5):
mm/vmalloc.c: change purge_ndoes as local static variable mm/vmalloc.c: find the vmap of vmap_nodes in reverse order mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit mm/vmalloc: optimize function vm_unmap_aliases() mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 6 lines
 mm/vmalloc.c | 68 +++++++++++++++++++++++++---------------------------
 1 file changed, 32 insertions(+), 36 deletions(-)
Signature · 1 line
2.41.0

I have review some patches, the rest i will check tomorrow!

--
Uladzislau Rezki

author
#30 lore raw
Quoted 10 lines · Uladzislau Rezki
Quoted 9 lines · Baoquan He

These were made from code inspection in mm/vmalloc.c.

Baoquan He (5):
mm/vmalloc.c: change purge_ndoes as local static variable mm/vmalloc.c: find the vmap of vmap_nodes in reverse order mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit mm/vmalloc: optimize function vm_unmap_aliases() mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 7 lines
Quoted 6 lines
 mm/vmalloc.c | 68 +++++++++++++++++++++++++---------------------------
 1 file changed, 32 insertions(+), 36 deletions(-)
Signature · 1 line
2.41.0

I have review some patches, the rest i will check tomorrow!

Thanks a lot for your quick and careful reviewing.

Reviewed-by Tested-by on the v1 series
#31 lore raw
In reply to Baoquan He
Quoted 9 lines · Baoquan He

These were made from code inspection in mm/vmalloc.c.

Baoquan He (5):
mm/vmalloc.c: change purge_ndoes as local static variable mm/vmalloc.c: find the vmap of vmap_nodes in reverse order mm/vmalloc.c: optimize code in decay_va_pool_node() a little bit mm/vmalloc: optimize function vm_unmap_aliases() mm/vmalloc.c: return explicit error value in alloc_vmap_area()

Quoted 3 lines
 mm/vmalloc.c | 68 +++++++++++++++++++++++++---------------------------
 1 file changed, 32 insertions(+), 36 deletions(-)

I've tested this patch-series on my AMD EPYC ZEN 3 machine using

1. Using lib/test_vmalloc.c: modprobe test_vmalloc
2. stress-ng: stress-ng --class memory,vm --seq 0

I did not find any issues.

Thanks,
Shivank