Skip to contentSkip to conversation

mm/slub: batch-detach node partial slabs

PATCH Superseded Hao Li 3 participants 5 messages May 25 – May 27

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Hao Li

author reviewer
#0 lore raw

PATCH 1/1 mm/slub: batch-detach node partial slabs

get_partial_node_bulk() used to move each selected slab from the node partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.

Instead, track contiguous runs of matching slabs and move each run with list_bulk_move_tail() in one operation. This reduces list pointer churn inside the lock critical section.

The mmap2 testcase shows a 5% improvement after applying this patch.

---
 mm/slub.c | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)
mm/slub.c +18 -4
index 04692a6f9128..180973a4a3d2 100644
@@ -3739,6 +3739,7 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 				  bool allow_spin)
 {
 	struct slab *slab, *slab2;
+	struct slab *first = NULL, *last = NULL;
 	unsigned int total_free = 0;
 	unsigned long flags;
 
@@ -3757,8 +3758,15 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 		struct freelist_counters flc;
 		unsigned int slab_free;
 
-		if (!pfmemalloc_match(slab, pc->flags))
+		if (!pfmemalloc_match(slab, pc->flags)) {
+			if (first) {
+				list_bulk_move_tail(&pc->slabs,
+						    &first->slab_list,
+						    &last->slab_list);
+				first = NULL;
+			}
 			continue;
+		}
 
 		/*
 		 * determine the number of free objects in the slab racily
@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 		    && total_free + slab_free > pc->max_objects)
 			break;
 
-		remove_partial(n, slab);
-
-		list_add(&slab->slab_list, &pc->slabs);
+		if (!first)
+			first = slab;
+		last = slab;
+		slab_clear_node_partial(slab);
+		n->nr_partial--;
 
 		total_free += slab_free;
 		if (total_free >= pc->max_objects)
 			break;
 	}
 
+	if (first)
+		list_bulk_move_tail(&pc->slabs, &first->slab_list,
+				    &last->slab_list);
+
 	spin_unlock_irqrestore(&n->list_lock, flags);
 	return total_free > 0;
 }
Signature · 1 line
2.54.0
maintainer
#1 lore raw
Quoted 8 lines · Hao Li

get_partial_node_bulk() used to move each selected slab from the node partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.

Instead, track contiguous runs of matching slabs and move each run with list_bulk_move_tail() in one operation.

TIL list_bulk_move_tail() :D

Quoted 1 line

This reduces list pointer churn> inside the lock critical section.

Similar to this, can we return all slabs in pc->slabs at once when returning those slabs to the list? ... I see Vlastimil removed 'nr of empty slabs' check in the other series already.

Now that it inserts slabs to the tail with Vlastimil's patchset, let's do a list_splice_tail() instead?

Quoted 14 lines

The mmap2 testcase shows a 5% improvement after applying this patch.

---
  mm/slub.c | 22 ++++++++++++++++++----
  1 file changed, 18 insertions(+), 4 deletions(-)
mm/slub.c +0 -0
index 04692a6f9128..180973a4a3d2 100644
@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
  		    && total_free + slab_free > pc->max_objects)
  			break;

mm/slub.c static bool get_partial_node_bulk(struct kmem_cache *s, · line 3791

Quoted 10 lines
mm/slub.c +5 -3
 
-		remove_partial(n, slab);
-
-		list_add(&slab->slab_list, &pc->slabs);
+		if (!first)
+			first = slab;
+		last = slab;
 
+		slab_clear_node_partial(slab);
+		n->nr_partial--;

Perhaps factor out those two statements into to a common function and call it in get_partial_node_bulk() and remove_partial()?

Quoted 12 lines
mm/slub.c +4 -0
  		total_free += slab_free;
  		if (total_free >= pc->max_objects)
  			break;
  	}
 
+	if (first)
+		list_bulk_move_tail(&pc->slabs, &first->slab_list,
+				    &last->slab_list);
+
  	spin_unlock_irqrestore(&n->list_lock, flags);
  	return total_free > 0;
  }
Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#2 lore raw
Quoted 9 lines · Harry Yoo
Quoted 6 lines · Hao Li

get_partial_node_bulk() used to move each selected slab from the node partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.

Quoted 4 lines
Quoted 2 lines

Instead, track contiguous runs of matching slabs and move each run with list_bulk_move_tail() in one operation.

TIL list_bulk_move_tail() :D

I had to dig through list.h for ages just to find it :P

Quoted 9 lines
Quoted 2 lines

This reduces list pointer churn> inside the lock critical section.

Similar to this, can we return all slabs in pc->slabs at once when returning those slabs to the list? ... I see Vlastimil removed 'nr of empty slabs' check in the other series already.

Now that it inserts slabs to the tail with Vlastimil's patchset, let's do a list_splice_tail() instead?

Great idea! then both get and put operations will be highly efficient.

Quoted 22 lines
Quoted 21 lines

The mmap2 testcase shows a 5% improvement after applying this patch.

---
  mm/slub.c | 22 ++++++++++++++++++----
  1 file changed, 18 insertions(+), 4 deletions(-)
mm/slub.c +3 -3
index 04692a6f9128..180973a4a3d2 100644
@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
  		    && total_free + slab_free > pc->max_objects)
  			break;
-		remove_partial(n, slab);
-
-		list_add(&slab->slab_list, &pc->slabs);
+		if (!first)
+			first = slab;
+		last = slab;
Quoted 5 lines

mm/slub.c static bool get_partial_node_bulk(struct kmem_cache *s, · line 3790

Quoted 2 lines
mm/slub.c +2 -0
+		slab_clear_node_partial(slab);
+		n->nr_partial--;

Perhaps factor out those two statements into to a common function and call it in get_partial_node_bulk() and remove_partial()?

Agreed, this is very reasonable.

Quoted 12 lines

mm/slub.c static bool get_partial_node_bulk(struct kmem_cache *s, · line 3803

Quoted 11 lines
mm/slub.c +4 -0
  		total_free += slab_free;
  		if (total_free >= pc->max_objects)
  			break;
  	}
+	if (first)
+		list_bulk_move_tail(&pc->slabs, &first->slab_list,
+				    &last->slab_list);
+
  	spin_unlock_irqrestore(&n->list_lock, flags);
  	return total_free > 0;
  }

Thanks for the review!

Signature · 2 lines
Thanks,
Hao
#3 lore raw
In reply to Harry Yoo
Quoted 9 lines · Harry Yoo
Quoted 6 lines · Hao Li

get_partial_node_bulk() used to move each selected slab from the node partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.

Quoted 6 lines
Quoted 2 lines

Instead, track contiguous runs of matching slabs and move each run with list_bulk_move_tail() in one operation.

TIL list_bulk_move_tail() :D

Quoted 1 line

This reduces list pointer churn> inside the lock critical section.

Nice!

Quoted 6 lines

Similar to this, can we return all slabs in pc->slabs at once when returning those slabs to the list? ... I see Vlastimil removed 'nr of empty slabs' check in the other series already.

Now that it inserts slabs to the tail with Vlastimil's patchset, let's do a list_splice_tail() instead?

Why not, although it should be rare to return more than one slab in that path. So in case this change gets tricky for some reason, we can leave it.

BTW, if you also get the same idea as I had and try to replace the current "remove from pc.slabs initially, reinsert to pc.slabs if it was a partial refill" with a "keep in pc.slabs and only remove if the refill was full" to reduce pointer churn, don't try that, it crashes rather quickly :D

Quoted 39 lines
Quoted 14 lines

The mmap2 testcase shows a 5% improvement after applying this patch.

---
  mm/slub.c | 22 ++++++++++++++++++----

  1 file changed, 18 insertions(+), 4 deletions(-)
mm/slub.c +0 -0
index 04692a6f9128..180973a4a3d2 100644
@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct
 kmem_cache *s,
               && total_free + slab_free > pc->max_objects)

mm/slub.c static bool get_partial_node_bulk(struct · line 3792

Quoted 10 lines
mm/slub.c +5 -2
               break;
   -        remove_partial(n, slab);
-
-        list_add(&slab->slab_list, &pc->slabs);
+        if (!first)
+            first = slab;
+        last = slab;
 
+        slab_clear_node_partial(slab);
+        n->nr_partial--;

Perhaps factor out those two statements into to a common function and call it in get_partial_node_bulk() and remove_partial()?

Quoted 11 lines
mm/slub.c +3 -0
           total_free += slab_free;
           if (total_free >= pc->max_objects)
               break;
       }
   +    if (first)
+        list_bulk_move_tail(&pc->slabs, &first->slab_list,
+                    &last->slab_list);
+
       spin_unlock_irqrestore(&n->list_lock, flags);
       return total_free > 0;
   }
author reviewer
#4 lore raw
Quoted 23 lines · Vlastimil Babka (SUSE)
Quoted 9 lines · Harry Yoo
Quoted 6 lines · Hao Li

get_partial_node_bulk() used to move each selected slab from the node partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.

Quoted 6 lines
Quoted 2 lines

Instead, track contiguous runs of matching slabs and move each run with list_bulk_move_tail() in one operation.

TIL list_bulk_move_tail() :D

Quoted 1 line

This reduces list pointer churn> inside the lock critical section.

Nice!

Quoted 4 lines

Similar to this, can we return all slabs in pc->slabs at once when returning those slabs to the list? ... I see Vlastimil removed 'nr of empty slabs' check in the other series already.

Quoted 5 lines
Quoted 2 lines

Now that it inserts slabs to the tail with Vlastimil's patchset, let's do a list_splice_tail() instead?

Why not, although it should be rare to return more than one slab in that path.

Yeah, it's definitely still worth doing.

Quoted 1 line

So in case this change gets tricky for some reason, we can leave it.

sure, I will try and test.

Quoted 6 lines

BTW, if you also get the same idea as I had and try to replace the current "remove from pc.slabs initially, reinsert to pc.slabs if it was a partial refill" with a "keep in pc.slabs and only remove if the refill was full" to reduce pointer churn, don't try that, it crashes rather quickly :D

Ah, fair enough. But honestly, it did seem like a good idea initially.

Quoted 41 lines
Quoted 40 lines
Quoted 15 lines

The mmap2 testcase shows a 5% improvement after applying this patch.

---
  mm/slub.c | 22 ++++++++++++++++++----

  1 file changed, 18 insertions(+), 4 deletions(-)
mm/slub.c +0 -0
index 04692a6f9128..180973a4a3d2 100644
@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct
 kmem_cache *s,
               && total_free + slab_free > pc->max_objects)

mm/slub.c static bool get_partial_node_bulk(struct · line 3792

Quoted 10 lines
mm/slub.c +5 -2
               break;
   -        remove_partial(n, slab);
-
-        list_add(&slab->slab_list, &pc->slabs);
+        if (!first)
+            first = slab;
+        last = slab;
 
+        slab_clear_node_partial(slab);
+        n->nr_partial--;

Perhaps factor out those two statements into to a common function and call it in get_partial_node_bulk() and remove_partial()?

Quoted 11 lines
mm/slub.c +3 -0
           total_free += slab_free;
           if (total_free >= pc->max_objects)
               break;
       }
   +    if (first)
+        list_bulk_move_tail(&pc->slabs, &first->slab_list,
+                    &last->slab_list);
+
       spin_unlock_irqrestore(&n->list_lock, flags);
       return total_free > 0;
   }
Signature · 2 lines
Thanks,
Hao