Skip to contentSkip to conversation

mm: unqueue deferred split folios outside the lruvec lock

PATCH Applied Joanne Koong 8 participants 12 messages Oct 19 – Nov 6

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 mm: unqueue deferred split folios outside the lruvec lock

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

This came out of the discussion on v2 of the mTHP shrinker series [1], where queueing small mTHP folios on the deferred split queue was found to regress Barry's fault microbenchmark. With [1] applied and khugepaged/max_ptes_none=0 so that every folio is queued, perf lock contention showed that the extra cost came from waiting on the lruvec lock in the free path.

Test setup: 2-socket Intel Skylake, 40 cores / 80 threads, memory bound to node 0 so that all threads contend on one lruvec, with each of 80 threads faulting in and unmapping 16 MB of anonymous memory 200 times. With only 16K mTHP enabled, this patch reduces the queueing cost from 15% runtime to noise, and lruvec lock wait time (summed across threads) drops from 491 seconds to 38 seconds. Most of the remaining contention moves to the deferred split list_lru lock, but runtime still drops back to the unqueued baseline. This test uses a worst-case configuration ([1] is expected to only queue folios of 2M and larger).

[1] https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/

---
v2 -> v3:
- Move benchmark results into the commit message (Lance, Barry)
v2 link: https://lore.kernel.org/linux-mm/[email protected]/

v1 -> v2:
- Reword comment in folio_batch_move_lru() (David)
v1 link: https://lore.kernel.org/linux-mm/[email protected]/

 mm/folio.c    | 24 +++++++++++++++++++-----
 mm/internal.h |  1 +
 mm/vmscan.c   |  3 ++-
 3 files changed, 22 insertions(+), 6 deletions(-)
mm/folio.c +19 -5
index 35e242b48870..45c814f3857b 100644
@@ -109,6 +109,19 @@ EXPORT_SYMBOL(__folio_put);
 
 typedef void (*move_fn_t)(struct lruvec *lruvec, struct folio *folio);
 
+/*
+ * Unqueue a batch of folios from the deferred split queue. This must be
+ * called after the folios' refcounts have dropped to zero and before they
+ * are uncharged (see comment above __folio_unqueue_deferred_split() for why).
+ */
+void folio_batch_unqueue_deferred_split(struct folio_batch *fbatch)
+{
+	unsigned int i;
+
+	for (i = 0; i < folio_batch_count(fbatch); i++)
+		folio_unqueue_deferred_split(fbatch->folios[i]);
+}
+
 static void lru_add(struct lruvec *lruvec, struct folio *folio)
 {
 	int was_unevictable = folio_test_clear_unevictable(folio);
@@ -177,14 +190,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 		 *
 		 * We're bypassing normal cleanup. Clear flags that are not
 		 * applicable to dead folios.
-		 *
-		 * Since the folio may be part of a huge page, unqueue from
-		 * deferred split list to avoid a dangling list entry.
 		 */
 		if (is_lru_add && folio_ref_freeze(folio, 1)) {
 			__folio_clear_active(folio);
 			__folio_clear_unevictable(folio);
-			folio_unqueue_deferred_split(folio);
 			fbatch->folios[i] = NULL;
 			folio_batch_add(&free_fbatch, folio);
 			continue;
@@ -201,6 +210,11 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
 
 	/* Cleanup filtered dead folios. */
 	if (is_lru_add) {
+		/*
+		 * Unqueue any large folios from the deferred split queue to
+		 * avoid a dangling list entry once the folios are freed.
+		 */
+		folio_batch_unqueue_deferred_split(&free_fbatch);
 		mem_cgroup_uncharge_folios(&free_fbatch);
 		free_unref_folios(&free_fbatch);
 	}
@@ -1020,7 +1034,6 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 			free_huge_folio(folio);
 			continue;
 		}
-		folio_unqueue_deferred_split(folio);
 		__page_cache_release(folio, &lruvec, &flags);
 
 		if (j != i)
@@ -1035,6 +1048,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 	}
 
 	folios->nr = j;
+	folio_batch_unqueue_deferred_split(folios);
 	mem_cgroup_uncharge_folios(folios);
 	free_unref_folios(folios);
 }
mm/internal.h +1 -0
index d497c35ee783..33ae388feb33 100644
@@ -761,6 +761,7 @@ static inline bool folio_unqueue_deferred_split(struct folio *folio)
 
 	return __folio_unqueue_deferred_split(folio);
 }
+void folio_batch_unqueue_deferred_split(struct folio_batch *fbatch);
 
 static inline struct folio *page_rmappable_folio(struct page *page)
 {
mm/vmscan.c +2 -1
index daa7e37c2236..9c71a50309e8 100644
@@ -2007,9 +2007,9 @@ static unsigned int move_folios_to_lru(struct list_head *list, bool do_rotate)
 		if (unlikely(folio_put_testzero(folio))) {
 			__folio_clear_lru_flags(folio);
 
-			folio_unqueue_deferred_split(folio);
 			if (folio_batch_add(&free_folios, folio) == 0) {
 				lruvec_unlock_irq(lruvec);
+				folio_batch_unqueue_deferred_split(&free_folios);
 				mem_cgroup_uncharge_folios(&free_folios);
 				free_unref_folios(&free_folios);
 				lruvec = NULL;
@@ -2041,6 +2041,7 @@ static unsigned int move_folios_to_lru(struct list_head *list, bool do_rotate)
 		lruvec_unlock_irq(lruvec);
 
 	if (free_folios.nr) {
+		folio_batch_unqueue_deferred_split(&free_folios);
 		mem_cgroup_uncharge_folios(&free_folios);
 		free_unref_folios(&free_folios);
 	}
Signature · 1 line
2.52.0
reviewer Acked-by on v3 1/1
#1 lore raw
Quoted 42 lines · Joanne Koong

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

This came out of the discussion on v2 of the mTHP shrinker series [1], where queueing small mTHP folios on the deferred split queue was found to regress Barry's fault microbenchmark. With [1] applied and khugepaged/max_ptes_none=0 so that every folio is queued, perf lock contention showed that the extra cost came from waiting on the lruvec lock in the free path.

Test setup: 2-socket Intel Skylake, 40 cores / 80 threads, memory bound to node 0 so that all threads contend on one lruvec, with each of 80 threads faulting in and unmapping 16 MB of anonymous memory 200 times. With only 16K mTHP enabled, this patch reduces the queueing cost from 15% runtime to noise, and lruvec lock wait time (summed across threads) drops from 491 seconds to 38 seconds. Most of the remaining contention moves to the deferred split list_lru lock, but runtime still drops back to the unqueued baseline. This test uses a worst-case configuration ([1] is expected to only queue folios of 2M and larger).

Quoted 4 lines

[1] https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/

Acked-by on v3 1/1
#2 lore raw
In reply to Joanne Koong
Quoted 46 lines · Joanne Koong

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

This came out of the discussion on v2 of the mTHP shrinker series [1], where queueing small mTHP folios on the deferred split queue was found to regress Barry's fault microbenchmark. With [1] applied and khugepaged/max_ptes_none=0 so that every folio is queued, perf lock contention showed that the extra cost came from waiting on the lruvec lock in the free path.

Test setup: 2-socket Intel Skylake, 40 cores / 80 threads, memory bound to node 0 so that all threads contend on one lruvec, with each of 80 threads faulting in and unmapping 16 MB of anonymous memory 200 times. With only 16K mTHP enabled, this patch reduces the queueing cost from 15% runtime to noise, and lruvec lock wait time (summed across threads) drops from 491 seconds to 38 seconds. Most of the remaining contention moves to the deferred split list_lru lock, but runtime still drops back to the unqueued baseline. This test uses a worst-case configuration ([1] is expected to only queue folios of 2M and larger).

Quoted 4 lines

[1] https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/

Thanks.

maintainer Acked-by on v3 1/1
#3 lore raw
In reply to Joanne Koong
Quoted 42 lines · Joanne Koong

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

This came out of the discussion on v2 of the mTHP shrinker series [1], where queueing small mTHP folios on the deferred split queue was found to regress Barry's fault microbenchmark. With [1] applied and khugepaged/max_ptes_none=0 so that every folio is queued, perf lock contention showed that the extra cost came from waiting on the lruvec lock in the free path.

Test setup: 2-socket Intel Skylake, 40 cores / 80 threads, memory bound to node 0 so that all threads contend on one lruvec, with each of 80 threads faulting in and unmapping 16 MB of anonymous memory 200 times. With only 16K mTHP enabled, this patch reduces the queueing cost from 15% runtime to noise, and lruvec lock wait time (summed across threads) drops from 491 seconds to 38 seconds. Most of the remaining contention moves to the deferred split list_lru lock, but runtime still drops back to the unqueued baseline. This test uses a worst-case configuration ([1] is expected to only queue folios of 2M and larger).

Quoted 5 lines

[1] https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/

---

Thanks Joanne!

Signature · 3 lines
Cheers,

David
reviewer Reviewed-by on v3 1/1
#4 lore raw
In reply to Joanne Koong
Quoted 43 lines · Joanne Koong

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

This came out of the discussion on v2 of the mTHP shrinker series [1], where queueing small mTHP folios on the deferred split queue was found to regress Barry's fault microbenchmark. With [1] applied and khugepaged/max_ptes_none=0 so that every folio is queued, perf lock contention showed that the extra cost came from waiting on the lruvec lock in the free path.

Test setup: 2-socket Intel Skylake, 40 cores / 80 threads, memory bound to node 0 so that all threads contend on one lruvec, with each of 80 threads faulting in and unmapping 16 MB of anonymous memory 200 times. With only 16K mTHP enabled, this patch reduces the queueing cost from 15% runtime to noise, and lruvec lock wait time (summed across threads) drops from 491 seconds to 38 seconds. Most of the remaining contention moves to the deferred split list_lru lock, but runtime still drops back to the unqueued baseline. This test uses a worst-case configuration ([1] is expected to only queue folios of 2M and larger).

Quoted 5 lines

[1] https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/

---

maintainer
#5 lore raw
In reply to Joanne Koong
Quoted 17 lines · On Mon, 19 Oct 2026 16:06:46 -0700 Joanne Koong <[email protected]> wrote:

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Quoted 7 lines

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

Thanks.

This is a speedup and not a correctness thing, yes? Let's make it a 7.4-rcX thing, for the 7.5 merge window.

I'll park it in the mm-new branch for now.

If people think the speedup is sufficiently worthwhile then I can add a cc:stable to it, if clear justification is included in the changelog.

reviewer
#6 lore raw
Quoted 30 lines · Andrew Morton
Quoted 17 lines · On Mon, 19 Oct 2026 16:06:46 -0700 Joanne Koong <[email protected]> wrote:

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Quoted 7 lines

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

Thanks.

Quoted 7 lines

This is a speedup and not a correctness thing, yes? Let's make it a 7.4-rcX thing, for the 7.5 merge window.

I'll park it in the mm-new branch for now.

If people think the speedup is sufficiently worthwhile then I can add a cc:stable to it, if clear justification is included in the changelog.

This isn't even a speedup at the moment, as no users have adopted it yet.

mTHP folios aren't currently added to the deferred split queue. Joanne is about to add >=2MB mTHP folios to the deferred split queue again, but even then, contention on the queue shouldn't be an issue.

It may become a problem someday when we add smaller large folios, such as 16KB folios, to the deferred split queue. So this patch is more about future-proofing.

Best Regards
Barry

maintainer
#7 lore raw
Quoted 11 lines · On Wed, 21 Oct 2026 06:40:23 +0800 Barry Song <[email protected]> wrote:
Quoted 7 lines

This is a speedup and not a correctness thing, yes? Let's make it a 7.4-rcX thing, for the 7.5 merge window.

I'll park it in the mm-new branch for now.

If people think the speedup is sufficiently worthwhile then I can add a cc:stable to it, if clear justification is included in the changelog.

This isn't even a speedup at the moment, as no users have adopted it yet.

Quoted 8 lines

mTHP folios aren't currently added to the deferred split queue. Joanne is about to add >=2MB mTHP folios to the deferred split queue again, but even then, contention on the queue shouldn't be an issue.

It may become a problem someday when we add smaller large folios, such as 16KB folios, to the deferred split queue. So this patch is more about future-proofing.

Oh. This:

: This came out of the discussion on v2 of the mTHP shrinker series [1], : where queueing small mTHP folios on the deferred split queue was found : to regress Barry's fault microbenchmark. With [1] applied and : khugepaged/max_ptes_none=0 so that every folio is queued, perf lock : contention showed that the extra cost came from waiting on the lruvec : lock in the free path.

made it sound like an unquantified speedup.

Joanne, please clarify the expected impact (always) in the changelogging if there are reasons for a v4.

#8 lore raw
In reply to Andrew Morton
Quoted 29 lines · Andrew Morton
Quoted 17 lines · On Mon, 19 Oct 2026 16:06:46 -0700 Joanne Koong <[email protected]> wrote:

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Quoted 7 lines

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

Thanks.

Quoted 7 lines

This is a speedup and not a correctness thing, yes? Let's make it a 7.4-rcX thing, for the 7.5 merge window.

I'll park it in the mm-new branch for now.

If people think the speedup is sufficiently worthwhile then I can add a cc:stable to it, if clear justification is included in the changelog.

I'll pick this up for the next merge window.

Signature · 3 lines
Cheers,

David
Reviewed-by on v3 1/1
#9 lore raw
In reply to Joanne Koong
Quoted 51 lines · Joanne Koong

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

This came out of the discussion on v2 of the mTHP shrinker series [1], where queueing small mTHP folios on the deferred split queue was found to regress Barry's fault microbenchmark. With [1] applied and khugepaged/max_ptes_none=0 so that every folio is queued, perf lock contention showed that the extra cost came from waiting on the lruvec lock in the free path.

Test setup: 2-socket Intel Skylake, 40 cores / 80 threads, memory bound to node 0 so that all threads contend on one lruvec, with each of 80 threads faulting in and unmapping 16 MB of anonymous memory 200 times. With only 16K mTHP enabled, this patch reduces the queueing cost from 15% runtime to noise, and lruvec lock wait time (summed across threads) drops from 491 seconds to 38 seconds. Most of the remaining contention moves to the deferred split list_lru lock, but runtime still drops back to the unqueued baseline. This test uses a worst-case configuration ([1] is expected to only queue folios of 2M and larger).

[1] https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/

---
v2 -> v3:
- Move benchmark results into the commit message (Lance, Barry)
v2 link: https://lore.kernel.org/linux-mm/[email protected]/

Quoted 9 lines

v1 -> v2:
- Reword comment in folio_batch_move_lru() (David)
v1 link: https://lore.kernel.org/linux-mm/[email protected]/

 mm/folio.c    | 24 +++++++++++++++++++-----
 mm/internal.h |  1 +
 mm/vmscan.c   |  3 ++-
 3 files changed, 22 insertions(+), 6 deletions(-)

LGTM.

Best Regards,
Yan, Zi

reviewer Reviewed-by on v3 1/1
#10 lore raw
In reply to Joanne Koong
Quoted 42 lines · Joanne Koong

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.

There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).

Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.

This came out of the discussion on v2 of the mTHP shrinker series [1], where queueing small mTHP folios on the deferred split queue was found to regress Barry's fault microbenchmark. With [1] applied and khugepaged/max_ptes_none=0 so that every folio is queued, perf lock contention showed that the extra cost came from waiting on the lruvec lock in the free path.

Test setup: 2-socket Intel Skylake, 40 cores / 80 threads, memory bound to node 0 so that all threads contend on one lruvec, with each of 80 threads faulting in and unmapping 16 MB of anonymous memory 200 times. With only 16K mTHP enabled, this patch reduces the queueing cost from 15% runtime to noise, and lruvec lock wait time (summed across threads) drops from 491 seconds to 38 seconds. Most of the remaining contention moves to the deferred split list_lru lock, but runtime still drops back to the unqueued baseline. This test uses a worst-case configuration ([1] is expected to only queue folios of 2M and larger).

Quoted 5 lines

[1] https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/

---

LGTM.

maintainer Applied
#11 lore raw
In reply to Joanne Koong
Quoted 9 lines · Joanne Koong

move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.

[...]

Applied, thanks!

tree: mm/core.git
branch: core-7.5.misc

Applied patches will appear soon in mm-next, followed by linux-next.

Please report any outstanding bugs that were missed during review, so we can either drop the patch series or decide how to fix it up.

It is encouraged to provide Acked-bys, Reviewed-bys, and Tested-bys even though the patches have already been applied. If possible, patch trailers will be updated.

Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. Further, this is not a guarantee that the patches will go upstream. If in doubt, please check the listed branch.

[1/1] mm: unqueue deferred split folios outside the lruvec lock

      https://git.kernel.org/mm/core/c/8c38bf2a
Signature · 3 lines
Cheers,

David