mm: unqueue deferred split folios outside the lruvec lock
Subsystems (4)
PATCH 1/1 mm: unqueue deferred split folios outside the lruvec lock
move_folios_to_lru(), folio_batch_move_lru() and folios_put_refs() call folio_unqueue_deferred_split() on folios where the refcount has dropped to zero. In these paths, folio_unqueue_deferred_split() is called with the lruvec lock often held (move_folios_to_lru() always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.
The lruvec lock protects folio->lru and the reclaim LRU lists whereas the deferred split queue lives on folio->_deferred_list in deferred_split_lru. The deferred_split_lru is a list_lru with its own per-memcg per-node lock.
There are already other existing callers that call folio_unqueue_deferred_split() without holding the lruvec lock (eg __folio_put(), shrink_folio_list(), __folio_migrate_mapping(), ...).
Add folio_batch_unqueue_deferred_split() and use it in move_folios_to_lru(), folio_batch_move_lru(), and folios_put_refs() to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and before mem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.
--- mm/folio.c | 24 +++++++++++++++++++----- mm/internal.h | 1 + mm/vmscan.c | 3 ++- 3 files changed, 22 insertions(+), 6 deletions(-)
mm/folio.c +19 -5
@@ -109,6 +109,19 @@ EXPORT_SYMBOL(__folio_put);typedef void (*move_fn_t )(struct lruvec *lruvec,struct folio *folio); +/* +* Unqueue a batch of folios from the deferred split queue. This must be +* called after the folios' refcounts have dropped to zero and before they +* are uncharged (see comment above __folio_unqueue_deferred_split() for why). +*/ +void folio_batch_unqueue_deferred_split (struct folio_batch *fbatch) +{ +unsigned int i; + +for (i =0 ; i <folio_batch_count (fbatch); i++) +folio_unqueue_deferred_split (fbatch->folios[i]); +} +static void lru_add (struct lruvec *lruvec,struct folio *folio) {int was_unevictable =folio_test_clear_unevictable (folio);
@@ -177,14 +190,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)* * We're bypassing normal cleanup. Clear flags that are not * applicable to dead folios. -* -* Since the folio may be part of a huge page, unqueue from -* deferred split list to avoid a dangling list entry. */ if (is_lru_add &&folio_ref_freeze (folio,1 )) {__folio_clear_active (folio);__folio_clear_unevictable (folio); -folio_unqueue_deferred_split (folio); fbatch->folios[i] =NULL ;folio_batch_add (&free_fbatch, folio);continue ;
@@ -201,6 +210,11 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)/* Cleanup filtered dead folios. */ if (is_lru_add) { +/* +* Since the folios may be part of a huge page, unqueue them +* from the deferred split list to avoid a dangling list entry. +*/ +folio_batch_unqueue_deferred_split (&free_fbatch);mem_cgroup_uncharge_folios (&free_fbatch);free_unref_folios (&free_fbatch); }
@@ -1020,7 +1034,6 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)free_huge_folio (folio);continue ; } -folio_unqueue_deferred_split (folio);__page_cache_release (folio, &lruvec, &flags);if (j != i)
@@ -1035,6 +1048,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) } folios->nr = j; +folio_batch_unqueue_deferred_split (folios);mem_cgroup_uncharge_folios (folios);free_unref_folios (folios); }
mm/internal.h +1 -0
@@ -760,6 +760,7 @@ static inline bool folio_unqueue_deferred_split(struct folio *folio)return __folio_unqueue_deferred_split (folio); } +void folio_batch_unqueue_deferred_split (struct folio_batch *fbatch);static inline struct folio *page_rmappable_folio (struct page *page) {
mm/vmscan.c +2 -1
@@ -2007,9 +2007,9 @@ static unsigned int move_folios_to_lru(struct list_head *list, bool do_rotate)if (unlikely (folio_put_testzero (folio))) {__folio_clear_lru_flags (folio); -folio_unqueue_deferred_split (folio);if (folio_batch_add (&free_folios, folio) ==0 ) {lruvec_unlock_irq (lruvec); +folio_batch_unqueue_deferred_split (&free_folios);mem_cgroup_uncharge_folios (&free_folios);free_unref_folios (&free_folios); lruvec =NULL ;
@@ -2041,6 +2041,7 @@ static unsigned int move_folios_to_lru(struct list_head *list, bool do_rotate)lruvec_unlock_irq (lruvec);if (free_folios.nr) { +folio_batch_unqueue_deferred_split (&free_folios);mem_cgroup_uncharge_folios (&free_folios);free_unref_folios (&free_folios); }
Signature · 1 line
2.52.0
Quoted 17 lines · Joanne Koong
move_folios_to_lru(),folio_batch_move_lru()andfolios_put_refs()callfolio_unqueue_deferred_split()on folios where the refcount has dropped to zero. In these paths,folio_unqueue_deferred_split()is called with the lruvec lock often held (move_folios_to_lru()always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.The lruvec lock protects
folio->lruand the reclaim LRU lists whereas the deferred split queue lives onfolio->_deferred_listindeferred_split_lru. Thedeferred_split_lruis alist_lruwith its own per-memcg per-node lock.There are already other existing callers that call
folio_unqueue_deferred_split()without holding the lruvec lock (eg__folio_put(),shrink_folio_list(),__folio_migrate_mapping(), ...).
Quoted 7 lines
Add
folio_batch_unqueue_deferred_split()and use it inmove_folios_to_lru(),folio_batch_move_lru(), andfolios_put_refs()to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and beforemem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.
Nice work! Thanks!
BTW, have you seen measurable overhead from unqueueing deferred split folios under the lruvec lock?
Any before/after numbers for this change, including the cost of the extra pass over the batch?
Cheers, Lance
Quoted 113 lines
Signed-off-by Joanne Koong <[email protected]>---
mm/folio.c | 24 +++++++++++++++++++----- mm/internal.h | 1 + mm/vmscan.c | 3 ++- 3 files changed, 22 insertions(+), 6 deletions(-)mm/folio.c +19
-5@@ -109,6 +109,19 @@ EXPORT_SYMBOL(__folio_put);typedef void (*move_fn_t )(struct lruvec *lruvec,struct folio *folio); +/* +* Unqueue a batch of folios from the deferred split queue. This must be +* called after the folios' refcounts have dropped to zero and before they +* are uncharged (see comment above __folio_unqueue_deferred_split() for why). +*/ +void folio_batch_unqueue_deferred_split (struct folio_batch *fbatch) +{ +unsigned int i; + +for (i =0 ; i <folio_batch_count (fbatch); i++) +folio_unqueue_deferred_split (fbatch->folios[i]); +} +static void lru_add (struct lruvec *lruvec,struct folio *folio) {int was_unevictable =folio_test_clear_unevictable (folio);@@ -177,14 +190,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)* * We're bypassing normal cleanup. Clear flags that are not * applicable to dead folios. -* -* Since the folio may be part of a huge page, unqueue from -* deferred split list to avoid a dangling list entry. */ if (is_lru_add &&folio_ref_freeze (folio,1 )) {__folio_clear_active (folio);__folio_clear_unevictable (folio); -folio_unqueue_deferred_split (folio); fbatch->folios[i] =NULL ;folio_batch_add (&free_fbatch, folio);continue ;@@ -201,6 +210,11 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)/* Cleanup filtered dead folios. */ if (is_lru_add) { +/* +* Since the folios may be part of a huge page, unqueue them +* from the deferred split list to avoid a dangling list entry. +*/ +folio_batch_unqueue_deferred_split (&free_fbatch);mem_cgroup_uncharge_folios (&free_fbatch);free_unref_folios (&free_fbatch); }@@ -1020,7 +1034,6 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)free_huge_folio (folio);continue ; } -folio_unqueue_deferred_split (folio);__page_cache_release (folio, &lruvec, &flags);if (j != i)@@ -1035,6 +1048,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) } folios->nr = j; +folio_batch_unqueue_deferred_split (folios);mem_cgroup_uncharge_folios (folios);free_unref_folios (folios); }mm/internal.h +1
-0@@ -760,6 +760,7 @@ static inline bool folio_unqueue_deferred_split(struct folio *folio)return __folio_unqueue_deferred_split (folio); } +void folio_batch_unqueue_deferred_split (struct folio_batch *fbatch);static inline struct folio *page_rmappable_folio (struct page *page) {mm/vmscan.c +2
-1@@ -2007,9 +2007,9 @@ static unsigned int move_folios_to_lru(struct list_head *list, bool do_rotate)if (unlikely (folio_put_testzero (folio))) {__folio_clear_lru_flags (folio); -folio_unqueue_deferred_split (folio);if (folio_batch_add (&free_folios, folio) ==0 ) {lruvec_unlock_irq (lruvec); +folio_batch_unqueue_deferred_split (&free_folios);mem_cgroup_uncharge_folios (&free_folios);free_unref_folios (&free_folios); lruvec =NULL ;@@ -2041,6 +2041,7 @@ static unsigned int move_folios_to_lru(struct list_head *list, bool do_rotate)lruvec_unlock_irq (lruvec);if (free_folios.nr) { +folio_batch_unqueue_deferred_split (&free_folios);mem_cgroup_uncharge_folios (&free_folios);free_unref_folios (&free_folios); }Signature · 1 line
2.52.0
Barry Song
reviewerQuoted 28 lines · Lance Yang
Quoted 17 lines · Joanne Koong
move_folios_to_lru(),folio_batch_move_lru()andfolios_put_refs()callfolio_unqueue_deferred_split()on folios where the refcount has dropped to zero. In these paths,folio_unqueue_deferred_split()is called with the lruvec lock often held (move_folios_to_lru()always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.The lruvec lock protects
folio->lruand the reclaim LRU lists whereas the deferred split queue lives onfolio->_deferred_listindeferred_split_lru. Thedeferred_split_lruis alist_lruwith its own per-memcg per-node lock.There are already other existing callers that call
folio_unqueue_deferred_split()without holding the lruvec lock (eg__folio_put(),shrink_folio_list(),__folio_migrate_mapping(), ...).Quoted 7 lines
Add
folio_batch_unqueue_deferred_split()and use it inmove_folios_to_lru(),folio_batch_move_lru(), andfolios_put_refs()to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and beforemem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.
Quoted 7 lines
Nice work! Thanks!
BTW, have you seen measurable overhead from unqueueing deferred split folios under the lruvec lock?
Any before/after numbers for this change, including the cost of the extra pass over the batch?
I recall Joanne mentioning some data on this in the link below. Is that right?
https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/
Best Regards
Barry
Quoted 34 lines · Barry Song
Quoted 33 lines · Lance Yang
Quoted 17 lines · Joanne Koong
move_folios_to_lru(),folio_batch_move_lru()andfolios_put_refs()callfolio_unqueue_deferred_split()on folios where the refcount has dropped to zero. In these paths,folio_unqueue_deferred_split()is called with the lruvec lock often held (move_folios_to_lru()always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.The lruvec lock protects
folio->lruand the reclaim LRU lists whereas the deferred split queue lives onfolio->_deferred_listindeferred_split_lru. Thedeferred_split_lruis alist_lruwith its own per-memcg per-node lock.There are already other existing callers that call
folio_unqueue_deferred_split()without holding the lruvec lock (eg__folio_put(),shrink_folio_list(),__folio_migrate_mapping(), ...).Quoted 7 lines
Add
folio_batch_unqueue_deferred_split()and use it inmove_folios_to_lru(),folio_batch_move_lru(), andfolios_put_refs()to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and beforemem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.Nice work! Thanks!
BTW, have you seen measurable overhead from unqueueing deferred split folios under the lruvec lock?
Quoted 6 lines
Quoted 2 lines
Any before/after numbers for this change, including the cost of the extra pass over the batch?
I recall Joanne mentioning some data on this in the link below. Is that right?
https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/
Thanks, Barry! Yes, those numbers answer my question about the observed overhead :)
Joanne, have you rerun that benchmark with this patch? I'd be interested in the before/after runtime and lruvec lock wait.
Cheers, Lance
Quoted 43 lines · Lance Yang
Quoted 39 lines · Barry Song
Quoted 28 lines · Lance Yang
Quoted 17 lines · Joanne Koong
move_folios_to_lru(),folio_batch_move_lru()andfolios_put_refs()callfolio_unqueue_deferred_split()on folios where the refcount has dropped to zero. In these paths,folio_unqueue_deferred_split()is called with the lruvec lock often held (move_folios_to_lru()always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.The lruvec lock protects
folio->lruand the reclaim LRU lists whereas the deferred split queue lives onfolio->_deferred_listindeferred_split_lru. Thedeferred_split_lruis alist_lruwith its own per-memcg per-node lock.There are already other existing callers that call
folio_unqueue_deferred_split()without holding the lruvec lock (eg__folio_put(),shrink_folio_list(),__folio_migrate_mapping(), ...).Quoted 7 lines
Add
folio_batch_unqueue_deferred_split()and use it inmove_folios_to_lru(),folio_batch_move_lru(), andfolios_put_refs()to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and beforemem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.Quoted 7 lines
Nice work! Thanks!
BTW, have you seen measurable overhead from unqueueing deferred split folios under the lruvec lock?
Any before/after numbers for this change, including the cost of the extra pass over the batch?
I recall Joanne mentioning some data on this in the link below. Is that right?
Quoted 7 lines
Quoted 1 line
https://lore.kernel.org/linux-mm/CAJnrk1a2PpUGTsCgS7jct2KET4K2zAL+_h+te8G1MaU9bXs99w@mail.gmail.com/
Thanks, Barry! Yes, those numbers answer my question about the observed overhead :)
Joanne, have you rerun that benchmark with this patch? I'd be interested in the before/after runtime and lruvec lock wait.
Hi Lance,
(Thanks Barry for providing the link to the earlier discussion!)
I have not run those benchmarks with this patch yet but I will get those numbers and add them to v2 when I send that out.
Thanks,
Joanne
David Hildenbrand (Arm)
maintainerQuoted 72 lines · Joanne Koong
move_folios_to_lru(),folio_batch_move_lru()andfolios_put_refs()callfolio_unqueue_deferred_split()on folios where the refcount has dropped to zero. In these paths,folio_unqueue_deferred_split()is called with the lruvec lock often held (move_folios_to_lru()always holds it and folio_batch_move_lru()/folios_put_refs() usually hold it for every folio in the batch except the first). However, it's not needed to hold the lruvec lock.The lruvec lock protects
folio->lruand the reclaim LRU lists whereas the deferred split queue lives onfolio->_deferred_listindeferred_split_lru. Thedeferred_split_lruis alist_lruwith its own per-memcg per-node lock.There are already other existing callers that call
folio_unqueue_deferred_split()without holding the lruvec lock (eg__folio_put(),shrink_folio_list(),__folio_migrate_mapping(), ...).Add
folio_batch_unqueue_deferred_split()and use it inmove_folios_to_lru(),folio_batch_move_lru(), andfolios_put_refs()to unqueue each batch of zero-refcount folios from the deferred split list after the lruvec lock is released and beforemem_cgroup_uncharge_folios(). This helps shorten the lruvec lock's critical sections, which run with IRQs disabled and are highly contended.Signed-off-by Joanne Koong <[email protected]>--- mm/folio.c | 24 +++++++++++++++++++----- mm/internal.h | 1 + mm/vmscan.c | 3 ++- 3 files changed, 22 insertions(+), 6 deletions(-)mm/folio.c +13
-4@@ -109,6 +109,19 @@ EXPORT_SYMBOL(__folio_put);typedef void (*move_fn_t )(struct lruvec *lruvec,struct folio *folio); +/* +* Unqueue a batch of folios from the deferred split queue. This must be +* called after the folios' refcounts have dropped to zero and before they +* are uncharged (see comment above __folio_unqueue_deferred_split() for why). +*/ +void folio_batch_unqueue_deferred_split (struct folio_batch *fbatch) +{ +unsigned int i; + +for (i =0 ; i <folio_batch_count (fbatch); i++) +folio_unqueue_deferred_split (fbatch->folios[i]); +} +static void lru_add (struct lruvec *lruvec,struct folio *folio) {int was_unevictable =folio_test_clear_unevictable (folio);@@ -177,14 +190,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)* * We're bypassing normal cleanup. Clear flags that are not * applicable to dead folios. -* -* Since the folio may be part of a huge page, unqueue from -* deferred split list to avoid a dangling list entry. */ if (is_lru_add &&folio_ref_freeze (folio,1 )) {__folio_clear_active (folio);__folio_clear_unevictable (folio); -folio_unqueue_deferred_split (folio); fbatch->folios[i] =NULL ;folio_batch_add (&free_fbatch, folio);continue ;
mm/folio.c static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) · line 216
Quoted 7 lines
mm/folio.c +4
-0@@ -201,6 +210,11 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)/* Cleanup filtered dead folios. */ if (is_lru_add) { +/* +* Since the folios may be part of a huge page, unqueue them +* from the deferred split list to avoid a dangling list entry. +*/
In general, this lgtm, thanks. I am confused about this existing comment you are moving.
folios being *part of* a huge page? Huh? A folio might represent a huge page, but what does the "part of" thing mean here.
Is it simply "Dequeue any large folios" ?
Signature · 3 lines
Cheers, David
Quoted 28 lines · David Hildenbrand (Arm)
Quoted 17 lines
mm/folio.c +0
-4@@ -177,14 +190,10 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)* * We're bypassing normal cleanup. Clear flags that are not * applicable to dead folios. -* -* Since the folio may be part of a huge page, unqueue from -* deferred split list to avoid a dangling list entry. */ if (is_lru_add &&folio_ref_freeze (folio,1 )) {__folio_clear_active (folio);__folio_clear_unevictable (folio); -folio_unqueue_deferred_split (folio); fbatch->folios[i] =NULL ;folio_batch_add (&free_fbatch, folio);continue ;mm/folio.c static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) · line 216
Quoted 7 lines
mm/folio.c +4
-0@@ -201,6 +210,11 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)/* Cleanup filtered dead folios. */ if (is_lru_add) { +/* +* Since the folios may be part of a huge page, unqueue them +* from the deferred split list to avoid a dangling list entry. +*/ In general, this lgtm, thanks. I am confused about this existing comment you are moving.
Quoted 4 lines
folios being *part of* a huge page? Huh? A folio might represent a huge page, but what does the "part of" thing mean here.
Is it simply "Dequeue any large folios" ?
Agreed, I'll reword this to "Unqueue any large folios from the deferred split queue to avoid a dangling list entry once the folios are freed".
Thanks,
Joanne