mm/mglru: yield the lru_lock earlier in fill/drain_evictable()
Subsystems (3)
PATCH v2 1/1 mm/mglru: yield the lru_lock earlier in fill/drain_evictable()
From: Hui Zhu <[email protected]>
lru_gen_change_state() migrates folios in batches of up to MAX_LRU_BATCH (4096) folios under lruvec_lock_irq(). A full batch keeps local interrupts off for up to about a millisecond, and CPUs contending for the same lru_lock spin with interrupts disabled for that long.
The caller already reschedules between batches, but a batch itself only ends after 4096 folios. Stop it early when a resched is pending, or when the lock has waiters and the current hold has moved at least MIN_LRU_BATCH folios; the caller restarts where it left off. TIF_NEED_RESCHED can still be set with interrupts off: a remote resched_curr() sets it directly in the target's thread_info. The minimum batch avoids lock bouncing under sustained contention, while a waiter now waits ~15us instead of ~1ms.
This follows lru_gen_look_around() in this file, security/keys/gc.c and jbd2_log_do_checkpoint(), and commit 6e9be217a3ce ("mm/mglru:
use a smaller batch for reclaim").
---
Changlog:
v2:
Also yield on lock contention after a minimum batch (Kairui Song).
mm/vmscan.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
mm/vmscan.c +8 -2
@@ -5533,7 +5533,10 @@ static bool fill_evictable(struct lruvec *lruvec) success =lru_gen_add_folio (lruvec, folio, !active);VM_WARN_ON_ONCE (!success); -if (!--remaining) +/* stop early to shorten the irq-off section */ +if (!--remaining ||need_resched () || + (MAX_LRU_BATCH - remaining >=MIN_LRU_BATCH && +spin_is_contended (&lruvec->lru_lock)))return false; } }
@@ -5562,7 +5565,10 @@ static bool drain_evictable(struct lruvec *lruvec)VM_WARN_ON_ONCE (!success);lruvec_add_folio (lruvec, folio); -if (!--remaining) +/* stop early to shorten the irq-off section */ +if (!--remaining ||need_resched () || + (MAX_LRU_BATCH - remaining >=MIN_LRU_BATCH && +spin_is_contended (&lruvec->lru_lock)))return false; } }
Signature · 1 line
2.53.0
Baolin Wang
reviewerQuoted 35 lines · Hui Zhu
From: Hui Zhu <[email protected]>
lru_gen_change_state()migrates folios in batches of up toMAX_LRU_BATCH(4096) folios underlruvec_lock_irq(). A full batch keeps local interrupts off for up to about a millisecond, and CPUs contending for the samelru_lockspin with interrupts disabled for that long.The caller already reschedules between batches, but a batch itself only ends after 4096 folios. Stop it early when a resched is pending, or when the lock has waiters and the current hold has moved at least
MIN_LRU_BATCHfolios; the caller restarts where it left off.TIF_NEED_RESCHEDcan still be set with interrupts off: a remoteresched_curr()sets it directly in the target'sthread_info. The minimum batch avoids lock bouncing under sustained contention, while a waiter now waits ~15us instead of ~1ms.This follows
lru_gen_look_around()in this file,security/keys/gc.candjbd2_log_do_checkpoint(), and commit6e9be217a3ce("mm/mglru:
use a smaller batch for reclaim").Signed-off-by Hui Zhu <[email protected]>---
Changlog:
v2:
Also yield on lock contention after a minimum batch (Kairui Song).mm/vmscan.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)mm/vmscan.c +0
-0
mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5540
Quoted 9 lines
mm/vmscan.c +4
-1@@ -5533,7 +5533,10 @@ static bool fill_evictable(struct lruvec *lruvec) success =lru_gen_add_folio (lruvec, folio, !active);VM_WARN_ON_ONCE (!success); -if (!--remaining) +/* stop early to shorten the irq-off section */ +if (!--remaining ||need_resched () || + (MAX_LRU_BATCH - remaining >=MIN_LRU_BATCH && +spin_is_contended (&lruvec->lru_lock)))return false;
IIUC, Kairui's point[1] is to just drop the 'MAX_LRU_BATCH - remaining >= MIN_LRU_BATCH' condition, no?
if (!--remaining || need_resched() || spin_is_contended(&lruvec->lru_lock))
[1] https://lore.kernel.org/all/CAMgjq7CUMuNSdCXZ7_kDdo9+0BFvPNHuO=mq=ct_Q5YUyyckvQ@mail.gmail.com/
2026/11/13 10:50, Baolin Wang :
Quoted 48 lines · Hui Zhu
Quoted 35 lines
From: Hui Zhu <[email protected]>
lru_gen_change_state()migrates folios in batches of up toMAX_LRU_BATCH(4096) folios under lruvec_lock_irq(). A full batch keeps local interrupts off for up to about a millisecond, and CPUs contending for the samelru_lockspin with interrupts disabled for that long.The caller already reschedules between batches, but a batch itself only ends after 4096 folios. Stop it early when a resched is pending, or when the lock has waiters and the current hold has moved at least
MIN_LRU_BATCHfolios; the caller restarts where it left off.TIF_NEED_RESCHEDcan still be set with interrupts off: a remoteresched_curr()sets it directly in the target'sthread_info. The minimum batch avoids lock bouncing under sustained contention, while a waiter now waits ~15us instead of ~1ms.This follows
lru_gen_look_around()in this file,security/keys/gc.candjbd2_log_do_checkpoint(), and commit6e9be217a3ce("mm/mglru:
use a smaller batch for reclaim").Signed-off-by Hui Zhu <[email protected]>---
Changlog:
v2:
Also yield on lock contention after a minimum batch (Kairui Song).
mm/vmscan.c| 10 ++++++++--1 file changed, 8 insertions(+), 2 deletions(-)mm/vmscan.c +0
-0mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5540
Quoted 8 lines
mm/vmscan.c +4
-0@@ -5533,7 +5533,10 @@ static bool fill_evictable(struct lruvec *lruvec) success = lru_gen_add_folio(lruvec, folio, !active); VM_WARN_ON_ONCE(!success); - if (!--remaining) + /* stop early to shorten the irq-off section */ + if (!--remaining || need_resched() || + (MAX_LRU_BATCH - remaining >= MIN_LRU_BATCH && + spin_is_contended(&lruvec->lru_lock))) return false;IIUC, Kairui's point[1] is to just drop the '
MAX_LRU_BATCH- remaining
Quoted 6 lines
Quoted 1 line
=
MIN_LRU_BATCH' condition, no?if (!--remaining ||
need_resched()||spin_is_contended(&lruvec->lru_lock))[1] https://lore.kernel.org/all/CAMgjq7CUMuNSdCXZ7_kDdo9+0BFvPNHuO=mq=ct_Q5YUyyckvQ@mail.gmail.com/
Not quite. [1] was about the extra moved counter in the version discussed on the v1 thread; Kairui was fine with the rest. Since remaining counts down from MAX_LRU_BATCH and one call is exactly one lock hold, MAX_LRU_BATCH - remaining is the number of folios moved in the current hold, so the condition is just moved >= MIN_LRU_BATCH without the extra variable.
The minimum-batch test itself is a response to Michal's lock bouncing concern [2]. Without it, under sustained contention the loop could yield after every single folio: each yield costs an unlock, a requeue and a restart (microseconds), while one folio move costs a few hundred nanoseconds, so the migration could slow down by tens of times, with cgroup_lock and cpus_read_lock held throughout. With it, each hold moves at least MIN_LRU_BATCH (64) folios, i.e. does ~15us of work before it may hand the lock over, while a waiter still waits only ~15us instead of the ~1ms of a full MAX_LRU_BATCH batch.
[2] https://lore.kernel.org/all/avV0IbvSx4iuv0tF@tiehlicka/
Best,
Hui
Baolin Wang
reviewerQuoted 55 lines · Hui Zhu
2026/11/13 10:50, Baolin Wang :
Quoted 53 lines · Hui Zhu
Quoted 35 lines
From: Hui Zhu <[email protected]>
lru_gen_change_state()migrates folios in batches of up toMAX_LRU_BATCH(4096) folios under lruvec_lock_irq(). A full batch keeps local interrupts off for up to about a millisecond, and CPUs contending for the samelru_lockspin with interrupts disabled for that long.The caller already reschedules between batches, but a batch itself only ends after 4096 folios. Stop it early when a resched is pending, or when the lock has waiters and the current hold has moved at least
MIN_LRU_BATCHfolios; the caller restarts where it left off.TIF_NEED_RESCHEDcan still be set with interrupts off: a remoteresched_curr()sets it directly in the target'sthread_info. The minimum batch avoids lock bouncing under sustained contention, while a waiter now waits ~15us instead of ~1ms.This follows
lru_gen_look_around()in this file,security/keys/gc.candjbd2_log_do_checkpoint(), and commit6e9be217a3ce("mm/mglru:
use a smaller batch for reclaim").Signed-off-by Hui Zhu <[email protected]>---
Changlog:
v2:
Also yield on lock contention after a minimum batch (Kairui Song).
mm/vmscan.c| 10 ++++++++--1 file changed, 8 insertions(+), 2 deletions(-)mm/vmscan.c +0
-0mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5540
Quoted 8 lines
mm/vmscan.c +4
-0@@ -5533,7 +5533,10 @@ static bool fill_evictable(struct lruvec *lruvec) success = lru_gen_add_folio(lruvec, folio, !active); VM_WARN_ON_ONCE(!success); - if (!--remaining) + /* stop early to shorten the irq-off section */ + if (!--remaining || need_resched() || + (MAX_LRU_BATCH - remaining >= MIN_LRU_BATCH && + spin_is_contended(&lruvec->lru_lock))) return false;IIUC, Kairui's point[1] is to just drop the '
MAX_LRU_BATCH- remainingQuoted 1 line
=
MIN_LRU_BATCH' condition, no?if (!--remaining ||
need_resched()|| spin_is_contended(&lruvec-Quoted 1 line
lru_lock))
Quoted 8 lines
Quoted 1 line
[1] https://lore.kernel.org/all/ CAMgjq7CUMuNSdCXZ7_kDdo9+0BFvPNHuO=mq=ct_Q5YUyyckvQ@mail.gmail.com/
Not quite. [1] was about the extra
movedcounter in the version discussed on the v1 thread; Kairui was fine with the rest. Sinceremainingcounts down fromMAX_LRU_BATCHand one call is exactly one lock hold,MAX_LRU_BATCH - remainingis the number of folios moved in the current hold, so the condition is justmoved >= MIN_LRU_BATCHwithout the extra variable.
If so, why not just set:
int remaining = MIN_LRU_BATCH;
?
Quoted 9 lines
The minimum-batch test itself is a response to Michal's lock bouncing concern [2]. Without it, under sustained contention the loop could yield after every single folio: each yield costs an unlock, a requeue and a restart (microseconds), while one folio move costs a few hundred nanoseconds, so the migration could slow down by tens of times, with
cgroup_lockandcpus_read_lockheld throughout. With it, each hold moves at leastMIN_LRU_BATCH(64) folios, i.e. does ~15us of work before it may hand the lock over, while a waiter still waits only ~15us instead of the ~1ms of a fullMAX_LRU_BATCHbatch.
fill_evictable()/drain_evictable() are really cold paths, I really don't see any practical impact from this. At least we (Alibaba) won't be dynamically switching MGLRU state in practice.
Quoted 51 lines · Baolin Wang
Quoted 50 lines · Hui Zhu
2026/11/13 10:50, Baolin Wang :
Quoted 48 lines · Hui Zhu
Quoted 35 lines
From: Hui Zhu <[email protected]>
lru_gen_change_state()migrates folios in batches of up toMAX_LRU_BATCH(4096) folios underlruvec_lock_irq(). A full batch keeps local interrupts off for up to about a millisecond, and CPUs contending for the samelru_lockspin with interrupts disabled for that long.The caller already reschedules between batches, but a batch itself only ends after 4096 folios. Stop it early when a resched is pending, or when the lock has waiters and the current hold has moved at least
MIN_LRU_BATCHfolios; the caller restarts where it left off.TIF_NEED_RESCHEDcan still be set with interrupts off: a remoteresched_curr()sets it directly in the target'sthread_info. The minimum batch avoids lock bouncing under sustained contention, while a waiter now waits ~15us instead of ~1ms.This follows
lru_gen_look_around()in this file,security/keys/gc.candjbd2_log_do_checkpoint(), and commit6e9be217a3ce("mm/mglru:
use a smaller batch for reclaim").Signed-off-by Hui Zhu <[email protected]>---
Changlog:
v2:
Also yield on lock contention after a minimum batch (Kairui Song).mm/vmscan.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)mm/vmscan.c +0
-0mm/vmscan.c static bool fill_evictable(struct lruvec *lruvec) · line 5540
Quoted 8 lines
mm/vmscan.c +4
-0@@ -5533,7 +5533,10 @@ static bool fill_evictable(struct lruvec *lruvec) success =lru_gen_add_folio (lruvec, folio, !active);VM_WARN_ON_ONCE (!success); - if (!--remaining) + /* stop early to shorten the irq-off section */ +if (!--remaining ||need_resched () || + (MAX_LRU_BATCH - remaining >=MIN_LRU_BATCH && +spin_is_contended (&lruvec->lru_lock)))return false;IIUC, Kairui's point[1] is to just drop the '
MAX_LRU_BATCH- remaining
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
=
MIN_LRU_BATCH' condition, no?if (!--remaining ||
need_resched()|| spin_is_contended(&lruvec-Quoted 1 line
lru_lock))
Right that's my point, we can keep the checking simpler.
Quoted 14 lines
Quoted 10 lines
Quoted 3 lines
[1] https://lore.kernel.org/all/
CAMgjq7CUMuNSdCXZ7_kDdo9+0BFvPNHuO=mq=ct_Q5YUyyckvQ@mail.gmail.com/Not quite. [1] was about the extra
movedcounter in the version discussed on the v1 thread; Kairui was fine with the rest. Sinceremainingcounts down fromMAX_LRU_BATCHand one call is exactly one lock hold,MAX_LRU_BATCH - remainingis the number of folios moved in the current hold, so the condition is justmoved >= MIN_LRU_BATCHwithout the extra variable.If so, why not just set:
int remaining =
MIN_LRU_BATCH;
...
Quoted 3 lines
fill_evictable()/drain_evictable() are really cold paths, I really don't see any practical impact from this. At least we (Alibaba) won't be dynamically switching MGLRU state in practice.
I'm not sure about this either, but if we want to reduce the lock holding time for the cold patch, setting the batch directly to MIN_LRU_BATCH seems cleaner. However, combining this with the newly added checks, I'm unsure if we've become too aggressive, potentially causing the switching process to take too long.