mm/swap_state: remove unnecessary lru_add_drain() from readahead
Subsystems (2)
PATCH 1/1 mm/swap_state: remove unnecessary lru_add_drain() from readahead
swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:
- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.
- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.
- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.
Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.
On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.
This is a direct continuation of the cleanup started in commit 1aa43598c03b ("mm: remove unnecessary calls to lru_add_drain") which removed the equivalent drain from free_pages_and_swap_cache() with the same rationale. A detailed reasoning for this is present in [1].
Remove both drains.
[1] https://lore.kernel.org/all/[email protected]/T/#u
--- mm/swap_state.c | 2 -- 1 file changed, 2 deletions(-)
mm/swap_state.c +0 -2
@@ -836,7 +836,6 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, }blk_finish_plug (&plug);swap_read_unplug (splug); -lru_add_drain ();/* Push any new pages onto the LRU now */ skip:/* The page was likely read above, so no need for plugging here */ return swap_cache_read_folio (entry, gfp_mask, mpol, ilx,NULL , false);
@@ -951,7 +950,6 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,pte_unmap (pte);blk_finish_plug (&plug);swap_read_unplug (splug); -lru_add_drain (); skip:/* The folio was likely read above, so no need for plugging here */ folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx,
Signature · 1 line
2.52.0
Quoted 34 lines · Usama Arif
swap_cluster_readahead()andswap_vma_readahead()end the readahead loop with an explicitlru_add_drain()call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:-
do_swap_page()ignores LRU residency for the readahead folios; it only needs the target folio it calledswapin_readahead()for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its ownlru_add_drain()at thewp_can_reuse_anon_folio()anddo_swap_page()sites.-
shmem_swapin_cluster()immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.-
try_to_unuse()likewise locks the folio and callsunuse_pte()without depending on LRU presence.Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU
folio_batchand will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compactionlru_add_drain_all()and so on. The unconditional drain only synchronously flushes a partial batch and forces contention onlruvec_lock.On a 176-CPU production host running a memory-pressured workload, this path was observed to call
folio_batch_move_lru()fromswap_cluster_readahead()~28K/min, a very large source of LRU lock traffic.This is a direct continuation of the cleanup started in commit
1aa43598c03b("mm: remove unnecessary calls tolru_add_drain") which removed the equivalent drain fromfree_pages_and_swap_cache()with the same rationale. A detailed reasoning for this is present in [1].
Quoted 5 lines
Remove both drains.
[1] https://lore.kernel.org/all/[email protected]/T/#u
Signed-off-by Usama Arif <[email protected]>
Thanks for pushing this. JP was also looking into LRU lock contention sources. Particularly we lack visibiluty into the lru_add_drain_all() callers. The idea was to add tracepoints to tracks such callers. (Just nudging you towards it :P)
Quoted 39 lines · Shakeel Butt
Quoted 38 lines · Usama Arif
swap_cluster_readahead()andswap_vma_readahead()end the readahead loop with an explicitlru_add_drain()call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:-
do_swap_page()ignores LRU residency for the readahead folios; it only needs the target folio it calledswapin_readahead()for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its ownlru_add_drain()at thewp_can_reuse_anon_folio()anddo_swap_page()sites.-
shmem_swapin_cluster()immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.-
try_to_unuse()likewise locks the folio and callsunuse_pte()without depending on LRU presence.Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU
folio_batchand will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compactionlru_add_drain_all()and so on. The unconditional drain only synchronously flushes a partial batch and forces contention onlruvec_lock.On a 176-CPU production host running a memory-pressured workload, this path was observed to call
folio_batch_move_lru()fromswap_cluster_readahead()~28K/min, a very large source of LRU lock traffic.This is a direct continuation of the cleanup started in commit
1aa43598c03b("mm: remove unnecessary calls tolru_add_drain") which removed the equivalent drain fromfree_pages_and_swap_cache()with the same rationale. A detailed reasoning for this is present in [1].Remove both drains.
Quoted 7 lines
Quoted 1 line
Signed-off-by Usama Arif <[email protected]>Acked-by Shakeel Butt <[email protected]>Thanks for pushing this. JP was also looking into LRU lock contention sources. Particularly we lack visibiluty into the
lru_add_drain_all()callers. The idea was to add tracepoints to tracks such callers. (Just nudging you towards it :P)
Yup, I'll send a patch for that one.
Barry Song
reviewerQuoted 10 lines · Usama Arif
swap_cluster_readahead()andswap_vma_readahead()end the readahead loop with an explicitlru_add_drain()call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:-
do_swap_page()ignores LRU residency for the readahead folios; it only needs the target folio it calledswapin_readahead()for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its ownlru_add_drain()at thewp_can_reuse_anon_folio()anddo_swap_page()sites.
right. as i can see the below in do_swap_page():
/*
* If we want to map a page that's in the swapcache writable, we
* have to detect via the refcount if we're really the exclusive
* owner. Try removing the extra reference from the local LRU
* caches if required.
*/
if ((vmf->flags & FAULT_FLAG_WRITE) &&
!folio_test_ksm(folio) && !folio_test_lru(folio))
lru_add_drain();and the below in wp_can_reuse_anon_folio():
if (!folio_test_lru(folio))
/*
* We cannot easily detect+handle references from
* remote LRU caches or references to LRU folios.
*/
lru_add_drain();Quoted 15 lines
-
shmem_swapin_cluster()immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.-
try_to_unuse()likewise locks the folio and callsunuse_pte()without depending on LRU presence.Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU
folio_batchand will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compactionlru_add_drain_all()and so on. The unconditional drain only synchronously flushes a partial batch and forces contention onlruvec_lock.
Quoted 5 lines
On a 176-CPU production host running a memory-pressured workload, this path was observed to call
folio_batch_move_lru()fromswap_cluster_readahead()~28K/min, a very large source of LRU lock traffic.
Do we see a workload improvement? If yes, can we put the data?
Thanks
Barry
Quoted 48 lines · Barry Song
Quoted 10 lines · Usama Arif
swap_cluster_readahead()andswap_vma_readahead()end the readahead loop with an explicitlru_add_drain()call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:-
do_swap_page()ignores LRU residency for the readahead folios; it only needs the target folio it calledswapin_readahead()for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its ownlru_add_drain()at thewp_can_reuse_anon_folio()anddo_swap_page()sites.right. as i can see the below in
do_swap_page():/* * If we want to map a page that's in the swapcache writable, we * have to detect via the refcount if we're really the exclusive * owner. Try removing the extra reference from the local LRU * caches if required. */ if ((vmf->flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && !folio_test_lru(folio)) lru_add_drain();and the below in
wp_can_reuse_anon_folio():if (!folio_test_lru(folio)) /* * We cannot easily detect+handle references from * remote LRU caches or references to LRU folios. */ lru_add_drain();Quoted 15 lines
-
shmem_swapin_cluster()immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.-
try_to_unuse()likewise locks the folio and callsunuse_pte()without depending on LRU presence.Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU
folio_batchand will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compactionlru_add_drain_all()and so on. The unconditional drain only synchronously flushes a partial batch and forces contention onlruvec_lock.
Quoted 8 lines
Quoted 5 lines
On a 176-CPU production host running a memory-pressured workload, this path was observed to call
folio_batch_move_lru()fromswap_cluster_readahead()~28K/min, a very large source of LRU lock traffic.Do we see a workload improvement? If yes, can we put the data?
Hello Barry!
So lru lock contention is a source of issue in the meta fleet.
This problem was specifically seen when I ran perf lock contention -a -b in production on a workload that has a really big anon heap and heavy swap activity.
When I tried to trace with bpftrace who was the biggest consumer, it was readahead.
It is easy to run perf and bpftrace on prod on this specific workload, but more difficult to flash a new kernel and see results. The easiest would be when kernel upgrade happens and this patch lands to see the difference and I can report back.
Thanks,
Usama
Quoted 2 lines
Thanks
Barry
Quoted 69 lines · Usama Arif
Quoted 48 lines · Barry Song
Quoted 10 lines · Usama Arif
swap_cluster_readahead()andswap_vma_readahead()end the readahead loop with an explicitlru_add_drain()call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:-
do_swap_page()ignores LRU residency for the readahead folios; it only needs the target folio it calledswapin_readahead()for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its ownlru_add_drain()at thewp_can_reuse_anon_folio()anddo_swap_page()sites.right. as i can see the below in
do_swap_page():/* * If we want to map a page that's in the swapcache writable, we * have to detect via the refcount if we're really the exclusive * owner. Try removing the extra reference from the local LRU * caches if required. */ if ((vmf->flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && !folio_test_lru(folio)) lru_add_drain();and the below in
wp_can_reuse_anon_folio():if (!folio_test_lru(folio)) /* * We cannot easily detect+handle references from * remote LRU caches or references to LRU folios. */ lru_add_drain();Quoted 15 lines
-
shmem_swapin_cluster()immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.-
try_to_unuse()likewise locks the folio and callsunuse_pte()without depending on LRU presence.Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU
folio_batchand will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compactionlru_add_drain_all()and so on. The unconditional drain only synchronously flushes a partial batch and forces contention onlruvec_lock.Quoted 8 lines
Quoted 5 lines
On a 176-CPU production host running a memory-pressured workload, this path was observed to call
folio_batch_move_lru()fromswap_cluster_readahead()~28K/min, a very large source of LRU lock traffic.Do we see a workload improvement? If yes, can we put the data?
Hello Barry!
So lru lock contention is a source of issue in the meta fleet.
This problem was specifically seen when I ran
perf lock contention -a -bin production on a workload that has a really big anon heap and heavy swap activity.
Quoted 7 lines
When I tried to trace with bpftrace who was the biggest consumer, it was readahead.
It is easy to run perf and bpftrace on prod on this specific workload, but more difficult to flash a new kernel and see results. The easiest would be when kernel upgrade happens and this patch lands to see the difference and I can report back.
Yes, it seems fairly straightforward to reduce LRU lock contention. From the review, the patch looks good. I’m just not certain whether we might be missing any corner cases.
I would be happy to see it queued for testing once the mm tree is ready to accept new patches. Is it too late at the moment?
Feel free to add:
Thanks,
Barry
Quoted 65 lines · Usama Arif
swap_cluster_readahead()andswap_vma_readahead()end the readahead loop with an explicitlru_add_drain()call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:-
do_swap_page()ignores LRU residency for the readahead folios; it only needs the target folio it calledswapin_readahead()for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its ownlru_add_drain()at thewp_can_reuse_anon_folio()anddo_swap_page()sites.-
shmem_swapin_cluster()immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.
`
-try_to_unuse()likewise locks the folio and callsunuse_pte()without depending on LRU presence.Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU
folio_batchand will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compactionlru_add_drain_all()and so on. The unconditional drain only synchronously flushes a partial batch and forces contention onlruvec_lock.On a 176-CPU production host running a memory-pressured workload, this path was observed to call
folio_batch_move_lru()fromswap_cluster_readahead()~28K/min, a very large source of LRU lock traffic.This is a direct continuation of the cleanup started in commit
1aa43598c03b("mm: remove unnecessary calls tolru_add_drain") which removed the equivalent drain fromfree_pages_and_swap_cache()with the same rationale. A detailed reasoning for this is present in [1].Remove both drains.
[1] https://lore.kernel.org/all/[email protected]/T/#u
Signed-off-by Usama Arif <[email protected]>--- mm/swap_state.c | 2 -- 1 file changed, 2 deletions(-)mm/swap_state.c +0
-3@@ -836,7 +836,6 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, }blk_finish_plug (&plug);swap_read_unplug (splug); -lru_add_drain ();/* Push any new pages onto the LRU now */ skip:/* The page was likely read above, so no need for plugging here */ return swap_cache_read_folio (entry, gfp_mask, mpol, ilx,NULL , false);@@ -951,7 +950,6 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,pte_unmap (pte);blk_finish_plug (&plug);swap_read_unplug (splug); -lru_add_drain (); skip:/* The folio was likely read above, so no need for plugging here */ folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx, --
Quoted 1 line
2.52.0
Thanks!
Quoted 34 lines · Usama Arif
swap_cluster_readahead()andswap_vma_readahead()end the readahead loop with an explicitlru_add_drain()call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:-
do_swap_page()ignores LRU residency for the readahead folios; it only needs the target folio it calledswapin_readahead()for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its ownlru_add_drain()at thewp_can_reuse_anon_folio()anddo_swap_page()sites.-
shmem_swapin_cluster()immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.-
try_to_unuse()likewise locks the folio and callsunuse_pte()without depending on LRU presence.Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU
folio_batchand will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compactionlru_add_drain_all()and so on. The unconditional drain only synchronously flushes a partial batch and forces contention onlruvec_lock.On a 176-CPU production host running a memory-pressured workload, this path was observed to call
folio_batch_move_lru()fromswap_cluster_readahead()~28K/min, a very large source of LRU lock traffic.This is a direct continuation of the cleanup started in commit
1aa43598c03b("mm: remove unnecessary calls tolru_add_drain") which removed the equivalent drain fromfree_pages_and_swap_cache()with the same rationale. A detailed reasoning for this is present in [1].
Quoted 5 lines
Remove both drains.
[1] https://lore.kernel.org/all/[email protected]/T/#u
Signed-off-by Usama Arif <[email protected]>