Skip to contentSkip to conversation

mm/swap_state: remove unnecessary lru_add_drain() from readahead

PATCH Applied Usama Arif 6 participants 8 messages Jun 8 – Jun 10

Subsystems (2)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

PATCH 1/1 mm/swap_state: remove unnecessary lru_add_drain() from readahead

swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:

- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.

- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.

- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.

Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.

On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.

This is a direct continuation of the cleanup started in commit 1aa43598c03b ("mm: remove unnecessary calls to lru_add_drain") which removed the equivalent drain from free_pages_and_swap_cache() with the same rationale. A detailed reasoning for this is present in [1].

Remove both drains.

[1] https://lore.kernel.org/all/[email protected]/T/#u

---
 mm/swap_state.c | 2 --
 1 file changed, 2 deletions(-)
mm/swap_state.c +0 -2
index 9c3a5cf99778..6fd6e3415b71 100644
@@ -836,7 +836,6 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
 	}
 	blk_finish_plug(&plug);
 	swap_read_unplug(splug);
-	lru_add_drain();	/* Push any new pages onto the LRU now */
 skip:
 	/* The page was likely read above, so no need for plugging here */
 	return swap_cache_read_folio(entry, gfp_mask, mpol, ilx, NULL, false);
@@ -951,7 +950,6 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
 		pte_unmap(pte);
 	blk_finish_plug(&plug);
 	swap_read_unplug(splug);
-	lru_add_drain();
 skip:
 	/* The folio was likely read above, so no need for plugging here */
 	folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx,
Signature · 1 line
2.52.0
Acked-by on v1 1/1
#1 lore raw
Quoted 34 lines · Usama Arif

swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:

- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.

- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.

- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.

Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.

On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.

This is a direct continuation of the cleanup started in commit 1aa43598c03b ("mm: remove unnecessary calls to lru_add_drain") which removed the equivalent drain from free_pages_and_swap_cache() with the same rationale. A detailed reasoning for this is present in [1].

Quoted 5 lines

Remove both drains.

[1] https://lore.kernel.org/all/[email protected]/T/#u

Thanks for pushing this. JP was also looking into LRU lock contention sources. Particularly we lack visibiluty into the lru_add_drain_all() callers. The idea was to add tracepoints to tracks such callers. (Just nudging you towards it :P)

#2 lore raw
Quoted 39 lines · Shakeel Butt
Quoted 38 lines · Usama Arif

swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:

- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.

- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.

- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.

Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.

On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.

This is a direct continuation of the cleanup started in commit 1aa43598c03b ("mm: remove unnecessary calls to lru_add_drain") which removed the equivalent drain from free_pages_and_swap_cache() with the same rationale. A detailed reasoning for this is present in [1].

Remove both drains.

[1] https://lore.kernel.org/all/[email protected]/T/#u

Quoted 7 lines
Quoted 1 line

Thanks for pushing this. JP was also looking into LRU lock contention sources. Particularly we lack visibiluty into the lru_add_drain_all() callers. The idea was to add tracepoints to tracks such callers. (Just nudging you towards it :P)

Yup, I'll send a patch for that one.

reviewer
#3 lore raw
In reply to Usama Arif
Quoted 10 lines · Usama Arif

swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:

- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.

right. as i can see the below in do_swap_page():

        /*
         * If we want to map a page that's in the swapcache writable, we
         * have to detect via the refcount if we're really the exclusive
         * owner. Try removing the extra reference from the local LRU
         * caches if required.
         */
        if ((vmf->flags & FAULT_FLAG_WRITE) &&
            !folio_test_ksm(folio) && !folio_test_lru(folio))
                lru_add_drain();

and the below in wp_can_reuse_anon_folio():

        if (!folio_test_lru(folio))
                /*
                 * We cannot easily detect+handle references from
                 * remote LRU caches or references to LRU folios.
                 */
                lru_add_drain();
Quoted 15 lines

- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.

- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.

Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.

Quoted 5 lines

On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.

Do we see a workload improvement? If yes, can we put the data?

Thanks
Barry

author
#4 lore raw
Quoted 48 lines · Barry Song
Quoted 10 lines · Usama Arif

swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:

- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.

right. as i can see the below in do_swap_page():

        /*
         * If we want to map a page that's in the swapcache writable, we
         * have to detect via the refcount if we're really the exclusive
         * owner. Try removing the extra reference from the local LRU
         * caches if required.
         */
        if ((vmf->flags & FAULT_FLAG_WRITE) &&
            !folio_test_ksm(folio) && !folio_test_lru(folio))
                lru_add_drain();

and the below in wp_can_reuse_anon_folio():

        if (!folio_test_lru(folio))
                /*
                 * We cannot easily detect+handle references from
                 * remote LRU caches or references to LRU folios.
                 */
                lru_add_drain();
Quoted 15 lines

- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.

- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.

Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.

Quoted 8 lines
Quoted 5 lines

On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.

Do we see a workload improvement? If yes, can we put the data?

Hello Barry!

So lru lock contention is a source of issue in the meta fleet.

This problem was specifically seen when I ran perf lock contention -a -b in production on a workload that has a really big anon heap and heavy swap activity.

When I tried to trace with bpftrace who was the biggest consumer, it was readahead.

It is easy to run perf and bpftrace on prod on this specific workload, but more difficult to flash a new kernel and see results. The easiest would be when kernel upgrade happens and this patch lands to see the difference and I can report back.

Thanks,
Usama

Quoted 2 lines

Thanks
Barry

reviewer Reviewed-by
#5 lore raw
Quoted 69 lines · Usama Arif
Quoted 48 lines · Barry Song
Quoted 10 lines · Usama Arif

swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:

- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.

right. as i can see the below in do_swap_page():

        /*
         * If we want to map a page that's in the swapcache writable, we
         * have to detect via the refcount if we're really the exclusive
         * owner. Try removing the extra reference from the local LRU
         * caches if required.
         */
        if ((vmf->flags & FAULT_FLAG_WRITE) &&
            !folio_test_ksm(folio) && !folio_test_lru(folio))
                lru_add_drain();

and the below in wp_can_reuse_anon_folio():

        if (!folio_test_lru(folio))
                /*
                 * We cannot easily detect+handle references from
                 * remote LRU caches or references to LRU folios.
                 */
                lru_add_drain();
Quoted 15 lines

- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.

- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.

Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.

Quoted 8 lines
Quoted 5 lines

On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.

Do we see a workload improvement? If yes, can we put the data?

Hello Barry!

So lru lock contention is a source of issue in the meta fleet.

This problem was specifically seen when I ran perf lock contention -a -b in production on a workload that has a really big anon heap and heavy swap activity.

Quoted 7 lines

When I tried to trace with bpftrace who was the biggest consumer, it was readahead.

It is easy to run perf and bpftrace on prod on this specific workload, but more difficult to flash a new kernel and see results. The easiest would be when kernel upgrade happens and this patch lands to see the difference and I can report back.

Yes, it seems fairly straightforward to reduce LRU lock contention. From the review, the patch looks good. I’m just not certain whether we might be missing any corner cases.

I would be happy to see it queued for testing once the mm tree is ready to accept new patches. Is it too late at the moment?

Feel free to add:

Thanks,
Barry

Reviewed-by on v1 1/1
#6 lore raw
In reply to Usama Arif
Quoted 65 lines · Usama Arif

swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:

- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.

- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.
`
- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.

Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.

On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.

This is a direct continuation of the cleanup started in commit 1aa43598c03b ("mm: remove unnecessary calls to lru_add_drain") which removed the equivalent drain from free_pages_and_swap_cache() with the same rationale. A detailed reasoning for this is present in [1].

Remove both drains.

[1] https://lore.kernel.org/all/[email protected]/T/#u

---
 mm/swap_state.c | 2 --
 1 file changed, 2 deletions(-)
mm/swap_state.c +0 -3
index 9c3a5cf99778..6fd6e3415b71 100644
@@ -836,7 +836,6 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
        }
        blk_finish_plug(&plug);
        swap_read_unplug(splug);
-       lru_add_drain();        /* Push any new pages onto the LRU now */
 skip:
        /* The page was likely read above, so no need for plugging here */
        return swap_cache_read_folio(entry, gfp_mask, mpol, ilx, NULL, false);
@@ -951,7 +950,6 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
                pte_unmap(pte);
        blk_finish_plug(&plug);
        swap_read_unplug(splug);
-       lru_add_drain();
 skip:
        /* The folio was likely read above, so no need for plugging here */
        folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx,
--
Quoted 1 line

2.52.0

Thanks!

Acked-by on v1 1/1
#7 lore raw
In reply to Usama Arif
Quoted 34 lines · Usama Arif

swap_cluster_readahead() and swap_vma_readahead() end the readahead loop with an explicit lru_add_drain() call. That drain is a leftover from 2.6.12 era code and serves no functional purpose for the callers:

- do_swap_page() ignores LRU residency for the readahead folios; it only needs the target folio it called swapin_readahead() for, and if the write-fault path needs the target folio on the LRU to count references accurately, it runs its own lru_add_drain() at the wp_can_reuse_anon_folio() and do_swap_page() sites.

- shmem_swapin_cluster() immediately locks the returned folio, waits for writeback, then operates on it - LRU residency of either the target or the readahead folios is irrelevant.

- try_to_unuse() likewise locks the folio and calls unuse_pte() without depending on LRU presence.

Folios newly added to the swap cache by the readahead loop sit in the per-CPU LRU folio_batch and will be drained naturally as the batch fills (FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and so on. The unconditional drain only synchronously flushes a partial batch and forces contention on lruvec_lock.

On a 176-CPU production host running a memory-pressured workload, this path was observed to call folio_batch_move_lru() from swap_cluster_readahead() ~28K/min, a very large source of LRU lock traffic.

This is a direct continuation of the cleanup started in commit 1aa43598c03b ("mm: remove unnecessary calls to lru_add_drain") which removed the equivalent drain from free_pages_and_swap_cache() with the same rationale. A detailed reasoning for this is present in [1].

Quoted 5 lines

Remove both drains.

[1] https://lore.kernel.org/all/[email protected]/T/#u