mm, madvise: fix potential workingset node list_lru leaks
Subsystems (4)
PATCH 1/1 mm, madvise: fix potential workingset node list_lru leaks
From: Kairui Song <[email protected]>
Since commit 5abc1e37afa0 ("mm: list_lru: allocate list_lru_one only when needed"), all list_lru users need to allocate the items using the new infrastructure that provides list_lru info for slab allocation, ensuring that the corresponding memcg list_lru is allocated before use.
For workingset shadow nodes (which are xa_node), users are converted to use the new infrastructure by commit 9bbdc0f32409 ("xarray: use kmem_cache_alloc_lru to allocate xa_node"). The xas->xa_lru will be set correctly for filemap users. However, there is a missing case:xa_node allocations caused by madvise(..., MADV_COLLAPSE).
madvise(..., MADV_COLLAPSE) will also read in the absent parts of file map, and there will be xa_nodes allocated for the caller's memcg (assuming it's not rootcg). However, these allocations won't trigger memcg list_lru allocation because the proper xas info was not set.
If nothing else has allocated other xa_nodes for that memcg to trigger list_lru creation, and memory pressure starts to evict file pages, workingset_update_node will try to add these xa_nodes to their corresponding memcg list_lru, and it does not exist (NULL). So they will be added to rootcg's list_lru instead.
This shouldn't be a significant issue in practice, but it is indeed unexpected behavior, and these xa_nodes will not be reclaimed effectively. And may lead to incorrect counting of the list_lru->nr_items counter.
This problem wasn't exposed until recent commit 28e98022b31ef ("mm/list_lru: simplify reparenting and initial allocation") added a sanity check: only dying memcg could have a NULL list_lru when list_lru_{add,del} is called. This problem triggered this WARNING.
So make madvise(..., MADV_COLLAPSE) also call xas_set_lru() to pass the list_lru which we may want to insert xa_node into later. And move mapping_set_update to mm/internal.h, and turn into a macro to avoid including extra headers in mm/internal.h.
--- mm/filemap.c | 9 --------- mm/internal.h | 6 ++++++ mm/khugepaged.c | 3 +++ 3 files changed, 9 insertions(+), 9 deletions(-)
mm/filemap.c +0 -9
@@ -124,15 +124,6 @@* ->private_lock (zap_pte_range->block_dirty_folio) */ -static void mapping_set_update (struct xa_state *xas, -struct address_space *mapping) -{ -if (dax_mapping (mapping) ||shmem_mapping (mapping)) -return ; -xas_set_update (xas, workingset_update_node); -xas_set_lru (xas, &shadow_nodes); -} -static void page_cache_delete (struct address_space *mapping,struct folio *folio,void *shadow) {
mm/internal.h +6 -0
@@ -1510,6 +1510,12 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,/* Only track the nodes of mappings with shadow entries */ void workingset_update_node (struct xa_node *node);extern struct list_lru shadow_nodes; +#define mapping_set_update (xas, mapping) do { \ + if (!dax_mapping(mapping) && !shmem_mapping(mapping)) { \ + xas_set_update(xas, workingset_update_node); \ + xas_set_lru(xas, &shadow_nodes); \ + } \ +} while (0)/* mremap.c */ unsigned long move_page_tables(struct vm_area_struct *vma,
mm/khugepaged.c +3 -0
@@ -19,6 +19,7 @@#include <linux/rcupdate_wait.h> #include <linux/swapops.h> #include <linux/shmem_fs.h> +#include <linux/dax.h> #include <linux/ksm.h> #include <asm/tlb.h>
@@ -1837,6 +1838,8 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,if (result !=SCAN_SUCCEED ) goto out; +mapping_set_update (&xas, mapping); +__folio_set_locked (new_folio);if (is_shmem)__folio_set_swapbacked (new_folio);
Signature · 1 line
2.47.1
Quoted 38 lines · Kairui Song
From: Kairui Song <[email protected]>
Since commit
5abc1e37afa0("mm:list_lru: allocatelist_lru_oneonly when needed"), alllist_lruusers need to allocate the items using the new infrastructure that provideslist_lruinfo for slab allocation, ensuring that the corresponding memcglist_lruis allocated before use.For workingset shadow nodes (which are
xa_node), users are converted to use the new infrastructure by commit9bbdc0f32409("xarray: usekmem_cache_alloc_lruto allocatexa_node"). Thexas->xa_lruwill be set correctly for filemap users. However, there is a missing case:xa_nodeallocations caused bymadvise(..., MADV_COLLAPSE).
madvise(..., MADV_COLLAPSE)will also read in the absent parts of file map, and there will bexa_nodesallocated for the caller's memcg (assuming it's not rootcg). However, these allocations won't trigger memcglist_lruallocation because the proper xas info was not set.If nothing else has allocated other
xa_nodesfor that memcg to triggerlist_lrucreation, and memory pressure starts to evict file pages,workingset_update_nodewill try to add thesexa_nodesto their corresponding memcglist_lru, and it does not exist (NULL). So they will be added to rootcg'slist_lruinstead.This shouldn't be a significant issue in practice, but it is indeed unexpected behavior, and these
xa_nodeswill not be reclaimed effectively. And may lead to incorrect counting of thelist_lru->nr_itemscounter.This problem wasn't exposed until recent commit
28e98022b31ef("mm/list_lru: simplify reparenting and initial allocation") added a sanity check: only dying memcg could have a NULLlist_lruwhen list_lru_{add,del} is called. This problem triggered this WARNING.
Quoted 6 lines
So make
madvise(..., MADV_COLLAPSE)also callxas_set_lru()to pass thelist_lruwhich we may want to insertxa_nodeinto later. And movemapping_set_updatetomm/internal.h, and turn into a macro to avoid including extra headers inmm/internal.h.Fixes 9bbdc0f32409 ("xarray: use kmem_cache_alloc_lru to allocate xa_node")
Hi Kairui,
This should be 7d8faaf15545 introducing MADV_COLLAPSE, right? Because it is developed after 9bbdc0f32409.
Thanks.
Quoted 70 lines
Reported-by [email protected]Signed-off-by Kairui Song <[email protected]>---
mm/filemap.c | 9 --------- mm/internal.h | 6 ++++++ mm/khugepaged.c | 3 +++ 3 files changed, 9 insertions(+), 9 deletions(-)mm/filemap.c +0
-9@@ -124,15 +124,6 @@* ->private_lock (zap_pte_range->block_dirty_folio) */ -static void mapping_set_update (struct xa_state *xas, -struct address_space *mapping) -{ -if (dax_mapping (mapping) ||shmem_mapping (mapping)) -return ; -xas_set_update (xas, workingset_update_node); -xas_set_lru (xas, &shadow_nodes); -} -static void page_cache_delete (struct address_space *mapping,struct folio *folio,void *shadow) {mm/internal.h +6
-0@@ -1510,6 +1510,12 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry,/* Only track the nodes of mappings with shadow entries */ void workingset_update_node (struct xa_node *node);extern struct list_lru shadow_nodes; +#define mapping_set_update (xas, mapping) do { \ + if (!dax_mapping(mapping) && !shmem_mapping(mapping)) { \ + xas_set_update(xas, workingset_update_node); \ + xas_set_lru(xas, &shadow_nodes); \ + } \ +} while (0)/* mremap.c */ unsigned long move_page_tables(struct vm_area_struct *vma,mm/khugepaged.c +3
-1@@ -19,6 +19,7 @@#include <linux/rcupdate_wait.h> #include <linux/swapops.h> #include <linux/shmem_fs.h> +#include <linux/dax.h> #include <linux/ksm.h> #include <asm/tlb.h> @@ -1837,6 +1838,8 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,if (result !=SCAN_SUCCEED ) goto out; +mapping_set_update (&xas, mapping); +__folio_set_locked (new_folio);if (is_shmem)__folio_set_swapbacked (new_folio); --2.47.1