FAILED: patch "[PATCH] mm/huge_memory: use folio's memcg inside __folio_split()" failed to apply to 6.18-stable tree
Subsystems (3)
[email protected]
maintainerThe patch below does not apply to the 6.18-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.18.y
git checkout FETCH_HEAD
git cherry-pick -x c299a2285d9d8bda4da024455de65e3d00de6f17
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2026090356-glacial-cornball-be8c@gregkh' --subject-prefix 'PATCH 6.18.y' 'HEAD^..'
Possible dependencies:
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From c299a2285d9d8bda4da024455de65e3d00de6f17 Mon Sep 17 00:00:00 2001
From: Zi Yan <[email protected]>
Date: Tue, 4 Aug 2026 17:04:27 -0400
Subject: [PATCH] mm/huge_memory: use folio's memcg inside __folio_split()
Patch series "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to folio's memcg", v3.
__GFP_ACCOUNT is needed for xarray node allocation accounting when XA_FLAGS_ACCOUNT is set. Commit 7b785645e8f13 ("mm: fix page cache convergence regression") fixed a workingset regression with it. xas_split_alloc() does not have it and needs to be fixed.
In addition, based on Sashiko's review[1] and Johannes' confirmation[2], to charge the right memcg, folio's memcg needs to be active during folio split. Add that before adding __GFP_ACCOUNT.
There is no workingset convergence regression related to missing __GFP_ACCOUNT in xas_split_alloc() and the impact to userspace should be minor.
This patch (of 2):
During a pagecache folio split, an xarray node allocation can happen and needs to charge at folio's memcg instead of folio split invoker's memcg, because for example folio split can happen during reclaim and reclaim's active memcg might not be folio's memcg. Switch to folio's memcg at the beginning and switch back afterwards.
mm/huge_memory.c +16 -4
@@ -4105,34 +4105,42 @@ static int __folio_split(struct folio *folio, unsigned int new_order,XA_STATE (xas, &folio->mapping->i_pages, folio->index);struct folio *end_folio =folio_next (folio);bool is_anon =folio_test_anon (folio); +struct mem_cgroup *memcg, *old_memcg;struct address_space *mapping =NULL ;struct anon_vma *anon_vma =NULL ;int old_order =folio_order (folio);struct folio *new_folio, *next;int nr_shmem_dropped =0 ;enum ttu_flags ttu_flags =0 ; -int ret;pgoff_t end =0 ; +int ret;VM_WARN_ON_ONCE_FOLIO (!folio_test_locked (folio), folio);VM_WARN_ON_ONCE_FOLIO (!folio_test_large (folio), folio);if (folio !=page_folio (split_at) || folio !=page_folio (lock_at)) { ret = -EINVAL ; - goto out; + goto out_no_memcg; }if (new_order >= old_order) { ret = -EINVAL ; - goto out; + goto out_no_memcg; } ret =folio_check_splittable (folio, new_order, split_type);if (ret) {VM_WARN_ONCE (ret == -EINVAL ,"Tried to split an unsplittable folio" ); - goto out; + goto out_no_memcg; } +/* +* switch to folio's memcg as xarray node allocation can happen and +* needs to charge to it. +*/ + memcg =get_mem_cgroup_from_folio (folio); + old_memcg =set_active_memcg (memcg); +if (is_anon) {/* * The caller does not necessarily hold an mmap_lock that would
@@ -4275,6 +4283,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order,if (mapping)i_mmap_unlock_read (mapping); out: +/* restore to caller's old_memcg */ +set_active_memcg (old_memcg); +mem_cgroup_put (memcg); +out_no_memcg:xas_destroy (&xas);if (is_pmd_order (old_order))count_vm_event (!ret ?THP_SPLIT_PAGE :THP_SPLIT_PAGE_FAILED );
Sasha Levin
maintainerPATCH 1/6 mm: thp: introduce folio_split_queue_lock and its variants
From: Muchun Song <[email protected]>
[ Upstream commit ad7c7f4576a5977b4ec4ac5dd090ab3f81ca7c6f ]
In future memcg removal, the binding between a folio and a memcg may change, making the split lock within the memcg unstable when held.
A new approach is required to reparent the split queue to its parent. This patch starts introducing a unified way to acquire the split lock for future work.
It's a code-only refactoring with no functional changes.
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
--- include/linux/memcontrol.h | 10 ++++ mm/huge_memory.c | 119 ++++++++++++++++++++++++++----------- 2 files changed, 94 insertions(+), 35 deletions(-)
include/linux/memcontrol.h +10 -0
@@ -1674,6 +1674,11 @@ int alloc_shrinker_info(struct mem_cgroup *memcg);void free_shrinker_info (struct mem_cgroup *memcg);void set_shrinker_bit (struct mem_cgroup *memcg,int nid,int shrinker_id);void reparent_shrinker_deferred (struct mem_cgroup *memcg); + +static inline int shrinker_id (struct shrinker *shrinker) +{ +return shrinker->id; +}#else #define mem_cgroup_sockets_enabled 0
@@ -1705,6 +1710,11 @@ static inline void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { } + +static inline int shrinker_id (struct shrinker *shrinker) +{ +return -1 ; +}#endif #ifdef CONFIG_MEMCG
mm/huge_memory.c +84 -35
213 lines not on this page · raw message
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 2/6 mm: thp: use folio_batch to handle THP splitting in deferred_split_scan()
From: Muchun Song <[email protected]>
[ Upstream commit 776bde7caf80f6af72b087cafe7d9f607b14716d ]
The maintenance of the folio->_deferred_list is intricate because it's reused in a local list.
Here are some peculiarities:
1) When a folio is removed from its split queue and added to a local on-stack list in deferred_split_scan(), the ->split_queue_len isn't updated, leading to an inconsistency between it and the actual number of folios in the split queue.
2) When the folio is split via split_folio() later, it's removed from the local list while holding the split queue lock. At this time, the lock is not needed as it is not protecting anything.
3) To handle the race condition with a third-party freeing or migrating the preceding folio, we must ensure there's always one safe (with raised refcount) folio before by delaying its folio_put(). More details can be found in commit e66f3185fa04 ("mm/thp: fix deferred split queue not partially_mapped"). It's rather tricky.
We can use the folio_batch infrastructure to handle this clearly. In this case, ->split_queue_len will be consistent with the real number of folios in the split queue. If list_empty(&folio->_deferred_list) returns false, it's clear the folio must be in its split queue (not in a local list anymore).
In the future, we will reparent LRU folios during memcg offline to eliminate dying memory cgroups, which requires reparenting the split queue to its parent first. So this patch prepares for using folio_split_queue_lock_irqsave() as the memcg may change then.
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
--- mm/huge_memory.c | 130 ++++++++++++++++++++++++----------------------- 1 file changed, 66 insertions(+), 64 deletions(-)
mm/huge_memory.c +66 -64
@@ -3686,19 +3686,21 @@ static int __folio_split(struct folio *folio, unsigned int new_order,bool is_anon =folio_test_anon (folio);struct address_space *mapping =NULL ;struct anon_vma *anon_vma =NULL ; -int order =folio_order (folio); +int old_order =folio_order (folio);struct folio *new_folio, *next; +int extra_pins;int nr_shmem_dropped =0 ; -int remap_flags =0 ; -int extra_pins, ret; -pgoff_t end; -bool is_hzp; +enum ttu_flags ttu_flags =0 ; +pgoff_t end =0 ; +int ret;VM_WARN_ON_ONCE_FOLIO (!folio_test_locked (folio), folio);VM_WARN_ON_ONCE_FOLIO (!folio_test_large (folio), folio); -if (folio !=page_folio (split_at) || folio !=page_folio (lock_at)) -return -EINVAL ; +if (folio !=page_folio (split_at) || folio !=page_folio (lock_at)) { + ret = -EINVAL ; + goto out; + }/* * Folios that just got truncated cannot get split. Signal to the
@@ -3710,14 +3712,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,if (!is_anon && !folio->mapping)return -EBUSY ; -if (new_order >=folio_order (folio)) -return -EINVAL ; - -if (!folio_split_supported (folio, new_order, uniform_split,/* warn = */ true)) -return -EINVAL ; +if (new_order >= old_order) { + ret = -EINVAL ; + goto out; + } - is_hzp =is_huge_zero_folio (folio); -if (is_hzp) { +if (is_huge_zero_folio (folio)) {pr_warn_ratelimited ("Called split_huge_page for huge zero page\n" );return -EBUSY ; }
@@ -3725,6 +3725,13 @@ static int __folio_split(struct folio *folio, unsigned int new_order,if (folio_test_writeback (folio))return -EBUSY ; + ret =folio_split_supported (folio, new_order, uniform_split, +/* warns = */ false) ?0 : -EINVAL ; +if (ret) { +VM_WARN_ONCE (ret == -EINVAL ,"Tried to split an unsplittable folio" ); + goto out; + } +if (is_anon) {/* * The caller does not necessarily hold an mmap_lock that would
@@ -3817,21 +3824,22 @@ static int __folio_split(struct folio *folio, unsigned int new_order,struct lruvec *lruvec;int expected_refs; -if (folio_order (folio) >1 && - !list_empty (&folio->_deferred_list)) { - ds_queue->split_queue_len--; +if (folio_order (folio) >1 ) { +if (!list_empty (&folio->_deferred_list)) { + ds_queue->split_queue_len--; +/* +* Reinitialize page_deferred_list after removing the +* page from the split_queue, otherwise a subsequent +* split will see list corruption when checking the +* page_deferred_list. +*/ +list_del_init (&folio->_deferred_list); + }if (folio_test_partially_mapped (folio)) {folio_clear_partially_mapped (folio);mod_mthp_stat (folio_order (folio),MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ,-1 ); } -/* -* Reinitialize page_deferred_list after removing the -* page from the split_queue, otherwise a subsequent -* split will see list corruption when checking the -* page_deferred_list. -*/ -list_del_init (&folio->_deferred_list); }split_queue_unlock (ds_queue);if (mapping) {
@@ -3941,8 +3949,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,shmem_uncharge (mapping->host, nr_shmem_dropped);if (!ret && is_anon) - remap_flags =RMP_USE_SHARED_ZEROPAGE ; -remap_page (folio,1 << order, remap_flags); + ttu_flags = (enum ttu_flags )RMP_USE_SHARED_ZEROPAGE ; +remap_page (folio,1 << old_order, ttu_flags);/* * Drop the mapping while the inode is still pinned. @folio stays
@@ -3985,9 +3993,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,i_mmap_unlock_read (mapping); out:xas_destroy (&xas); -if (order ==HPAGE_PMD_ORDER ) +if (old_order ==HPAGE_PMD_ORDER )count_vm_event (!ret ?THP_SPLIT_PAGE :THP_SPLIT_PAGE_FAILED ); -count_mthp_stat (order, !ret ?MTHP_STAT_SPLIT :MTHP_STAT_SPLIT_FAILED ); +count_mthp_stat (old_order, !ret ?MTHP_STAT_SPLIT :MTHP_STAT_SPLIT_FAILED );return ret; }
@@ -4230,35 +4238,40 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, {struct deferred_split *ds_queue;unsigned long flags; -LIST_HEAD (list); -struct folio *folio, *next, *prev =NULL ; -int split =0 , removed =0 ; +struct folio *folio, *next; +int split =0 , i; +struct folio_batch fbatch; + +folio_batch_init (&fbatch); +retry: ds_queue =split_queue_lock_irqsave (sc->nid, sc->memcg, &flags);/* Take pin on all head pages to avoid freeing them under us */ list_for_each_entry_safe (folio, next, &ds_queue->split_queue, _deferred_list) {if (folio_try_get (folio)) { -list_move (&folio->_deferred_list, &list); - }else { +folio_batch_add (&fbatch, folio); + }else if (folio_test_partially_mapped (folio)) {/* We lost race with folio_put() */ -if (folio_test_partially_mapped (folio)) { -folio_clear_partially_mapped (folio); -mod_mthp_stat (folio_order (folio), -MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ,-1 ); - } -list_del_init (&folio->_deferred_list); - ds_queue->split_queue_len--; +folio_clear_partially_mapped (folio); +mod_mthp_stat (folio_order (folio), +MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ,-1 ); } +list_del_init (&folio->_deferred_list); + ds_queue->split_queue_len--;if (!--sc->nr_to_scan)break ; +if (!folio_batch_space (&fbatch)) +break ; }split_queue_unlock_irqrestore (ds_queue, flags); -list_for_each_entry_safe (folio, next, &list, _deferred_list) { +for (i =0 ; i <folio_batch_count (&fbatch); i++) {bool did_split = false;bool underused = false; +struct deferred_split *fqueue; + folio = fbatch.folios[i];if (!folio_test_partially_mapped (folio)) {/* * See try_to_map_unused_to_zeropage(): we cannot
@@ -4281,38 +4294,27 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, }folio_unlock (folio); next: +if (did_split || !folio_test_partially_mapped (folio)) +continue ;/* -* split_folio() removes folio from list on success. * Only add back to the queue if folio is partially mapped. * If thp_underused returns false, or if split_folio fails * in the case it was underused, then consider it used and * don't add it back to split_queue. */ -if (did_split) { - ;/* folio already removed from list */ - }else if (!folio_test_partially_mapped (folio)) { -list_del_init (&folio->_deferred_list); - removed++; - }else { -/* -* That unlocked list_del_init() above would be unsafe, -* unless its folio is separated from any earlier folios -* left on the list (which may be concurrently unqueued) -* by one safe folio with refcount still raised. -*/ -swap (folio, prev); + fqueue =folio_split_queue_lock_irqsave (folio, &flags); +if (list_empty (&folio->_deferred_list)) { +list_add_tail (&folio->_deferred_list, &fqueue->split_queue); + fqueue->split_queue_len++; } -if (folio) -folio_put (folio); +split_queue_unlock_irqrestore (fqueue, flags); } +folios_put (&fbatch); -spin_lock_irqsave (&ds_queue->split_queue_lock, flags); -list_splice_tail (&list, &ds_queue->split_queue); - ds_queue->split_queue_len -= removed; -spin_unlock_irqrestore (&ds_queue->split_queue_lock, flags); - -if (prev) -folio_put (prev); +if (sc->nr_to_scan && !list_empty (&ds_queue->split_queue)) { +cond_resched (); + goto retry; + }/* * Stop shrinker if we didn't split any page, but the queue is empty.
Signature · 1 line
2.53.0
Re: [PATCH 6.18.y 2/6] mm: thp: use folio_batch to handle THP splitting in deferred_split_scan()
Quoted 57 lines · Sasha Levin
From: Muchun Song <[email protected]>
[ Upstream commit
776bde7caf80f6af72b087cafe7d9f607b14716d]The maintenance of the
folio->_deferred_listis intricate because it's reused in a local list.Here are some peculiarities:
1) When a folio is removed from its split queue and added to a local on-stack list in
deferred_split_scan(), the ->split_queue_len isn't updated, leading to an inconsistency between it and the actual number of folios in the split queue.2) When the folio is split via
split_folio()later, it's removed from the local list while holding the split queue lock. At this time, the lock is not needed as it is not protecting anything.3) To handle the race condition with a third-party freeing or migrating the preceding folio, we must ensure there's always one safe (with raised refcount) folio before by delaying its
folio_put(). More details can be found in commite66f3185fa04("mm/thp: fix deferred split queue notpartially_mapped"). It's rather tricky.We can use the
folio_batchinfrastructure to handle this clearly. In this case, ->split_queue_len will be consistent with the real number of folios in the split queue. Iflist_empty(&folio->_deferred_list)returns false, it's clear the folio must be in its split queue (not in a local list anymore).In the future, we will reparent LRU folios during memcg offline to eliminate dying memory cgroups, which requires reparenting the split queue to its parent first. So this patch prepares for using
folio_split_queue_lock_irqsave()as the memcg may change then.Link https://lkml.kernel.org/r/59cb6b6fb5ffcff9d23b81890b252960139ad8e7.1762762324.git.zhengqi.arch@bytedance.com open hereSigned-off-by Muchun Song <[email protected]>Signed-off-by Qi Zheng <[email protected]>Reviewed-by Zi Yan <[email protected]>Acked-by Shakeel Butt <[email protected]>Reviewed-by Wei Yang <[email protected]>Reviewed-by Harry Yoo <[email protected]>Cc Baolin Wang <[email protected]>Cc Barry Song <[email protected]>Cc Dev Jain <[email protected]>Cc Hugh Dickins <[email protected]>Cc Johannes Weiner <[email protected]>Cc Lance Yang <[email protected]>Cc Liam Howlett <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Michal Hocko <[email protected]>Cc Muchun Song <[email protected]>Cc Nico Pache <[email protected]>Cc Roman Gushchin <[email protected]>Cc Ryan Roberts <[email protected]>Signed-off-by Andrew Morton <[email protected]>
Quoted 2 lines
Stable-dep-of:
c299a2285d9d("mm/huge_memory: use folio's memcg inside__folio_split()")Signed-off-by Sasha Levin <[email protected]>
Oh dear, more of these Stable-dep-ofs. I just happened to be Cc'ed on this one, and after commenting on the pincount ones, I'd better comment on this too: we definitely want Zi Yan's XA_FLAGS_ACCOUNT fix, but deferred split queue is tricky: so I'd judge that this should not go forward without explicit Acks from... whoever knows best.
Hugh
Quoted 236 lines
--- mm/huge_memory.c | 130 ++++++++++++++++++++++++----------------------- 1 file changed, 66 insertions(+), 64 deletions(-)mm/huge_memory.c +66
-64@@ -3686,19 +3686,21 @@ static int __folio_split(struct folio *folio, unsigned int new_order,bool is_anon =folio_test_anon (folio);struct address_space *mapping =NULL ;struct anon_vma *anon_vma =NULL ; -int order =folio_order (folio); +int old_order =folio_order (folio);struct folio *new_folio, *next; +int extra_pins;int nr_shmem_dropped =0 ; -int remap_flags =0 ; -int extra_pins, ret; -pgoff_t end; -bool is_hzp; +enum ttu_flags ttu_flags =0 ; +pgoff_t end =0 ; +int ret;VM_WARN_ON_ONCE_FOLIO (!folio_test_locked (folio), folio);VM_WARN_ON_ONCE_FOLIO (!folio_test_large (folio), folio); -if (folio !=page_folio (split_at) || folio !=page_folio (lock_at)) -return -EINVAL ; +if (folio !=page_folio (split_at) || folio !=page_folio (lock_at)) { + ret = -EINVAL ; + goto out; + }/* * Folios that just got truncated cannot get split. Signal to the @@ -3710,14 +3712,12 @@ static int __folio_split(struct folio *folio, unsigned int new_order,if (!is_anon && !folio->mapping)return -EBUSY ; -if (new_order >=folio_order (folio)) -return -EINVAL ; - -if (!folio_split_supported (folio, new_order, uniform_split,/* warn = */ true)) -return -EINVAL ; +if (new_order >= old_order) { + ret = -EINVAL ; + goto out; + } - is_hzp =is_huge_zero_folio (folio); -if (is_hzp) { +if (is_huge_zero_folio (folio)) {pr_warn_ratelimited ("Called split_huge_page for huge zero page\n" );return -EBUSY ; }@@ -3725,6 +3725,13 @@ static int __folio_split(struct folio *folio, unsigned int new_order,if (folio_test_writeback (folio))return -EBUSY ; + ret =folio_split_supported (folio, new_order, uniform_split, +/* warns = */ false) ?0 : -EINVAL ; +if (ret) { +VM_WARN_ONCE (ret == -EINVAL ,"Tried to split an unsplittable folio" ); + goto out; + } +if (is_anon) {/* * The caller does not necessarily hold an mmap_lock that would @@ -3817,21 +3824,22 @@ static int __folio_split(struct folio *folio, unsigned int new_order,struct lruvec *lruvec;int expected_refs; -if (folio_order (folio) >1 && - !list_empty (&folio->_deferred_list)) { - ds_queue->split_queue_len--; +if (folio_order (folio) >1 ) { +if (!list_empty (&folio->_deferred_list)) { + ds_queue->split_queue_len--; +/* +* Reinitialize page_deferred_list after removing the +* page from the split_queue, otherwise a subsequent +* split will see list corruption when checking the +* page_deferred_list. +*/ +list_del_init (&folio->_deferred_list); + }if (folio_test_partially_mapped (folio)) {folio_clear_partially_mapped (folio);mod_mthp_stat (folio_order (folio),MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ,-1 ); } -/* -* Reinitialize page_deferred_list after removing the -* page from the split_queue, otherwise a subsequent -* split will see list corruption when checking the -* page_deferred_list. -*/ -list_del_init (&folio->_deferred_list); }split_queue_unlock (ds_queue);if (mapping) {@@ -3941,8 +3949,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,shmem_uncharge (mapping->host, nr_shmem_dropped);if (!ret && is_anon) - remap_flags =RMP_USE_SHARED_ZEROPAGE ; -remap_page (folio,1 << order, remap_flags); + ttu_flags = (enum ttu_flags )RMP_USE_SHARED_ZEROPAGE ; +remap_page (folio,1 << old_order, ttu_flags);/* * Drop the mapping while the inode is still pinned. @folio stays @@ -3985,9 +3993,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,i_mmap_unlock_read (mapping); out:xas_destroy (&xas); -if (order ==HPAGE_PMD_ORDER ) +if (old_order ==HPAGE_PMD_ORDER )count_vm_event (!ret ?THP_SPLIT_PAGE :THP_SPLIT_PAGE_FAILED ); -count_mthp_stat (order, !ret ?MTHP_STAT_SPLIT :MTHP_STAT_SPLIT_FAILED ); +count_mthp_stat (old_order, !ret ?MTHP_STAT_SPLIT :MTHP_STAT_SPLIT_FAILED );return ret; }@@ -4230,35 +4238,40 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, {struct deferred_split *ds_queue;unsigned long flags; -LIST_HEAD (list); -struct folio *folio, *next, *prev =NULL ; -int split =0 , removed =0 ; +struct folio *folio, *next; +int split =0 , i; +struct folio_batch fbatch; + +folio_batch_init (&fbatch); +retry: ds_queue =split_queue_lock_irqsave (sc->nid, sc->memcg, &flags);/* Take pin on all head pages to avoid freeing them under us */ list_for_each_entry_safe (folio, next, &ds_queue->split_queue, _deferred_list) {if (folio_try_get (folio)) { -list_move (&folio->_deferred_list, &list); - }else { +folio_batch_add (&fbatch, folio); + }else if (folio_test_partially_mapped (folio)) {/* We lost race with folio_put() */ -if (folio_test_partially_mapped (folio)) { -folio_clear_partially_mapped (folio); -mod_mthp_stat (folio_order (folio), -MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ,-1 ); - } -list_del_init (&folio->_deferred_list); - ds_queue->split_queue_len--; +folio_clear_partially_mapped (folio); +mod_mthp_stat (folio_order (folio), +MTHP_STAT_NR_ANON_PARTIALLY_MAPPED ,-1 ); } +list_del_init (&folio->_deferred_list); + ds_queue->split_queue_len--;if (!--sc->nr_to_scan)break ; +if (!folio_batch_space (&fbatch)) +break ; }split_queue_unlock_irqrestore (ds_queue, flags); -list_for_each_entry_safe (folio, next, &list, _deferred_list) { +for (i =0 ; i <folio_batch_count (&fbatch); i++) {bool did_split = false;bool underused = false; +struct deferred_split *fqueue; + folio = fbatch.folios[i];if (!folio_test_partially_mapped (folio)) {/* * See try_to_map_unused_to_zeropage(): we cannot @@ -4281,38 +4294,27 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, }folio_unlock (folio); next: +if (did_split || !folio_test_partially_mapped (folio)) +continue ;/* -* split_folio() removes folio from list on success. * Only add back to the queue if folio is partially mapped. * If thp_underused returns false, or if split_folio fails * in the case it was underused, then consider it used and * don't add it back to split_queue. */ -if (did_split) { - ;/* folio already removed from list */ - }else if (!folio_test_partially_mapped (folio)) { -list_del_init (&folio->_deferred_list); - removed++; - }else { -/* -* That unlocked list_del_init() above would be unsafe, -* unless its folio is separated from any earlier folios -* left on the list (which may be concurrently unqueued) -* by one safe folio with refcount still raised. -*/ -swap (folio, prev); + fqueue =folio_split_queue_lock_irqsave (folio, &flags); +if (list_empty (&folio->_deferred_list)) { +list_add_tail (&folio->_deferred_list, &fqueue->split_queue); + fqueue->split_queue_len++; } -if (folio) -folio_put (folio); +split_queue_unlock_irqrestore (fqueue, flags); } +folios_put (&fbatch); -spin_lock_irqsave (&ds_queue->split_queue_lock, flags); -list_splice_tail (&list, &ds_queue->split_queue); - ds_queue->split_queue_len -= removed; -spin_unlock_irqrestore (&ds_queue->split_queue_lock, flags); - -if (prev) -folio_put (prev); +if (sc->nr_to_scan && !list_empty (&ds_queue->split_queue)) { +cond_resched (); + goto retry; + }/* * Stop shrinker if we didn't split any page, but the queue is empty. Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 3/6 mm/huge_memory: change folio_split_supported() to folio_check_splittable()
In reply to Sasha LevinFrom: Zi Yan <[email protected]>
[ Upstream commit bdd0d69a32c2aa6437d23e35acc705758b835a75 ]
Patch series "Improve folio split related functions", v4.
This patchset improves several folio split related functions to avoid future misuse. The changes are:
1. Consolidated folio splittable checks by moving truncated folio check, huge zero folio check, and writeback folio check into folio_split_supported(). Changed the function return type. Renamed it to folio_check_splittable() for clarification.
2. Replaced can_split_folio() with open coded folio_expected_ref_count() and folio_ref_count() and introduced folio_cache_ref_count().
3. Changed min_order_for_split() to always return an order.
4. Fixed folio split stats counting.
Motivation
==========
This is based on Wei's observation[1] and solves several potential issues:
1. Dereferencing NULL folio->mapping in try_folio_split_to_order() if it is called on truncated folios.
2. Not handling of negative return value of min_order_for_split() in mm/memory-failure.c
There is no bug in the current code.
This patch (of 4):
folio_split_supported() used in try_folio_split_to_order() requires folio->mapping to be non NULL, but current try_folio_split_to_order() does not check it. There is no issue in the current code, since try_folio_split_to_order() is only used in truncate_inode_partial_folio(), where folio->mapping is not NULL.
To prevent future misuse, move folio->mapping NULL check (i.e., folio is truncated) into folio_split_supported(). Since folio->mapping NULL check returns -EBUSY and folio_split_supported() == false means -EINVAL, change folio_split_supported() return type from bool to int and return error numbers accordingly. Rename folio_split_supported() to folio_check_splittable() to match the return type change.
While at it, move is_huge_zero_folio() check and folio_test_writeback() check into folio_check_splittable() and add kernel-doc.
Remove all warnings inside folio_check_splittable() and give warnings in __folio_split() instead, so that bool warns parameter can be removed.
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
--- include/linux/huge_mm.h | 6 ++-- mm/huge_memory.c | 72 +++++++++++++++++++++++------------------ 2 files changed, 43 insertions(+), 35 deletions(-)
include/linux/huge_mm.h +3 -3
@@ -369,8 +369,8 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, unsigned int new_order);int min_order_for_split (struct folio *folio);int split_folio_to_list (struct folio *folio,struct list_head *list); -bool folio_split_supported (struct folio *folio,unsigned int new_order, -bool uniform_split,bool warns); +int folio_check_splittable (struct folio *folio,unsigned int new_order, +bool uniform_split);int folio_split (struct folio *folio,unsigned int new_order,struct page *page,struct list_head *list);/*
@@ -390,7 +390,7 @@ int folio_split(struct folio *folio, unsigned int new_order, struct page *page,static inline int try_folio_split_to_order (struct folio *folio,struct page *page,unsigned int new_order) { -if (!folio_split_supported (folio, new_order, false,/* warns= */ false)) +if (folio_check_splittable (folio, new_order, false))return split_huge_page_to_list_to_order (&folio->page,NULL , new_order);return folio_split (folio, new_order, page,NULL );
mm/huge_memory.c +40 -32
107 lines not on this page · raw message
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 4/6 mm/huge_memory: replace can_split_folio() with direct refcount calculation
In reply to Sasha LevinFrom: Zi Yan <[email protected]>
[ Upstream commit 5842bcbfc316738cbfcbdb4def5a7592aa03ebf2 ]
can_split_folio() is just a refcount comparison, making sure only the split caller holds an extra pin. Open code it with folio_expected_ref_count() != folio_ref_count() - 1. For the extra_pins used by folio_ref_freeze(), add folio_cache_ref_count() to calculate it. Also replace folio_expected_ref_count() with folio_cache_ref_count() used by folio_ref_unfreeze(), since they are returning the same values when a folio is frozen and folio_cache_ref_count() does not have unnecessary folio_mapcount() in its implementation.
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
--- include/linux/huge_mm.h | 1 - mm/huge_memory.c | 35 ++++++++++++----------------------- mm/vmscan.c | 3 ++- 3 files changed, 14 insertions(+), 25 deletions(-)
include/linux/huge_mm.h +0 -1
@@ -364,7 +364,6 @@ unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long add unsigned long len, unsigned long pgoff, unsigned long flags, vm_flags_t vm_flags); -bool can_split_folio (struct folio *folio,int caller_pins,int *pextra_pins);int split_huge_page_to_list_to_order (struct page *page,struct list_head *list,unsigned int new_order);int min_order_for_split (struct folio *folio);
mm/huge_memory.c +12 -23
@@ -3326,23 +3326,6 @@ static void lru_add_split_folio(struct folio *folio, struct folio *new_folio, } } -/* Racy check whether the huge page can be split */ -bool can_split_folio (struct folio *folio,int caller_pins,int *pextra_pins) -{ -int extra_pins; - -/* Additional pins from page cache */ -if (folio_test_anon (folio)) - extra_pins =folio_test_swapcache (folio) ? -folio_nr_pages (folio) :0 ; -else - extra_pins =folio_nr_pages (folio); -if (pextra_pins) - *pextra_pins = extra_pins; -return folio_mapcount (folio) ==folio_ref_count (folio) - extra_pins - - caller_pins; -} -static bool page_range_has_hwpoisoned (struct page *page,long nr_pages) {for (; nr_pages; page++, nr_pages--)
@@ -3715,7 +3698,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order,struct anon_vma *anon_vma =NULL ;int old_order =folio_order (folio);struct folio *new_folio, *next; -int extra_pins;int nr_shmem_dropped =0 ;enum ttu_flags ttu_flags =0 ;pgoff_t end =0 ;
@@ -3803,7 +3785,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,* Racy check if we can split the page, before unmap_folio() will * split PMDs */ -if (!can_split_folio (folio,1 , &extra_pins)) { +if (folio_expected_ref_count (folio) !=folio_ref_count (folio) -1 ) { ret = -EAGAIN ; goto out_unlock; }
@@ -3827,7 +3809,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order,/* Prevent deferred_split_scan() touching ->_refcount */ ds_queue =folio_split_queue_lock (folio); -if (folio_ref_freeze (folio,1 + extra_pins)) { +if (folio_ref_freeze (folio, + (folio_test_anon (folio) && + !folio_test_swapcache (folio) ? +0 :folio_nr_pages (folio)) +1 )) {struct swap_cluster_info *ci =NULL ;struct lruvec *lruvec;int expected_refs;
@@ -3896,7 +3881,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order, next =folio_next (new_folio); - expected_refs =folio_expected_ref_count (new_folio) +1 ; + expected_refs = (folio_test_anon (new_folio) && + !folio_test_swapcache (new_folio) ? +0 :folio_nr_pages (new_folio)) +1 ;folio_ref_unfreeze (new_folio, expected_refs);lru_add_split_folio (folio, new_folio, lruvec, list);
@@ -3936,7 +3923,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,* Otherwise, a parallel folio_try_get() can grab @folio * and its caller can see stale page cache entries. */ - expected_refs =folio_expected_ref_count (folio) +1 ; + expected_refs = (folio_test_anon (folio) && + !folio_test_swapcache (folio) ? +0 :folio_nr_pages (folio)) +1 ;folio_ref_unfreeze (folio, expected_refs);unlock_page_lruvec (lruvec);
@@ -4465,7 +4454,7 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start,* can be split or not. So skip the check here. */ if (!folio_test_private (folio) && - !can_split_folio (folio,0 ,NULL )) +folio_expected_ref_count (folio) !=folio_ref_count (folio)) goto next;if (!folio_trylock (folio))
mm/vmscan.c +2 -1
@@ -1317,7 +1317,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, goto keep_locked;if (folio_test_large (folio)) {/* cannot split folio, skip it */ -if (!can_split_folio (folio,1 ,NULL )) +if (folio_expected_ref_count (folio) != +folio_ref_count (folio) -1 ) goto activate_locked;/* * Split partially mapped folios right away.
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 5/6 migrate: replace RMP_ flags with TTU_ flags
In reply to Sasha LevinFrom: "Matthew Wilcox (Oracle)" <[email protected]>
[ Upstream commit 832d95b5314eea558cf4cc9ca40db10122ce8f63 ]
Instead of translating between RMP_ and TTU_ flags, remove the RMP_ flags and just use the TTU_ flag space; there's plenty available.
Possibly we should rename these to RMAP_ flags, and maybe even pass them in through rmap_walk_arg, but that can be done later.
Stable-dep-of: c299a2285d9d ("mm/huge_memory: use folio's memcg inside __folio_split()")
--- include/linux/rmap.h | 9 +++------ mm/huge_memory.c | 6 +++--- mm/migrate.c | 12 ++++++------ 3 files changed, 12 insertions(+), 15 deletions(-)
include/linux/rmap.h +3 -6
@@ -92,6 +92,7 @@ struct anon_vma_chain { };enum ttu_flags { +TTU_USE_SHARED_ZEROPAGE =0x2 ,/* for unused pages of large folios */ TTU_SPLIT_HUGE_PMD =0x4 ,/* split huge PMD if any */ TTU_IGNORE_MLOCK =0x8 ,/* ignore mlock */ TTU_SYNC =0x10 ,/* avoid racy checks with PVMW_SYNC */
@@ -1000,12 +1001,8 @@ int mapping_wrprotect_range(struct address_space *mapping, pgoff_t pgoff,int pfn_mkclean_range (unsigned long pfn,unsigned long nr_pages,pgoff_t pgoff,struct vm_area_struct *vma); -enum rmp_flags { -RMP_LOCKED =1 <<0 , -RMP_USE_SHARED_ZEROPAGE =1 <<1 , -}; - -void remove_migration_ptes (struct folio *src,struct folio *dst,int flags); +void remove_migration_ptes (struct folio *src,struct folio *dst, +enum ttu_flags flags);/* * rmap_walk_control: To control rmap traversing for specific needs
mm/huge_memory.c +3 -3
@@ -3296,7 +3296,7 @@ static void remap_page(struct folio *folio, unsigned long nr, int flags)if (!folio_test_anon (folio))return ;for (;;) { -remove_migration_ptes (folio, folio,RMP_LOCKED | flags); +remove_migration_ptes (folio, folio,TTU_RMAP_LOCKED | flags); i +=folio_nr_pages (folio);if (i >= nr)break ;
@@ -3700,8 +3700,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,struct folio *new_folio, *next;int nr_shmem_dropped =0 ;enum ttu_flags ttu_flags =0 ; -pgoff_t end =0 ;int ret; +pgoff_t end =0 ;VM_WARN_ON_ONCE_FOLIO (!folio_test_locked (folio), folio);VM_WARN_ON_ONCE_FOLIO (!folio_test_large (folio), folio);
@@ -3946,7 +3946,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,shmem_uncharge (mapping->host, nr_shmem_dropped);if (!ret && is_anon) - ttu_flags = (enum ttu_flags )RMP_USE_SHARED_ZEROPAGE ; + ttu_flags =TTU_USE_SHARED_ZEROPAGE ;remap_page (folio,1 << old_order, ttu_flags);/*
mm/migrate.c +6 -6
@@ -451,11 +451,12 @@ static bool remove_migration_pte(struct folio *folio,* Get rid of all migration entries and replace them by * references to the indicated page. */ -void remove_migration_ptes (struct folio *src,struct folio *dst,int flags) +void remove_migration_ptes (struct folio *src,struct folio *dst, +enum ttu_flags flags) {struct rmap_walk_arg rmap_walk_arg = { .folio = src, - .map_unused_to_zeropage = flags &RMP_USE_SHARED_ZEROPAGE , + .map_unused_to_zeropage = flags &TTU_USE_SHARED_ZEROPAGE , };struct rmap_walk_control rwc = {
@@ -463,9 +464,9 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, int flags) .arg = &rmap_walk_arg, }; -VM_BUG_ON_FOLIO ((flags &RMP_USE_SHARED_ZEROPAGE ) && (src != dst), src); +VM_BUG_ON_FOLIO ((flags &TTU_USE_SHARED_ZEROPAGE ) && (src != dst), src); -if (flags &RMP_LOCKED ) +if (flags &TTU_RMAP_LOCKED )rmap_walk_locked (dst, &rwc);else rmap_walk (dst, &rwc);
@@ -1530,8 +1531,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio, rc =move_to_new_folio (dst, src, mode);if (page_was_mapped) -remove_migration_ptes (src, !rc ? dst : src, - ttu ?RMP_LOCKED :0 ); +remove_migration_ptes (src, !rc ? dst : src, ttu);if (ttu &TTU_RMAP_LOCKED )i_mmap_unlock_write (mapping);
Signature · 1 line
2.53.0
Sasha Levin
maintainerPATCH 6/6 mm/huge_memory: use folio's memcg inside __folio_split()
In reply to Sasha LevinFrom: Zi Yan <[email protected]>
[ Upstream commit c299a2285d9d8bda4da024455de65e3d00de6f17 ]
Patch series "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to folio's memcg", v3.
__GFP_ACCOUNT is needed for xarray node allocation accounting when XA_FLAGS_ACCOUNT is set. Commit 7b785645e8f13 ("mm: fix page cache convergence regression") fixed a workingset regression with it. xas_split_alloc() does not have it and needs to be fixed.
In addition, based on Sashiko's review[1] and Johannes' confirmation[2], to charge the right memcg, folio's memcg needs to be active during folio split. Add that before adding __GFP_ACCOUNT.
There is no workingset convergence regression related to missing __GFP_ACCOUNT in xas_split_alloc() and the impact to userspace should be minor.
This patch (of 2):
During a pagecache folio split, an xarray node allocation can happen and needs to charge at folio's memcg instead of folio split invoker's memcg, because for example folio split can happen during reclaim and reclaim's active memcg might not be folio's memcg. Switch to folio's memcg at the beginning and switch back afterwards.
--- mm/huge_memory.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
mm/huge_memory.c +16 -4
@@ -3694,34 +3694,42 @@ static int __folio_split(struct folio *folio, unsigned int new_order,XA_STATE (xas, &folio->mapping->i_pages, folio->index);struct folio *end_folio =folio_next (folio);bool is_anon =folio_test_anon (folio); +struct mem_cgroup *memcg, *old_memcg;struct address_space *mapping =NULL ;struct anon_vma *anon_vma =NULL ;int old_order =folio_order (folio);struct folio *new_folio, *next;int nr_shmem_dropped =0 ;enum ttu_flags ttu_flags =0 ; -int ret;pgoff_t end =0 ; +int ret;VM_WARN_ON_ONCE_FOLIO (!folio_test_locked (folio), folio);VM_WARN_ON_ONCE_FOLIO (!folio_test_large (folio), folio);if (folio !=page_folio (split_at) || folio !=page_folio (lock_at)) { ret = -EINVAL ; - goto out; + goto out_no_memcg; }if (new_order >= old_order) { ret = -EINVAL ; - goto out; + goto out_no_memcg; } ret =folio_check_splittable (folio, new_order, uniform_split);if (ret) {VM_WARN_ONCE (ret == -EINVAL ,"Tried to split an unsplittable folio" ); - goto out; + goto out_no_memcg; } +/* +* switch to folio's memcg as xarray node allocation can happen and +* needs to charge to it. +*/ + memcg =get_mem_cgroup_from_folio (folio); + old_memcg =set_active_memcg (memcg); +if (is_anon) {/* * The caller does not necessarily hold an mmap_lock that would
@@ -3989,6 +3997,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order,if (mapping)i_mmap_unlock_read (mapping); out: +/* restore to caller's old_memcg */ +set_active_memcg (old_memcg); +mem_cgroup_put (memcg); +out_no_memcg:xas_destroy (&xas);if (old_order ==HPAGE_PMD_ORDER )count_vm_event (!ret ?THP_SPLIT_PAGE :THP_SPLIT_PAGE_FAILED );
Signature · 1 line
2.53.0