mm/hugetlb: folio and migration cleanups
Subsystems (7)
cover mm/hugetlb: folio and migration cleanups
Some cleanups around more folio conversion and migration handling that I collected working on random stuff.
Patch #2->#6 were previous sent as part of [1], but I think they make sense independent of that.
[1] https://lkml.kernel.org/r/[email protected]
David Hildenbrand (6):
mm/huge_memory: convert has_hwpoisoned into a pure folio flag mm/hugetlb: rename isolate_hugetlb() to folio_isolate_hugetlb() mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb
folio
mm/hugetlb: rename folio_putback_active_hugetlb() to
folio_putback_hugetlb()
mm/hugetlb-cgroup: convert hugetlb_cgroup_css_offline() to work on
folios
mm/hugetlb: use folio->lru int demote_free_hugetlb_folios()
include/linux/hugetlb.h | 8 +++---- include/linux/page-flags.h | 6 ++--- mm/gup.c | 2 +- mm/huge_memory.c | 2 +- mm/hugetlb.c | 47 +++++++++++++++++++++++++++++++++----- mm/hugetlb_cgroup.c | 17 +++++++------- mm/mempolicy.c | 2 +- mm/migrate.c | 20 ++++++++-------- 8 files changed, 68 insertions(+), 36 deletions(-)
base-commit: 0703fa3785f1b969a3a98fc9bb3e4ae5062684ea
Signature · 1 line
2.47.1
PATCH 1/6 mm/huge_memory: convert has_hwpoisoned into a pure folio flag
Let's stop setting it on pages, there is no need to anymore.
--- include/linux/page-flags.h | 6 ++---- mm/huge_memory.c | 2 +- 2 files changed, 3 insertions(+), 5 deletions(-)
include/linux/page-flags.h +2 -4
@@ -948,11 +948,9 @@ TESTPAGEFLAG_FALSE(TransCompound, transcompound)* * This flag is set by hwpoison handler. Cleared by THP split or free page. */ -PAGEFLAG(HasHWPoisoned, has_hwpoisoned, PF_SECOND) - TESTSCFLAG(HasHWPoisoned, has_hwpoisoned, PF_SECOND) +FOLIO_FLAG(has_hwpoisoned, FOLIO_SECOND_PAGE)#else -PAGEFLAG_FALSE(HasHWPoisoned, has_hwpoisoned) - TESTSCFLAG_FALSE(HasHWPoisoned, has_hwpoisoned) +FOLIO_FLAG_FALSE(has_hwpoisoned)#endif /*
mm/huge_memory.c +1 -1
@@ -3290,7 +3290,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,/* lock lru list/PageCompound, ref frozen by page_ref_freeze */ lruvec =folio_lruvec_lock (folio); -ClearPageHasHWPoisoned (head); +folio_clear_has_hwpoisoned (folio);for (i = nr - new_nr; i >= new_nr; i -= new_nr) {struct folio *tail;
Signature · 1 line
2.47.1
Re: [PATCH v1 1/6] mm/huge_memory: convert has_hwpoisoned into a pure folio flag
Quoted 3 lines · David Hildenbrand
Let's stop setting it on pages, there is no need to anymore.
Signed-off-by David Hildenbrand <[email protected]>
Yes!
PATCH 2/6 mm/hugetlb: rename isolate_hugetlb() to folio_isolate_hugetlb()
In reply to David HildenbrandLet's make the function name match "folio_isolate_lru()", and add some kernel doc.
--- include/linux/hugetlb.h | 4 ++-- mm/gup.c | 2 +- mm/hugetlb.c | 23 ++++++++++++++++++++--- mm/mempolicy.c | 2 +- mm/migrate.c | 6 +++--- 5 files changed, 27 insertions(+), 10 deletions(-)
include/linux/hugetlb.h +2 -2
@@ -153,7 +153,7 @@ bool hugetlb_reserve_pages(struct inode *inode, long from, long to, vm_flags_t vm_flags);long hugetlb_unreserve_pages (struct inode *inode,long start,long end,long freed); -bool isolate_hugetlb (struct folio *folio,struct list_head *list); +bool folio_isolate_hugetlb (struct folio *folio,struct list_head *list);int get_hwpoison_hugetlb_folio (struct folio *folio,bool *hugetlb,bool unpoison);int get_huge_page_for_hwpoison (unsigned long pfn,int flags,bool *migratable_cleared);
@@ -414,7 +414,7 @@ static inline pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr,return NULL ; } -static inline bool isolate_hugetlb (struct folio *folio,struct list_head *list) +static inline bool folio_isolate_hugetlb (struct folio *folio,struct list_head *list) {return false; }
mm/gup.c +1 -1
@@ -2344,7 +2344,7 @@ static unsigned long collect_longterm_unpinnable_folios(continue ;if (folio_test_hugetlb (folio)) { -isolate_hugetlb (folio, movable_folio_list); +folio_isolate_hugetlb (folio, movable_folio_list);continue ; }
mm/hugetlb.c +20 -3
@@ -2808,7 +2808,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct hstate *h,* Fail with -EBUSY if not possible. */ spin_unlock_irq (&hugetlb_lock); - isolated =isolate_hugetlb (old_folio, list); + isolated =folio_isolate_hugetlb (old_folio, list); ret = isolated ?0 : -EBUSY ;spin_lock_irq (&hugetlb_lock); goto free_new;
@@ -2893,7 +2893,7 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list)if (hstate_is_gigantic (h))return -ENOMEM ; -if (folio_ref_count (folio) &&isolate_hugetlb (folio, list)) +if (folio_ref_count (folio) &&folio_isolate_hugetlb (folio, list)) ret =0 ;else if (!folio_ref_count (folio)) ret =alloc_and_dissolve_hugetlb_folio (h, folio, list);
@@ -7417,7 +7417,24 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ -bool isolate_hugetlb (struct folio *folio,struct list_head *list) +/** +* folio_isolate_hugetlb: try to isolate an allocated hugetlb folio +* @folio: the folio to isolate +* @list: the list to add the folio to on success +* +* Isolate an allocated (refcount > 0) hugetlb folio, marking it as +* isolated/non-migratable, and moving it from the active list to the +* given list. +* +* Isolation will fail if @folio is not an allocated hugetlb folio, or if +* it is already isolated/non-migratable. +* +* On success, an additional folio reference is taken that must be dropped +* using folio_putback_active_hugetlb() to undo the isolation. +* +* Return: True if isolation worked, otherwise False. +*/ +bool folio_isolate_hugetlb (struct folio *folio,struct list_head *list) {bool ret = true;
mm/mempolicy.c +1 -1
@@ -647,7 +647,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,*/ if ((flags &MPOL_MF_MOVE_ALL ) || (!folio_likely_mapped_shared (folio) && !hugetlb_pmd_shared (pte))) -if (!isolate_hugetlb (folio, qp->pagelist)) +if (!folio_isolate_hugetlb (folio, qp->pagelist)) qp->nr_failed++; unlock:spin_unlock (ptl);
mm/migrate.c +3 -3
@@ -128,7 +128,7 @@ static void putback_movable_folio(struct folio *folio)* * This function shall be used whenever the isolated pageset has been * built from lru, balloon, hugetlbfs page. See isolate_migratepages_range() -* and isolate_hugetlb(). +* and folio_isolate_hugetlb(). */ void putback_movable_pages (struct list_head *l) {
@@ -169,7 +169,7 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list)bool isolated, lru;if (folio_test_hugetlb (folio)) -return isolate_hugetlb (folio, list); +return folio_isolate_hugetlb (folio, list); lru = !__folio_test_movable (folio);if (lru)
@@ -2203,7 +2203,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,return -EACCES ;if (folio_test_hugetlb (folio)) { -if (isolate_hugetlb (folio, pagelist)) +if (folio_isolate_hugetlb (folio, pagelist))return 1 ; }else if (folio_isolate_lru (folio)) {list_add_tail (&folio->lru, pagelist);
Signature · 1 line
2.47.1
Re: [PATCH v1 2/6] mm/hugetlb: rename isolate_hugetlb() to folio_isolate_hugetlb()
Quoted 2 lines · David Hildenbrand
+/**
+ *folio_isolate_hugetlb: try to isolate an allocated hugetlb folio
s/:/ -/
Re: [PATCH v1 2/6] mm/hugetlb: rename isolate_hugetlb() to folio_isolate_hugetlb()
In reply to David HildenbrandQuoted 4 lines · David Hildenbrand
Let's make the function name match "
folio_isolate_lru()", and add some kernel doc.Signed-off-by David Hildenbrand <[email protected]>
LGTM.
PATCH 3/6 mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio
In reply to David HildenbrandWe replaced a simple put_page() by a putback_active_hugepage() call in commit 3aaa76e125c1 ("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.
Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".
Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed and we just want to free the freshly allocated hugetlb folio.
Let's simply set the "migratable" flag in move_hugetlb_state(), where we know that allocation succeeded, and use simple folio_put() to return our reference.
Do we need the hugetlb_lock for setting that flag? Staring at other users of folio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.
--- mm/hugetlb.c | 5 +++++ mm/migrate.c | 8 ++++---- 2 files changed, 9 insertions(+), 4 deletions(-)
mm/hugetlb.c +5 -0
@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re }spin_unlock_irq (&hugetlb_lock); } +/* +* Our old folio is isolated and has "migratable" cleared until it +* is putback. As migration succeeded, set the new folio "migratable". +*/ +folio_set_hugetlb_migratable (new_folio); } static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
mm/migrate.c +4 -4
@@ -1542,14 +1542,14 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,list_move_tail (&src->lru, ret);/* -* If migration was not successful and there's a freeing callback, use -* it. Otherwise, put_page() will drop the reference grabbed during -* isolation. +* If migration was not successful and there's a freeing callback, +* return the folio to that special allocator. Otherwise, simply drop +* our additional reference. */ if (put_new_folio)put_new_folio (dst, private);else -folio_putback_active_hugetlb (dst); +folio_put (dst);return rc; }
Signature · 1 line
2.47.1
Baolin Wang
reviewerRe: [PATCH v1 3/6] mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio
Quoted 52 lines · David Hildenbrand
We replaced a simple
put_page()by aputback_active_hugepage()call in commit3aaa76e125c1("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".
Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed and we just want to free the freshly allocated hugetlb folio.
Let's simply set the "migratable" flag in
move_hugetlb_state(), where we know that allocation succeeded, and use simplefolio_put()to return our reference.Do we need the
hugetlb_lockfor setting that flag? Staring at other users offolio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.Signed-off-by David Hildenbrand <[email protected]>--- mm/hugetlb.c | 5 +++++ mm/migrate.c | 8 ++++---- 2 files changed, 9 insertions(+), 4 deletions(-)mm/hugetlb.c +5
-0@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re }spin_unlock_irq (&hugetlb_lock); } +/* +* Our old folio is isolated and has "migratable" cleared until it +* is putback. As migration succeeded, set the new folio "migratable". +*/ +folio_set_hugetlb_migratable (new_folio); } static void hugetlb_unshare_pmds(struct vm_area_struct *vma,mm/migrate.c +0
-2@@ -1542,14 +1542,14 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,list_move_tail (&src->lru, ret);/* -* If migration was not successful and there's a freeing callback, use -* it. Otherwise, put_page() will drop the reference grabbed during
mm/migrate.c static int unmap_and_move_huge_page(new_folio_t get_new_folio, · line 1552
Quoted 10 lines
mm/migrate.c +4
-2-* isolation. +* If migration was not successful and there's a freeing callback, +* return the folio to that special allocator. Otherwise, simply drop +* our additional reference. */ if (put_new_folio)put_new_folio (dst, private);else -folio_putback_active_hugetlb (dst); +folio_put (dst);
IIUC, after the changes, so the 'dst' folio might not be added into the 'h->hugepage_activelist' list (if the 'dst' is temporarily allocated), Could this cause any side effects?
Re: [PATCH v1 3/6] mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio
Quoted 52 lines · Baolin Wang
Quoted 49 lines · David Hildenbrand
We replaced a simple
put_page()by aputback_active_hugepage()call in commit3aaa76e125c1("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".
Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed and we just want to free the freshly allocated hugetlb folio.
Let's simply set the "migratable" flag in
move_hugetlb_state(), where we know that allocation succeeded, and use simplefolio_put()to return our reference.Do we need the
hugetlb_lockfor setting that flag? Staring at other users offolio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.Signed-off-by David Hildenbrand <[email protected]>--- mm/hugetlb.c | 5 +++++ mm/migrate.c | 8 ++++---- 2 files changed, 9 insertions(+), 4 deletions(-)mm/hugetlb.c +5
-0@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re }spin_unlock_irq (&hugetlb_lock); } +/* +* Our old folio is isolated and has "migratable" cleared until it +* is putback. As migration succeeded, set the new folio "migratable". +*/ +folio_set_hugetlb_migratable (new_folio); } static void hugetlb_unshare_pmds(struct vm_area_struct *vma,mm/migrate.c +0
-0@@ -1542,14 +1542,14 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,list_move_tail (&src->lru, ret);
Quoted 14 lines
mm/migrate.c static int unmap_and_move_huge_page(new_folio_t get_new_folio, · line 1552
Quoted 13 lines
mm/migrate.c +4
-4/* -* If migration was not successful and there's a freeing callback, use -* it. Otherwise, put_page() will drop the reference grabbed during -* isolation. +* If migration was not successful and there's a freeing callback, +* return the folio to that special allocator. Otherwise, simply drop +* our additional reference. */ if (put_new_folio)put_new_folio (dst, private);else -folio_putback_active_hugetlb (dst); +folio_put (dst);
Hi Baolin,
thanks for the review!
Quoted 3 lines
IIUC, after the changes, so the 'dst' folio might not be added into the '
h->hugepage_activelist' list (if the 'dst' i:s temporarily allocated), Could this cause any side effects?
Good point, so far I was under the assumption that the dst folio would already be in the active list.
alloc_migration_target() and friends call alloc_hugetlb_folio_nodemask().
There are two cases:
1) We call dequeue_hugetlb_folio_nodemask() -> dequeue_hugetlb_folio_node_exact() where we add the folio to the hugepage_activelist.
2) We call alloc_migrate_hugetlb_folio()
It indeed looks like we don't add them to the active list. So likely we should do:
mm/hugetlb.c +3 -0
@@ -7546,7 +7546,10 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re* Our old folio is isolated and has "migratable" cleared until it * is putback. As migration succeeded, set the new folio "migratable". */ +spin_lock_irq (&hugetlb_lock);folio_set_hugetlb_migratable (new_folio); +list_move_tail (&new_folio->lru, &(folio_hstate (new_folio))->hugepage_activelist); +spin_unlock_irq (&hugetlb_lock); } static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
move_hugetlb_state() also takes care of that "temporary" handling.
(I wonder if in case 1) it was a problem that the folio was already on the active list)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 3/6] mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio
Quoted 113 lines · David Hildenbrand
Quoted 61 lines · Baolin Wang
Quoted 58 lines · David Hildenbrand
We replaced a simple
put_page()by aputback_active_hugepage()call in commit3aaa76e125c1("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".
Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed and we just want to free the freshly allocated hugetlb folio.
Let's simply set the "migratable" flag in
move_hugetlb_state(), where we know that allocation succeeded, and use simplefolio_put()to return our reference.Do we need the
hugetlb_lockfor setting that flag? Staring at other users offolio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.Signed-off-by David Hildenbrand <[email protected]>---
mm/hugetlb.c| 5 +++++
mm/migrate.c| 8 ++++----2 files changed, 9 insertions(+), 4 deletions(-)mm/hugetlb.c +3
-0@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re } spin_unlock_irq(&hugetlb_lock); } + /* +* Our old folio is isolated and has "migratable" cleared until it +* is putback. As migration succeeded, set the new folio "migratable".
+ */
+folio_set_hugetlb_migratable(new_folio);
}
static void hugetlb_unshare_pmds(structvm_area_struct*vma,mm/migrate.c +0
-1@@ -1542,14 +1542,14 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio, list_move_tail(&src->lru, ret); /* -* If migration was not successful and there's a freeing callback, use
- * it. Otherwise,put_page()will drop the reference grabbed during
- * isolation.
+ * If migration was not successful and there's a freeing callback, + * return the folio to that special allocator. Otherwise, simplyQuoted 9 lines
Quoted 8 lines
drop
+ * our additional reference.
*/
if (put_new_folio)
put_new_folio(dst, private);
else
-folio_putback_active_hugetlb(dst);
+folio_put(dst);Hi Baolin,
thanks for the review!
Quoted 3 lines
IIUC, after the changes, so the 'dst' folio might not be added into the '
h->hugepage_activelist' list (if the 'dst' i:s temporarily allocated), Could this cause any side effects?Good point, so far I was under the assumption that the dst folio would already be in the active list.
alloc_migration_target()and friends callalloc_hugetlb_folio_nodemask().There are two cases:
1) We call
dequeue_hugetlb_folio_nodemask()->dequeue_hugetlb_folio_node_exact()where we add the folio to thehugepage_activelist.2) We call
alloc_migrate_hugetlb_folio()It indeed looks like we don't add them to the active list. So likely we should do:
mm/hugetlb.c +0
-0@@ -7546,7 +7546,10 @@ void move_hugetlb_state(struct folio *old_folio,struct folio
*new_folio, int re
* Our old folio is isolated and has "migratable" cleared until it
* is putback. As migration succeeded, set the new folio
"migratable".
*/
+spin_lock_irq(&hugetlb_lock);
folio_set_hugetlb_migratable(new_folio);
+list_move_tail(&new_folio->lru, &(folio_hstate(new_folio))->hugepage_activelist);
+spin_unlock_irq(&hugetlb_lock);
}
Quoted 4 lines
static void hugetlb_unshare_pmds(struct
vm_area_struct*vma,
move_hugetlb_state()also takes care of that "temporary" handling.
LGTM. With the changes:
Quoted 2 lines
(I wonder if in case 1) it was a problem that the folio was already on the active list)
Seems harmless, and putback_active_hugepage() did the same before.
Re: [PATCH v1 3/6] mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio
Quoted 119 lines · Baolin Wang
Quoted 116 lines · David Hildenbrand
Quoted 62 lines · Baolin Wang
Quoted 59 lines · David Hildenbrand
We replaced a simple
put_page()by aputback_active_hugepage()call in commit3aaa76e125c1("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".
Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed
and we just want to free the freshly allocated hugetlb folio.Let's simply set the "migratable" flag in
move_hugetlb_state(), where we know that allocation succeeded, and use simplefolio_put()to return our reference.Do we need the
hugetlb_lockfor setting that flag? Staring at other users offolio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.Signed-off-by David Hildenbrand <[email protected]>--- mm/hugetlb.c | 5 +++++ mm/migrate.c | 8 ++++---- 2 files changed, 9 insertions(+), 4 deletions(-)mm/hugetlb.c +3
-0@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re } spin_unlock_irq(&hugetlb_lock); } + /* +* Our old folio is isolated and has "migratable" cleared until it +* is putback. As migration succeeded, set the new folio "migratable".
+ */
+folio_set_hugetlb_migratable(new_folio);
}
static void hugetlb_unshare_pmds(structvm_area_struct*vma,mm/migrate.c +0
-1@@ -1542,14 +1542,14 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio, list_move_tail(&src->lru, ret); /* -* If migration was not successful and there's a freeing callback, use
- * it. Otherwise,put_page()will drop the reference grabbed during
- * isolation.
+ * If migration was not successful and there's a freeing callback, + * return the folio to that special allocator. Otherwise, simplyQuoted 9 lines
Quoted 8 lines
drop
+ * our additional reference.
*/
if (put_new_folio)
put_new_folio(dst, private);
else
-folio_putback_active_hugetlb(dst);
+folio_put(dst);Hi Baolin,
thanks for the review!
Quoted 3 lines
IIUC, after the changes, so the 'dst' folio might not be added into the '
h->hugepage_activelist' list (if the 'dst' i:s temporarily allocated), Could this cause any side effects?Good point, so far I was under the assumption that the dst folio would already be in the active list.
alloc_migration_target()and friends callalloc_hugetlb_folio_nodemask().There are two cases:
1) We call
dequeue_hugetlb_folio_nodemask()->dequeue_hugetlb_folio_node_exact()
where we add the folio to thehugepage_activelist.2) We call
alloc_migrate_hugetlb_folio()It indeed looks like we don't add them to the active list. So likely we should do:
mm/hugetlb.c +0
-0@@ -7546,7 +7546,10 @@ void move_hugetlb_state(struct folio *old_folio,struct folio
*new_folio, int re
* Our old folio is isolated and has "migratable" cleared until it
* is putback. As migration succeeded, set the new folio
"migratable".
*/
+spin_lock_irq(&hugetlb_lock);
folio_set_hugetlb_migratable(new_folio);
+list_move_tail(&new_folio->lru, &(folio_hstate(new_folio))->hugepage_activelist);
+spin_unlock_irq(&hugetlb_lock);
}
Quoted 8 lines
Quoted 4 lines
static void hugetlb_unshare_pmds(struct
vm_area_struct*vma,
move_hugetlb_state()also takes care of that "temporary" handling.LGTM. With the changes:
Reviewed-by Baolin Wang <[email protected]>
Thanks!
Quoted 5 lines
Quoted 3 lines
(I wonder if in case 1) it was a problem that the folio was already on the active list)
Seems harmless, and
putback_active_hugepage()did the same before.
Yes, I think as we free it, we'll put it automatically back to the freelist using enqueue_hugetlb_folio(). I suspect the temporary-active-although-not-active is no real problem.
Signature · 3 lines
Cheers, David / dhildenb
PATCH 4/6 mm/hugetlb: rename folio_putback_active_hugetlb() to folio_putback_hugetlb()
In reply to David HildenbrandNow that folio_putback_hugetlb() is only called on folios that were previously isolated through folio_isolate_hugetlb(), let's rename it to match folio_putback_lru().
Add some kernel doc to clarify how this function is supposed to be used.
--- include/linux/hugetlb.h | 4 ++-- mm/hugetlb.c | 15 +++++++++++++-- mm/migrate.c | 6 +++--- 3 files changed, 18 insertions(+), 7 deletions(-)
include/linux/hugetlb.h +2 -2
@@ -157,7 +157,7 @@ bool folio_isolate_hugetlb(struct folio *folio, struct list_head *list);int get_hwpoison_hugetlb_folio (struct folio *folio,bool *hugetlb,bool unpoison);int get_huge_page_for_hwpoison (unsigned long pfn,int flags,bool *migratable_cleared); -void folio_putback_active_hugetlb (struct folio *folio); +void folio_putback_hugetlb (struct folio *folio);void move_hugetlb_state (struct folio *old_folio,struct folio *new_folio,int reason);void hugetlb_fix_reserve_counts (struct inode *inode);extern struct mutex *hugetlb_fault_mutex_table;
@@ -430,7 +430,7 @@ static inline int get_huge_page_for_hwpoison(unsigned long pfn, int flags,return 0 ; } -static inline void folio_putback_active_hugetlb (struct folio *folio) +static inline void folio_putback_hugetlb (struct folio *folio) { }
mm/hugetlb.c +13 -2
@@ -7430,7 +7430,7 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)* it is already isolated/non-migratable. * * On success, an additional folio reference is taken that must be dropped -* using folio_putback_active_hugetlb() to undo the isolation. +* using folio_putback_hugetlb() to undo the isolation. * * Return: True if isolation worked, otherwise False. */
@@ -7482,7 +7482,18 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags,return ret; } -void folio_putback_active_hugetlb (struct folio *folio) +/** +* folio_putback_hugetlb: unisolate a hugetlb folio +* @folio: the isolated hugetlb folio +* +* Putback/un-isolate the hugetlb folio that was previous isolated using +* folio_isolate_hugetlb(): marking it non-isolated/migratable and putting it +* back onto the active list. +* +* Will drop the additional folio reference obtained through +* folio_isolate_hugetlb(). +*/ +void folio_putback_hugetlb (struct folio *folio) {spin_lock_irq (&hugetlb_lock);folio_set_hugetlb_migratable (folio);
mm/migrate.c +3 -3
@@ -137,7 +137,7 @@ void putback_movable_pages(struct list_head *l)list_for_each_entry_safe (folio, folio2, l, lru) {if (unlikely (folio_test_hugetlb (folio))) { -folio_putback_active_hugetlb (folio); +folio_putback_hugetlb (folio);continue ; }list_del (&folio->lru);
@@ -1454,7 +1454,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,if (folio_ref_count (src) ==1 ) {/* page was freed from under us. So we are done. */ -folio_putback_active_hugetlb (src); +folio_putback_hugetlb (src);return MIGRATEPAGE_SUCCESS ; }
@@ -1537,7 +1537,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,folio_unlock (src); out:if (rc ==MIGRATEPAGE_SUCCESS ) -folio_putback_active_hugetlb (src); +folio_putback_hugetlb (src);else if (rc != -EAGAIN )list_move_tail (&src->lru, ret);
Signature · 1 line
2.47.1
Re: [PATCH v1 4/6] mm/hugetlb: rename folio_putback_active_hugetlb() to folio_putback_hugetlb()
Quoted 7 lines · David Hildenbrand
Now that
folio_putback_hugetlb()is only called on folios that were previously isolated throughfolio_isolate_hugetlb(), let's rename it to matchfolio_putback_lru().Add some kernel doc to clarify how this function is supposed to be used.
Signed-off-by David Hildenbrand <[email protected]>
Yes, makes sense. Thanks.
PATCH 5/6 mm/hugetlb-cgroup: convert hugetlb_cgroup_css_offline() to work on folios
In reply to David HildenbrandLet's convert hugetlb_cgroup_css_offline() and hugetlb_cgroup_move_parent() to work on folios. hugepage_activelist contains folios, not pages.
While at it, rename page_hcg simply to hcg, removing most of the "page" terminology.
--- mm/hugetlb_cgroup.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
mm/hugetlb_cgroup.c +8 -9
@@ -195,24 +195,23 @@ static void hugetlb_cgroup_css_free(struct cgroup_subsys_state *css)* cannot fail. */ static void hugetlb_cgroup_move_parent (int idx,struct hugetlb_cgroup *h_cg, -struct page *page) +struct folio *folio) {unsigned int nr_pages;struct page_counter *counter; -struct hugetlb_cgroup *page_hcg; +struct hugetlb_cgroup *hcg;struct hugetlb_cgroup *parent =parent_hugetlb_cgroup (h_cg); -struct folio *folio =page_folio (page); - page_hcg =hugetlb_cgroup_from_folio (folio); + hcg =hugetlb_cgroup_from_folio (folio);/* * We can have pages in active list without any cgroup * ie, hugepage with less than 3 pages. We can safely * ignore those pages. */ -if (!page_hcg || page_hcg != h_cg) +if (!hcg || hcg != h_cg) goto out; - nr_pages =compound_nr (page); + nr_pages =folio_nr_pages (folio);if (!parent) { parent = root_h_cgroup;/* root has no limit */
@@ -235,13 +234,13 @@ static void hugetlb_cgroup_css_offline(struct cgroup_subsys_state *css) {struct hugetlb_cgroup *h_cg =hugetlb_cgroup_from_css (css);struct hstate *h; -struct page *page; +struct folio *folio; do { for_each_hstate(h) { spin_lock_irq(&hugetlb_lock); - list_for_each_entry(page, &h->hugepage_activelist, lru) - hugetlb_cgroup_move_parent(hstate_index(h), h_cg, page); + list_for_each_entry(folio, &h->hugepage_activelist, lru) + hugetlb_cgroup_move_parent(hstate_index(h), h_cg, folio); spin_unlock_irq(&hugetlb_lock); }
Signature · 1 line
2.47.1
Re: [PATCH v1 5/6] mm/hugetlb-cgroup: convert hugetlb_cgroup_css_offline() to work on folios
Quoted 8 lines · David Hildenbrand
Let's convert
hugetlb_cgroup_css_offline()andhugetlb_cgroup_move_parent()to work on folios.hugepage_activelistcontains folios, not pages.While at it, rename
page_hcgsimply to hcg, removing most of the "page" terminology.Signed-off-by David Hildenbrand <[email protected]>
I might also note that this removes an unnecessary call to compound_head().
PATCH 6/6 mm/hugetlb: use folio->lru int demote_free_hugetlb_folios()
In reply to David HildenbrandWe are demoting hugetlb folios to smaller hugetlb folios; let's avoid messing with pages where avoidable.
--- mm/hugetlb.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
mm/hugetlb.c +4 -2
@@ -3822,13 +3822,15 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst,for (i =0 ; i <pages_per_huge_page (src); i +=pages_per_huge_page (dst)) {struct page *page =folio_page (folio, i); +struct folio *new_folio; page->mapping =NULL ;clear_compound_head (page);prep_compound_page (page, dst->order); + new_folio =page_folio (page); -init_new_hugetlb_folio (dst,page_folio (page)); -list_add (&page->lru, &dst_list); +init_new_hugetlb_folio (dst, new_folio); +list_add (&new_folio->lru, &dst_list); } }
Signature · 1 line
2.47.1
Re: [PATCH v1 6/6] mm/hugetlb: use folio->lru int demote_free_hugetlb_folios()
Quoted 4 lines · David Hildenbrand
We are demoting hugetlb folios to smaller hugetlb folios; let's avoid messing with pages where avoidable.
Signed-off-by David Hildenbrand <[email protected]>
Good stuff. I have questions.
mm/hugetlb.c static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, · line 3825
Quoted 6 lines
+++
b/mm/hugetlb.cmm/hugetlb.c +1
-0@@ -3822,13 +3822,15 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst,for (i =0 ; i <pages_per_huge_page (src); i +=pages_per_huge_page (dst)) {struct page *page =folio_page (folio, i); +struct folio *new_folio;
I'm usually very against casting from page to folio, but I think it might be the better option in this case ...
mm/hugetlb.c static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, · line 3827
Quoted 1 line
mm/hugetlb.c +0
-0page->mapping =NULL ;
because then we could do new_folio->mapping = NULL.
We're going to have to do serious changes to this function anyway to convert from Ottawa to the New York interpretation, so the cast doesn't give me the feeling of danger that it would elsewhere.
Quoted 14 lines
mm/hugetlb.c +3
-2clear_compound_head (page);prep_compound_page (page, dst->order); + new_folio =page_folio (page); -init_new_hugetlb_folio (dst,page_folio (page)); -list_add (&page->lru, &dst_list); +init_new_hugetlb_folio (dst, new_folio); +list_add (&new_folio->lru, &dst_list); } }Signature · 1 line
2.47.1
Re: [PATCH v1 6/6] mm/hugetlb: use folio->lru int demote_free_hugetlb_folios()
Quoted 18 lines · Matthew Wilcox
Quoted 4 lines · David Hildenbrand
We are demoting hugetlb folios to smaller hugetlb folios; let's avoid messing with pages where avoidable.
Signed-off-by David Hildenbrand <[email protected]>Good stuff. I have questions.
Review static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, · line 3825
Quoted 6 lines
+++
b/mm/hugetlb.c(quoted diff) +1
-0@@ -3822,13 +3822,15 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, for (i = 0; i < pages_per_huge_page(src); i += pages_per_huge_page(dst)) { struct page *page = folio_page(folio, i); + struct folio *new_folio;I'm usually very against casting from page to folio, but I think it might be the better option in this case ...
Quoted 7 lines
Review static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, · line 3826
Quoted 1 line
(quoted diff) +0
-0page->mapping = NULL;because then we could do
new_folio->mapping= NULL.We're going to have to do serious changes to this function anyway to convert from Ottawa to the New York interpretation, so the cast doesn't give me the feeling of danger that it would elsewhere.
Hm, that makes me wonder if we should do it even more similar like our other split function (__split_huge_page_tail)?
mm/hugetlb.c +5 -3
@@ -3821,14 +3821,18 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst,pgalloc_tag_split (folio,huge_page_order (src),huge_page_order (dst));for (i =0 ; i <pages_per_huge_page (src); i +=pages_per_huge_page (dst)) {struct page *page =folio_page (folio, i); +/* Careful: see __split_huge_page_tail() */ +struct folio *new_folio = (struct folio *)page; - page->mapping =NULL ;clear_compound_head (page);prep_compound_page (page, dst->order); -init_new_hugetlb_folio (dst,page_folio (page)); -list_add (&page->lru, &dst_list); + new_folio->mapping =NULL ; +init_new_hugetlb_folio (dst, new_folio); +list_add (&new_folio->lru, &dst_list); } }
I was even wondering if we should be using nth_page() instead of folio_page() -- similar to __split_huge_page_tail.
If we'd add sanity checking in current code to folio_page() to verify that i falls inside the folio, the current code would blow up as we modify the folio using prep_compound_page().
Signature · 3 lines
Cheers, David / dhildenb
Quoted 5 lines · David Hildenbrand
Some cleanups around more folio conversion and migration handling that I collected working on random stuff.
Patch #2->#6 were previous sent as part of [1], but I think they make sense independent of that.
Heh, sending it out now I realize that #1 is a THP cleanup. Anyhow, not the end of the world for this series :)
Signature · 3 lines
Cheers, David / dhildenb
Quoted 9 lines · David Hildenbrand
Quoted 5 lines · David Hildenbrand
Some cleanups around more folio conversion and migration handling that I collected working on random stuff.
Patch #2->#6 were previous sent as part of [1], but I think they make sense independent of that.
Heh, sending it out now I realize that #1 is a THP cleanup. Anyhow, not the end of the world for this series :)
I've reviewed the ones I feel comfortable with. The others are a bit more hugetlb-centric than I'm expert in.
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · David Hildenbrand
Some cleanups around more folio conversion and migration handling that I collected working on random stuff.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
Patch #2->#6 were previous sent as part of [1], but I think they make sense independent of that.
Heh, sending it out now I realize that #1 is a THP cleanup. Anyhow, not the end of the world for this series :)
I've reviewed the ones I feel comfortable with. The others are a bit more hugetlb-centric than I'm expert in.
Thanks Willy!
Signature · 3 lines
Cheers, David / dhildenb