Skip to contentSkip to conversation

mm/hugetlb: folio and migration cleanups

PATCH Superseded David Hildenbrand 3 participants 21 messages Jan 10, 2025 – Jan 13, 2025

Subsystems (7)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover mm/hugetlb: folio and migration cleanups

Some cleanups around more folio conversion and migration handling that I collected working on random stuff.

Patch #2->#6 were previous sent as part of [1], but I think they make sense independent of that.

[1] https://lkml.kernel.org/r/[email protected]

Cc Andrew Morton <[email protected]>
Cc Muchun Song <[email protected]>
Cc "Matthew Wilcox (Oracle)" <[email protected]>

David Hildenbrand (6):
mm/huge_memory: convert has_hwpoisoned into a pure folio flag mm/hugetlb: rename isolate_hugetlb() to folio_isolate_hugetlb() mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb
folio
mm/hugetlb: rename folio_putback_active_hugetlb() to
folio_putback_hugetlb()
mm/hugetlb-cgroup: convert hugetlb_cgroup_css_offline() to work on
folios
mm/hugetlb: use folio->lru int demote_free_hugetlb_folios()

 include/linux/hugetlb.h    |  8 +++----
 include/linux/page-flags.h |  6 ++---
 mm/gup.c                   |  2 +-
 mm/huge_memory.c           |  2 +-
 mm/hugetlb.c               | 47 +++++++++++++++++++++++++++++++++-----
 mm/hugetlb_cgroup.c        | 17 +++++++-------
 mm/mempolicy.c             |  2 +-
 mm/migrate.c               | 20 ++++++++--------
 8 files changed, 68 insertions(+), 36 deletions(-)

base-commit: 0703fa3785f1b969a3a98fc9bb3e4ae5062684ea

Signature · 1 line
2.47.1
#1 lore raw

PATCH 1/6 mm/huge_memory: convert has_hwpoisoned into a pure folio flag

Let's stop setting it on pages, there is no need to anymore.

---
 include/linux/page-flags.h | 6 ++----
 mm/huge_memory.c           | 2 +-
 2 files changed, 3 insertions(+), 5 deletions(-)
include/linux/page-flags.h +2 -4
index 14226d6bd6f84..3f6a64ff968a7 100644
@@ -948,11 +948,9 @@ TESTPAGEFLAG_FALSE(TransCompound, transcompound)
  *
  * This flag is set by hwpoison handler.  Cleared by THP split or free page.
  */
-PAGEFLAG(HasHWPoisoned, has_hwpoisoned, PF_SECOND)
-	TESTSCFLAG(HasHWPoisoned, has_hwpoisoned, PF_SECOND)
+FOLIO_FLAG(has_hwpoisoned, FOLIO_SECOND_PAGE)
 #else
-PAGEFLAG_FALSE(HasHWPoisoned, has_hwpoisoned)
-	TESTSCFLAG_FALSE(HasHWPoisoned, has_hwpoisoned)
+FOLIO_FLAG_FALSE(has_hwpoisoned)
 #endif
 
 /*
mm/huge_memory.c +1 -1
index 2654a95487499..3d3ebdc002d59 100644
@@ -3290,7 +3290,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 	/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
 	lruvec = folio_lruvec_lock(folio);
 
-	ClearPageHasHWPoisoned(head);
+	folio_clear_has_hwpoisoned(folio);
 
 	for (i = nr - new_nr; i >= new_nr; i -= new_nr) {
 		struct folio *tail;
Signature · 1 line
2.47.1
#3 lore raw

PATCH 2/6 mm/hugetlb: rename isolate_hugetlb() to folio_isolate_hugetlb()

In reply to David Hildenbrand

Let's make the function name match "folio_isolate_lru()", and add some kernel doc.

---
 include/linux/hugetlb.h |  4 ++--
 mm/gup.c                |  2 +-
 mm/hugetlb.c            | 23 ++++++++++++++++++++---
 mm/mempolicy.c          |  2 +-
 mm/migrate.c            |  6 +++---
 5 files changed, 27 insertions(+), 10 deletions(-)
include/linux/hugetlb.h +2 -2
index 49ec2362ce926..c95ad5cd7894d 100644
@@ -153,7 +153,7 @@ bool hugetlb_reserve_pages(struct inode *inode, long from, long to,
 						vm_flags_t vm_flags);
 long hugetlb_unreserve_pages(struct inode *inode, long start, long end,
 						long freed);
-bool isolate_hugetlb(struct folio *folio, struct list_head *list);
+bool folio_isolate_hugetlb(struct folio *folio, struct list_head *list);
 int get_hwpoison_hugetlb_folio(struct folio *folio, bool *hugetlb, bool unpoison);
 int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
 				bool *migratable_cleared);
@@ -414,7 +414,7 @@ static inline pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr,
 	return NULL;
 }
 
-static inline bool isolate_hugetlb(struct folio *folio, struct list_head *list)
+static inline bool folio_isolate_hugetlb(struct folio *folio, struct list_head *list)
 {
 	return false;
 }
mm/gup.c +1 -1
index 00a1269cbee0a..2cc3a9d28e70e 100644
@@ -2344,7 +2344,7 @@ static unsigned long collect_longterm_unpinnable_folios(
 			continue;
 
 		if (folio_test_hugetlb(folio)) {
-			isolate_hugetlb(folio, movable_folio_list);
+			folio_isolate_hugetlb(folio, movable_folio_list);
 			continue;
 		}
 
mm/hugetlb.c +20 -3
index 9a5596022c4b3..da98d671088d0 100644
@@ -2808,7 +2808,7 @@ static int alloc_and_dissolve_hugetlb_folio(struct hstate *h,
 		 * Fail with -EBUSY if not possible.
 		 */
 		spin_unlock_irq(&hugetlb_lock);
-		isolated = isolate_hugetlb(old_folio, list);
+		isolated = folio_isolate_hugetlb(old_folio, list);
 		ret = isolated ? 0 : -EBUSY;
 		spin_lock_irq(&hugetlb_lock);
 		goto free_new;
@@ -2893,7 +2893,7 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list)
 	if (hstate_is_gigantic(h))
 		return -ENOMEM;
 
-	if (folio_ref_count(folio) && isolate_hugetlb(folio, list))
+	if (folio_ref_count(folio) && folio_isolate_hugetlb(folio, list))
 		ret = 0;
 	else if (!folio_ref_count(folio))
 		ret = alloc_and_dissolve_hugetlb_folio(h, folio, list);
@@ -7417,7 +7417,24 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)
 
 #endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */
 
-bool isolate_hugetlb(struct folio *folio, struct list_head *list)
+/**
+ * folio_isolate_hugetlb: try to isolate an allocated hugetlb folio
+ * @folio: the folio to isolate
+ * @list: the list to add the folio to on success
+ *
+ * Isolate an allocated (refcount > 0) hugetlb folio, marking it as
+ * isolated/non-migratable, and moving it from the active list to the
+ * given list.
+ *
+ * Isolation will fail if @folio is not an allocated hugetlb folio, or if
+ * it is already isolated/non-migratable.
+ *
+ * On success, an additional folio reference is taken that must be dropped
+ * using folio_putback_active_hugetlb() to undo the isolation.
+ *
+ * Return: True if isolation worked, otherwise False.
+ */
+bool folio_isolate_hugetlb(struct folio *folio, struct list_head *list)
 {
 	bool ret = true;
 
mm/mempolicy.c +1 -1
index f83b73236ffe7..bbaadbeeb2919 100644
@@ -647,7 +647,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
 	 */
 	if ((flags & MPOL_MF_MOVE_ALL) ||
 	    (!folio_likely_mapped_shared(folio) && !hugetlb_pmd_shared(pte)))
-		if (!isolate_hugetlb(folio, qp->pagelist))
+		if (!folio_isolate_hugetlb(folio, qp->pagelist))
 			qp->nr_failed++;
 unlock:
 	spin_unlock(ptl);
mm/migrate.c +3 -3
index caadbe393aa21..80887cadb2774 100644
@@ -128,7 +128,7 @@ static void putback_movable_folio(struct folio *folio)
  *
  * This function shall be used whenever the isolated pageset has been
  * built from lru, balloon, hugetlbfs page. See isolate_migratepages_range()
- * and isolate_hugetlb().
+ * and folio_isolate_hugetlb().
  */
 void putback_movable_pages(struct list_head *l)
 {
@@ -169,7 +169,7 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list)
 	bool isolated, lru;
 
 	if (folio_test_hugetlb(folio))
-		return isolate_hugetlb(folio, list);
+		return folio_isolate_hugetlb(folio, list);
 
 	lru = !__folio_test_movable(folio);
 	if (lru)
@@ -2203,7 +2203,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,
 		return -EACCES;
 
 	if (folio_test_hugetlb(folio)) {
-		if (isolate_hugetlb(folio, pagelist))
+		if (folio_isolate_hugetlb(folio, pagelist))
 			return 1;
 	} else if (folio_isolate_lru(folio)) {
 		list_add_tail(&folio->lru, pagelist);
Signature · 1 line
2.47.1
#6 lore raw

PATCH 3/6 mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio

In reply to David Hildenbrand

We replaced a simple put_page() by a putback_active_hugepage() call in commit 3aaa76e125c1 ("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.

Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".

Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed and we just want to free the freshly allocated hugetlb folio.

Let's simply set the "migratable" flag in move_hugetlb_state(), where we know that allocation succeeded, and use simple folio_put() to return our reference.

Do we need the hugetlb_lock for setting that flag? Staring at other users of folio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.

---
 mm/hugetlb.c | 5 +++++
 mm/migrate.c | 8 ++++----
 2 files changed, 9 insertions(+), 4 deletions(-)
mm/hugetlb.c +5 -0
index da98d671088d0..b24ccf8ecbf38 100644
@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re
 		}
 		spin_unlock_irq(&hugetlb_lock);
 	}
+	/*
+	 * Our old folio is isolated and has "migratable" cleared until it
+	 * is putback. As migration succeeded, set the new folio "migratable".
+	 */
+	folio_set_hugetlb_migratable(new_folio);
 }
 
 static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
mm/migrate.c +4 -4
index 80887cadb2774..7e23e78f1e57b 100644
@@ -1542,14 +1542,14 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 		list_move_tail(&src->lru, ret);
 
 	/*
-	 * If migration was not successful and there's a freeing callback, use
-	 * it.  Otherwise, put_page() will drop the reference grabbed during
-	 * isolation.
+	 * If migration was not successful and there's a freeing callback,
+	 * return the folio to that special allocator. Otherwise, simply drop
+	 * our additional reference.
 	 */
 	if (put_new_folio)
 		put_new_folio(dst, private);
 	else
-		folio_putback_active_hugetlb(dst);
+		folio_put(dst);
 
 	return rc;
 }
Signature · 1 line
2.47.1
reviewer
#7 lore raw

Re: [PATCH v1 3/6] mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio

Quoted 52 lines · David Hildenbrand

We replaced a simple put_page() by a putback_active_hugepage() call in commit 3aaa76e125c1 ("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.

Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".

Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed and we just want to free the freshly allocated hugetlb folio.

Let's simply set the "migratable" flag in move_hugetlb_state(), where we know that allocation succeeded, and use simple folio_put() to return our reference.

Do we need the hugetlb_lock for setting that flag? Staring at other users of folio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.

---
  mm/hugetlb.c | 5 +++++
  mm/migrate.c | 8 ++++----
  2 files changed, 9 insertions(+), 4 deletions(-)
mm/hugetlb.c +5 -0
index da98d671088d0..b24ccf8ecbf38 100644
@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re
  		}
  		spin_unlock_irq(&hugetlb_lock);
  	}
+	/*
+	 * Our old folio is isolated and has "migratable" cleared until it
+	 * is putback. As migration succeeded, set the new folio "migratable".
+	 */
+	folio_set_hugetlb_migratable(new_folio);
  }
 
  static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
mm/migrate.c +0 -2
index 80887cadb2774..7e23e78f1e57b 100644
@@ -1542,14 +1542,14 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
  		list_move_tail(&src->lru, ret);
 
  	/*
-	 * If migration was not successful and there's a freeing callback, use
-	 * it.  Otherwise, put_page() will drop the reference grabbed during

mm/migrate.c static int unmap_and_move_huge_page(new_folio_t get_new_folio, · line 1552

Quoted 10 lines
mm/migrate.c +4 -2
-	 * isolation.
+	 * If migration was not successful and there's a freeing callback,
+	 * return the folio to that special allocator. Otherwise, simply drop
+	 * our additional reference.
  	 */
  	if (put_new_folio)
  		put_new_folio(dst, private);
  	else
-		folio_putback_active_hugetlb(dst);
+		folio_put(dst);

IIUC, after the changes, so the 'dst' folio might not be added into the 'h->hugepage_activelist' list (if the 'dst' is temporarily allocated), Could this cause any side effects?

#8 lore raw

Re: [PATCH v1 3/6] mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio

Quoted 52 lines · Baolin Wang
Quoted 49 lines · David Hildenbrand

We replaced a simple put_page() by a putback_active_hugepage() call in commit 3aaa76e125c1 ("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.

Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".

Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed and we just want to free the freshly allocated hugetlb folio.

Let's simply set the "migratable" flag in move_hugetlb_state(), where we know that allocation succeeded, and use simple folio_put() to return our reference.

Do we need the hugetlb_lock for setting that flag? Staring at other users of folio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.

---
   mm/hugetlb.c | 5 +++++
   mm/migrate.c | 8 ++++----
   2 files changed, 9 insertions(+), 4 deletions(-)
mm/hugetlb.c +5 -0
index da98d671088d0..b24ccf8ecbf38 100644
@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re
   		}
   		spin_unlock_irq(&hugetlb_lock);
   	}
+	/*
+	 * Our old folio is isolated and has "migratable" cleared until it
+	 * is putback. As migration succeeded, set the new folio "migratable".
+	 */
+	folio_set_hugetlb_migratable(new_folio);
   }
 
   static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
mm/migrate.c +0 -0
index 80887cadb2774..7e23e78f1e57b 100644
@@ -1542,14 +1542,14 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
   		list_move_tail(&src->lru, ret);
 
Quoted 14 lines

mm/migrate.c static int unmap_and_move_huge_page(new_folio_t get_new_folio, · line 1552

Quoted 13 lines
mm/migrate.c +4 -4
   	/*
-	 * If migration was not successful and there's a freeing callback, use
-	 * it.  Otherwise, put_page() will drop the reference grabbed during
-	 * isolation.
+	 * If migration was not successful and there's a freeing callback,
+	 * return the folio to that special allocator. Otherwise, simply drop
+	 * our additional reference.
   	 */
   	if (put_new_folio)
   		put_new_folio(dst, private);
   	else
-		folio_putback_active_hugetlb(dst);
+		folio_put(dst);

Hi Baolin,

thanks for the review!

Quoted 3 lines

IIUC, after the changes, so the 'dst' folio might not be added into the 'h->hugepage_activelist' list (if the 'dst' i:s temporarily allocated), Could this cause any side effects?

Good point, so far I was under the assumption that the dst folio would already be in the active list.

alloc_migration_target() and friends call alloc_hugetlb_folio_nodemask().

There are two cases:

1) We call dequeue_hugetlb_folio_nodemask() -> dequeue_hugetlb_folio_node_exact() where we add the folio to the hugepage_activelist.

2) We call alloc_migrate_hugetlb_folio()

It indeed looks like we don't add them to the active list. So likely we should do:

mm/hugetlb.c +3 -0
index dca4f310617a2..c6463dd7a1fc8 100644
@@ -7546,7 +7546,10 @@ void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int re
         * Our old folio is isolated and has "migratable" cleared until it
         * is putback. As migration succeeded, set the new folio "migratable".
         */
+       spin_lock_irq(&hugetlb_lock);
        folio_set_hugetlb_migratable(new_folio);
+       list_move_tail(&new_folio->lru, &(folio_hstate(new_folio))->hugepage_activelist);
+       spin_unlock_irq(&hugetlb_lock);
 }
 
 static void hugetlb_unshare_pmds(struct vm_area_struct *vma,

move_hugetlb_state() also takes care of that "temporary" handling.

(I wonder if in case 1) it was a problem that the folio was already on the active list)

Signature · 3 lines
Cheers,

David / dhildenb
reviewer Reviewed-by
#9 lore raw

Re: [PATCH v1 3/6] mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio

Quoted 113 lines · David Hildenbrand
Quoted 61 lines · Baolin Wang
Quoted 58 lines · David Hildenbrand

We replaced a simple put_page() by a putback_active_hugepage() call in commit 3aaa76e125c1 ("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.

Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".

Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed and we just want to free the freshly allocated hugetlb folio.

Let's simply set the "migratable" flag in move_hugetlb_state(), where we know that allocation succeeded, and use simple folio_put() to return our reference.

Do we need the hugetlb_lock for setting that flag? Staring at other users of folio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.

---
   mm/hugetlb.c | 5 +++++
   mm/migrate.c | 8 ++++----

   2 files changed, 9 insertions(+), 4 deletions(-)
mm/hugetlb.c +3 -0
index da98d671088d0..b24ccf8ecbf38 100644
@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio 
 *old_folio, struct folio *new_folio, int re
            }
            spin_unlock_irq(&hugetlb_lock);
        }
+    /*
+     * Our old folio is isolated and has "migratable" cleared until it
+     * is putback. As migration succeeded, set the new folio 

"migratable".
+     */
+    folio_set_hugetlb_migratable(new_folio);
   }
   static void hugetlb_unshare_pmds(struct vm_area_struct *vma,

mm/migrate.c +0 -1
index 80887cadb2774..7e23e78f1e57b 100644
@@ -1542,14 +1542,14 @@ static int 
 unmap_and_move_huge_page(new_folio_t get_new_folio,
            list_move_tail(&src->lru, ret);
        /*
-     * If migration was not successful and there's a freeing 

callback, use
-     * it.  Otherwise, put_page() will drop the reference grabbed during
-     * isolation.
+     * If migration was not successful and there's a freeing callback, +     * return the folio to that special allocator. Otherwise, simply

Quoted 9 lines
Quoted 8 lines

drop
+     * our additional reference.
        */
       if (put_new_folio)
           put_new_folio(dst, private);
       else
-        folio_putback_active_hugetlb(dst);
+        folio_put(dst);

Hi Baolin,

thanks for the review!

Quoted 3 lines

IIUC, after the changes, so the 'dst' folio might not be added into the 'h->hugepage_activelist' list (if the 'dst' i:s temporarily allocated), Could this cause any side effects?

Good point, so far I was under the assumption that the dst folio would already be in the active list.

alloc_migration_target() and friends call alloc_hugetlb_folio_nodemask().

There are two cases:

1) We call dequeue_hugetlb_folio_nodemask() -> dequeue_hugetlb_folio_node_exact() where we add the folio to the hugepage_activelist.

2) We call alloc_migrate_hugetlb_folio()

It indeed looks like we don't add them to the active list. So likely we should do:

mm/hugetlb.c +0 -0
index dca4f310617a2..c6463dd7a1fc8 100644
@@ -7546,7 +7546,10 @@ void move_hugetlb_state(struct folio *old_folio, 

struct folio *new_folio, int re
         * Our old folio is isolated and has "migratable" cleared until it
         * is putback. As migration succeeded, set the new folio
"migratable".
         */
+       spin_lock_irq(&hugetlb_lock);
        folio_set_hugetlb_migratable(new_folio);
+       list_move_tail(&new_folio->lru, &(folio_hstate(new_folio))->hugepage_activelist);
+       spin_unlock_irq(&hugetlb_lock);
 }

Quoted 4 lines

 static void hugetlb_unshare_pmds(struct vm_area_struct *vma,

move_hugetlb_state() also takes care of that "temporary" handling.

LGTM. With the changes:

Quoted 2 lines

(I wonder if in case 1) it was a problem that the folio was already on the active list)

Seems harmless, and putback_active_hugepage() did the same before.

#10 lore raw

Re: [PATCH v1 3/6] mm/migrate: don't call folio_putback_active_hugetlb() on dst hugetlb folio

Quoted 119 lines · Baolin Wang
Quoted 116 lines · David Hildenbrand
Quoted 62 lines · Baolin Wang
Quoted 59 lines · David Hildenbrand

We replaced a simple put_page() by a putback_active_hugepage() call in commit 3aaa76e125c1 ("mm: migrate: hugetlb: putback destination hugepage to active list"), to set the "active" flag on the dst hugetlb folio.

Nowadays, we decoupled the "active" list from the flag, by calling the flag "migratable".

Calling "putback" on something that wasn't allocated is weird and not future proof, especially if we might reach that path when migration failed
and we just want to free the freshly allocated hugetlb folio.

Let's simply set the "migratable" flag in move_hugetlb_state(), where we know that allocation succeeded, and use simple folio_put() to return our reference.

Do we need the hugetlb_lock for setting that flag? Staring at other users of folio_set_hugetlb_migratable(), it does not look like it. After all, the dst folio should already be on the active list, and we are not modifying that list.

---
    mm/hugetlb.c | 5 +++++
    mm/migrate.c | 8 ++++----
    2 files changed, 9 insertions(+), 4 deletions(-)
mm/hugetlb.c +3 -0
index da98d671088d0..b24ccf8ecbf38 100644
@@ -7529,6 +7529,11 @@ void move_hugetlb_state(struct folio
 *old_folio, struct folio *new_folio, int re
            }
            spin_unlock_irq(&hugetlb_lock);
        }
+    /*
+     * Our old folio is isolated and has "migratable" cleared until it
+     * is putback. As migration succeeded, set the new folio

"migratable".
+     */
+    folio_set_hugetlb_migratable(new_folio);
   }
   static void hugetlb_unshare_pmds(struct vm_area_struct *vma,

mm/migrate.c +0 -1
index 80887cadb2774..7e23e78f1e57b 100644
@@ -1542,14 +1542,14 @@ static int
 unmap_and_move_huge_page(new_folio_t get_new_folio,
            list_move_tail(&src->lru, ret);
        /*
-     * If migration was not successful and there's a freeing

callback, use
-     * it.  Otherwise, put_page() will drop the reference grabbed during
-     * isolation.
+     * If migration was not successful and there's a freeing callback, +     * return the folio to that special allocator. Otherwise, simply

Quoted 9 lines
Quoted 8 lines

drop
+     * our additional reference.
        */
       if (put_new_folio)
           put_new_folio(dst, private);
       else
-        folio_putback_active_hugetlb(dst);
+        folio_put(dst);

Hi Baolin,

thanks for the review!

Quoted 3 lines

IIUC, after the changes, so the 'dst' folio might not be added into the 'h->hugepage_activelist' list (if the 'dst' i:s temporarily allocated), Could this cause any side effects?

Good point, so far I was under the assumption that the dst folio would already be in the active list.

alloc_migration_target() and friends call alloc_hugetlb_folio_nodemask().

There are two cases:

1) We call dequeue_hugetlb_folio_nodemask() -> dequeue_hugetlb_folio_node_exact()
where we add the folio to the hugepage_activelist.

2) We call alloc_migrate_hugetlb_folio()

It indeed looks like we don't add them to the active list. So likely we should do:

mm/hugetlb.c +0 -0
index dca4f310617a2..c6463dd7a1fc8 100644
@@ -7546,7 +7546,10 @@ void move_hugetlb_state(struct folio *old_folio,

struct folio *new_folio, int re
         * Our old folio is isolated and has "migratable" cleared until it
         * is putback. As migration succeeded, set the new folio
"migratable".
         */
+       spin_lock_irq(&hugetlb_lock);
        folio_set_hugetlb_migratable(new_folio);
+       list_move_tail(&new_folio->lru, &(folio_hstate(new_folio))->hugepage_activelist);
+       spin_unlock_irq(&hugetlb_lock);
 }

Quoted 8 lines
Quoted 4 lines

 static void hugetlb_unshare_pmds(struct vm_area_struct *vma,

move_hugetlb_state() also takes care of that "temporary" handling.

LGTM. With the changes:

Thanks!

Quoted 5 lines
Quoted 3 lines

(I wonder if in case 1) it was a problem that the folio was already on the active list)

Seems harmless, and putback_active_hugepage() did the same before.

Yes, I think as we free it, we'll put it automatically back to the freelist using enqueue_hugetlb_folio(). I suspect the temporary-active-although-not-active is no real problem.

Signature · 3 lines
Cheers,

David / dhildenb
#11 lore raw

PATCH 4/6 mm/hugetlb: rename folio_putback_active_hugetlb() to folio_putback_hugetlb()

In reply to David Hildenbrand

Now that folio_putback_hugetlb() is only called on folios that were previously isolated through folio_isolate_hugetlb(), let's rename it to match folio_putback_lru().

Add some kernel doc to clarify how this function is supposed to be used.

---
 include/linux/hugetlb.h |  4 ++--
 mm/hugetlb.c            | 15 +++++++++++++--
 mm/migrate.c            |  6 +++---
 3 files changed, 18 insertions(+), 7 deletions(-)
include/linux/hugetlb.h +2 -2
index c95ad5cd7894d..ec8c0ccc8f959 100644
@@ -157,7 +157,7 @@ bool folio_isolate_hugetlb(struct folio *folio, struct list_head *list);
 int get_hwpoison_hugetlb_folio(struct folio *folio, bool *hugetlb, bool unpoison);
 int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
 				bool *migratable_cleared);
-void folio_putback_active_hugetlb(struct folio *folio);
+void folio_putback_hugetlb(struct folio *folio);
 void move_hugetlb_state(struct folio *old_folio, struct folio *new_folio, int reason);
 void hugetlb_fix_reserve_counts(struct inode *inode);
 extern struct mutex *hugetlb_fault_mutex_table;
@@ -430,7 +430,7 @@ static inline int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
 	return 0;
 }
 
-static inline void folio_putback_active_hugetlb(struct folio *folio)
+static inline void folio_putback_hugetlb(struct folio *folio)
 {
 }
 
mm/hugetlb.c +13 -2
index b24ccf8ecbf38..60617eecb99dd 100644
@@ -7430,7 +7430,7 @@ __weak unsigned long hugetlb_mask_last_page(struct hstate *h)
  * it is already isolated/non-migratable.
  *
  * On success, an additional folio reference is taken that must be dropped
- * using folio_putback_active_hugetlb() to undo the isolation.
+ * using folio_putback_hugetlb() to undo the isolation.
  *
  * Return: True if isolation worked, otherwise False.
  */
@@ -7482,7 +7482,18 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
 	return ret;
 }
 
-void folio_putback_active_hugetlb(struct folio *folio)
+/**
+ * folio_putback_hugetlb: unisolate a hugetlb folio
+ * @folio: the isolated hugetlb folio
+ *
+ * Putback/un-isolate the hugetlb folio that was previous isolated using
+ * folio_isolate_hugetlb(): marking it non-isolated/migratable and putting it
+ * back onto the active list.
+ *
+ * Will drop the additional folio reference obtained through
+ * folio_isolate_hugetlb().
+ */
+void folio_putback_hugetlb(struct folio *folio)
 {
 	spin_lock_irq(&hugetlb_lock);
 	folio_set_hugetlb_migratable(folio);
mm/migrate.c +3 -3
index 7e23e78f1e57b..be9e3b48cd622 100644
@@ -137,7 +137,7 @@ void putback_movable_pages(struct list_head *l)
 
 	list_for_each_entry_safe(folio, folio2, l, lru) {
 		if (unlikely(folio_test_hugetlb(folio))) {
-			folio_putback_active_hugetlb(folio);
+			folio_putback_hugetlb(folio);
 			continue;
 		}
 		list_del(&folio->lru);
@@ -1454,7 +1454,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 
 	if (folio_ref_count(src) == 1) {
 		/* page was freed from under us. So we are done. */
-		folio_putback_active_hugetlb(src);
+		folio_putback_hugetlb(src);
 		return MIGRATEPAGE_SUCCESS;
 	}
 
@@ -1537,7 +1537,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio,
 	folio_unlock(src);
 out:
 	if (rc == MIGRATEPAGE_SUCCESS)
-		folio_putback_active_hugetlb(src);
+		folio_putback_hugetlb(src);
 	else if (rc != -EAGAIN)
 		list_move_tail(&src->lru, ret);
 
Signature · 1 line
2.47.1
reviewer Reviewed-by on v1 4/6
#12 lore raw

Re: [PATCH v1 4/6] mm/hugetlb: rename folio_putback_active_hugetlb() to folio_putback_hugetlb()

Quoted 7 lines · David Hildenbrand

Now that folio_putback_hugetlb() is only called on folios that were previously isolated through folio_isolate_hugetlb(), let's rename it to match folio_putback_lru().

Add some kernel doc to clarify how this function is supposed to be used.

Yes, makes sense. Thanks.

#13 lore raw

PATCH 5/6 mm/hugetlb-cgroup: convert hugetlb_cgroup_css_offline() to work on folios

In reply to David Hildenbrand

Let's convert hugetlb_cgroup_css_offline() and hugetlb_cgroup_move_parent() to work on folios. hugepage_activelist contains folios, not pages.

While at it, rename page_hcg simply to hcg, removing most of the "page" terminology.

---
 mm/hugetlb_cgroup.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
mm/hugetlb_cgroup.c +8 -9
index 89a8ad45a533d..bb9578bd99f98 100644
@@ -195,24 +195,23 @@ static void hugetlb_cgroup_css_free(struct cgroup_subsys_state *css)
  * cannot fail.
  */
 static void hugetlb_cgroup_move_parent(int idx, struct hugetlb_cgroup *h_cg,
-				       struct page *page)
+				       struct folio *folio)
 {
 	unsigned int nr_pages;
 	struct page_counter *counter;
-	struct hugetlb_cgroup *page_hcg;
+	struct hugetlb_cgroup *hcg;
 	struct hugetlb_cgroup *parent = parent_hugetlb_cgroup(h_cg);
-	struct folio *folio = page_folio(page);
 
-	page_hcg = hugetlb_cgroup_from_folio(folio);
+	hcg = hugetlb_cgroup_from_folio(folio);
 	/*
 	 * We can have pages in active list without any cgroup
 	 * ie, hugepage with less than 3 pages. We can safely
 	 * ignore those pages.
 	 */
-	if (!page_hcg || page_hcg != h_cg)
+	if (!hcg || hcg != h_cg)
 		goto out;
 
-	nr_pages = compound_nr(page);
+	nr_pages = folio_nr_pages(folio);
 	if (!parent) {
 		parent = root_h_cgroup;
 		/* root has no limit */
@@ -235,13 +234,13 @@ static void hugetlb_cgroup_css_offline(struct cgroup_subsys_state *css)
 {
 	struct hugetlb_cgroup *h_cg = hugetlb_cgroup_from_css(css);
 	struct hstate *h;
-	struct page *page;
+	struct folio *folio;
 
 	do {
 		for_each_hstate(h) {
 			spin_lock_irq(&hugetlb_lock);
-			list_for_each_entry(page, &h->hugepage_activelist, lru)
-				hugetlb_cgroup_move_parent(hstate_index(h), h_cg, page);
+			list_for_each_entry(folio, &h->hugepage_activelist, lru)
+				hugetlb_cgroup_move_parent(hstate_index(h), h_cg, folio);
 
 			spin_unlock_irq(&hugetlb_lock);
 		}
Signature · 1 line
2.47.1
Reviewed-by on v1 5/6
#14 lore raw

Re: [PATCH v1 5/6] mm/hugetlb-cgroup: convert hugetlb_cgroup_css_offline() to work on folios

Quoted 8 lines · David Hildenbrand

Let's convert hugetlb_cgroup_css_offline() and hugetlb_cgroup_move_parent() to work on folios. hugepage_activelist contains folios, not pages.

While at it, rename page_hcg simply to hcg, removing most of the "page" terminology.

I might also note that this removes an unnecessary call to compound_head().

#15 lore raw

PATCH 6/6 mm/hugetlb: use folio->lru int demote_free_hugetlb_folios()

In reply to David Hildenbrand

We are demoting hugetlb folios to smaller hugetlb folios; let's avoid messing with pages where avoidable.

---
 mm/hugetlb.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
mm/hugetlb.c +4 -2
index 60617eecb99dd..e872eff124abb 100644
@@ -3822,13 +3822,15 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst,
 
 		for (i = 0; i < pages_per_huge_page(src); i += pages_per_huge_page(dst)) {
 			struct page *page = folio_page(folio, i);
+			struct folio *new_folio;
 
 			page->mapping = NULL;
 			clear_compound_head(page);
 			prep_compound_page(page, dst->order);
+			new_folio = page_folio(page);
 
-			init_new_hugetlb_folio(dst, page_folio(page));
-			list_add(&page->lru, &dst_list);
+			init_new_hugetlb_folio(dst, new_folio);
+			list_add(&new_folio->lru, &dst_list);
 		}
 	}
 
Signature · 1 line
2.47.1
#16 lore raw

Re: [PATCH v1 6/6] mm/hugetlb: use folio->lru int demote_free_hugetlb_folios()

Quoted 4 lines · David Hildenbrand

We are demoting hugetlb folios to smaller hugetlb folios; let's avoid messing with pages where avoidable.

Good stuff. I have questions.

mm/hugetlb.c static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, · line 3825

Quoted 6 lines

+++ b/mm/hugetlb.c

mm/hugetlb.c +1 -0
@@ -3822,13 +3822,15 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst,
 
 		for (i = 0; i < pages_per_huge_page(src); i += pages_per_huge_page(dst)) {
 			struct page *page = folio_page(folio, i);
+			struct folio *new_folio;

I'm usually very against casting from page to folio, but I think it might be the better option in this case ...

mm/hugetlb.c static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, · line 3827

Quoted 1 line
mm/hugetlb.c +0 -0
 			page->mapping = NULL;

because then we could do new_folio->mapping = NULL.

We're going to have to do serious changes to this function anyway to convert from Ottawa to the New York interpretation, so the cast doesn't give me the feeling of danger that it would elsewhere.

Quoted 14 lines
mm/hugetlb.c +3 -2
 			clear_compound_head(page);
 			prep_compound_page(page, dst->order);
+			new_folio = page_folio(page);
 
-			init_new_hugetlb_folio(dst, page_folio(page));
-			list_add(&page->lru, &dst_list);
+			init_new_hugetlb_folio(dst, new_folio);
+			list_add(&new_folio->lru, &dst_list);
 		}
 	}
 
Signature · 1 line
2.47.1
#17 lore raw

Re: [PATCH v1 6/6] mm/hugetlb: use folio->lru int demote_free_hugetlb_folios()

Quoted 18 lines · Matthew Wilcox
Quoted 4 lines · David Hildenbrand

We are demoting hugetlb folios to smaller hugetlb folios; let's avoid messing with pages where avoidable.

Good stuff. I have questions.

Review static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, · line 3825

Quoted 6 lines

+++ b/mm/hugetlb.c

(quoted diff) +1 -0
@@ -3822,13 +3822,15 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst,
 
  		for (i = 0; i < pages_per_huge_page(src); i += pages_per_huge_page(dst)) {
  			struct page *page = folio_page(folio, i);
+			struct folio *new_folio;

I'm usually very against casting from page to folio, but I think it might be the better option in this case ...

Quoted 7 lines

Review static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, · line 3826

Quoted 1 line
(quoted diff) +0 -0
  			page->mapping = NULL;

because then we could do new_folio->mapping = NULL.

We're going to have to do serious changes to this function anyway to convert from Ottawa to the New York interpretation, so the cast doesn't give me the feeling of danger that it would elsewhere.

Hm, that makes me wonder if we should do it even more similar like our other split function (__split_huge_page_tail)?

mm/hugetlb.c +5 -3
index 60617eecb99dd..23fe5654f632c 100644
@@ -3821,14 +3821,18 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst,
                pgalloc_tag_split(folio, huge_page_order(src), huge_page_order(dst));
 
                for (i = 0; i < pages_per_huge_page(src); i += pages_per_huge_page(dst)) {
                        struct page *page = folio_page(folio, i);
+                       /* Careful: see __split_huge_page_tail() */
+                       struct folio *new_folio = (struct folio *)page;
 
-                       page->mapping = NULL;
                        clear_compound_head(page);
                        prep_compound_page(page, dst->order);
 
-                       init_new_hugetlb_folio(dst, page_folio(page));
-                       list_add(&page->lru, &dst_list);
+                       new_folio->mapping = NULL;
+                       init_new_hugetlb_folio(dst, new_folio);
+                       list_add(&new_folio->lru, &dst_list);
                }
        }
 

I was even wondering if we should be using nth_page() instead of folio_page() -- similar to __split_huge_page_tail.

If we'd add sanity checking in current code to folio_page() to verify that i falls inside the folio, the current code would blow up as we modify the folio using prep_compound_page().

Signature · 3 lines
Cheers,

David / dhildenb
#18 lore raw
In reply to David Hildenbrand
Quoted 5 lines · David Hildenbrand

Some cleanups around more folio conversion and migration handling that I collected working on random stuff.

Patch #2->#6 were previous sent as part of [1], but I think they make sense independent of that.

Heh, sending it out now I realize that #1 is a THP cleanup. Anyhow, not the end of the world for this series :)

Signature · 3 lines
Cheers,

David / dhildenb
#19 lore raw
Quoted 9 lines · David Hildenbrand
Quoted 5 lines · David Hildenbrand

Some cleanups around more folio conversion and migration handling that I collected working on random stuff.

Patch #2->#6 were previous sent as part of [1], but I think they make sense independent of that.

Heh, sending it out now I realize that #1 is a THP cleanup. Anyhow, not the end of the world for this series :)

I've reviewed the ones I feel comfortable with. The others are a bit more hugetlb-centric than I'm expert in.

#20 lore raw
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · David Hildenbrand

Some cleanups around more folio conversion and migration handling that I collected working on random stuff.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

Patch #2->#6 were previous sent as part of [1], but I think they make sense independent of that.

Heh, sending it out now I realize that #1 is a THP cleanup. Anyhow, not the end of the world for this series :)

I've reviewed the ones I feel comfortable with. The others are a bit more hugetlb-centric than I'm expert in.

Thanks Willy!

Signature · 3 lines
Cheers,

David / dhildenb