Skip to contentSkip to conversation

mm/collapse: separate a collapse from its callers

PATCH Applied Kiryl Shutsemau (Meta) 5 participants 22 messages Oct 14 – Oct 19

Subsystems (2)
lore
Share & tools
lore

Kiryl Shutsemau (Meta)

author reviewer
#0 lore raw

v5 cover mm/collapse: separate a collapse from its callers

There is no line between the collapse engine and the callers that ask for a collapse. khugepaged.c holds both, and they reach into each other.

- Sixteen tests through the collapse path read cc->is_khugepaged to work out what they are allowed to do, when every one of those decisions was made by the caller before it asked.

- collapse_single_pmd() does both halves of a collapse behind one call. Where it gives up mmap_lock between them is buried in the anonymous scan on one side and ahead of the file scan on the other.

Start drawing the line. The caller states what it allows in a policy, and the collapse code reads the policy instead of asking who called. The scan of a PTE table and the collapse that follows get a function each, with the mmap_lock drop between them in one place.

Changes since v4
================

https://lore.kernel.org/all/[email protected]/

- Patches 10-13 dropped.

- Patch 7: the changelog no longer mentions a release for the control, which v4 dropped.

- Patch 9: collapse_scan_anon(), collapse_pmd() and collapse_anon() replace collapse_scan_anon_pmd(), collapse_run_pmd() and mthp_collapse() (David). The scan_* fields in collapse_control are commented as anonymous-only and file-only (Zi). Comments that named collapse_scan_pmd() for the anonymous scan follow the rename.

- Acked-by from David Hildenbrand on 5 and 8; Reviewed-by from Zi Yan on 5 and 8 and from Baolin Wang on 5; Tested-by from Baolin Wang on 5; Acked-by from Zi Yan on 9.

Patches
=======

The first three stand alone and can be taken separately:

1 drop the mmgrab() MADV_COLLAPSE has held since 7d8faaf15545 2 count collapses in khugepaged's own walk, where the daemon's
bookkeeping belongs
3 rename cc->mthp_present_ptes to eligible_ptes, which is what a set
bit means

Then, in order:

4 add collapse.h, and move enum scan_result, struct collapse_control
and the two constants into it
5 struct collapse_policy, filled by the caller; the is_khugepaged
tests become field reads, and the flag goes
6 drop collapse_possible(), a wrapper that only turns a mask into a bool 7 collapse_control_init_scan() is a per-table reset, so name it
collapse_scan_reset()
8 call collapse_file() from collapse_single_pmd(), so the file scan
stops at the decision
9 give the scan and the collapse a function each: collapse_scan_pmd()
and collapse_pmd(), joined by collapse_single_pmd()

Behaviour
=========

No functional change is intended. Nothing here changes which tables get collapsed, into what, or what MADV_COLLAPSE returns. The tracepoints are the one place a change can be seen from outside; the rest is where work happens, not what it does.

- Patches 8 and 9: mm_khugepaged_scan_file and mm_khugepaged_scan_pmd fire before the collapse rather than after it. For an accepted table their status field reads SCAN_SUCCEED, where it used to carry what the collapse made of the table; that is now for mm_khugepaged_collapse_file and mm_collapse_huge_page to report.

Three things move that a reader should not have to find in the diff:

- Patch 5: khugepaged fills its policy once per scan pass, so the max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than once per table. A knob written mid-pass takes effect on the next pass instead of the next table.

- Patch 8: the writeback retry re-runs collapse_file() alone, where it used to rescan the page cache first.

- Patch 9: the file scan runs under mmap_lock, where before the lock was given up first. A file table the scan refuses no longer costs khugepaged an unlock, a trip back through khugepaged_do_scan(), a relock and a VMA lookup; only a table it goes on to collapse does.

selftests/mm khugepaged passes on x86-64 with a KASAN, lockdep and DEBUG_VM config, in the forms run_vmtests.sh runs it, with all:file on ext4 rather than XFS. Every patch builds, CONFIG_TRANSPARENT_HUGEPAGE=n included.

Kiryl Shutsemau (Meta) (9):
mm/khugepaged: drop redundant mm_struct pin in madvise_collapse() mm/khugepaged: count collapses where khugepaged makes them mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes mm/collapse: add collapse.h for the collapse interface mm/collapse: state what a collapse may do in the policy mm/collapse: drop the collapse_possible() wrapper mm/collapse: name the per-table scan reset for what it resets mm/collapse: call collapse_file() from collapse_single_pmd() mm/collapse: separate scanning a PTE table from collapsing it

 MAINTAINERS     |   1 +
 mm/collapse.h   | 121 +++++++++++++
 mm/khugepaged.c | 468 ++++++++++++++++++++++++------------------------
 3 files changed, 355 insertions(+), 235 deletions(-)
 create mode 100644 mm/collapse.h

Range-diff against v4:

This compares the first 9 patches of v4 with this version.

 1:  73d87f3454b0 =  1:  73d87f3454b0 mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()
 2:  095b43905bd6 =  2:  095b43905bd6 mm/khugepaged: count collapses where khugepaged makes them
 3:  66802b5676a3 =  3:  66802b5676a3 mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes
 4:  22f00b9a706d =  4:  22f00b9a706d mm/collapse: add collapse.h for the collapse interface
 5:  4b9e43984bc9 !  5:  f712b2ab6635 mm/collapse: state what a collapse may do in the policy

@@ Commit message
cc unconditionally, so the check was already dead.

Assisted-by LLM

+ Acked-by: David Hildenbrand (Arm) <[email protected]>
+ Reviewed-by: Zi Yan <[email protected]>
+ Reviewed-by: Baolin Wang <[email protected]>
+ Tested-by: Baolin Wang <[email protected]>

## mm/collapse.h ##

 6:  9ae83d07f1e0 =  6:  6fa4e02993e3 mm/collapse: drop the collapse_possible() wrapper
 7:  bfdf63c56c7e !  7:  48ea4107ec7b mm/collapse: name the per-table scan reset for what it resets

@@ Commit message

Call it collapse_scan_reset().

- Preparation for giving a control a real init and release, run once each
- for a whole series of scans. Two names a word apart would then stand for
- two different jobs.
+ Preparation for giving a control a real init, run once for a whole series
+ of scans. Two names a word apart would then stand for two different jobs.

No functional change.

8: 9b8038917bc5 ! 8: 7d5dfd5de145 mm/collapse: call collapse_file() from collapse_single_pmd()
@@ Commit message
run without it. The file scan has to stop where the anonymous one will.

Assisted-by LLM

+ Acked-by: David Hildenbrand (Arm) <[email protected]>
+ Reviewed-by: Zi Yan <[email protected]>

## mm/khugepaged.c ##
9: dd6d94fdad05 ! 9: 6cccb8e88539 mm/collapse: separate scanning a PTE table from collapsing it
@@ Commit message

Give each half its own function:

- - collapse_scan_pmd() scans one table. The anonymous scan that used to
- carry that name keeps its body as collapse_scan_anon_pmd(), and
- collapse_scan_pmd() is now the entry that picks the anonymous or the
- file side.
+ - collapse_scan_pmd() scans one table. It picks the side:
+ collapse_scan_anon(), the anonymous scan that used to carry the
+ collapse_scan_pmd() name, or collapse_scan_file().

- - collapse_run_pmd() does the collapse the scan asked for, and is
- handed what the scan returned. SCAN_SUCCEED means there is
- something to collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page
- cache already holds the PMD folio and only the PTE table is left to
- retract. Both are work for the run; anything else is why there is
- nothing to do.
+ - collapse_pmd() does the collapse the scan asked for, and is handed
+ what the scan returned. SCAN_SUCCEED means there is something to
+ collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache already
+ holds the PMD folio and only the PTE table is left to retract.
+ Anything else is why there is nothing to do. It calls
+ collapse_anon(), formerly mthp_collapse(), or collapse_file().

- collapse_single_pmd() is now the two of them with the mmap_lock drop in
- between, so its callers see what they saw before. collapse_control_init()
- sets a control up before its first scan.
+ For now collapse_single_pmd() stays, as the two of them with the
+ mmap_lock drop in between, so its callers see what they saw before.
+ collapse_control_init() sets a control up before its first scan.

- What the scan found and the run needs travels in collapse_control. For
- an anonymous table that is the orders and the referenced and swapped-out
- counts, which mthp_collapse() and collapse_huge_page() now read from
- there instead of taking as arguments. For a file it is the file itself
- and the offset in it: a file collapse works on the page cache and never
- sees a VMA, so the scan takes the reference while it still has one and
- the run gives it back.
+ What the scan found and the collapse needs travels in collapse_control.
+ For an anonymous table that is the orders and the referenced and
+ swapped-out counts. collapse_anon() and collapse_huge_page() now read
+ them from there instead of taking them as arguments.
+
+ For a file it is the file itself and the offset in it. A file collapse
+ works on the page cache and never sees a VMA, so the scan takes the
+ reference while it still has one and the collapse gives it back.

The file scan moves under mmap_lock with the anonymous one, where before the lock was given up first. The lock is now held over the page cache
@@ Commit message
only a table that goes on to be collapsed does.

Assisted-by LLM

+ Acked-by: Zi Yan <[email protected]>

## mm/collapse.h ##
@@ mm/collapse.h: struct collapse_control {
DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
+
+ /*
-+ * What a scan found and the run after it needs. Live only between the
-+ * two, and read by nobody else.
-+ *
-+ * The file side takes a reference while it still has the VMA, since a
-+ * file collapse works on the page cache and never sees one; the run is
-+ * what gives it back.
++ * What a scan found and the collapse after it needs. Live only between
++ * the two, and read by nobody else.
+ */
++
++ /* Anonymous only: orders to try, referenced and swap PTE counts */
+ unsigned long scan_orders;
+ int scan_referenced;
+ int scan_unmapped;
++
++ /*
++ * File only. A file collapse works on the page cache and never sees a
++ * VMA, so the scan takes a reference to the file while it still has
++ * one, and the collapse gives it back.
++ */
+ struct file *scan_file;
+ pgoff_t scan_pgoff;
};
@@ mm/collapse.h: struct collapse_control {
#endif /* __MM_COLLAPSE_H */

## mm/khugepaged.c ##
+@@ mm/khugepaged.c: static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
+ goto out;
+ }
+
+- /* See collapse_scan_pmd(). */
++ /* See collapse_scan_anon(). */
+ if (folio_maybe_mapped_shared(folio)) {
+ /*
+ * TODO: Support shared pages without leading to further
+@@ mm/khugepaged.c: static enum scan_result check_pmd_still_valid(struct mm_struct *mm, +
+ /*
+ * Bring missing pages in from swap, to complete THP collapse.
+- * Only done if collapse_scan_pmd() believes it is worthwhile.
++ * Only done if collapse_scan_anon() believes it is worthwhile.
+ *
+ * For mTHP orders the function bails on the first swap entry, because
+ * faulting pages back in during collapse could re-populate PTEs that
@@ mm/khugepaged.c: static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
* while allocating a THP, as that could trigger direct reclaim/compaction.
* Note that the VMA must be rechecked after grabbing the mmap_lock again.
@@ mm/khugepaged.c: static enum scan_result collapse_huge_page(struct mm_struct *mm
if (result != SCAN_SUCCEED)
goto out_nolock;
}
+@@ mm/khugepaged.c: static unsigned int max_order_from_offset(unsigned int offset)
+ }
+
+ /*
+- * mthp_collapse() consumes the bitmap that is generated during
+- * collapse_scan_pmd() to determine what regions and mTHP orders fit best.
++ * collapse_anon() consumes the bitmap that is generated during
++ * collapse_scan_anon() to determine what regions and mTHP orders fit best.
+ *
+ * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse
+ * source. We start at the PMD order and check if it is eligible for collapse;
@@ mm/khugepaged.c: static unsigned int max_order_from_offset(unsigned int offset)
* If a collapse is permitted, we attempt to collapse the PTE range into a
* mTHP.
@@ mm/khugepaged.c: static unsigned int max_order_from_offset(unsigned int offset)
-static enum scan_result mthp_collapse(struct mm_struct *mm,
- unsigned long address, int referenced, int unmapped,
- struct collapse_control *cc, unsigned long enabled_orders)
-+static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long address,
-+ struct collapse_control *cc)
++static enum scan_result collapse_anon(struct mm_struct *mm,
++ unsigned long address, struct collapse_control *cc)
{
unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none;
enum scan_result last_result = SCAN_FAIL;
@@ mm/khugepaged.c: static enum scan_result mthp_collapse(struct mm_struct *mm,
-static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
- struct vm_area_struct *vma, unsigned long start_addr,
- bool *lock_dropped, struct collapse_control *cc)
-+static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, ++static enum scan_result collapse_scan_anon(struct vm_area_struct *vma,
+ unsigned long start_addr, struct collapse_control *cc)
{
const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
@@ mm/khugepaged.c: static enum scan_result mthp_collapse(struct mm_struct *mm,
pmd_t *pmd;
pte_t *pte, *_pte, pteval;
int i;
+@@ mm/khugepaged.c: static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
+ /*
+ * If PMD is the only enabled order, enforce max_ptes_none, otherwise
+ * scan all pages to populate the bitmap for mTHP collapse. The bitmap
+- * is then checked again in mthp_collapse() for each attempted order.
++ * is then checked again in collapse_anon() for each attempted order.
+ */
+ if (enabled_orders != BIT(HPAGE_PMD_ORDER))
+ max_ptes_none = COLLAPSE_MAX_PTES_LIMIT;
@@ mm/khugepaged.c: static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
out_unmap:
pte_unmap_unlock(pte, ptl);
@@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm

- mmap_assert_locked(mm);
+ mmap_assert_locked(vma->vm_mm);
-+ /* Whatever the last scan found has to have been run by now */
++ /* A scan that found work has to have been followed by collapse_pmd() */
+ if (WARN_ON_ONCE(cc->scan_file)) {
+ fput(cc->scan_file);
+ cc->scan_file = NULL;
@@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm

if (vma_is_anonymous(vma))
- return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc);
-+ return collapse_scan_anon_pmd(vma, addr, cc);
++ return collapse_scan_anon(vma, addr, cc);

- file = get_file(vma->vm_file);
pgoff = linear_page_index(vma, addr);
@@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm
/*
* SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds
- * the PMD folio, and only the PTE table is left to retract.
-+ * the PMD folio, and retracting the PTE table is the run's job.
++ * the PMD folio, and collapse_pmd() still has to retract the PTE table.
*/
- result = collapse_scan_file(mm, addr, file, pgoff, cc);
- if (result != SCAN_SUCCEED)
@@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm
+ return result;
+}
+
-+static enum scan_result collapse_run_pmd(struct mm_struct *mm, ++static enum scan_result collapse_pmd(struct mm_struct *mm,
+ unsigned long addr, enum scan_result result,
+ struct collapse_control *cc)
+{
@@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm
+ pgoff_t pgoff;
+
+ if (!file)
-+ return mthp_collapse(mm, addr, cc);
++ return collapse_anon(mm, addr, cc);
+
+ cc->scan_file = NULL;
+ pgoff = cc->scan_pgoff;
@@ mm/khugepaged.c: static enum scan_result collapse_single_pmd(unsigned long addr,
+ mmap_read_unlock(mm);
+ *lock_dropped = true;
+
-+ return collapse_run_pmd(mm, addr, result, cc);
++ return collapse_pmd(mm, addr, result, cc);
+}
+
static void collapse_scan_mm_slot(unsigned int progress_max,

Signature · 1 line
2.54.0
author reviewer
#1 lore raw

PATCH v5 1/9 mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()

madvise_collapse() holds an mmgrab() reference across its work. It is redundant. Every caller already holds mm_users:

- madvise(2) works on current->mm, which lives as long as the task is in the syscall;
- process_madvise(2) reaches a remote mm through mm_access(), which takes an mm_users reference and holds it until the syscall returns;
- io_uring passes current->mm;
- DAMON takes one with get_task_mm() and drops it after the call.

Drop the mmgrab()/mmdrop() pair.

---
 mm/khugepaged.c | 2 --
 1 file changed, 2 deletions(-)
mm/khugepaged.c +0 -2
index e13d233b9967..75c876acf527 100644
@@ -3229,7 +3229,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	cc->is_khugepaged = false;
 	cc->progress = 0;
 
-	mmgrab(mm);
 	lru_add_drain_all();
 
 	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
@@ -3285,7 +3284,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	}
 out_nolock:
 	mmap_assert_locked(mm);
-	mmdrop(mm);
 	kfree(cc);
 
 	return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0
Signature · 1 line
2.54.0
author reviewer
#3 lore raw

PATCH v5 2/9 mm/khugepaged: count collapses where khugepaged makes them

In reply to Kiryl Shutsemau (Meta)

collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and tests cc->is_khugepaged to know whether it should: the counter belongs to the daemon, and MADV_COLLAPSE must not touch it.

The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called.

---
 mm/khugepaged.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
mm/khugepaged.c +4 -7
index 75c876acf527..4979a93e3648 100644
@@ -2815,10 +2815,8 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 
 	mmap_assert_locked(mm);
 
-	if (vma_is_anonymous(vma)) {
-		result = collapse_scan_pmd(mm, vma, addr, lock_dropped, cc);
-		goto end;
-	}
+	if (vma_is_anonymous(vma))
+		return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc);
 
 	file = get_file(vma->vm_file);
 	pgoff = linear_page_index(vma, addr);
@@ -2854,9 +2852,6 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 			result = SCAN_SUCCEED;
 		mmap_read_unlock(mm);
 	}
-end:
-	if (cc->is_khugepaged && result == SCAN_SUCCEED)
-		++khugepaged_pages_collapsed;
 	return result;
 }
 
@@ -2933,6 +2928,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
 
 			*result = collapse_single_pmd(khugepaged_scan.address,
 						      vma, &lock_dropped, cc);
+			if (*result == SCAN_SUCCEED)
+				khugepaged_pages_collapsed++;
 			/* move to next address */
 			khugepaged_scan.address += HPAGE_PMD_SIZE;
 			if (lock_dropped)
Signature · 1 line
2.54.0
author reviewer
#5 lore raw

PATCH v5 3/9 mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes

In reply to Kiryl Shutsemau (Meta)

The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source.

mthp_collapse() then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either.

Name it for what a set bit means, and update the comments that named it.

No functional change.

---
 mm/khugepaged.c | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)
mm/khugepaged.c +16 -16
index 4979a93e3648..081f705cfca2 100644
@@ -115,8 +115,8 @@ struct collapse_control {
 	/* nodemask for allocation fallback */
 	nodemask_t alloc_nmask;
 
-	/* Each bit represents a single occupied (!none/zero) page. */
-	DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE);
+	/* Each bit marks a PTE the scan accepted as a collapse source */
+	DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
 };
 
 /**
@@ -627,7 +627,7 @@ static void collapse_control_init_scan(struct collapse_control *cc)
 {
 	memset(cc->node_load, 0, sizeof(cc->node_load));
 	nodes_clear(cc->alloc_nmask);
-	bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
+	bitmap_zero(cc->eligible_ptes, MAX_PTRS_PER_PTE);
 }
 
 static void release_pte_folio(struct folio *folio)
@@ -1482,15 +1482,15 @@ static unsigned int max_order_from_offset(unsigned int offset)
  * mthp_collapse() consumes the bitmap that is generated during
  * collapse_scan_pmd() to determine what regions and mTHP orders fit best.
  *
- * Each bit in cc->mthp_present_ptes represents a single occupied (!none/zero)
- * page. We start at the PMD order and check if it is eligible for collapse;
+ * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse
+ * source. We start at the PMD order and check if it is eligible for collapse;
  * if not, we check the left and right halves of the PTE page table we are
  * examining at a lower order.
  *
- * For each of these, we determine how many PTE entries are occupied in the
- * range of PTE entries we propose to collapse, then we compare this to a
- * threshold number of PTE entries which would need to be occupied for a
- * collapse to be permitted at that order (accounting for max_ptes_none).
+ * For each of these, we count the eligible PTEs in the range we propose to
+ * collapse, then we compare this to the number of eligible PTEs the range
+ * would need for a collapse to be permitted at that order (accounting for
+ * max_ptes_none).
  *
  * If a collapse is permitted, we attempt to collapse the PTE range into a
  * mTHP.
@@ -1499,7 +1499,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 		unsigned long address, int referenced, int unmapped,
 		struct collapse_control *cc, unsigned long enabled_orders)
 {
-	unsigned int nr_occupied_ptes, nr_ptes, max_ptes_none;
+	unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none;
 	enum scan_result last_result = SCAN_FAIL;
 	int collapsed = 0;
 	bool alloc_failed = false;
@@ -1514,18 +1514,18 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 			goto next_order;
 
 		max_ptes_none = collapse_max_ptes_none(cc, NULL, order);
-		nr_occupied_ptes = bitmap_weight_from(cc->mthp_present_ptes, offset,
+		nr_eligible_ptes = bitmap_weight_from(cc->eligible_ptes, offset,
 						      offset + nr_ptes);
 
 		/*
 		 * Swap PTEs accepted during the scan are counted in @unmapped,
-		 * not in the present-PTE bitmap. Account them for the PMD-order
+		 * not in cc->eligible_ptes. Account them for the PMD-order
 		 * candidate.
 		 */
 		if (is_pmd_order(order))
-			nr_occupied_ptes += unmapped;
+			nr_eligible_ptes += unmapped;
 
-		if (nr_occupied_ptes >= nr_ptes - max_ptes_none) {
+		if (nr_eligible_ptes >= nr_ptes - max_ptes_none) {
 			enum scan_result ret;
 
 			collapse_address = address + offset * PAGE_SIZE;
@@ -1731,8 +1731,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 			}
 		}
 
-		/* Set bit for occupied pages */
-		__set_bit(i, cc->mthp_present_ptes);
+		/* The scan accepted this PTE as a collapse source */
+		__set_bit(i, cc->eligible_ptes);
 		/*
 		 * Record which node the original page is from and save this
 		 * information to cc->node_load[].
Signature · 1 line
2.54.0
author reviewer
#7 lore raw

PATCH v5 4/9 mm/collapse: add collapse.h for the collapse interface

In reply to Kiryl Shutsemau (Meta)

khugepaged.c holds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins.

Start drawing that line.

Add mm/collapse.h for what the two sides have to agree on:

- enum scan_result - what the engine hands back;
- struct collapse_control - the state a request carries.

And two constants move with them:

- KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT;
- KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER.

Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix.

No functional change.

---
 MAINTAINERS     |  1 +
 mm/collapse.h   | 64 +++++++++++++++++++++++++++++++++++++++
 mm/khugepaged.c | 79 ++++++++-----------------------------------------
 3 files changed, 78 insertions(+), 66 deletions(-)
 create mode 100644 mm/collapse.h
MAINTAINERS +1 -0
index 35262aa3e67e..36315972156b 100644
@@ -17434,6 +17434,7 @@ F:	Documentation/admin-guide/mm/transhuge.rst
 F:	include/linux/huge_mm.h
 F:	include/linux/khugepaged.h
 F:	include/trace/events/huge_memory.h
+F:	mm/collapse.h
 F:	mm/huge_memory.c
 F:	mm/khugepaged.c
 F:	mm/mm_slot.h
mm/collapse.h new file +64 -0

64 lines not on this page · raw message

mm/khugepaged.c +13 -66

150 lines not on this page · raw message

Signature · 1 line
2.54.0
author reviewer
#9 lore raw

PATCH v5 5/9 mm/collapse: state what a collapse may do in the policy

In reply to Kiryl Shutsemau (Meta)

Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle:

- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is left alone;
- whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying.

None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out.

Add struct collapse_policy for the caller to fill: khugepaged from its own settings, MADV_COLLAPSE from the fact that a user asked explicitly. Every test becomes a read of a field, and cc->is_khugepaged goes, having no reader left.

The PTE limits come as two sets, one for a PMD-sized window and one for anything smaller, so that the helpers pick a set for the order and read it. khugepaged takes no swapped-out or shared PTE into a sub-PMD window, and empty PTEs only when the knob says all or nothing; the warning for a knob value in between moves to where khugepaged fills its policy. The fields only one side reads say which: anon_ or file_. David Hildenbrand asked for both.

khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then treated alike.

collapse_file() also drops a NULL check on the collapse_control. It has one call site, reached only from collapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead.

---
 mm/collapse.h   |  41 +++++++++++++-
 mm/khugepaged.c | 147 +++++++++++++++++++++++++-----------------------
 2 files changed, 118 insertions(+), 70 deletions(-)
mm/collapse.h +40 -1
index b115034d9018..dcd117071955 100644
@@ -45,8 +45,47 @@ enum scan_result {
 	SCAN_PAGE_DIRTY_OR_WRITEBACK,
 };
 
+/* How many PTEs of a window may be missing, swapped out or shared */
+struct collapse_limits {
+	/* Counted over a PMD-sized window; HPAGE_PMD_NR means "no limit" */
+	unsigned int max_ptes_none;
+	unsigned int max_ptes_swap;
+	unsigned int max_ptes_shared;
+};
+
+/* What a collapse is allowed to do, decided by the caller that asks for it */
+struct collapse_policy {
+	/* Limits for a PMD-sized window */
+	struct collapse_limits pmd;
+
+	/*
+	 * Limits for a smaller window.  Its max_ptes_none is either 0 or
+	 * COLLAPSE_MAX_PTES_LIMIT, the latter meaning all but one PTE of the
+	 * window whatever its order; any other value counts as 0.
+	 */
+	struct collapse_limits sub_pmd;
+
+	/* Leave clean lazyfree folios to reclaim rather than collapse them */
+	bool anon_skip_lazyfree;
+
+	/* Refuse an anonymous range with no sign of use */
+	bool anon_require_referenced;
+
+	/* Map the PMD over a file collapse instead of leaving it to a fault */
+	bool file_install_pmd;
+
+	/* Write dirty pages back and retry once instead of refusing them */
+	bool file_writeback_dirty;
+
+	/* How hard to try for a destination folio */
+	gfp_t gfp;
+
+	/* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */
+	enum tva_type tva_type;
+};
+
 struct collapse_control {
-	bool is_khugepaged;
+	struct collapse_policy policy;
 
 	/* Num pages scanned per node */
 	u32 node_load[MAX_NUMNODES];
mm/khugepaged.c +78 -69

264 lines not on this page · raw message

Signature · 1 line
2.54.0
author reviewer
#11 lore raw

PATCH v5 6/9 mm/collapse: drop the collapse_possible() wrapper

In reply to Kiryl Shutsemau (Meta)

collapse_possible() only forwards to collapse_possible_orders() and turns its mask into a bool. Its three callers can test the mask themselves.

No functional change.

---
 mm/khugepaged.c | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)
mm/khugepaged.c +5 -10
index 667216ae15dd..cc69281c542f 100644
@@ -494,17 +494,11 @@ static unsigned long collapse_possible_orders(struct vm_area_struct *vma,
 	return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders);
 }
 
-static bool collapse_possible(struct vm_area_struct *vma,
-		vm_flags_t vm_flags, enum tva_type tva_flags)
-{
-	return collapse_possible_orders(vma, vm_flags, tva_flags);
-}
-
 void khugepaged_enter_vma(struct vm_area_struct *vma,
 			  vm_flags_t vm_flags)
 {
-	if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled()
-	    && collapse_possible(vma, vm_flags, TVA_KHUGEPAGED))
+	if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() &&
+	    collapse_possible_orders(vma, vm_flags, TVA_KHUGEPAGED))
 		__khugepaged_enter(vma->vm_mm);
 }
 
@@ -2855,7 +2849,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
 			cc->progress++;
 			break;
 		}
-		if (!collapse_possible(vma, vma->vm_flags, TVA_KHUGEPAGED)) {
+		if (!collapse_possible_orders(vma, vma->vm_flags,
+					      TVA_KHUGEPAGED)) {
 			cc->progress++;
 			continue;
 		}
@@ -3167,7 +3162,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	BUG_ON(vma->vm_start > start);
 	BUG_ON(vma->vm_end < end);
 
-	if (!collapse_possible(vma, vma->vm_flags, TVA_FORCED_COLLAPSE))
+	if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE))
 		return -EINVAL;
 
 	hstart = ALIGN(start, HPAGE_PMD_SIZE);
Signature · 1 line
2.54.0
author reviewer
#13 lore raw

PATCH v5 7/9 mm/collapse: name the per-table scan reset for what it resets

In reply to Kiryl Shutsemau (Meta)

collapse_control_init_scan() resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing.

Call it collapse_scan_reset().

Preparation for giving a control a real init, run once for a whole series of scans. Two names a word apart would then stand for two different jobs.

No functional change.

---
 mm/khugepaged.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/khugepaged.c +3 -3
index cc69281c542f..60ec7e80d554 100644
@@ -532,7 +532,7 @@ void __khugepaged_exit(struct mm_struct *mm)
 	}
 }
 
-static void collapse_control_init_scan(struct collapse_control *cc)
+static void collapse_scan_reset(struct collapse_control *cc)
 {
 	memset(cc->node_load, 0, sizeof(cc->node_load));
 	nodes_clear(cc->alloc_nmask);
@@ -1577,7 +1577,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		goto out;
 	}
 
-	collapse_control_init_scan(cc);
+	collapse_scan_reset(cc);
 
 	enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags);
 
@@ -2655,7 +2655,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
 
 	present = 0;
 	swap = 0;
-	collapse_control_init_scan(cc);
+	collapse_scan_reset(cc);
 	rcu_read_lock();
 	xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) {
 		if (xas_retry(&xas, folio))
Signature · 1 line
2.54.0
author reviewer
#15 lore raw

PATCH v5 8/9 mm/collapse: call collapse_file() from collapse_single_pmd()

In reply to Kiryl Shutsemau (Meta)

collapse_scan_file() reads the page cache to decide whether a table is worth collapsing and, when it is, calls collapse_file() itself. The caller cannot get between the decision and the collapse.

Move the collapse_file() call up into collapse_single_pmd(), so the scan stops at the decision.

Two things change with it. The writeback retry re-runs collapse_file() alone instead of rescanning first; collapse_file() repeats the scan's checks under the page cache lock anyway. And mm_khugepaged_scan_file fires before the collapse, so for an accepted table its status reads SCAN_SUCCEED; what the collapse made of the table is for mm_khugepaged_collapse_file to report.

Preparation for splitting a collapse into a scan under mmap_lock and a run without it. The file scan has to stop where the anonymous one will.

---
 mm/khugepaged.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
mm/khugepaged.c +13 -8
index 60ec7e80d554..89a4c3f5a91c 100644
@@ -2737,13 +2737,9 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
 	else
 		cc->progress += HPAGE_PMD_NR;
 
-	if (result == SCAN_SUCCEED) {
-		if (present < HPAGE_PMD_NR - max_ptes_none) {
-			result = SCAN_EXCEED_NONE_PTE;
-			count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
-		} else {
-			result = collapse_file(mm, addr, file, start, cc);
-		}
+	if (result == SCAN_SUCCEED && present < HPAGE_PMD_NR - max_ptes_none) {
+		result = SCAN_EXCEED_NONE_PTE;
+		count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
 	}
 
 	trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result);
@@ -2774,8 +2770,16 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 
 	mmap_read_unlock(mm);
 	*lock_dropped = true;
-retry:
+
+	/*
+	 * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds
+	 * the PMD folio, and only the PTE table is left to retract.
+	 */
 	result = collapse_scan_file(mm, addr, file, pgoff, cc);
+	if (result != SCAN_SUCCEED)
+		goto put;
+retry:
+	result = collapse_file(mm, addr, file, pgoff, cc);
 
 	/* Dirty pages are worth a writeback and one more try, if asked for */
 	if (cc->policy.file_writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
@@ -2787,6 +2791,7 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 		triggered_wb = true;
 		goto retry;
 	}
+put:
 	fput(file);
 
 	if (result == SCAN_PTE_MAPPED_HUGEPAGE) {
Signature · 1 line
2.54.0
author reviewer
#18 lore raw

PATCH v5 9/9 mm/collapse: separate scanning a PTE table from collapsing it

In reply to Kiryl Shutsemau (Meta)

A collapse is two jobs. One reads a PTE table under mmap_lock and decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first.

collapse_single_pmd() did both, so the boundary between them was somewhere in the middle of a function.

Give each half its own function:

- collapse_scan_pmd() scans one table. It picks the side:
collapse_scan_anon(), the anonymous scan that used to carry the collapse_scan_pmd() name, or collapse_scan_file().

- collapse_pmd() does the collapse the scan asked for, and is handed what the scan returned. SCAN_SUCCEED means there is something to collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache already holds the PMD folio and only the PTE table is left to retract. Anything else is why there is nothing to do. It calls collapse_anon(), formerly mthp_collapse(), or collapse_file().

For now collapse_single_pmd() stays, as the two of them with the mmap_lock drop in between, so its callers see what they saw before. collapse_control_init() sets a control up before its first scan.

What the scan found and the collapse needs travels in collapse_control. For an anonymous table that is the orders and the referenced and swapped-out counts. collapse_anon() and collapse_huge_page() now read them from there instead of taking them as arguments.

For a file it is the file itself and the offset in it. A file collapse works on the page cache and never sees a VMA, so the scan takes the reference while it still has one and the collapse gives it back.

The file scan moves under mmap_lock with the anonymous one, where before the lock was given up first. The lock is now held over the page cache walk, an RCU walk over one table's worth of slots with no PTL, and taken fewer times. collapse_scan_mm_slot() ends its walk whenever the lock was dropped, so a refused file table used to cost khugepaged an unlock, a trip back through khugepaged_do_scan(), a relock and a VMA lookup. Now only a table that goes on to be collapsed does.

---
 mm/collapse.h   |  18 ++++++
 mm/khugepaged.c | 151 +++++++++++++++++++++++++++++++-----------------
 2 files changed, 117 insertions(+), 52 deletions(-)
mm/collapse.h +18 -0
index dcd117071955..759ec9cd4541 100644
@@ -98,6 +98,24 @@ struct collapse_control {
 
 	/* Each bit marks a PTE the scan accepted as a collapse source */
 	DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
+
+	/*
+	 * What a scan found and the collapse after it needs.  Live only between
+	 * the two, and read by nobody else.
+	 */
+
+	/* Anonymous only: orders to try, referenced and swap PTE counts */
+	unsigned long scan_orders;
+	int scan_referenced;
+	int scan_unmapped;
+
+	/*
+	 * File only.  A file collapse works on the page cache and never sees a
+	 * VMA, so the scan takes a reference to the file while it still has
+	 * one, and the collapse gives it back.
+	 */
+	struct file *scan_file;
+	pgoff_t scan_pgoff;
 };
 
 #endif	/* __MM_COLLAPSE_H */
mm/khugepaged.c +99 -52

288 lines not on this page · raw message

Signature · 1 line
2.54.0
maintainer
#21 lore raw
In reply to Kiryl Shutsemau (Meta)
Quoted 7 lines · On Wed, 14 Oct 2026 18:11:33 +0100 "Kiryl Shutsemau (Meta)" <[email protected]>…

There is no line between the collapse engine and the callers that ask for a collapse. khugepaged.c holds both, and they reach into each other.

- Sixteen tests through the collapse path read cc->is_khugepaged to work out what they are allowed to do, when every one of those decisions was made by the caller before it asked.

Quoted 8 lines

- collapse_single_pmd() does both halves of a collapse behind one call. Where it gives up mmap_lock between them is buried in the anonymous scan on one side and ahead of the file scan on the other.

Start drawing the line. The caller states what it allows in a policy, and the collapse code reads the policy instead of asking who called. The scan of a PTE table and the collapse that follows get a function each, with the mmap_lock drop between them in one place.

Thanks, updated.

Quoted 16 lines

Changes since v4
================

https://lore.kernel.org/all/[email protected]/

- Patches 10-13 dropped.

- Patch 7: the changelog no longer mentions a release for the control, which v4 dropped.

- Patch 9: collapse_scan_anon(), collapse_pmd() and collapse_anon() replace collapse_scan_anon_pmd(), collapse_run_pmd() and mthp_collapse() (David). The scan_* fields in collapse_control are commented as anonymous-only and file-only (Zi). Comments that named collapse_scan_pmd() for the anonymous scan follow the rename.

Quoted 3 lines

- Acked-by from David Hildenbrand on 5 and 8; Reviewed-by from Zi Yan on 5 and 8 and from Baolin Wang on 5; Tested-by from Baolin Wang on 5; Acked-by from Zi Yan on 9.

Here are the v3->v4 changes for the nine remaining patches:

 mm/collapse.h   |   16 ++++++++++------
 mm/khugepaged.c |   28 ++++++++++++++--------------
 2 files changed, 24 insertions(+), 20 deletions(-)
mm/collapse.h renamed from mm/collapse.h~b +10 -6
@@ -100,16 +100,20 @@ struct collapse_control {
 	DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
 
 	/*
-	 * What a scan found and the run after it needs.  Live only between the
-	 * two, and read by nobody else.
-	 *
-	 * The file side takes a reference while it still has the VMA, since a
-	 * file collapse works on the page cache and never sees one; the run is
-	 * what gives it back.
+	 * What a scan found and the collapse after it needs.  Live only between
+	 * the two, and read by nobody else.
 	 */
+
+	/* Anonymous only: orders to try, referenced and swap PTE counts */
 	unsigned long scan_orders;
 	int scan_referenced;
 	int scan_unmapped;
+
+	/*
+	 * File only.  A file collapse works on the page cache and never sees a
+	 * VMA, so the scan takes a reference to the file while it still has
+	 * one, and the collapse gives it back.
+	 */
 	struct file *scan_file;
 	pgoff_t scan_pgoff;
 };
mm/khugepaged.c renamed from mm/khugepaged.c~b +14 -14
@@ -646,7 +646,7 @@ static enum scan_result __collapse_huge_
 			goto out;
 		}
 
-		/* See collapse_scan_pmd(). */
+		/* See collapse_scan_anon(). */
 		if (folio_maybe_mapped_shared(folio)) {
 			/*
 			 * TODO: Support shared pages without leading to further
@@ -1099,7 +1099,7 @@ static enum scan_result check_pmd_still_
 
 /*
  * Bring missing pages in from swap, to complete THP collapse.
- * Only done if collapse_scan_pmd() believes it is worthwhile.
+ * Only done if collapse_scan_anon() believes it is worthwhile.
  *
  * For mTHP orders the function bails on the first swap entry, because
  * faulting pages back in during collapse could re-populate PTEs that
@@ -1460,8 +1460,8 @@ static unsigned int max_order_from_offse
 }
 
 /*
- * mthp_collapse() consumes the bitmap that is generated during
- * collapse_scan_pmd() to determine what regions and mTHP orders fit best.
+ * collapse_anon() consumes the bitmap that is generated during
+ * collapse_scan_anon() to determine what regions and mTHP orders fit best.
  *
  * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse
  * source. We start at the PMD order and check if it is eligible for collapse;
@@ -1476,8 +1476,8 @@ static unsigned int max_order_from_offse
  * If a collapse is permitted, we attempt to collapse the PTE range into a
  * mTHP.
  */
-static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long address,
-		struct collapse_control *cc)
+static enum scan_result collapse_anon(struct mm_struct *mm,
+		unsigned long address, struct collapse_control *cc)
 {
 	unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none;
 	enum scan_result last_result = SCAN_FAIL;
@@ -1576,7 +1576,7 @@ done:
 	return last_result;
 }
 
-static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,
+static enum scan_result collapse_scan_anon(struct vm_area_struct *vma,
 		unsigned long start_addr, struct collapse_control *cc)
 {
 	const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
@@ -1612,7 +1612,7 @@ static enum scan_result collapse_scan_an
 	/*
 	 * If PMD is the only enabled order, enforce max_ptes_none, otherwise
 	 * scan all pages to populate the bitmap for mTHP collapse. The bitmap
-	 * is then checked again in mthp_collapse() for each attempted order.
+	 * is then checked again in collapse_anon() for each attempted order.
 	 */
 	if (enabled_orders != BIT(HPAGE_PMD_ORDER))
 		max_ptes_none = COLLAPSE_MAX_PTES_LIMIT;
@@ -2785,20 +2785,20 @@ static enum scan_result collapse_scan_pm
 	pgoff_t pgoff;
 
 	mmap_assert_locked(vma->vm_mm);
-	/* Whatever the last scan found has to have been run by now */
+	/* A scan that found work has to have been followed by collapse_pmd() */
 	if (WARN_ON_ONCE(cc->scan_file)) {
 		fput(cc->scan_file);
 		cc->scan_file = NULL;
 	}
 
 	if (vma_is_anonymous(vma))
-		return collapse_scan_anon_pmd(vma, addr, cc);
+		return collapse_scan_anon(vma, addr, cc);
 
 	pgoff = linear_page_index(vma, addr);
 	result = collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc);
 	/*
 	 * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds
-	 * the PMD folio, and retracting the PTE table is the run's job.
+	 * the PMD folio, and collapse_pmd() still has to retract the PTE table.
 	 */
 	if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
 		return result;
@@ -2812,7 +2812,7 @@ static enum scan_result collapse_scan_pm
 	return result;
 }
 
-static enum scan_result collapse_run_pmd(struct mm_struct *mm,
+static enum scan_result collapse_pmd(struct mm_struct *mm,
 		unsigned long addr, enum scan_result result,
 		struct collapse_control *cc)
 {
@@ -2821,7 +2821,7 @@ static enum scan_result collapse_run_pmd
 	pgoff_t pgoff;
 
 	if (!file)
-		return mthp_collapse(mm, addr, cc);
+		return collapse_anon(mm, addr, cc);
 
 	cc->scan_file = NULL;
 	pgoff = cc->scan_pgoff;
@@ -2882,7 +2882,7 @@ static enum scan_result collapse_single_
 	mmap_read_unlock(mm);
 	*lock_dropped = true;
 
-	return collapse_run_pmd(mm, addr, result, cc);
+	return collapse_pmd(mm, addr, result, cc);
 }
 
 static void collapse_scan_mm_slot(unsigned int progress_max,

_

reviewer Reviewed-by on v5 1/9
#2 lore raw

Re: [PATCH v5 1/9] mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()

In reply to Kiryl Shutsemau (Meta)
Quoted 10 lines · Kiryl Shutsemau (Meta)

madvise_collapse() holds an mmgrab() reference across its work. It is redundant. Every caller already holds mm_users:

- madvise(2) works on current->mm, which lives as long as the task is in the syscall;
- process_madvise(2) reaches a remote mm through mm_access(), which takes an mm_users reference and holds it until the syscall returns;
- io_uring passes current->mm;
- DAMON takes one with get_task_mm() and drops it after the call.

Quoted 8 lines

Drop the mmgrab()/mmdrop() pair.

---

reviewer Reviewed-by on v5 2/9
#4 lore raw

Re: [PATCH v5 2/9] mm/khugepaged: count collapses where khugepaged makes them

In reply to Kiryl Shutsemau (Meta)
Quoted 7 lines · Kiryl Shutsemau (Meta)

collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and tests cc->is_khugepaged to know whether it should: the counter belongs to the daemon, and MADV_COLLAPSE must not touch it.

The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called.

Quoted 6 lines

---

reviewer Reviewed-by on v5 3/9
#6 lore raw

Re: [PATCH v5 3/9] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes

In reply to Kiryl Shutsemau (Meta)
Quoted 10 lines · Kiryl Shutsemau (Meta)

The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source.

mthp_collapse() then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either.

Name it for what a set bit means, and update the comments that named it.

Quoted 8 lines

No functional change.

---

reviewer Reviewed-by on v5 4/9
#8 lore raw

Re: [PATCH v5 4/9] mm/collapse: add collapse.h for the collapse interface

In reply to Kiryl Shutsemau (Meta)
Quoted 20 lines · Kiryl Shutsemau (Meta)

khugepaged.c holds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins.

Start drawing that line.

Add mm/collapse.h for what the two sides have to agree on:

- enum scan_result - what the engine hands back;
- struct collapse_control - the state a request carries.

And two constants move with them:

- KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT;
- KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER.

Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix.

Quoted 8 lines

No functional change.

---

reviewer Reviewed-by on v5 5/9
#10 lore raw

Re: [PATCH v5 5/9] mm/collapse: state what a collapse may do in the policy

In reply to Kiryl Shutsemau (Meta)
Quoted 39 lines · Kiryl Shutsemau (Meta)

Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle:

- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is left alone;
- whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying.

None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out.

Add struct collapse_policy for the caller to fill: khugepaged from its own settings, MADV_COLLAPSE from the fact that a user asked explicitly. Every test becomes a read of a field, and cc->is_khugepaged goes, having no reader left.

The PTE limits come as two sets, one for a PMD-sized window and one for anything smaller, so that the helpers pick a set for the order and read it. khugepaged takes no swapped-out or shared PTE into a sub-PMD window, and empty PTEs only when the knob says all or nothing; the warning for a knob value in between moves to where khugepaged fills its policy. The fields only one side reads say which: anon_ or file_. David Hildenbrand asked for both.

khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then treated alike.

collapse_file() also drops a NULL check on the collapse_control. It has one call site, reached only from collapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead.

Quoted 7 lines

---

reviewer Reviewed-by on v5 6/9
#12 lore raw

Re: [PATCH v5 6/9] mm/collapse: drop the collapse_possible() wrapper

In reply to Kiryl Shutsemau (Meta)
Quoted 11 lines · Kiryl Shutsemau (Meta)

collapse_possible() only forwards to collapse_possible_orders() and turns its mask into a bool. Its three callers can test the mask themselves.

No functional change.

---

reviewer Reviewed-by on v5 7/9
#14 lore raw

Re: [PATCH v5 7/9] mm/collapse: name the per-table scan reset for what it resets

In reply to Kiryl Shutsemau (Meta)
Quoted 10 lines · Kiryl Shutsemau (Meta)

collapse_control_init_scan() resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing.

Call it collapse_scan_reset().

Preparation for giving a control a real init, run once for a whole series of scans. Two names a word apart would then stand for two different jobs.

Quoted 8 lines

No functional change.

---

reviewer Reviewed-by on v5 8/9
#16 lore raw

Re: [PATCH v5 8/9] mm/collapse: call collapse_file() from collapse_single_pmd()

In reply to Kiryl Shutsemau (Meta)
Quoted 14 lines · Kiryl Shutsemau (Meta)

collapse_scan_file() reads the page cache to decide whether a table is worth collapsing and, when it is, calls collapse_file() itself. The caller cannot get between the decision and the collapse.

Move the collapse_file() call up into collapse_single_pmd(), so the scan stops at the decision.

Two things change with it. The writeback retry re-runs collapse_file() alone instead of rescanning first; collapse_file() repeats the scan's checks under the page cache lock anyway. And mm_khugepaged_scan_file fires before the collapse, so for an accepted table its status reads SCAN_SUCCEED; what the collapse made of the table is for mm_khugepaged_collapse_file to report.

Quoted 8 lines

Preparation for splitting a collapse into a scan under mmap_lock and a run without it. The file scan has to stop where the anonymous one will.

---

reviewer Reviewed-by on v5 9/9
#19 lore raw

Re: [PATCH v5 9/9] mm/collapse: separate scanning a PTE table from collapsing it

In reply to Kiryl Shutsemau (Meta)
Quoted 41 lines · Kiryl Shutsemau (Meta)

A collapse is two jobs. One reads a PTE table under mmap_lock and decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first.

collapse_single_pmd() did both, so the boundary between them was somewhere in the middle of a function.

Give each half its own function:

- collapse_scan_pmd() scans one table. It picks the side:
collapse_scan_anon(), the anonymous scan that used to carry the collapse_scan_pmd() name, or collapse_scan_file().

- collapse_pmd() does the collapse the scan asked for, and is handed what the scan returned. SCAN_SUCCEED means there is something to collapse. SCAN_PTE_MAPPED_HUGEPAGE means the page cache already holds the PMD folio and only the PTE table is left to retract. Anything else is why there is nothing to do. It calls collapse_anon(), formerly mthp_collapse(), or collapse_file().

For now collapse_single_pmd() stays, as the two of them with the mmap_lock drop in between, so its callers see what they saw before. collapse_control_init() sets a control up before its first scan.

What the scan found and the collapse needs travels in collapse_control. For an anonymous table that is the orders and the referenced and swapped-out counts. collapse_anon() and collapse_huge_page() now read them from there instead of taking them as arguments.

For a file it is the file itself and the offset in it. A file collapse works on the page cache and never sees a VMA, so the scan takes the reference while it still has one and the collapse gives it back.

The file scan moves under mmap_lock with the anonymous one, where before the lock was given up first. The lock is now held over the page cache walk, an RCU walk over one table's worth of slots with no PTL, and taken fewer times. collapse_scan_mm_slot() ends its walk whenever the lock was dropped, so a refused file table used to cost khugepaged an unlock, a trip back through khugepaged_do_scan(), a relock and a VMA lookup. Now only a table that goes on to be collapsed does.

Quoted 4 lines

---

reviewer Reviewed-by on v5 8/9
#17 lore raw

Re: [PATCH v5 8/9] mm/collapse: call collapse_file() from collapse_single_pmd()

In reply to Kiryl Shutsemau (Meta)
Quoted 14 lines · Kiryl Shutsemau (Meta)

collapse_scan_file() reads the page cache to decide whether a table is worth collapsing and, when it is, calls collapse_file() itself. The caller cannot get between the decision and the collapse.

Move the collapse_file() call up into collapse_single_pmd(), so the scan stops at the decision.

Two things change with it. The writeback retry re-runs collapse_file() alone instead of rescanning first; collapse_file() repeats the scan's checks under the page cache lock anyway. And mm_khugepaged_scan_file fires before the collapse, so for an accepted table its status reads SCAN_SUCCEED; what the collapse made of the table is for mm_khugepaged_collapse_file to report.

Quoted 8 lines

Preparation for splitting a collapse into a scan under mmap_lock and a run without it. The file scan has to stop where the anonymous one will.

---

LGTM.

maintainer Acked-by on v5 9/9
#20 lore raw

Re: [PATCH v5 9/9] mm/collapse: separate scanning a PTE table from collapsing it

In reply to Kiryl Shutsemau (Meta)
Quoted 9 lines

/*
* Bring missing pages in from swap, to complete THP collapse.
- * Only done if collapse_scan_pmd() believes it is worthwhile.
+ * Only done if collapse_scan_anon() believes it is worthwhile. *
* For mTHP orders the function bails on the first swap entry, because
* faulting pages back in during collapse could re-populate PTEs that

mm/khugepaged.c static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru · line 1239

Quoted 5 lines
mm/khugepaged.c +0 -2
@@ -1235,8 +1235,8 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
  * while allocating a THP, as that could trigger direct reclaim/compaction.
  * Note that the VMA must be rechecked after grabbing the mmap_lock again.
  */
-static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long start_addr,
-		int referenced, int unmapped, struct collapse_control *cc,

Still some sub-optimally named functions around here, but we can tackle that later.

Signature · 3 lines
Cheers,

David