mm/collapse: separate a collapse from its callers
Subsystems (3)
cover mm/collapse: separate a collapse from its callers
From: "Kiryl Shutsemau (Meta)" <[email protected]>
[ This is the first of the cleanups I said I would front-load ]
There is no line between the collapse engine and the callers that ask for a collapse. khugepaged.c holds both, and they reach into each other.
- Sixteen tests through the collapse path read cc->is_khugepaged to work out what they are allowed to do, when every one of those decisions was made by the caller before it asked.
- collapse_single_pmd() does both halves of a collapse behind one call and drops mmap_lock somewhere in the middle. Which of its paths dropped it is not something a caller can see, so it hands back a bool and the caller keeps track.
- MADV_COLLAPSE's implementation -- the walk over the user's range, the per-PMD loop, the errno translation -- sits in khugepaged.c, which is the daemon's file.
So: draw the line. State what a caller allows in a policy, split the call in two with the lock as the boundary, and move the syscall to madvise.c. What the engine offers is then four calls, with the lock state written down against each, and a policy the caller fills for itself:
collapse_control_init(cc) once, before the first table
collapse_policy_*(&cc->policy) what this caller allows
collapse_scan_pmd(vma, addr, ...) per table, under mmap_lock
collapse_run_pmd(mm, addr, cc) when a scan found work, no mmap_lock
collapse_control_release(cc) once, when done
The engine stays in khugepaged.c for now; what changes is that it has an interface, and that neither half has to ask about the other. madvise.c gains the operation it should have had all along.
Patches
=======
The first three stand alone and can be taken separately:
1 drop the mmgrab() MADV_COLLAPSE has held since 7d8faaf15545 2 count collapses in khugepaged's own walk, where the daemon's
bookkeeping belongs
3 rename cc->mthp_present_ptes to eligible_ptes, which is what a set
bit means
Then the interface, in order:
4 add collapse.h, and move enum scan_result, struct collapse_control
and the two constants into it
5 struct collapse_policy, filled by the caller; the is_khugepaged
tests become field reads, and the flag goes
6 drop collapse_possible(), a wrapper that only turns a mask into a bool 7 collapse_control_init_scan() is a per-table reset, so name it
collapse_scan_reset()
8 give the scan and the collapse a function each: collapse_scan_pmd()
and collapse_run_pmd()
9 open-code the entry point that joined them, so each caller owns the
lock across the boundary and the bool goes
10 work out the orders a VMA allows once per VMA, not once per table 11 declare the four calls in collapse.h, with the lock rules 12 MADV_COLLAPSE moves to madvise.c
Behaviour
=========
Nothing here changes what gets collapsed. Three things a reader should not have to find in the diff:
- Patch 5: khugepaged fills its policy once per scan pass, so the max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than once per table. A table scanned early in a pass and one scanned late are then judged alike, where before a knob written mid-pass split them.
- Patch 8: mm_khugepaged_scan_pmd fires before mm_collapse_huge_page rather than after it, the scan having returned before the collapse runs. Its status field already reads SCAN_SUCCEED for an accepted table, so nothing changes there; what the collapse then made of the table is mm_collapse_huge_page's to report, per order.
- Patch 10: which orders a table is scanned for is sampled once per VMA rather than once per table. It cannot widen what a collapse does -- the order is tested again under the lock the collapse retakes.
The lock itself is given up and taken again at exactly the points it was before; the only difference is that the caller is the one doing it.
selftests/mm khugepaged passes on x86-64 with a KASAN, lockdep and DEBUG_VM config, and every patch builds, CONFIG_TRANSPARENT_HUGEPAGE=n included.
Kiryl Shutsemau (Meta) (12):
mm/khugepaged: drop redundant mm_struct pin in madvise_collapse() mm/khugepaged: count collapses where khugepaged makes them mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes mm/collapse: add collapse.h for the collapse interface mm/collapse: state what a collapse may do in the policy mm/collapse: drop the collapse_possible() wrapper mm/collapse: name the per-table scan reset for what it resets mm/collapse: separate scanning a PTE table from collapsing it mm/collapse: open-code collapse_single_pmd() in its two callers mm/collapse: work out the orders a VMA allows once per VMA mm/collapse: declare the collapse interface in collapse.h mm/collapse: implement MADV_COLLAPSE in madvise.c
MAINTAINERS | 1 + include/linux/huge_mm.h | 9 - mm/collapse.h | 163 ++++++++++++++ mm/khugepaged.c | 485 ++++++++++++++-------------------------- mm/madvise.c | 169 +++++++++++++- 5 files changed, 497 insertions(+), 330 deletions(-) create mode 100644 mm/collapse.h
base-commit: e3fc12b08aadde9cec7b3799ac0e0c9a1aa245c4
Signature · 1 line
2.54.0
PATCH 1/12 mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()
From: "Kiryl Shutsemau (Meta)" <[email protected]>
madvise_collapse() holds an mmgrab() reference across its work. It is redundant. Every caller already holds mm_users:
- madvise(2) works on current->mm, which lives as long as the task is in the syscall;
- process_madvise(2) reaches a remote mm through mm_access(), which takes an mm_users reference and holds it until the syscall returns;
- io_uring passes current->mm;
- DAMON takes one with get_task_mm() and drops it after the call.
Drop the mmgrab()/mmdrop() pair.
--- mm/khugepaged.c | 2 -- 1 file changed, 2 deletions(-)
mm/khugepaged.c +0 -2
@@ -3236,7 +3236,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, cc->is_khugepaged = false; cc->progress =0 ; -mmgrab (mm);lru_add_drain_all ();for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) {
@@ -3292,7 +3291,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, } out_nolock:mmap_assert_locked (mm); -mmdrop (mm);kfree (cc); return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0
Signature · 1 line
2.54.0
PATCH 2/12 mm/khugepaged: count collapses where khugepaged makes them
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and tests cc->is_khugepaged to know whether it should: the counter belongs to the daemon, and MADV_COLLAPSE must not touch it.
The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called.
--- mm/khugepaged.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
mm/khugepaged.c +4 -7
@@ -2822,10 +2822,8 @@ static enum scan_result collapse_single_pmd(unsigned long addr,mmap_assert_locked (mm); -if (vma_is_anonymous (vma)) { - result =collapse_scan_pmd (mm, vma, addr, lock_dropped, cc); - goto end; - } +if (vma_is_anonymous (vma)) +return collapse_scan_pmd (mm, vma, addr, lock_dropped, cc); file =get_file (vma->vm_file); pgoff =linear_page_index (vma, addr);
@@ -2861,9 +2859,6 @@ static enum scan_result collapse_single_pmd(unsigned long addr, result =SCAN_SUCCEED ;mmap_read_unlock (mm); } -end: -if (cc->is_khugepaged && result ==SCAN_SUCCEED ) - ++khugepaged_pages_collapsed;return result; }
@@ -2940,6 +2935,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max, *result =collapse_single_pmd (khugepaged_scan.address, vma, &lock_dropped, cc); +if (*result ==SCAN_SUCCEED ) + khugepaged_pages_collapsed++;/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ;if (lock_dropped)
Signature · 1 line
2.54.0
PATCH 3/12 mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source.
mthp_collapse() then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either.
Name it for what a set bit means, and update the comments that named it.
No functional change.
--- mm/khugepaged.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
mm/khugepaged.c +9 -9
@@ -115,8 +115,8 @@ struct collapse_control {/* nodemask for allocation fallback */ nodemask_t alloc_nmask; -/* Each bit represents a single occupied (!none/zero) page. */ -DECLARE_BITMAP (mthp_present_ptes,MAX_PTRS_PER_PTE ); +/* Each bit marks a PTE the scan accepted as a collapse source */ +DECLARE_BITMAP (eligible_ptes,MAX_PTRS_PER_PTE ); };/**
@@ -627,7 +627,7 @@ static void collapse_control_init_scan(struct collapse_control *cc) {memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); -bitmap_zero (cc->mthp_present_ptes,MAX_PTRS_PER_PTE ); +bitmap_zero (cc->eligible_ptes,MAX_PTRS_PER_PTE ); }static void release_pte_folio (struct folio *folio)
@@ -1482,8 +1482,8 @@ static unsigned int max_order_from_offset(unsigned int offset)* mthp_collapse() consumes the bitmap that is generated during * collapse_scan_pmd() to determine what regions and mTHP orders fit best. * -* Each bit in cc->mthp_present_ptes represents a single occupied (!none/zero) -* page. We start at the PMD order and check if it is eligible for collapse; +* Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse +* source. We start at the PMD order and check if it is eligible for collapse; * if not, we check the left and right halves of the PTE page table we are * examining at a lower order. *
@@ -1514,12 +1514,12 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, goto next_order; max_ptes_none =collapse_max_ptes_none (cc,NULL , order); - nr_occupied_ptes =bitmap_weight_from (cc->mthp_present_ptes, offset, + nr_occupied_ptes =bitmap_weight_from (cc->eligible_ptes, offset, offset + nr_ptes);/* * Swap PTEs accepted during the scan are counted in @unmapped, -* not in the present-PTE bitmap. Account them for the PMD-order +* not in cc->eligible_ptes. Account them for the PMD-order * candidate. */ if (is_pmd_order (order))
@@ -1731,8 +1731,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, } } -/* Set bit for occupied pages */ -__set_bit (i, cc->mthp_present_ptes); +/* The scan accepted this PTE as a collapse source */ +__set_bit (i, cc->eligible_ptes);/* * Record which node the original page is from and save this * information to cc->node_load[].
Signature · 1 line
2.54.0
PATCH 4/12 mm/collapse: add collapse.h for the collapse interface
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
khugepaged.c holds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins.
Start drawing that line.
Add mm/collapse.h for what the two sides have to agree on:
- enum scan_result - what the engine hands back;
- struct collapse_control - the state a request carries.
And two constants move with them:
- KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT;
- KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER.
Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix.
No functional change.
--- MAINTAINERS | 1 + mm/collapse.h | 67 +++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 79 ++++++++----------------------------------------- 3 files changed, 81 insertions(+), 66 deletions(-) create mode 100644 mm/collapse.h
MAINTAINERS +1 -0
@@ -17433,6 +17433,7 @@ F: Documentation/admin-guide/mm/transhuge.rst F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h +F: mm/collapse.h F: mm/huge_memory.c F: mm/khugepaged.c F: mm/mm_slot.h
mm/collapse.h new file +67 -0
67 lines not on this page · raw message
mm/khugepaged.c +13 -66
150 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 5/12 mm/collapse: state what a collapse may do in the policy
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle:
- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is left alone;
- whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying.
None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out.
Add struct collapse_policy for the caller to fill: khugepaged from its own settings, MADV_COLLAPSE from the fact that a user asked explicitly. Every test becomes a read of a field, and cc->is_khugepaged goes, having no reader left.
khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then judged alike.
collapse_file() also drops a NULL check on the collapse_control. It has one call site, reached only from collapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead.
--- mm/collapse.h | 40 ++++++++++++++++- mm/khugepaged.c | 114 ++++++++++++++++++++++++++---------------------- 2 files changed, 102 insertions(+), 52 deletions(-)
mm/collapse.h +39 -1
@@ -48,8 +48,46 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; +/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { +/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ +unsigned int max_ptes_none; +unsigned int max_ptes_swap; +unsigned int max_ptes_shared; + +/* +* Hold a sub-PMD window to a stricter rule than a PMD: no swapped-out +* and no shared PTEs at all, and max_ptes_none as +* collapse_max_ptes_none() scales it. +*/ +bool strict_sub_pmd; + +/* +* Collapse only where it looks worth doing: require some sign the +* range is in use, and leave clean lazyfree folios for reclaim rather +* than collapsing them into a folio that is not lazyfree. +*/ +bool skip_lazyfree; +bool require_referenced; + +/* +* Finish the job rather than leaving it half done for a fault to pick +* up: map the PMD over a file collapse before returning, and write +* dirty pages back and retry once instead of refusing them. Both cost +* latency the caller has to be willing to pay. +*/ +bool install_pmd; +bool writeback_dirty; + +/* How hard to try for a destination folio */ +gfp_t gfp; + +/* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ +enum tva_type tva_type; +}; +struct collapse_control { -bool is_khugepaged; +struct collapse_policy policy;/* Num pages scanned per node */ u32 node_load[MAX_NUMNODES ];
mm/khugepaged.c +63 -51
232 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 6/12 mm/collapse: drop the collapse_possible() wrapper
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_possible() only forwards to collapse_possible_orders() and turns its mask into a bool. Its three callers can test the mask themselves.
No functional change.
--- mm/khugepaged.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-)
mm/khugepaged.c +5 -10
@@ -512,17 +512,11 @@ static unsigned long collapse_possible_orders(struct vm_area_struct *vma,return thp_vma_allowable_orders (vma, vm_flags, tva_flags, orders); } -static bool collapse_possible (struct vm_area_struct *vma, -vm_flags_t vm_flags,enum tva_type tva_flags) -{ -return collapse_possible_orders (vma, vm_flags, tva_flags); -} -void khugepaged_enter_vma (struct vm_area_struct *vma,vm_flags_t vm_flags) { -if (!mm_flags_test (MMF_VM_HUGEPAGE , vma->vm_mm) &&hugepage_enabled () - &&collapse_possible (vma, vm_flags,TVA_KHUGEPAGED )) +if (!mm_flags_test (MMF_VM_HUGEPAGE , vma->vm_mm) &&hugepage_enabled () && +collapse_possible_orders (vma, vm_flags,TVA_KHUGEPAGED ))__khugepaged_enter (vma->vm_mm); }
@@ -2864,7 +2858,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max, cc->progress++;break ; } -if (!collapse_possible (vma, vma->vm_flags,TVA_KHUGEPAGED )) { +if (!collapse_possible_orders (vma, vma->vm_flags, +TVA_KHUGEPAGED )) { cc->progress++;continue ; }
@@ -3177,7 +3172,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,BUG_ON (vma->vm_start > start);BUG_ON (vma->vm_end < end); -if (!collapse_possible (vma, vma->vm_flags,TVA_FORCED_COLLAPSE )) +if (!collapse_possible_orders (vma, vma->vm_flags,TVA_FORCED_COLLAPSE ))return -EINVAL ; hstart =ALIGN (start,HPAGE_PMD_SIZE );
Signature · 1 line
2.54.0
PATCH 7/12 mm/collapse: name the per-table scan reset for what it resets
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_control_init_scan() resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing.
Call it collapse_scan_reset().
Preparation for giving a control a real init and release, run once each for a whole series of scans. Two names a word apart would then stand for two different jobs.
No functional change.
--- mm/khugepaged.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
mm/khugepaged.c +3 -3
@@ -550,7 +550,7 @@ void __khugepaged_exit(struct mm_struct *mm) } } -static void collapse_control_init_scan (struct collapse_control *cc) +static void collapse_scan_reset (struct collapse_control *cc) {memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask);
@@ -1579,7 +1579,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, goto out; } -collapse_control_init_scan (cc); +collapse_scan_reset (cc); enabled_orders =collapse_possible_orders (vma, vma->vm_flags, tva_flags);
@@ -2660,7 +2660,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, present =0 ; swap =0 ; -collapse_control_init_scan (cc); +collapse_scan_reset (cc);rcu_read_lock ();xas_for_each (&xas, folio, start +HPAGE_PMD_NR -1 ) {if (xas_retry (&xas, folio))
Signature · 1 line
2.54.0
PATCH 8/12 mm/collapse: separate scanning a PTE table from collapsing it
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
A collapse is two jobs. One reads a PTE table under mmap_lock and decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first.
collapse_single_pmd() did both, so the boundary between them was somewhere in the middle of a function.
Give each half its own function:
- collapse_scan_pmd() scans one table and only reads. The anonymous scan that used to carry that name keeps its body as collapse_scan_anon_pmd(), and collapse_scan_pmd() is now the entry that picks the anonymous or the file side.
- collapse_run_pmd() does the collapse the scan asked for. SCAN_SUCCEED from the scan means there is something to run; anything else is why there is not.
collapse_single_pmd() is now the two of them with the mmap_lock drop in between, so its callers see what they saw before.
Scan results (beyond SCAN_SUCCEED) communicated via collapse_control
structure: the orders, the referenced and swapped-out counts, and for a file the file itself and the offset in it.
A file collapse works on the page cache and never sees a VMA. The scan takes the file reference while it still has VMA and the run unpins it when it is done.
Tracing changes with it. mm_khugepaged_scan_pmd now fires before mm_collapse_huge_page instead of after it. Its status field already reads SCAN_SUCCEED for an accepted table, so what the collapse then made of that table is mm_collapse_huge_page's to report, per order.
The two calls to that tracepoint become one. They differed in what the collapse between them changed; with the collapse no longer here, both carry the same arguments. failed_pfn is set only where a PTE was refused, so it is -1 exactly when the result is SCAN_SUCCEED.
--- mm/collapse.h | 14 ++++++ mm/khugepaged.c | 121 ++++++++++++++++++++++++++++++++++-------------- 2 files changed, 100 insertions(+), 35 deletions(-)
mm/collapse.h +14 -0
@@ -100,6 +100,20 @@ struct collapse_control {/* Each bit marks a PTE the scan accepted as a collapse source */ DECLARE_BITMAP (eligible_ptes,MAX_PTRS_PER_PTE ); + +/* +* What a scan found and the run after it needs. Live only between the +* two, and read by nobody else. +* +* The file side takes a reference while it still has the VMA, since a +* file collapse works on the page cache and never sees one; the run is +* what gives it back. +*/ +unsigned long scan_orders; +int scan_referenced; +int scan_unmapped; +struct file *scan_file; + pgoff_t scan_pgoff; };#endif /* __MM_COLLAPSE_H */
mm/khugepaged.c +86 -35
185 lines not on this page · raw message
Signature · 1 line
2.54.0
PATCH 9/12 mm/collapse: open-code collapse_single_pmd() in its two callers
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_scan_pmd() and collapse_run_pmd() each have a clear locking contract. The scan is called with mmap_lock held for reading and returns with it still held. The collapse is called without it.
collapse_single_pmd() kept that boundary inside itself. It dropped the lock on some paths and not others, and reported which by way of a bool its callers had to carry along and then act on.
Open-code it in the two callers. Each scans under the lock it already holds and, on SCAN_SUCCEED, gives the lock up before running the collapse. khugepaged's lock_dropped and madvise_collapse()'s mmap_unlocked both go:
the code dropping the lock is now the code that wanted to know.
khugepaged's walk carries on to the next table while the scan keeps refusing, and ends once a collapse has taken the lock from under it. madvise_collapse() re-finds its VMA after a collapse, which it did before, and now uses a NULL vma to say that it has to. It still reports the drop to its own caller, from the line that does it.
The lock is given up and taken again at the same points as before. No functional change.
--- mm/khugepaged.c | 102 +++++++++++++++++++++++------------------------- 1 file changed, 49 insertions(+), 53 deletions(-)
mm/khugepaged.c +49 -53
@@ -2838,28 +2838,6 @@ static enum scan_result collapse_run_pmd(struct mm_struct *mm,return result; } -/* -* Try to collapse a single PMD starting at a PMD aligned addr, and return -* the results. -*/ -static enum scan_result collapse_single_pmd (unsigned long addr, -struct vm_area_struct *vma,bool *lock_dropped, -struct collapse_control *cc) -{ -struct mm_struct *mm = vma->vm_mm; -enum scan_result result; - - result =collapse_scan_pmd (vma, addr, cc); -if (result !=SCAN_SUCCEED ) -return result; - -/* The collapse takes its own locks, so give this up */ -mmap_read_unlock (mm); - *lock_dropped = true; - -return collapse_run_pmd (mm, addr, cc); -} - static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock)
@@ -2922,7 +2900,7 @@ static void collapse_scan_mm_slot(unsigned int progress_max,VM_BUG_ON (khugepaged_scan.address & ~HPAGE_PMD_MASK );while (khugepaged_scan.address < hend) { -bool lock_dropped = false; +unsigned long addr;cond_resched ();if (unlikely (collapse_test_exit_or_disable (mm)))
@@ -2932,23 +2910,29 @@ static void collapse_scan_mm_slot(unsigned int progress_max, khugepaged_scan.address +HPAGE_PMD_SIZE > hend); - *result =collapse_single_pmd (khugepaged_scan.address, - vma, &lock_dropped, cc); -if (*result ==SCAN_SUCCEED ) - khugepaged_pages_collapsed++; + addr = khugepaged_scan.address;/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ; -if (lock_dropped) -/* -* We released mmap_lock so break loop. Note -* that we drop mmap_lock before all hugepage -* allocations, so if allocation fails, we are -* guaranteed to break here and report the -* correct result back to caller. -*/ - goto breakouterloop_mmap_lock; -if (cc->progress >= progress_max) - goto breakouterloop; + + *result =collapse_scan_pmd (vma, addr, cc); +/* Nothing to collapse here, and the lock is still ours */ +if (*result !=SCAN_SUCCEED ) { +if (cc->progress >= progress_max) + goto breakouterloop; +continue ; + } + +/* +* A collapse takes its own locks and is slow enough +* that a writer should not wait behind it, so give the +* lock up. That ends this walk: vma and the mm are +* whatever the collapse leaves them. +*/ +mmap_read_unlock (mm); + *result =collapse_run_pmd (mm, addr, cc); +if (*result ==SCAN_SUCCEED ) + khugepaged_pages_collapsed++; + goto breakouterloop_mmap_lock; } } breakouterloop:
@@ -3217,7 +3201,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,unsigned long hstart, hend, addr;enum scan_result last_fail =SCAN_FAIL ;int thps =0 ; -bool mmap_unlocked = false;BUG_ON (vma->vm_start > start);BUG_ON (vma->vm_end < end);
@@ -3240,25 +3223,40 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,lru_add_drain_all ();for (addr = hstart; addr < hend; addr +=HPAGE_PMD_SIZE ) { -enum scan_result result =SCAN_FAIL ; +struct vm_area_struct *found; +enum scan_result result; -if (mmap_unlocked) { +/* +* A collapse gives the lock up, so the VMA has to be found +* again after one: it can shrink while nothing is held. A scan +* that finds nothing to collapse leaves the lock alone, so a +* range that is already collapsed walks on without relocking. +*/ +if (!vma) {cond_resched ();mmap_read_lock (mm); - mmap_unlocked = false; - *lock_dropped = true; - result =hugepage_vma_revalidate (mm, addr, false, &vma, + result =hugepage_vma_revalidate (mm, addr, false, &found, cc,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED ) { last_fail = result; - goto out_nolock; + goto out_locked; } - + vma = found; hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); } - result =collapse_single_pmd (addr, vma, &mmap_unlocked, cc); + result =collapse_scan_pmd (vma, addr, cc); +/* Nothing to collapse here, and the lock is still ours */ +if (result !=SCAN_SUCCEED ) + goto tally; +/* The collapse takes its own locks, so give this up */ +mmap_read_unlock (mm); + *lock_dropped = true; + vma =NULL ; + + result =collapse_run_pmd (mm, addr, cc); +tally:switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED :
@@ -3280,17 +3278,15 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,default : last_fail = result;/* Other error, exit */ - goto out_maybelock; + goto out; } } -out_maybelock: +out:/* Caller expects us to hold mmap_lock on return */ -if (mmap_unlocked) { - *lock_dropped = true; +if (!vma)mmap_read_lock (mm); - } -out_nolock: +out_locked:mmap_assert_locked (mm);collapse_control_release (cc);kfree (cc);
Signature · 1 line
2.54.0
PATCH 10/12 mm/collapse: work out the orders a VMA allows once per VMA
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
The scan asked collapse_possible_orders() for every PTE table, for an answer that is a property of the VMA. Both callers walk a VMA a table at a time, so let them work it out once and pass the mask in. It is only good while the lock that produced it is held, so madvise_collapse() takes it again after every collapse.
The mask is then sampled once per VMA rather than once per table. A thp enabled knob written during a walk takes effect one VMA later, and cannot widen a collapse: hugepage_vma_revalidate() tests the order again under the lock the collapse retakes.
--- mm/khugepaged.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-)
mm/khugepaged.c +18 -14
@@ -1551,12 +1551,12 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, }static enum scan_result collapse_scan_anon_pmd (struct vm_area_struct *vma, -unsigned long start_addr,struct collapse_control *cc) +unsigned long start_addr,struct collapse_control *cc, +unsigned long enabled_orders) {const unsigned int max_ptes_shared =collapse_max_ptes_shared (cc,HPAGE_PMD_ORDER );const unsigned int max_ptes_swap =collapse_max_ptes_swap (cc,HPAGE_PMD_ORDER );unsigned int max_ptes_none =collapse_max_ptes_none (cc, vma,HPAGE_PMD_ORDER ); -enum tva_type tva_flags = cc->policy.tva_type;struct mm_struct *mm = vma->vm_mm;pmd_t *pmd;pte_t *pte, *_pte, pteval;
@@ -1567,7 +1567,6 @@ static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,struct folio *folio =NULL ;unsigned long failed_pfn =-1 ;unsigned long addr; -unsigned long enabled_orders;spinlock_t *ptl;int node =NUMA_NO_NODE , unmapped =0 ;
@@ -1581,8 +1580,6 @@ static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,collapse_scan_reset (cc); - enabled_orders =collapse_possible_orders (vma, vma->vm_flags, tva_flags); -/* * If PMD is the only enabled order, enforce max_ptes_none, otherwise * scan all pages to populate the bitmap for mTHP collapse. The bitmap
@@ -2773,7 +2770,8 @@ static void collapse_control_release(struct collapse_control *cc) }static enum scan_result collapse_scan_pmd (struct vm_area_struct *vma, -unsigned long addr,struct collapse_control *cc) +unsigned long addr,struct collapse_control *cc, +unsigned long orders) {mmap_assert_locked (vma->vm_mm);/* Whatever the last scan found has to have been run by now */
@@ -2783,7 +2781,7 @@ static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, }if (vma_is_anonymous (vma)) -return collapse_scan_anon_pmd (vma, addr, cc); +return collapse_scan_anon_pmd (vma, addr, cc, orders);/* * A file collapse works on the page cache and never sees a VMA, so take
@@ -2877,15 +2875,17 @@ static void collapse_scan_mm_slot(unsigned int progress_max,vma_iter_init (&vmi, mm, khugepaged_scan.address);for_each_vma (vmi, vma) { -unsigned long hstart, hend; +unsigned long hstart, hend, orders;cond_resched ();if (unlikely (collapse_test_exit_or_disable (mm))) { cc->progress++;break ; } -if (!collapse_possible_orders (vma, vma->vm_flags, -TVA_KHUGEPAGED )) { +/* One mask for the whole VMA */ + orders =collapse_possible_orders (vma, vma->vm_flags, + cc->policy.tva_type); +if (!orders) { cc->progress++;continue ; }
@@ -2914,7 +2914,7 @@ static void collapse_scan_mm_slot(unsigned int progress_max,/* move to next address */ khugepaged_scan.address +=HPAGE_PMD_SIZE ; - *result =collapse_scan_pmd (vma, addr, cc); + *result =collapse_scan_pmd (vma, addr, cc, orders);/* Nothing to collapse here, and the lock is still ours */ if (*result !=SCAN_SUCCEED ) {if (cc->progress >= progress_max)
@@ -3198,14 +3198,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, {struct collapse_control *cc;struct mm_struct *mm = vma->vm_mm; -unsigned long hstart, hend, addr; +unsigned long hstart, hend, addr, orders;enum scan_result last_fail =SCAN_FAIL ;int thps =0 ;BUG_ON (vma->vm_start > start);BUG_ON (vma->vm_end < end); -if (!collapse_possible_orders (vma, vma->vm_flags,TVA_FORCED_COLLAPSE )) + orders =collapse_possible_orders (vma, vma->vm_flags, +TVA_FORCED_COLLAPSE ); +if (!orders)return -EINVAL ; hstart =ALIGN (start,HPAGE_PMD_SIZE );
@@ -3243,9 +3245,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, } vma = found; hend =min (hend, vma->vm_end &HPAGE_PMD_MASK ); + orders =collapse_possible_orders (vma, vma->vm_flags, + cc->policy.tva_type); } - result =collapse_scan_pmd (vma, addr, cc); + result =collapse_scan_pmd (vma, addr, cc, orders);/* Nothing to collapse here, and the lock is still ours */ if (result !=SCAN_SUCCEED ) goto tally;
Signature · 1 line
2.54.0
PATCH 11/12 mm/collapse: declare the collapse interface in collapse.h
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
A collapse takes four calls:
- collapse_control_init() - set up the control a caller carries;
- collapse_scan_pmd() - scan one PTE table, under mmap_lock;
- collapse_run_pmd() - collapse what the scan found, no mmap_lock;
- collapse_control_release() - done with the control.
All four are static in khugepaged.c, as are collapse_possible_orders(), which says what a VMA allows, and the revalidate a caller needs once a collapse has given the mmap_lock up. No other file can ask for a collapse without them.
Declare them in collapse.h, with a comment stating the order they are called in and who holds the lock over each step.
hugepage_vma_revalidate() becomes collapse_vma_revalidate(): it is part of what a collapse offers now, not a helper of the daemon.
Preparation for implementing MADV_COLLAPSE in madvise.c.
No functional change.
--- mm/collapse.h | 44 ++++++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 18 +++++++++--------- 2 files changed, 53 insertions(+), 9 deletions(-)
mm/collapse.h +44 -0
@@ -116,4 +116,48 @@ struct collapse_control { pgoff_t scan_pgoff; }; +/* Which orders a VMA may collapse to, zero when it may not collapse at all */ +unsigned long collapse_possible_orders (struct vm_area_struct *vma, +vm_flags_t vm_flags,enum tva_type tva_flags); + +/* +* A caller states what it allows in cc->policy and then hands over one PTE +* table's worth of a VMA at a time: +* +* collapse_control_init(cc) once, before the first table +* collapse_scan_pmd(vma, addr, ...) per table +* collapse_run_pmd(mm, addr, cc) when a scan found work +* collapse_control_release(cc) once, when done with the control +* +* The caller holds mmap_lock for reading over the scan and passes an address +* within @vma, aligned to the PTE table the scan is to judge. +* +* The scan returns with that lock still held. It only reads, and almost every +* table it is offered has nothing to collapse, so a caller walks a whole VMA +* under the one lock it took to get there. SCAN_SUCCEED means there is +* something to collapse; anything else is why there is not. +* +* The run is called without the lock and returns without it, taking what it +* needs in between: what it does -- allocate, isolate, copy, flush -- is slow +* enough that a writer would wait behind it. The caller gives the lock up +* first, and with it @vma and anything derived under it, so a caller carrying +* on has to look up again with collapse_vma_revalidate(). The run revalidates +* for itself rather than trusting what the scan saw. +* +* A scan that found something has to be run: the file side takes a reference on +* the file while it still has the VMA to take it from, and the run is what +* gives it back. +*/ +void collapse_control_init (struct collapse_control *cc); +void collapse_control_release (struct collapse_control *cc); +enum scan_result collapse_scan_pmd (struct vm_area_struct *vma, +unsigned long addr,struct collapse_control *cc, +unsigned long orders); +enum scan_result collapse_run_pmd (struct mm_struct *mm,unsigned long addr, +struct collapse_control *cc); +enum scan_result collapse_vma_revalidate (struct mm_struct *mm, +unsigned long address,bool expect_anon, +struct vm_area_struct **vmap,struct collapse_control *cc, +unsigned int order); +#endif /* __MM_COLLAPSE_H */
mm/khugepaged.c +9 -9
@@ -498,7 +498,7 @@ void __khugepaged_enter(struct mm_struct *mm)* Check what orders are possible based on the vma and collapse type. * This is used to determine if mTHP collapse is a viable option. */ -static unsigned long collapse_possible_orders (struct vm_area_struct *vma, +unsigned long collapse_possible_orders (struct vm_area_struct *vma,vm_flags_t vm_flags,enum tva_type tva_flags) {unsigned long orders;
@@ -1016,7 +1016,7 @@ static int collapse_find_target_node(struct collapse_control *cc)* Returns enum scan_result value. */ -static enum scan_result hugepage_vma_revalidate (struct mm_struct *mm,unsigned long address, +enum scan_result collapse_vma_revalidate (struct mm_struct *mm,unsigned long address,bool expect_anon,struct vm_area_struct **vmap,struct collapse_control *cc,unsigned int order) {
@@ -1264,7 +1264,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s }mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, pmd_addr,/*expect_anon=*/ true, + result =collapse_vma_revalidate (mm, pmd_addr,/*expect_anon=*/ true, &vma, cc, order);if (result !=SCAN_SUCCEED ) {mmap_read_unlock (mm);
@@ -1299,7 +1299,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s* mmap_lock. */ mmap_write_lock (mm); - result =hugepage_vma_revalidate (mm, pmd_addr,/*expect_anon=*/ true, + result =collapse_vma_revalidate (mm, pmd_addr,/*expect_anon=*/ true, &vma, cc, order);if (result !=SCAN_SUCCEED ) goto out_up_write;
@@ -2754,13 +2754,13 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,return result; } -static void collapse_control_init (struct collapse_control *cc) +void collapse_control_init (struct collapse_control *cc) { cc->progress =0 ; cc->scan_file =NULL ; } -static void collapse_control_release (struct collapse_control *cc) +void collapse_control_release (struct collapse_control *cc) {/* A scan that took a file reference should have been run */ if (WARN_ON_ONCE (cc->scan_file)) {
@@ -2769,7 +2769,7 @@ static void collapse_control_release(struct collapse_control *cc) } } -static enum scan_result collapse_scan_pmd (struct vm_area_struct *vma, +enum scan_result collapse_scan_pmd (struct vm_area_struct *vma,unsigned long addr,struct collapse_control *cc,unsigned long orders) {
@@ -2793,7 +2793,7 @@ static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,return SCAN_SUCCEED ; } -static enum scan_result collapse_run_pmd (struct mm_struct *mm, +enum scan_result collapse_run_pmd (struct mm_struct *mm,unsigned long addr,struct collapse_control *cc) {struct file *file = cc->scan_file;
@@ -3237,7 +3237,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,if (!vma) {cond_resched ();mmap_read_lock (mm); - result =hugepage_vma_revalidate (mm, addr, false, &found, + result =collapse_vma_revalidate (mm, addr, false, &found, cc,HPAGE_PMD_ORDER );if (result !=SCAN_SUCCEED ) { last_fail = result;
Signature · 1 line
2.54.0
PATCH 12/12 mm/collapse: implement MADV_COLLAPSE in madvise.c
In reply to Kiryl ShutsemauFrom: "Kiryl Shutsemau (Meta)" <[email protected]>
MADV_COLLAPSE is a madvise operation, but its implementation sat in khugepaged.c. The daemon's file therefore also held a syscall's worth of code that has nothing to do with the daemon: the walk over the user's range, the per-PMD loop, and the errno translation.
Move it to madvise.c, among the operations it belongs with, along with the errno map and the policy it states for itself. It takes a struct madvise_behavior like every one of those operations, which is where the range, the VMA and the lock-dropped flag it used to be handed separately already live.
It stays a caller of the interface khugepaged uses, so nothing about the collapse changes.
The !CONFIG_TRANSPARENT_HUGEPAGE stub moves in with it.
--- include/linux/huge_mm.h | 9 --- mm/khugepaged.c | 157 +------------------------------------ mm/madvise.c | 169 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 169 insertions(+), 166 deletions(-)
include/linux/huge_mm.h +0 -9
@@ -510,8 +510,6 @@ change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,int hugepage_madvise (struct vm_area_struct *vma,vm_flags_t *vm_flags,int advice); -int madvise_collapse (struct vm_area_struct *vma,unsigned long start, -unsigned long end,bool *lock_dropped);void vma_adjust_trans_huge (struct vm_area_struct *vma,unsigned long start,unsigned long end,struct vm_area_struct *next);spinlock_t *__pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma);
@@ -715,13 +713,6 @@ static inline int hugepage_madvise(struct vm_area_struct *vma,return -EINVAL ; } -static inline int madvise_collapse (struct vm_area_struct *vma, -unsigned long start, -unsigned long end,bool *lock_dropped) -{ -return -EINVAL ; -} - static inline void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end,
mm/khugepaged.c +2 -155
172 lines not on this page · raw message
mm/madvise.c +167 -2
187 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH 01/12] mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()
In reply to Kiryl ShutsemauQuoted 12 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
madvise_collapse()holds anmmgrab()reference across its work. It is redundant. Every caller already holdsmm_users:- madvise(2) works on
current->mm, which lives as long as the task is in the syscall;
-process_madvise(2)reaches a remote mm throughmm_access(), which takes anmm_usersreference and holds it until the syscall returns;
-io_uringpassescurrent->mm;
- DAMON takes one withget_task_mm()and drops it after the call.
Quoted 8 lines
Drop the mmgrab()/mmdrop() pair.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/khugepaged.c | 2 -- 1 file changed, 2 deletions(-)
Makes sense.
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH 02/12] mm/khugepaged: count collapses where khugepaged makes them
In reply to Kiryl ShutsemauQuoted 6 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_single_pmd()bumpskhugepaged_pages_collapsedfor its caller, and testscc->is_khugepagedto know whether it should: the counter belongs to the daemon, andMADV_COLLAPSEmust not touch it.
Quoted 9 lines
The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/khugepaged.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH 03/12] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes
In reply to Kiryl ShutsemauQuoted 20 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source.
mthp_collapse()then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either.Name it for what a set bit means, and update the comments that named it.
No functional change.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/khugepaged.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
<snip>
mm/khugepaged.c static enum scan_result mthp_collapse(struct mm_struct *mm, · line 1518
Quoted 7 lines
mm/khugepaged.c +1
-1@@ -1514,12 +1514,12 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, goto next_order; max_ptes_none =collapse_max_ptes_none (cc,NULL , order); - nr_occupied_ptes =bitmap_weight_from (cc->mthp_present_ptes, offset, + nr_occupied_ptes =bitmap_weight_from (cc->eligible_ptes, offset, offset + nr_ptes);
s/nr_occupied_ptes/nr_eligible_ptes too?
Otherwise, LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH 04/12] mm/collapse: add collapse.h for the collapse interface
In reply to Kiryl ShutsemauQuoted 33 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
khugepaged.cholds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins.Start drawing that line.
Add
mm/collapse.hfor what the two sides have to agree on:- enum
scan_result- what the engine hands back;
- structcollapse_control- the state a request carries.And two constants move with them:
-
KHUGEPAGED_MAX_PTES_LIMIT->COLLAPSE_MAX_PTES_LIMIT;
-KHUGEPAGED_MIN_MTHP_ORDER->COLLAPSE_MIN_MTHP_ORDER.Neither is a fact about the daemon, so both lose the
KHUGEPAGED_prefix.No functional change.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- MAINTAINERS | 1 + mm/collapse.h | 67 +++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 79 ++++++++----------------------------------------- 3 files changed, 81 insertions(+), 66 deletions(-) create mode 100644 mm/collapse.h
<snip>
Quoted 25 lines
mm/collapse.h new file +25
-0+ +enum scan_result { +SCAN_FAIL , +SCAN_SUCCEED , +SCAN_NO_PTE_TABLE , +SCAN_PMD_MAPPED , +SCAN_EXCEED_NONE_PTE , +SCAN_EXCEED_SWAP_PTE , +SCAN_EXCEED_SHARED_PTE , +SCAN_PTE_NON_PRESENT , +SCAN_PTE_UFFD , +SCAN_PTE_MAPPED_HUGEPAGE , +SCAN_LACK_REFERENCED_PAGE , +SCAN_PAGE_NULL , +SCAN_SCAN_ABORT , +SCAN_PAGE_COUNT , +SCAN_PAGE_LRU , +SCAN_PAGE_LOCK , +SCAN_PAGE_ANON , +SCAN_PAGE_LAZYFREE , +SCAN_PAGE_COMPOUND , +SCAN_ANY_PROCESS , +SCAN_VMA_NULL , +SCAN_VMA_CHECK , +SCAN_ADDRESS_RANGE ,
mm/collapse.h line 49
Quoted 10 lines
mm/collapse.h new file +10
-0+SCAN_DEL_PAGE_LRU , +SCAN_ALLOC_HUGE_PAGE_FAIL , +SCAN_CGROUP_CHARGE_FAIL , +SCAN_TRUNCATED , +SCAN_PAGE_HAS_PRIVATE , +SCAN_STORE_FAILED , +SCAN_COPY_MC , +SCAN_PAGE_FILLED , + SCAN_PAGE_DIRTY_OR_WRITEBACK, +};
Are you planning to clean up scan_result? SCAN_SCAN_ABORT sounds funny, SCAN_ANY_PROCESS means nothing, SCAN_COPY_MC might better be COLLAPSE_COPY_MC.
Anyway, for this patch,
Signature · 2 lines
Best Regards, Yan, Zi
Zi Yan
reviewerRe: [PATCH 05/12] mm/collapse: state what a collapse may do in the policy
In reply to Kiryl ShutsemauQuoted 79 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle:
- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is left alone;
- whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying.None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out.
Add struct
collapse_policyfor the caller to fill: khugepaged from its own settings,MADV_COLLAPSEfrom the fact that a user asked explicitly. Every test becomes a read of a field, andcc->is_khugepagedgoes, having no reader left.khugepaged fills the policy once per scan pass,
MADV_COLLAPSEonce per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then judged alike.
collapse_file()also drops a NULL check on thecollapse_control. It has one call site, reached only fromcollapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead.Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/collapse.h | 40 ++++++++++++++++- mm/khugepaged.c | 114 ++++++++++++++++++++++++++---------------------- 2 files changed, 102 insertions(+), 52 deletions(-)mm/collapse.h +31
-0@@ -48,8 +48,46 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; +/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { +/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ +unsigned int max_ptes_none; +unsigned int max_ptes_swap; +unsigned int max_ptes_shared; + +/* +* Hold a sub-PMD window to a stricter rule than a PMD: no swapped-out +* and no shared PTEs at all, and max_ptes_none as +* collapse_max_ptes_none() scales it. +*/ +bool strict_sub_pmd; + +/* +* Collapse only where it looks worth doing: require some sign the +* range is in use, and leave clean lazyfree folios for reclaim rather +* than collapsing them into a folio that is not lazyfree. +*/ +bool skip_lazyfree; +bool require_referenced; + +/* +* Finish the job rather than leaving it half done for a fault to pick +* up: map the PMD over a file collapse before returning, and write +* dirty pages back and retry once instead of refusing them. Both cost +* latency the caller has to be willing to pay. +*/ +bool install_pmd; +bool writeback_dirty; +
mm/collapse.h enum scan_result { · line 90
Quoted 10 lines
mm/collapse.h +8
-1+/* How hard to try for a destination folio */ +gfp_t gfp; + +/* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ +enum tva_type tva_type; +}; +struct collapse_control { -bool is_khugepaged; +struct collapse_policy policy;
Can it be made const since it seems to be read-only?
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH 06/12] mm/collapse: drop the collapse_possible() wrapper
In reply to Kiryl ShutsemauQuoted 5 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_possible()only forwards tocollapse_possible_orders()and turns its mask into a bool. Its three callers can test the mask themselves.
Quoted 8 lines
No functional change.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/khugepaged.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-)
LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH 07/12] mm/collapse: name the per-table scan reset for what it resets
In reply to Kiryl ShutsemauQuoted 13 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_control_init_scan()resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing.Call it
collapse_scan_reset().Preparation for giving a control a real init and release, run once each for a whole series of scans. Two names a word apart would then stand for two different jobs.
Quoted 8 lines
No functional change.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/khugepaged.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
LGTM.
Signature · 2 lines
Best Regards, Yan, Zi
Andrew Morton
maintainerQuoted 45 lines · On Fri, 4 Sep 2026 16:10:14 +0100 Kiryl Shutsemau <[email protected]> wrote:
From: "Kiryl Shutsemau (Meta)" <[email protected]>
[ This is the first of the cleanups I said I would front-load ]
There is no line between the collapse engine and the callers that ask for a collapse.
khugepaged.cholds both, and they reach into each other.- Sixteen tests through the collapse path read
cc->is_khugepagedto work out what they are allowed to do, when every one of those decisions was made by the caller before it asked.-
collapse_single_pmd()does both halves of a collapse behind one call and dropsmmap_locksomewhere in the middle. Which of its paths dropped it is not something a caller can see, so it hands back a bool and the caller keeps track.- MADV_COLLAPSE's implementation -- the walk over the user's range, the per-PMD loop, the errno translation -- sits in
khugepaged.c, which is the daemon's file.So: draw the line. State what a caller allows in a policy, split the call in two with the lock as the boundary, and move the syscall to
madvise.c. What the engine offers is then four calls, with the lock state written down against each, and a policy the caller fills for itself:...
Behaviour
=========Nothing here changes what gets collapsed. Three things a reader should not have to find in the diff:
- Patch 5: khugepaged fills its policy once per scan pass, so the max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than once per table. A table scanned early in a pass and one scanned late are then judged alike, where before a knob written mid-pass split them.
- Patch 8:
mm_khugepaged_scan_pmdfires beforemm_collapse_huge_pagerather than after it, the scan having returned before the collapse runs. Its status field already readsSCAN_SUCCEEDfor an accepted table, so nothing changes there; what the collapse then made of the table is mm_collapse_huge_page's to report, per order.
Quoted 3 lines
- Patch 10: which orders a table is scanned for is sampled once per VMA rather than once per table. It cannot widen what a collapse does -- the order is tested again under the lock the collapse retakes.
So is *any* functional change expected with this series?
I'm reluctant to merge an unreviewed v1, but this comes close! I'll sit on it for a week or less, see what people say. Please poke me if/when you think it's good for mm-new.
Zi Yan
reviewerRe: [PATCH 08/12] mm/collapse: separate scanning a PTE table from collapsing it
In reply to Kiryl ShutsemauQuoted 48 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
A collapse is two jobs. One reads a PTE table under
mmap_lockand decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first.
collapse_single_pmd()did both, so the boundary between them was somewhere in the middle of a function.Give each half its own function:
-
collapse_scan_pmd()scans one table and only reads. The anonymous scan that used to carry that name keeps its body ascollapse_scan_anon_pmd(), andcollapse_scan_pmd()is now the entry that picks the anonymous or the file side.-
collapse_run_pmd()does the collapse the scan asked for.SCAN_SUCCEEDfrom the scan means there is something to run; anything else is why there is not.
collapse_single_pmd()is now the two of them with themmap_lockdrop in between, so its callers see what they saw before.Scan results (beyond
SCAN_SUCCEED) communicated viacollapse_control
structure: the orders, the referenced and swapped-out counts, and for a file the file itself and the offset in it.A file collapse works on the page cache and never sees a VMA. The scan takes the file reference while it still has VMA and the run unpins it when it is done.
Tracing changes with it.
mm_khugepaged_scan_pmdnow fires beforemm_collapse_huge_pageinstead of after it. Its status field already readsSCAN_SUCCEEDfor an accepted table, so what the collapse then made of that table is mm_collapse_huge_page's to report, per order.The two calls to that tracepoint become one. They differed in what the collapse between them changed; with the collapse no longer here, both carry the same arguments.
failed_pfnis set only where a PTE was refused, so it is -1 exactly when the result isSCAN_SUCCEED.Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/collapse.h | 14 ++++++ mm/khugepaged.c | 121 ++++++++++++++++++++++++++++++++++-------------- 2 files changed, 100 insertions(+), 35 deletions(-)
<snip>
Quoted 9 lines
mm/khugepaged.c +6
-2-mmap_read_unlock (mm); - *lock_dropped = true; +static enum scan_result collapse_run_pmd (struct mm_struct *mm, +unsigned long addr,struct collapse_control *cc) +{ +struct file *file = cc->scan_file; +bool triggered_wb = false; +enum scan_result result;
mm/khugepaged.c static enum scan_result collapse_scan_file(struct mm_struct *mm, · line 2813
Quoted 10 lines
mm/khugepaged.c +8
-0+pgoff_t pgoff; + +if (!file) +return mthp_collapse (mm, addr, cc->scan_referenced, + cc->scan_unmapped, cc, cc->scan_orders); + + cc->scan_file =NULL ; + pgoff = cc->scan_pgoff; retry: result =collapse_scan_file (mm, addr, file, pgoff, cc);
In the commit message, collapse_run_pmd() is said to do the collapse work, but collapse_scan_file() is scanning, right?
It seems that the code only separate anonymous scan and collapse. Why cannot pagecache code be separated in a similar way?
Signature · 2 lines
Best Regards, Yan, Zi
Re: [PATCH 01/12] mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()
In reply to Kiryl ShutsemauQuoted 9 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
madvise_collapse()holds anmmgrab()reference across its work. It is redundant. Every caller already holdsmm_users:- madvise(2) works on
current->mm, which lives as long as the task is inthe syscall; - process_madvise(2) reaches a remote mm through mm_access(), which takes an mm_users reference and holds it until the syscall returns;
Quoted 8 lines
-
io_uringpassescurrent->mm;
- DAMON takes one withget_task_mm()and drops it after the call.Drop the mmgrab()/mmdrop() pair.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>---
LGTM.
Re: [PATCH 02/12] mm/khugepaged: count collapses where khugepaged makes them
In reply to Kiryl ShutsemauQuoted 6 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_single_pmd()bumpskhugepaged_pages_collapsedfor its caller, and testscc->is_khugepagedto know whether it should: the counter belongs to the daemon, andMADV_COLLAPSEmust not touch it.
Quoted 6 lines
The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>---
LGTM.
Re: [PATCH 03/12] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes
In reply to Kiryl ShutsemauQuoted 45 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source.
mthp_collapse()then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either.Name it for what a set bit means, and update the comments that named it.
No functional change.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/khugepaged.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)mm/khugepaged.c +3
-3@@ -115,8 +115,8 @@ struct collapse_control {/* nodemask for allocation fallback */ nodemask_t alloc_nmask; -/* Each bit represents a single occupied (!none/zero) page. */ -DECLARE_BITMAP (mthp_present_ptes,MAX_PTRS_PER_PTE ); +/* Each bit marks a PTE the scan accepted as a collapse source */ +DECLARE_BITMAP (eligible_ptes,MAX_PTRS_PER_PTE ); };/** @@ -627,7 +627,7 @@ static void collapse_control_init_scan(struct collapse_control *cc) {memset (cc->node_load,0 ,sizeof (cc->node_load));nodes_clear (cc->alloc_nmask); -bitmap_zero (cc->mthp_present_ptes,MAX_PTRS_PER_PTE ); +bitmap_zero (cc->eligible_ptes,MAX_PTRS_PER_PTE ); }static void release_pte_folio (struct folio *folio)
mm/khugepaged.c static unsigned int max_order_from_offset(unsigned int offset) · line 1489
Quoted 10 lines
mm/khugepaged.c +2
-2@@ -1482,8 +1482,8 @@ static unsigned int max_order_from_offset(unsigned int offset)* mthp_collapse() consumes the bitmap that is generated during * collapse_scan_pmd() to determine what regions and mTHP orders fit best. * -* Each bit in cc->mthp_present_ptes represents a single occupied (!none/zero) -* page. We start at the PMD order and check if it is eligible for collapse; +* Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse +* source. We start at the PMD order and check if it is eligible for collapse; * if not, we check the left and right halves of the PTE page table we are * examining at a lower order. *
Should also update the following comments: s/occupied/accepted as a collapse source ?
"
* For each of these, we determine how many PTE entries are occupied in the
* range of PTE entries we propose to collapse, then we compare this to a
* threshold number of PTE entries which would need to be occupied for a
* collapse to be permitted at that order (accounting for max_ptes_none).
"
With that and Zi's comments addressed,
Re: [PATCH 04/12] mm/collapse: add collapse.h for the collapse interface
In reply to Kiryl ShutsemauQuoted 55 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
khugepaged.cholds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins.Start drawing that line.
Add
mm/collapse.hfor what the two sides have to agree on:- enum
scan_result- what the engine hands back;
- structcollapse_control- the state a request carries.And two constants move with them:
-
KHUGEPAGED_MAX_PTES_LIMIT->COLLAPSE_MAX_PTES_LIMIT;
-KHUGEPAGED_MIN_MTHP_ORDER->COLLAPSE_MIN_MTHP_ORDER.Neither is a fact about the daemon, so both lose the
KHUGEPAGED_prefix.No functional change.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- MAINTAINERS | 1 + mm/collapse.h | 67 +++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 79 ++++++++----------------------------------------- 3 files changed, 81 insertions(+), 66 deletions(-)create mode 100644
mm/collapse.hMAINTAINERS +1
-0@@ -17433,6 +17433,7 @@ F: Documentation/admin-guide/mm/transhuge.rst F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h +F: mm/collapse.h F: mm/huge_memory.c F: mm/khugepaged.c F: mm/mm_slot.hmm/collapse.h new file +4
-0@@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_COLLAPSE_H +#define __MM_COLLAPSE_H +
mm/collapse.h line 14
Quoted 10 lines
mm/collapse.h new file +10
-0+#include <linux/mm.h> +#include <linux/nodemask.h> +#include <linux/pgtable.h> +#include <linux/types.h> + +/* The most the max_ptes_* tunables accept */ +#define COLLAPSE_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) + +/* The smallest order a collapse will build */ +#define COLLAPSE_MIN_MTHP_ORDER 2
These 2 macros are already self-explanatory, and the comments are over-documented. I'd prefer to drop them. With that,
Re: [PATCH 06/12] mm/collapse: drop the collapse_possible() wrapper
In reply to Kiryl ShutsemauQuoted 10 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_possible()only forwards tocollapse_possible_orders()and turns its mask into a bool. Its three callers can test the mask themselves.No functional change.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>---
LGTM.
Re: [PATCH 07/12] mm/collapse: name the per-table scan reset for what it resets
In reply to Kiryl ShutsemauQuoted 13 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
collapse_control_init_scan()resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing.Call it
collapse_scan_reset().Preparation for giving a control a real init and release, run once each for a whole series of scans. Two names a word apart would then stand for two different jobs.
Quoted 5 lines
No functional change.
Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>---
LGTM.
Baolin Wang
reviewerRe: [PATCH 05/12] mm/collapse: state what a collapse may do in the policy
In reply to Kiryl ShutsemauQuoted 51 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle:
- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, andwhether a sub-PMD window is held to a stricter rule than a PMD; - whether a range has to look used, and whether a MADV_FREE'd page is left alone;- whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying.
None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out.
Add struct
collapse_policyfor the caller to fill: khugepaged from its own settings,MADV_COLLAPSEfrom the fact that a user asked explicitly. Every test becomes a read of a field, andcc->is_khugepagedgoes, having no reader left.khugepaged fills the policy once per scan pass,
MADV_COLLAPSEonce per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then judged alike.
collapse_file()also drops a NULL check on thecollapse_control. It has one call site, reached only fromcollapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead.Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/collapse.h | 40 ++++++++++++++++- mm/khugepaged.c | 114 ++++++++++++++++++++++++++---------------------- 2 files changed, 102 insertions(+), 52 deletions(-)mm/collapse.h +3
-0@@ -48,8 +48,46 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; +/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { +/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */
mm/collapse.h enum scan_result { · line 63
Quoted 10 lines
mm/collapse.h +10
-0+unsigned int max_ptes_none; +unsigned int max_ptes_swap; +unsigned int max_ptes_shared; + +/* +* Hold a sub-PMD window to a stricter rule than a PMD: no swapped-out +* and no shared PTEs at all, and max_ptes_none as +* collapse_max_ptes_none() scales it. +*/ +bool strict_sub_pmd;
This is a bit confusing to me. Actually, the check for mTHP collapse is stricter.
How about naming it 'allow_mthp_collapse'? That way we can keep the most original comments for the collapse_max_ptes_xxx() functions, which is clearer to me.
If others have a better name, please ignore my comment.
Quoted 4 lines
mm/collapse.h +4
-0+/* +* Collapse only where it looks worth doing: require some sign the +* range is in use, and leave clean lazyfree folios for reclaim rather +* than collapsing them into a folio that is not lazyfree.
mm/collapse.h enum scan_result { · line 78
Quoted 10 lines
mm/collapse.h +10
-0+*/ +bool skip_lazyfree; +bool require_referenced; + +/* +* Finish the job rather than leaving it half done for a fault to pick +* up: map the PMD over a file collapse before returning, and write +* dirty pages back and retry once instead of refusing them. Both cost +* latency the caller has to be willing to pay. +*/
Can you simplify theses comments? I think the 'install_pmd' is easy to expalain. :)
Quoted 9 lines
mm/collapse.h +9
-0+bool install_pmd; +bool writeback_dirty; + +/* How hard to try for a destination folio */ +gfp_t gfp; + +/* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ +enum tva_type tva_type; +};
Quoted 18 lines · Andrew Morton
Quoted 18 lines
Behaviour
=========Nothing here changes what gets collapsed. Three things a reader should not have to find in the diff:
- Patch 5: khugepaged fills its policy once per scan pass, so the max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than once per table. A table scanned early in a pass and one scanned late are then judged alike, where before a knob written mid-pass split them.
- Patch 8:
mm_khugepaged_scan_pmdfires beforemm_collapse_huge_pagerather than after it, the scan having returned before the collapse runs. Its status field already readsSCAN_SUCCEEDfor an accepted table, so nothing changes there; what the collapse then made of the table is mm_collapse_huge_page's to report, per order.
Quoted 5 lines
Quoted 3 lines
- Patch 10: which orders a table is scanned for is sampled once per VMA rather than once per table. It cannot widen what a collapse does -- the order is tested again under the lock the collapse retakes.
So is *any* functional change expected with this series?
No.
The three items here are so nobody has to find them in the diff, and calling them behaviour changes is a stretch: a sysfs knob written mid-pass takes effect on the next pass instead of the next table, and a tracepoint fires a few lines earlier with the same arguments.
Quoted 3 lines
I'm reluctant to merge an unreviewed v1, but this comes close! I'll sit on it for a week or less, see what people say. Please poke me if/when you think it's good for mm-new.
Zi and Baolin have already provided enough feedback to justify v2.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH 03/12] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes
In reply to Baolin WangQuoted 2 lines · Baolin Wang
Should also update the following comments: s/occupied/accepted as a collapse source ?
Yes. v2 renames nr_occupied_ptes to nr_eligible_ptes as Zi asked and rewrites that paragraph in the same terms, so the comment, the bitmap and the count all say eligible.
Quoted 9 lines
"
* For each of these, we determine how many PTE entries are occupied in the
* range of PTE entries we propose to collapse, then we compare this to a
* threshold number of PTE entries which would need to be occupied for a
* collapse to be permitted at that order (accounting formax_ptes_none).
"With that and Zi's comments addressed,
Reviewed-by Baolin Wang <[email protected]>
Thanks you, both.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH 04/12] mm/collapse: add collapse.h for the collapse interface
In reply to Zi YanQuoted 3 lines · Zi Yan
Are you planning to clean up
scan_result?SCAN_SCAN_ABORTsounds funny,SCAN_ANY_PROCESSmeans nothing,SCAN_COPY_MCmight better beCOLLAPSE_COPY_MC.
I was not, but it is on my list now. Not as part of this series: the rename touches every result site and the tracepoint's EM() table, and this patch only moves the enum.
Quoted 3 lines
Anyway, for this patch,
Reviewed-by Zi Yan <[email protected]>
Thanks!
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH 05/12] mm/collapse: state what a collapse may do in the policy
In reply to Zi YanQuoted 84 lines · Zi Yan
Quoted 83 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle:
- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is left alone;
- whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying.None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out.
Add struct
collapse_policyfor the caller to fill: khugepaged from its own settings,MADV_COLLAPSEfrom the fact that a user asked explicitly. Every test becomes a read of a field, andcc->is_khugepagedgoes, having no reader left.khugepaged fills the policy once per scan pass,
MADV_COLLAPSEonce per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then judged alike.
collapse_file()also drops a NULL check on thecollapse_control. It has one call site, reached only fromcollapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead.Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/collapse.h | 40 ++++++++++++++++- mm/khugepaged.c | 114 ++++++++++++++++++++++++++---------------------- 2 files changed, 102 insertions(+), 52 deletions(-)mm/collapse.h +35
-0@@ -48,8 +48,46 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; +/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { +/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ +unsigned int max_ptes_none; +unsigned int max_ptes_swap; +unsigned int max_ptes_shared; + +/* +* Hold a sub-PMD window to a stricter rule than a PMD: no swapped-out +* and no shared PTEs at all, and max_ptes_none as +* collapse_max_ptes_none() scales it. +*/ +bool strict_sub_pmd; + +/* +* Collapse only where it looks worth doing: require some sign the +* range is in use, and leave clean lazyfree folios for reclaim rather +* than collapsing them into a folio that is not lazyfree. +*/ +bool skip_lazyfree; +bool require_referenced; + +/* +* Finish the job rather than leaving it half done for a fault to pick +* up: map the PMD over a file collapse before returning, and write +* dirty pages back and retry once instead of refusing them. Both cost +* latency the caller has to be willing to pay. +*/ +bool install_pmd; +bool writeback_dirty; + +/* How hard to try for a destination folio */ +gfp_t gfp; + +/* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */
Quoted 8 lines
mm/collapse.h enum scan_result { · line 90
Quoted 6 lines
mm/collapse.h +4
-1+enum tva_type tva_type; +}; +struct collapse_control { -bool is_khugepaged; +struct collapse_policy policy;Can it be made const since it seems to be read-only?
Not as a member: khugepaged fills it once per pass from the sysfs knobs, so there is no constant object to point at. And madvise has to initialize it somehow after cc is allocated.
It could become a const pointer to a struct the caller owns, but that adds a lifetime to track for a struct that is written in one place and read everywhere else. I would rather keep it embedded.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH 05/12] mm/collapse: state what a collapse may do in the policy
In reply to Baolin WangQuoted 17 lines · Baolin Wang
mm/collapse.h enum scan_result { · line 63
Quoted 13 lines
mm/collapse.h +13
-0+/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { +/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ +unsigned int max_ptes_none; +unsigned int max_ptes_swap; +unsigned int max_ptes_shared; + +/* +* Hold a sub-PMD window to a stricter rule than a PMD: no swapped-out +* and no shared PTEs at all, and max_ptes_none as +* collapse_max_ptes_none() scales it. +*/ +bool strict_sub_pmd;This is a bit confusing to me. Actually, the check for mTHP collapse is stricter.
Quoted 5 lines
How about naming it '
allow_mthp_collapse'? That way we can keep the most original comments for thecollapse_max_ptes_xxx()functions, which is clearer to me.If others have a better name, please ignore my comment.
I would rather keep strict_sub_pmd.
Which orders a caller asks for is already decided elsewhere:collapse_possible_orders() hands khugepaged every anonymous order and MADV_COLLAPSE the PMD order only, from tva_type.
A flag called allow_mthp_collapse next to that would read as a second place deciding the same thing, and flipping it would not change which orders get collapsed. What it does change is how a sub-PMD window is judged once one is asked for, and it is read at exactly the three places that judge one.
I will do:
/* Take no swapped-out or shared PTE into a sub-PMD collapse */
bool strict_sub_pmd;Quoted 10 lines
Quoted 10 lines
mm/collapse.h +9
-0+/* +* Collapse only where it looks worth doing: require some sign the +* range is in use, and leave clean lazyfree folios for reclaim rather +* than collapsing them into a folio that is not lazyfree. +*/ +bool skip_lazyfree; +bool require_referenced; + +/*
Quoted 8 lines
mm/collapse.h enum scan_result { · line 78
Quoted 5 lines
mm/collapse.h +5
-0+* Finish the job rather than leaving it half done for a fault to pick +* up: map the PMD over a file collapse before returning, and write +* dirty pages back and retry once instead of refusing them. Both cost +* latency the caller has to be willing to pay. +*/ Can you simplify theses comments? I think the '
install_pmd' is easy to expalain. :)
Done for v2.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH 08/12] mm/collapse: separate scanning a PTE table from collapsing it
In reply to Zi YanQuoted 75 lines · Zi Yan
Quoted 48 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
A collapse is two jobs. One reads a PTE table under
mmap_lockand decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first.
collapse_single_pmd()did both, so the boundary between them was somewhere in the middle of a function.Give each half its own function:
-
collapse_scan_pmd()scans one table and only reads. The anonymous scan that used to carry that name keeps its body ascollapse_scan_anon_pmd(), andcollapse_scan_pmd()is now the entry that picks the anonymous or the file side.-
collapse_run_pmd()does the collapse the scan asked for.SCAN_SUCCEEDfrom the scan means there is something to run; anything else is why there is not.
collapse_single_pmd()is now the two of them with themmap_lockdrop in between, so its callers see what they saw before.Scan results (beyond
SCAN_SUCCEED) communicated viacollapse_control
structure: the orders, the referenced and swapped-out counts, and for a file the file itself and the offset in it.A file collapse works on the page cache and never sees a VMA. The scan takes the file reference while it still has VMA and the run unpins it when it is done.
Tracing changes with it.
mm_khugepaged_scan_pmdnow fires beforemm_collapse_huge_pageinstead of after it. Its status field already readsSCAN_SUCCEEDfor an accepted table, so what the collapse then made of that table is mm_collapse_huge_page's to report, per order.The two calls to that tracepoint become one. They differed in what the collapse between them changed; with the collapse no longer here, both carry the same arguments.
failed_pfnis set only where a PTE was refused, so it is -1 exactly when the result isSCAN_SUCCEED.Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/collapse.h | 14 ++++++ mm/khugepaged.c | 121 ++++++++++++++++++++++++++++++++++-------------- 2 files changed, 100 insertions(+), 35 deletions(-)<snip>
Quoted 9 lines
(quoted diff) +6
-2- mmap_read_unlock(mm); - *lock_dropped = true; +static enum scan_result collapse_run_pmd(struct mm_struct *mm, + unsigned long addr, struct collapse_control *cc) +{ + struct file *file = cc->scan_file; + bool triggered_wb = false; + enum scan_result result;Review
Quoted 10 lines
(quoted diff) +8
-0+ pgoff_t pgoff; + + if (!file) + return mthp_collapse(mm, addr, cc->scan_referenced, + cc->scan_unmapped, cc, cc->scan_orders); + + cc->scan_file = NULL; + pgoff = cc->scan_pgoff; retry: result = collapse_scan_file(mm, addr, file, pgoff, cc);In the commit message,
collapse_run_pmd()is said to do the collapse work, butcollapse_scan_file()is scanning, right?
Quoted 2 lines
It seems that the code only separate anonymous scan and collapse. Why cannot pagecache code be separated in a similar way?
Will fold the patch below into v2:
mm/collapse.h +3 -1
@@ -95,13 +95,15 @@ struct collapse_control {* * The file side takes a reference while it still has the VMA, since a * file collapse works on the page cache and never sees one; the run is -* what gives it back. +* what gives it back. A scan that found the PMD folio already in the +* cache leaves only the PTE table to retract. */ unsigned long scan_orders;int scan_referenced;int scan_unmapped;struct file *scan_file;pgoff_t scan_pgoff; +bool scan_retract_only; };/* Which orders a VMA may collapse to, zero when it may not collapse at all */
mm/khugepaged.c +40 -17
96 lines not on this page · raw message
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Zi Yan
reviewerRe: [PATCH 05/12] mm/collapse: state what a collapse may do in the policy
In reply to Kiryl ShutsemauQuoted 94 lines · Kiryl Shutsemau
Quoted 84 lines · Zi Yan
Quoted 83 lines · Kiryl Shutsemau
From: "Kiryl Shutsemau (Meta)" <[email protected]>
Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle:
- which VMAs are eligible, and how hard to try for a folio;
- how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD;
- whether a range has to look used, and whether a MADV_FREE'd page is left alone;
- whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying.None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out.
Add struct
collapse_policyfor the caller to fill: khugepaged from its own settings,MADV_COLLAPSEfrom the fact that a user asked explicitly. Every test becomes a read of a field, andcc->is_khugepagedgoes, having no reader left.khugepaged fills the policy once per scan pass,
MADV_COLLAPSEonce per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then judged alike.
collapse_file()also drops a NULL check on thecollapse_control. It has one call site, reached only fromcollapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead.Assisted-by Claude-Code:claude-opus-5Signed-off-by Kiryl Shutsemau (Meta) <[email protected]>--- mm/collapse.h | 40 ++++++++++++++++- mm/khugepaged.c | 114 ++++++++++++++++++++++++++---------------------- 2 files changed, 102 insertions(+), 52 deletions(-)mm/collapse.h +35
-0@@ -48,8 +48,46 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; +/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { +/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ +unsigned int max_ptes_none; +unsigned int max_ptes_swap; +unsigned int max_ptes_shared; + +/* +* Hold a sub-PMD window to a stricter rule than a PMD: no swapped-out +* and no shared PTEs at all, and max_ptes_none as +* collapse_max_ptes_none() scales it. +*/ +bool strict_sub_pmd; + +/* +* Collapse only where it looks worth doing: require some sign the +* range is in use, and leave clean lazyfree folios for reclaim rather +* than collapsing them into a folio that is not lazyfree. +*/ +bool skip_lazyfree; +bool require_referenced; + +/* +* Finish the job rather than leaving it half done for a fault to pick +* up: map the PMD over a file collapse before returning, and write +* dirty pages back and retry once instead of refusing them. Both cost +* latency the caller has to be willing to pay. +*/ +bool install_pmd; +bool writeback_dirty; + +/* How hard to try for a destination folio */ +gfp_t gfp; + +/* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ Quoted 8 lines
mm/collapse.h enum scan_result { · line 90
Quoted 6 lines
mm/collapse.h +4
-1+enum tva_type tva_type; +}; +struct collapse_control { -bool is_khugepaged; +struct collapse_policy policy;Can it be made const since it seems to be read-only?
Quoted 7 lines
Not as a member: khugepaged fills it once per pass from the sysfs knobs, so there is no constant object to point at. And madvise has to initialize it somehow after cc is allocated.
It could become a const pointer to a struct the caller owns, but that adds a lifetime to track for a struct that is written in one place and read everywhere else. I would rather keep it embedded.
Got it. Thanks for the explanation.
Signature · 2 lines
Best Regards, Yan, Zi
Baolin Wang
reviewerRe: [PATCH 05/12] mm/collapse: state what a collapse may do in the policy
In reply to Kiryl ShutsemauQuoted 36 lines · Kiryl Shutsemau
Quoted 17 lines · Baolin Wang
mm/collapse.h enum scan_result { · line 63
Quoted 13 lines
mm/collapse.h +13
-0+/* What a collapse is allowed to do, decided by the caller that asks for it */ +struct collapse_policy { +/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ +unsigned int max_ptes_none; +unsigned int max_ptes_swap; +unsigned int max_ptes_shared; + +/* +* Hold a sub-PMD window to a stricter rule than a PMD: no swapped-out +* and no shared PTEs at all, and max_ptes_none as +* collapse_max_ptes_none() scales it. +*/ +bool strict_sub_pmd;This is a bit confusing to me. Actually, the check for mTHP collapse is stricter.
Quoted 5 lines
How about naming it '
allow_mthp_collapse'? That way we can keep the most original comments for thecollapse_max_ptes_xxx()functions, which is clearer to me.If others have a better name, please ignore my comment.
I would rather keep
strict_sub_pmd.Which orders a caller asks for is already decided elsewhere:
collapse_possible_orders()hands khugepaged every anonymous order andMADV_COLLAPSEthe PMD order only, fromtva_type.A flag called
allow_mthp_collapsenext to that would read as a second place deciding the same thing, and flipping it would not change which orders get collapsed. What it does change is how a sub-PMD window is judged once one is asked for, and it is read at exactly the three places that judge one.
Quoted 4 lines
I will do:
/* Take no swapped-out or shared PTE into a sub-PMD collapse */ bool strict_sub_pmd;
OK. Fair enough. Thanks.