Skip to contentSkip to conversation

Small-sized THP for anonymous memory

PATCH RESEND Superseded Ryan Roberts 7 participants 63 messages Nov 22, 2023 – Nov 30, 2023

Subsystems (9)

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Ryan Roberts

author reviewer
#0 lore raw

v7 cover Small-sized THP for anonymous memory

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

Hi All,

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:

1) Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, reduced lru list, etc. In short, we reduce kernel overhead. This should benefit all architectures.
2) Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).

The major change in this revision is the migration to a new sysfs interface as recommended by David Hildenbrand - thanks to David for the suggestion! This interface is inspired by the existing per-hugepage-size sysfs interface used by hugetlb, provides full backwards compatibility with the existing PMD-size THP interface, and provides a base for future extensibility. See [7] for detailed discussion of the interface.

By default, the existing behaviour (and performance) is maintained. The user must explicitly enable small-sized THP to see the performance benefit.

The series has also become heavy with mm selftest changes: These all relate to enlightenment of cow and khugepaged tests to explicitly test with small-sized THP.

This series is based on mm-unstable (60df8b4235f5).

Prerequisites
=============

Some work items identified as being prerequisites are listed on page 3 at [8]. The summary is:

| item                          | status                  |
|:------------------------------|:------------------------|
| mlock                         | In mainline (v6.7)      |
| madvise                       | In mainline (v6.6)      |
| compaction                    | v1 posted [9]           |
| numa balancing                | Investigated: see below |
| user-triggered page migration | In mainline (v6.7)      |
| khugepaged collapse           | In mainline (NOP)       |

On NUMA balancing, which currently ignores any PTE-mapped THPs it encounters, John Hubbard has investigated this and concluded that it is A) not clear at the moment what a better policy might be for PTE-mapped THP and B) questions whether this should really be considered a prerequisite given no regression is caused for the default "small-sized THP disabled" case, and there is no correctness issue when it is enabled - its just a potential for non-optimal performance. (John please do elaborate if I haven't captured this correctly!)

If there are no disagreements about removing numa balancing from the list, then that just leaves compaction which is in review on list at the moment.

I really would like to get this series (and its remaining comapction prerequisite) in for v6.8. I accept that it may be a bit optimistic at this point, but lets see where we get to with review?

Testing
=======

The series includes patches for mm selftests to enlighten the cow and khugepaged tests to explicitly test with small-order THP, in the same way that PMD-order THP is tested. The new tests all pass, and no regressions are observed in the mm selftest suite. I've also run my usual kernel compilation and java script benchmarks without any issues.

Refer to my performance numbers posted with v6 [6]. (These are for small-sized THP only - they do not include the arm64 contpte follow-on series).

John Hubbard at Nvidia has indicated dramatic 10x performance improvements for some workloads at [10]. (Observed using v6 of this series as well as the arm64 contpte series).

Kefeng Wang at Huawei has also indicated he sees improvements at [11] although there are some latency regressions also.

Changes since v6 [6]
====================

- Refactored vmf_pte_range_changed() to remove uffd special-case (suggested by JohnH)
- Dropped accounting patch (#3 in v6) (suggested by DavidH)
- Continue to account *PMD-sized* THP only for now
- Can add more counters in future if needed
- Page cache large folios haven't needed any new counters yet
- Pivot to sysfs ABI proposed by DavidH
- per-size directories in a similar shape to that used by hugetlb
- Dropped "recommend" keyword patch (#6 in v6) (suggested by DavidH, Yu Zhou)
- For now, users need to understand implicitly which sizes are beneficial to their HW/SW
- Dropped arch_wants_pte_order() patch (#7 in v6)
- No longer needed due to dropping patch "recommend" keyword patch
- Enlightened khugepaged mm selftest to explicitly test with small-size THP
- Scrubbed commit logs to use "small-sized THP" consistently (suggested by DavidH)

Changes since v5 [5]
====================

- Added accounting for PTE-mapped THPs (patch 3)
- Added runtime control mechanism via sysfs as extension to THP (patch 4)
- Minor refactoring of alloc_anon_folio() to integrate with runtime controls
- Stripped out hardcoded policy for allocation order; its now all user space controlled (although user space can request "recommend" which will configure the HW-preferred order)

Changes since v4 [4]
====================

- Removed "arm64: mm: Override arch_wants_pte_order()" patch; arm64 now uses the default order-3 size. I have moved this patch over to the contpte series.
- Added "mm: Allow deferred splitting of arbitrary large anon folios" back into series. I originally removed this at v2 to add to a separate series, but that series has transformed significantly and it no longer fits, so bringing it back here.
- Reintroduced dependency on set_ptes(); Originally dropped this at v2, but set_ptes() is in mm-unstable now.
- Updated policy for when to allocate LAF; only fallback to order-0 if MADV_NOHUGEPAGE is present or if THP disabled via prctl; no longer rely on sysfs's never/madvise/always knob.
- Fallback to order-0 whenever uffd is armed for the vma, not just when uffd-wp is set on the pte.
- alloc_anon_folio() now returns struct folio *, where errors are encoded

    with ERR_PTR().

The last 3 changes were proposed by Yu Zhao - thanks!

Changes since v3 [3]
====================

- Renamed feature from FLEXIBLE_THP to LARGE_ANON_FOLIO.
- Removed flexthp_unhinted_max boot parameter. Discussion concluded that a sysctl is preferable but we will wait until real workload needs it.
- Fixed uninitialized addr on read fault path in do_anonymous_page().
- Added mm selftests for large anon folios in cow test suite.

Changes since v2 [2]
====================

- Dropped commit "Allow deferred splitting of arbitrary large anon folios"
- Huang, Ying suggested the "batch zap" work (which I dropped from this series after v1) is a prerequisite for merging FLXEIBLE_THP, so I've moved the deferred split patch to a separate series along with the batch zap changes. I plan to submit this series early next week.
- Changed folio order fallback policy
- We no longer iterate from preferred to 0 looking for acceptable policy
- Instead we iterate through preferred, PAGE_ALLOC_COSTLY_ORDER and 0 only
- Removed vma parameter from arch_wants_pte_order()
- Added command line parameter flexthp_unhinted_max
- clamps preferred order when vma hasn't explicitly opted-in to THP
- Never allocate large folio for MADV_NOHUGEPAGE vma (or when THP is disabled for process or system).
- Simplified implementation and integration with do_anonymous_page()
- Removed dependency on set_ptes()

Changes since v1 [1]
====================

- removed changes to arch-dependent vma_alloc_zeroed_movable_folio()
- replaced with arch-independent alloc_anon_folio()
- follows THP allocation approach
- no longer retry with intermediate orders if allocation fails
- fallback directly to order-0
- remove folio_add_new_anon_rmap_range() patch
- instead add its new functionality to folio_add_new_anon_rmap()
- remove batch-zap pte mappings optimization patch
- remove enabler folio_remove_rmap_range() patch too
- These offer real perf improvement so will submit separately
- simplify Kconfig
- single FLEXIBLE_THP option, which is independent of arch
- depends on TRANSPARENT_HUGEPAGE
- when enabled default to max anon folio size of 64K unless arch explicitly overrides
- simplify changes to do_anonymous_page():
- no more retry loop

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://lore.kernel.org/linux-mm/[email protected]/
[6] https://lore.kernel.org/linux-mm/[email protected]/
[7] https://lore.kernel.org/linux-mm/[email protected]/
[8] https://drive.google.com/file/d/1GnfYFpr7_c1kA41liRUW5YtCb8Cj18Ud/view?usp=sharing&resourcekey=0-U1Mj3-RhLD1JV6EThpyPyA
[9] https://lore.kernel.org/linux-mm/[email protected]/
[10] https://lore.kernel.org/linux-mm/[email protected]/
[11] https://lore.kernel.org/linux-mm/[email protected]/

Thanks,
Ryan

Ryan Roberts (10):
mm: Allow deferred splitting of arbitrary anon large folios
mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
mm: thp: Introduce per-size thp sysfs interface
mm: thp: Support allocation of anonymous small-sized THP selftests/mm/kugepaged: Restore thp settings at exit selftests/mm: Factor out thp settings management selftests/mm: Support small-sized THP interface in thp_settings selftests/mm/khugepaged: Enlighten for small-sized THP selftests/mm/cow: Generalize do_run_with_thp() helper selftests/mm/cow: Add tests for anonymous small-sized THP

 Documentation/admin-guide/mm/transhuge.rst |  74 +++-
 Documentation/filesystems/proc.rst         |   6 +-
 fs/proc/task_mmu.c                         |   3 +-
 include/linux/huge_mm.h                    | 102 +++--
 mm/huge_memory.c                           | 263 +++++++++++--
 mm/khugepaged.c                            |  16 +-
 mm/memory.c                                | 112 +++++-
 mm/page_vma_mapped.c                       |   3 +-
 mm/rmap.c                                  |  32 +-
 tools/testing/selftests/mm/Makefile        |   4 +-
 tools/testing/selftests/mm/cow.c           | 215 +++++++----
 tools/testing/selftests/mm/khugepaged.c    | 410 ++++-----------------
 tools/testing/selftests/mm/run_vmtests.sh  |   2 +
 tools/testing/selftests/mm/thp_settings.c  | 349 ++++++++++++++++++
 tools/testing/selftests/mm/thp_settings.h  |  80 ++++
 15 files changed, 1160 insertions(+), 511 deletions(-)
 create mode 100644 tools/testing/selftests/mm/thp_settings.c
 create mode 100644 tools/testing/selftests/mm/thp_settings.h

--
2.25.1

author reviewer
#1 lore raw

PATCH RESEND v7 1/10 mm: Allow deferred splitting of arbitrary anon large folios

In preparation for the introduction of anonymous small-sized THP, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.

---
 mm/rmap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/rmap.c +2 -2
index 7a27a2b41802..49e4d86a4f70 100644
@@ -1488,11 +1488,11 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,
 		__lruvec_stat_mod_folio(folio, idx, -nr);
 
 		/*
-		 * Queue anon THP for deferred split if at least one
+		 * Queue anon large folio for deferred split if at least one
 		 * page of the folio is unmapped and at least one page
 		 * is still mapped.
 		 */
-		if (folio_test_pmd_mappable(folio) && folio_test_anon(folio))
+		if (folio_test_large(folio) && folio_test_anon(folio))
 			if (!compound || nr < nr_pmdmapped)
 				deferred_split_folio(folio);
 	}

--
2.25.1

Reviewed-by on v7 1/10
#2 lore raw

Re: [RESEND PATCH v7 01/10] mm: Allow deferred splitting of arbitrary anon large folios

Quoted 6 lines

In preparation for the introduction of anonymous small-sized THP, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.

Quoted 6 lines

---

Quoted 22 lines
 mm/rmap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/rmap.c +2 -3
index 7a27a2b41802..49e4d86a4f70 100644
@@ -1488,11 +1488,11 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,
 		__lruvec_stat_mod_folio(folio, idx, -nr);
 
 		/*
-		 * Queue anon THP for deferred split if at least one
+		 * Queue anon large folio for deferred split if at least one
 		 * page of the folio is unmapped and at least one page
 		 * is still mapped.
 		 */
-		if (folio_test_pmd_mappable(folio) && folio_test_anon(folio))
+		if (folio_test_large(folio) && folio_test_anon(folio))
 			if (!compound || nr < nr_pmdmapped)
 				deferred_split_folio(folio);
 	}
--
Quoted 1 line

2.25.1

Thanks
Barry

author reviewer
#3 lore raw

PATCH RESEND v7 2/10 mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

In reply to Ryan Roberts

In preparation for supporting anonymous small-sized THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
 mm/rmap.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +20 -8
index 49e4d86a4f70..b086dc957b0c 100644
@@ -1305,32 +1305,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
  * This means the inc-and-test can be bypassed.
  * The folio does not have to be locked.
  *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio
  * is new, it's assumed to be mapped exclusively by a single process.
  */
 void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 		unsigned long address)
 {
-	int nr;
+	int nr = folio_nr_pages(folio);
 
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
 	__folio_set_swapbacked(folio);
+	__folio_set_anon(folio, vma, address, true);
 
-	if (likely(!folio_test_pmd_mappable(folio))) {
+	if (likely(!folio_test_large(folio))) {
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_mapcount, 0);
-		nr = 1;
+		SetPageAnonExclusive(&folio->page);
+	} else if (!folio_test_pmd_mappable(folio)) {
+		int i;
+
+		for (i = 0; i < nr; i++) {
+			struct page *page = folio_page(folio, i);
+
+			/* increment count (starts at -1) */
+			atomic_set(&page->_mapcount, 0);
+			SetPageAnonExclusive(page);
+		}
+
+		atomic_set(&folio->_nr_pages_mapped, nr);
 	} else {
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_entire_mapcount, 0);
 		atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED);
-		nr = folio_nr_pages(folio);
+		SetPageAnonExclusive(&folio->page);
 		__lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr);
 	}
 
 	__lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr);
-	__folio_set_anon(folio, vma, address, true);
-	SetPageAnonExclusive(&folio->page);
 }
 
 /**

--
2.25.1

Reviewed-by on v7 2/10
#4 lore raw

Re: [RESEND PATCH v7 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 25 lines · Ryan Roberts

In preparation for supporting anonymous small-sized THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +1 -1
index 49e4d86a4f70..b086dc957b0c 100644
@@ -1305,32 +1305,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,
   * This means the inc-and-test can be bypassed.
   * The folio does not have to be locked.
   *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio
   * is new, it's assumed to be mapped exclusively by a single process.
   */
  void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1319

Quoted 10 lines
mm/rmap.c +4 -2
  		unsigned long address)
  {
-	int nr;
+	int nr = folio_nr_pages(folio);
 
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
  	__folio_set_swapbacked(folio);
+	__folio_set_anon(folio, vma, address, true);

Likely the changed order doesn't matter.

LGTM

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#5 lore raw

Re: [RESEND PATCH v7 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 32 lines · David Hildenbrand
Quoted 31 lines · Ryan Roberts

In preparation for supporting anonymous small-sized THP, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.

---
  mm/rmap.c | 28 ++++++++++++++++++++--------

  1 file changed, 20 insertions(+), 8 deletions(-)
mm/rmap.c +2 -2
index 49e4d86a4f70..b086dc957b0c 100644
@@ -1305,32 +1305,44 @@ void page_add_anon_rmap(struct page *page, struct
 vm_area_struct *vma,
    * This means the inc-and-test can be bypassed.
    * The folio does not have to be locked.
    *
- * If the folio is large, it is accounted as a THP.  As the folio
+ * If the folio is pmd-mappable, it is accounted as a THP.  As the folio
    * is new, it's assumed to be mapped exclusively by a single process.
    */
   void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
           unsigned long address)
   {
-    int nr;
+    int nr = folio_nr_pages(folio);
 
Quoted 7 lines

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1319

Quoted 5 lines
mm/rmap.c +3 -1
-    VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+    VM_BUG_ON_VMA(address < vma->vm_start ||
+            address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
       __folio_set_swapbacked(folio);
+    __folio_set_anon(folio, vma, address, true);

Likely the changed order doesn't matter.

Yes; the reason I moved __folio_set_anon() up here is because SetPageAnonExclusive() asserts that the page is anon, and SetPageAnonExclusive() has to be called differently for the 3 cases. I couldn't see any reason why it wouldn't be safe to call __folio_set_anon() before setting up the mapcounts.

Quoted 5 lines

LGTM

Thanks!

#6 lore raw

Re: [RESEND PATCH v7 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

In reply to Ryan Roberts
Quoted 17 lines

void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,

mm/rmap.c +5 -4
 		unsigned long address)
 {
-	int nr;
+	int nr = folio_nr_pages(folio);
 
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
 	__folio_set_swapbacked(folio);
+	__folio_set_anon(folio, vma, address, true);
 
-	if (likely(!folio_test_pmd_mappable(folio))) {
+	if (likely(!folio_test_large(folio))) {
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_mapcount, 0);
-		nr = 1;

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1333

Quoted 10 lines
mm/rmap.c +10 -0
+		SetPageAnonExclusive(&folio->page);
+	} else if (!folio_test_pmd_mappable(folio)) {
+		int i;
+
+		for (i = 0; i < nr; i++) {
+			struct page *page = folio_page(folio, i);
+
+			/* increment count (starts at -1) */
+			atomic_set(&page->_mapcount, 0);
+			SetPageAnonExclusive(page);

Hi Ryan,

we are doing an entire mapping, right? what is the reason to increase mapcount for each subpage? shouldn't we only increase mapcount of subpage in either split or doublemap case?

in page_add_anon_rmap(), are we also increasing mapcount of each subpage for fork() case where the entire large folio is inheritted by child processes?

Quoted 6 lines
mm/rmap.c +3 -0
+		}
+
+		atomic_set(&folio->_nr_pages_mapped, nr);
 	} else {
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_entire_mapcount, 0);

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +1 -3
 		atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED);
-		nr = folio_nr_pages(folio);
+		SetPageAnonExclusive(&folio->page);
 		__lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr);
 	}
 
 	__lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr);
-	__folio_set_anon(folio, vma, address, true);
-	SetPageAnonExclusive(&folio->page);
 }

Thanks
Barry

author reviewer
#7 lore raw

Re: [RESEND PATCH v7 02/10] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()

Quoted 30 lines · Barry Song
Quoted 17 lines

void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,

mm/rmap.c +5 -4
 		unsigned long address)
 {
-	int nr;
+	int nr = folio_nr_pages(folio);
 
-	VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+	VM_BUG_ON_VMA(address < vma->vm_start ||
+			address + (nr << PAGE_SHIFT) > vma->vm_end, vma);
 	__folio_set_swapbacked(folio);
+	__folio_set_anon(folio, vma, address, true);
 
-	if (likely(!folio_test_pmd_mappable(folio))) {
+	if (likely(!folio_test_large(folio))) {
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_mapcount, 0);
-		nr = 1;

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1333

Quoted 10 lines
mm/rmap.c +10 -0
+		SetPageAnonExclusive(&folio->page);
+	} else if (!folio_test_pmd_mappable(folio)) {
+		int i;
+
+		for (i = 0; i < nr; i++) {
+			struct page *page = folio_page(folio, i);
+
+			/* increment count (starts at -1) */
+			atomic_set(&page->_mapcount, 0);
+			SetPageAnonExclusive(page);

Hi Ryan,

Quoted 7 lines

we are doing an entire mapping, right? what is the reason to increase mapcount for each subpage? shouldn't we only increase mapcount of subpage in either split or doublemap case?

in page_add_anon_rmap(), are we also increasing mapcount of each subpage for fork() case where the entire large folio is inheritted by child processes?

I think this is all answered by the conversation we just had in the context of the contpte series? Let me know if you still have concerns.

Quoted 21 lines
Quoted 7 lines
mm/rmap.c +3 -0
+		}
+
+		atomic_set(&folio->_nr_pages_mapped, nr);
 	} else {
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_entire_mapcount, 0);

mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1346

Quoted 10 lines
mm/rmap.c +1 -3
 		atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED);
-		nr = folio_nr_pages(folio);
+		SetPageAnonExclusive(&folio->page);
 		__lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr);
 	}
 
 	__lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr);
-	__folio_set_anon(folio, vma, address, true);
-	SetPageAnonExclusive(&folio->page);
 }

Thanks
Barry

author reviewer
#8 lore raw

PATCH RESEND v7 3/10 mm: thp: Introduce per-size thp sysfs interface

In reply to Ryan Roberts

In preparation for adding support for anonymous small-sized THP, introduce new sysfs structure that will be used to control the new behaviours. A new directory is added under transparent_hugepage for each supported THP size, and contains an enabled file, which can be set to "global" (to inherrit the global setting), "always", "madvise" or "never". For now, the kernel still only supports PMD-sized anonymous THP, so only 1 directory is populated.

The first half of the change converts transhuge_vma_suitable() and hugepage_vma_check() so that they take a bitfield of orders for which the user wants to determine support, and the functions filter out all the orders that can't be supported, given the current sysfs configuration and the VMA dimensions. If there is only 1 order set in the input then the output can continue to be treated like a boolean; this is the case for most call sites.

The second half of the change implements the new sysfs interface. It has been done so that each supported THP size has a struct thpsize, which describes the relevant metadata and is itself a kobject. This is pretty minimal for now, but should make it easy to add new per-thpsize files to the interface if needed in future (e.g. per-size defrag). Rather than keep the enabled state directly in the struct thpsize, I've elected to directly encode it into huge_anon_orders_[always|madvise|global] bitfields since this reduces the amount of work required in transhuge_vma_suitable() which is called for every page fault.

The remainder is copied from Documentation/admin-guide/mm/transhuge.rst, as modified by this commit. See that file for further details.

Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled system wide. This can be achieved per-supported-THP-size with one of::

echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

where <size> is the hugepage size being addressed, the available sizes for which vary by system. Alternatively it is possible to specify that a given hugepage size will inherrit the global enabled setting::

echo global >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

The global (legacy) enabled setting can be set as follows::

echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled

By default, PMD-sized hugepages have enabled="global" and all other hugepage sizes have enabled="never". If enabling multiple hugepage sizes, the kernel will select the most appropriate enabled size for a given allocation.

---
 Documentation/admin-guide/mm/transhuge.rst |  74 ++++--
 Documentation/filesystems/proc.rst         |   6 +-
 fs/proc/task_mmu.c                         |   3 +-
 include/linux/huge_mm.h                    | 100 +++++---
 mm/huge_memory.c                           | 263 +++++++++++++++++++--
 mm/khugepaged.c                            |  16 +-
 mm/memory.c                                |   6 +-
 mm/page_vma_mapped.c                       |   3 +-
 8 files changed, 387 insertions(+), 84 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +59 -15
index b0cc8243e093..52565e0bd074 100644
@@ -45,10 +45,23 @@ components:
    the two is using hugepages just because of the fact the TLB miss is
    going to run faster.
 
+As well as PMD-sized THP described above, it is also possible to
+configure the system to allocate "small-sized THP" to back anonymous
+memory (for example 16K, 32K, 64K, etc). These THPs continue to be
+PTE-mapped, but in many cases can still provide similar benefits to
+those outlined above: Page faults are significantly reduced (by a
+factor of e.g. 4, 8, 16, etc), but latency spikes are much less
+prominent because the size of each page isn't as huge as the PMD-sized
+variant and there is less memory to clear in each page fault. Some
+architectures also employ TLB compression mechanisms to squeeze more
+entries in when a set of PTEs are virtually and physically contiguous
+and approporiately aligned. In this case, TLB misses will occur less
+often.
+
 THP can be enabled system wide or restricted to certain tasks or even
 memory ranges inside task's address space. Unless THP is completely
 disabled, there is ``khugepaged`` daemon that scans memory and
-collapses sequences of basic pages into huge pages.
+collapses sequences of basic pages into PMD-sized huge pages.
 
 The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
 interface and using madvise(2) and prctl(2) system calls.
@@ -95,12 +108,29 @@ Global THP controls
 Transparent Hugepage Support for anonymous memory can be entirely disabled
 (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE
 regions (to avoid the risk of consuming more memory resources) or enabled
-system wide. This can be achieved with one of::
+system wide. This can be achieved per-supported-THP-size with one of::
+
+	echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+	echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+	echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+where <size> is the hugepage size being addressed, the available sizes
+for which vary by system. Alternatively it is possible to specify that
+a given hugepage size will inherrit the global enabled setting::
+
+	echo global >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+The global (legacy) enabled setting can be set as follows::
 
 	echo always >/sys/kernel/mm/transparent_hugepage/enabled
 	echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
 	echo never >/sys/kernel/mm/transparent_hugepage/enabled
 
+By default, PMD-sized hugepages have enabled="global" and all other
+hugepage sizes have enabled="never". If enabling multiple hugepage
+sizes, the kernel will select the most appropriate enabled size for a
+given allocation.
+
 It's also possible to limit defrag efforts in the VM to generate
 anonymous hugepages in case they're not immediately free to madvise
 regions or to never try to defrag memory and simply fallback to regular
@@ -146,25 +176,34 @@ madvise
 never
 	should be self-explanatory.
 
-By default kernel tries to use huge zero page on read page fault to
-anonymous mapping. It's possible to disable huge zero page by writing 0
-or enable it back by writing 1::
+By default kernel tries to use huge, PMD-mappable zero page on read
+page fault to anonymous mapping. It's possible to disable huge zero
+page by writing 0 or enable it back by writing 1::
 
 	echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
 	echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
 
-Some userspace (such as a test program, or an optimized memory allocation
-library) may want to know the size (in bytes) of a transparent hugepage::
+Some userspace (such as a test program, or an optimized memory
+allocation library) may want to know the size (in bytes) of a
+PMD-mappable transparent hugepage::
 
 	cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
 
-khugepaged will be automatically started when
-transparent_hugepage/enabled is set to "always" or "madvise, and it'll
-be automatically shutdown if it's set to "never".
+khugepaged will be automatically started when one or more hugepage
+sizes are enabled (either by directly setting "always" or "madvise",
+or by setting "global" while the global enabled is set to "always" or
+"madvise"), and it'll be automatically shutdown when the last hugepage
+size is disabled (either by directly setting "never", or by setting
+"global" while the global enabled is set to "never").
 
 Khugepaged controls
 -------------------
 
+.. note::
+   khugepaged currently only searches for opportunities to collapse to
+   PMD-sized THP and no attempt is made to collapse to small-sized
+   THP.
+
 khugepaged runs usually at low frequency so while one may not want to
 invoke defrag algorithms synchronously during the page faults, it
 should be worth invoking defrag at least in khugepaged. However it's
@@ -282,10 +321,11 @@ force
 Need of application restart
 ===========================
 
-The transparent_hugepage/enabled values and tmpfs mount option only affect
-future behavior. So to make them effective you need to restart any
-application that could have been using hugepages. This also applies to the
-regions registered in khugepaged.
+The transparent_hugepage/enabled and
+transparent_hugepage/hugepages-<size>kB/enabled values and tmpfs mount
+option only affect future behavior. So to make them effective you need
+to restart any application that could have been using hugepages. This
+also applies to the regions registered in khugepaged.
 
 Monitoring usage
 ================
@@ -308,6 +348,10 @@ frequently will incur overhead.
 There are a number of counters in ``/proc/vmstat`` that may be used to
 monitor how successfully the system is providing huge pages for use.
 
+.. note::
+   Currently the below counters only record events relating to
+   PMD-sized THP. Events relating to small-sized THP are not included.
+
 thp_fault_alloc
 	is incremented every time a huge page is successfully
 	allocated to handle a page fault.
@@ -413,7 +457,7 @@ for huge pages.
 Optimizing the applications
 ===========================
 
-To be guaranteed that the kernel will map a 2M page immediately in any
+To be guaranteed that the kernel will map a THP immediately in any
 memory region, the mmap region has to be hugepage naturally
 aligned. posix_memalign() can provide that guarantee.
 
Documentation/filesystems/proc.rst +3 -3
index 49ef12df631b..f8e8dd1fd148 100644
@@ -528,9 +528,9 @@ replaced by copy-on-write) part of the underlying shmem object out on swap.
 does not take into account swapped out page of underlying shmem objects.
 "Locked" indicates whether the mapping is locked in memory or not.
 
-"THPeligible" indicates whether the mapping is eligible for allocating THP
-pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise.
-It just shows the current status.
+"THPeligible" indicates whether the mapping is eligible for allocating
+naturally aligned THP pages of any currently enabled size. 1 if true, 0
+otherwise. It just shows the current status.
 
 "VmFlags" field deserves a separate description. This member represents the
 kernel flags associated with the particular virtual memory area in two letter
fs/proc/task_mmu.c +2 -1
index 51e0ec658457..2e25362ca9fa 100644
@@ -865,7 +865,8 @@ static int show_smap(struct seq_file *m, void *v)
 	__show_smap(m, &mss, false);
 
 	seq_printf(m, "THPeligible:    %8u\n",
-		   hugepage_vma_check(vma, vma->vm_flags, true, false, true));
+		   !!hugepage_vma_check(vma, vma->vm_flags, true, false, true,
+					THP_ORDERS_ALL));
 
 	if (arch_pkeys_enabled())
 		seq_printf(m, "ProtectionKey:  %8u\n", vma_pkey(vma));
include/linux/huge_mm.h +69 -31

136 lines not on this page · raw message

mm/huge_memory.c +239 -24

351 lines not on this page · raw message

mm/khugepaged.c +10 -6
index 064654717843..c3a874c0d601 100644
@@ -446,7 +446,8 @@ void khugepaged_enter_vma(struct vm_area_struct *vma,
 {
 	if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) &&
 	    hugepage_flags_enabled()) {
-		if (hugepage_vma_check(vma, vm_flags, false, false, true))
+		if (hugepage_vma_check(vma, vm_flags, false, false, true,
+				       BIT(PMD_ORDER)))
 			__khugepaged_enter(vma->vm_mm);
 	}
 }
@@ -922,10 +923,10 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
 	if (!vma)
 		return SCAN_VMA_NULL;
 
-	if (!transhuge_vma_suitable(vma, address))
+	if (!transhuge_vma_suitable(vma, address, BIT(PMD_ORDER)))
 		return SCAN_ADDRESS_RANGE;
 	if (!hugepage_vma_check(vma, vma->vm_flags, false, false,
-				cc->is_khugepaged))
+				cc->is_khugepaged, BIT(PMD_ORDER)))
 		return SCAN_VMA_CHECK;
 	/*
 	 * Anon VMA expected, the address may be unmapped then
@@ -1503,7 +1504,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
 	 * and map it by a PMD, regardless of sysfs THP settings. As such, let's
 	 * analogously elide sysfs THP settings here.
 	 */
-	if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false))
+	if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false,
+				BIT(PMD_ORDER)))
 		return SCAN_VMA_CHECK;
 
 	/* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */
@@ -2368,7 +2370,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
 			progress++;
 			break;
 		}
-		if (!hugepage_vma_check(vma, vma->vm_flags, false, false, true)) {
+		if (!hugepage_vma_check(vma, vma->vm_flags, false, false, true,
+					BIT(PMD_ORDER))) {
 skip:
 			progress++;
 			continue;
@@ -2705,7 +2708,8 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev,
 
 	*prev = vma;
 
-	if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false))
+	if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false,
+				BIT(PMD_ORDER)))
 		return -EINVAL;
 
 	cc = kmalloc(sizeof(*cc), GFP_KERNEL);
mm/memory.c +3 -3
index c32954e16b28..9d5e61d6d859 100644
@@ -4319,7 +4319,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)
 	pmd_t entry;
 	vm_fault_t ret = VM_FAULT_FALLBACK;
 
-	if (!transhuge_vma_suitable(vma, haddr))
+	if (!transhuge_vma_suitable(vma, haddr, BIT(PMD_ORDER)))
 		return ret;
 
 	page = compound_head(page);
@@ -5117,7 +5117,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 		return VM_FAULT_OOM;
 retry_pud:
 	if (pud_none(*vmf.pud) &&
-	    hugepage_vma_check(vma, vm_flags, false, true, true)) {
+	    hugepage_vma_check(vma, vm_flags, false, true, true, BIT(PUD_ORDER))) {
 		ret = create_huge_pud(&vmf);
 		if (!(ret & VM_FAULT_FALLBACK))
 			return ret;
@@ -5151,7 +5151,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 		goto retry_pud;
 
 	if (pmd_none(*vmf.pmd) &&
-	    hugepage_vma_check(vma, vm_flags, false, true, true)) {
+	    hugepage_vma_check(vma, vm_flags, false, true, true, BIT(PMD_ORDER))) {
 		ret = create_huge_pmd(&vmf);
 		if (!(ret & VM_FAULT_FALLBACK))
 			return ret;
mm/page_vma_mapped.c +2 -1
index e0b368e545ed..5f7e89c5b595 100644
@@ -268,7 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
 			 * cleared *pmd but not decremented compound_mapcount().
 			 */
 			if ((pvmw->flags & PVMW_SYNC) &&
-			    transhuge_vma_suitable(vma, pvmw->address) &&
+			    transhuge_vma_suitable(vma, pvmw->address,
+						   BIT(PMD_ORDER)) &&
 			    (pvmw->nr_pages >= HPAGE_PMD_NR)) {
 				spinlock_t *ptl = pmd_lock(mm, pvmw->pmd);
 

--
2.25.1

#9 lore raw

Re: [RESEND PATCH v7 03/10] mm: thp: Introduce per-size thp sysfs interface

Quoted 73 lines · Ryan Roberts

In preparation for adding support for anonymous small-sized THP, introduce new sysfs structure that will be used to control the new behaviours. A new directory is added under transparent_hugepage for each supported THP size, and contains an enabled file, which can be set to "global" (to inherrit the global setting), "always", "madvise" or "never". For now, the kernel still only supports PMD-sized anonymous THP, so only 1 directory is populated.

The first half of the change converts transhuge_vma_suitable() and hugepage_vma_check() so that they take a bitfield of orders for which the user wants to determine support, and the functions filter out all the orders that can't be supported, given the current sysfs configuration and the VMA dimensions. If there is only 1 order set in the input then the output can continue to be treated like a boolean; this is the case for most call sites.

The second half of the change implements the new sysfs interface. It has been done so that each supported THP size has a struct thpsize, which describes the relevant metadata and is itself a kobject. This is pretty minimal for now, but should make it easy to add new per-thpsize files to the interface if needed in future (e.g. per-size defrag). Rather than keep the enabled state directly in the struct thpsize, I've elected to directly encode it into huge_anon_orders_[always|madvise|global] bitfields since this reduces the amount of work required in transhuge_vma_suitable() which is called for every page fault.

The remainder is copied from Documentation/admin-guide/mm/transhuge.rst, as modified by this commit. See that file for further details.

Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled system wide. This can be achieved per-supported-THP-size with one of::

echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

where <size> is the hugepage size being addressed, the available sizes for which vary by system. Alternatively it is possible to specify that a given hugepage size will inherrit the global enabled setting::

echo global >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

The global (legacy) enabled setting can be set as follows::

echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled

By default, PMD-sized hugepages have enabled="global" and all other hugepage sizes have enabled="never". If enabling multiple hugepage sizes, the kernel will select the most appropriate enabled size for a given allocation.

---
  Documentation/admin-guide/mm/transhuge.rst |  74 ++++--
  Documentation/filesystems/proc.rst         |   6 +-
  fs/proc/task_mmu.c                         |   3 +-
  include/linux/huge_mm.h                    | 100 +++++---
  mm/huge_memory.c                           | 263 +++++++++++++++++++--
  mm/khugepaged.c                            |  16 +-
  mm/memory.c                                |   6 +-
  mm/page_vma_mapped.c                       |   3 +-
  8 files changed, 387 insertions(+), 84 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +0 -0
index b0cc8243e093..52565e0bd074 100644

Documentation/admin-guide/mm/transhuge.rst components: · line 49

Quoted 5 lines
Documentation/admin-guide/mm/transhuge.rst +2 -0
@@ -45,10 +45,23 @@ components:
     the two is using hugepages just because of the fact the TLB miss is
     going to run faster.
 
+As well as PMD-sized THP described above, it is also possible to
+configure the system to allocate "small-sized THP" to back anonymous

Here's one of the places to change to the new name, which lately is "multi-size THP", or mTHP or m_thp for short. (I've typed "multi-size" instead of "multi-sized", because the 'd' doesn't add significantly to the meaning, and if in doubt, shorter is better.

Documentation/admin-guide/mm/transhuge.rst components: · line 60

Quoted 11 lines
Documentation/admin-guide/mm/transhuge.rst +11 -0
+memory (for example 16K, 32K, 64K, etc). These THPs continue to be
+PTE-mapped, but in many cases can still provide similar benefits to
+those outlined above: Page faults are significantly reduced (by a
+factor of e.g. 4, 8, 16, etc), but latency spikes are much less
+prominent because the size of each page isn't as huge as the PMD-sized
+variant and there is less memory to clear in each page fault. Some
+architectures also employ TLB compression mechanisms to squeeze more
+entries in when a set of PTEs are virtually and physically contiguous
+and approporiately aligned. In this case, TLB misses will occur less
+often.
+

OK, all of the above still seems like it can remain the same.

Quoted 12 lines
Documentation/admin-guide/mm/transhuge.rst +1 -1
  THP can be enabled system wide or restricted to certain tasks or even
  memory ranges inside task's address space. Unless THP is completely
  disabled, there is ``khugepaged`` daemon that scans memory and
-collapses sequences of basic pages into huge pages.
+collapses sequences of basic pages into PMD-sized huge pages.
 
  The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
  interface and using madvise(2) and prctl(2) system calls.
@@ -95,12 +108,29 @@ Global THP controls
  Transparent Hugepage Support for anonymous memory can be entirely disabled
  (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE
  regions (to avoid the risk of consuming more memory resources) or enabled

Documentation/admin-guide/mm/transhuge.rst Global THP controls · line 119

Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +9 -1
-system wide. This can be achieved with one of::
+system wide. This can be achieved per-supported-THP-size with one of::
+
+	echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+	echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+	echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+where <size> is the hugepage size being addressed, the available sizes
+for which vary by system. Alternatively it is possible to specify that
+a given hugepage size will inherrit the global enabled setting::

typo: inherrit

Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +4 -0
+
+	echo global >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+The global (legacy) enabled setting can be set as follows::

Documentation/admin-guide/mm/transhuge.rst Global THP controls · line 133

Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +5 -0
 
  	echo always >/sys/kernel/mm/transparent_hugepage/enabled
  	echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
  	echo never >/sys/kernel/mm/transparent_hugepage/enabled
 
+By default, PMD-sized hugepages have enabled="global" and all other
+hugepage sizes have enabled="never". If enabling multiple hugepage
+sizes, the kernel will select the most appropriate enabled size for a
+given allocation.
+

This is slightly murky. I wonder if "inherited" is a little more directly informative than global; it certainly felt that way my first time running this and poking at it.

And a few trivial examples would be a nice touch.

And so overall with a few other minor tweaks, I'd suggest this:

...
where <size> is the hugepage size being addressed, the available sizes for which vary by system.

For example:
echo always >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled

Alternatively it is possible to specify that a given hugepage size will inherit the top-level "enabled" value:

echo inherited >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

For example:
echo inherited >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled

The top-level setting (for use with "inherited") can be by issuing one of the following commands::

echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled

By default, PMD-sized hugepages have enabled="inherited" and all other hugepage sizes have enabled="never".

Quoted 63 lines
Documentation/admin-guide/mm/transhuge.rst +22 -12
  It's also possible to limit defrag efforts in the VM to generate
  anonymous hugepages in case they're not immediately free to madvise
  regions or to never try to defrag memory and simply fallback to regular
@@ -146,25 +176,34 @@ madvise
  never
  	should be self-explanatory.
 
-By default kernel tries to use huge zero page on read page fault to
-anonymous mapping. It's possible to disable huge zero page by writing 0
-or enable it back by writing 1::
+By default kernel tries to use huge, PMD-mappable zero page on read
+page fault to anonymous mapping. It's possible to disable huge zero
+page by writing 0 or enable it back by writing 1::
 
  	echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
  	echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
 
-Some userspace (such as a test program, or an optimized memory allocation
-library) may want to know the size (in bytes) of a transparent hugepage::
+Some userspace (such as a test program, or an optimized memory
+allocation library) may want to know the size (in bytes) of a
+PMD-mappable transparent hugepage::
 
  	cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
 
-khugepaged will be automatically started when
-transparent_hugepage/enabled is set to "always" or "madvise, and it'll
-be automatically shutdown if it's set to "never".
+khugepaged will be automatically started when one or more hugepage
+sizes are enabled (either by directly setting "always" or "madvise",
+or by setting "global" while the global enabled is set to "always" or
+"madvise"), and it'll be automatically shutdown when the last hugepage
+size is disabled (either by directly setting "never", or by setting
+"global" while the global enabled is set to "never").
 
  Khugepaged controls
  -------------------
 
+.. note::
+   khugepaged currently only searches for opportunities to collapse to
+   PMD-sized THP and no attempt is made to collapse to small-sized
+   THP.
+
  khugepaged runs usually at low frequency so while one may not want to
  invoke defrag algorithms synchronously during the page faults, it
  should be worth invoking defrag at least in khugepaged. However it's
@@ -282,10 +321,11 @@ force
  Need of application restart
  ===========================
 
-The transparent_hugepage/enabled values and tmpfs mount option only affect
-future behavior. So to make them effective you need to restart any
-application that could have been using hugepages. This also applies to the
-regions registered in khugepaged.
+The transparent_hugepage/enabled and
+transparent_hugepage/hugepages-<size>kB/enabled values and tmpfs mount
+option only affect future behavior. So to make them effective you need
+to restart any application that could have been using hugepages. This
+also applies to the regions registered in khugepaged.
 
  Monitoring usage
  ================

Documentation/admin-guide/mm/transhuge.rst frequently will incur overhead. · line 353

Quoted 6 lines
Documentation/admin-guide/mm/transhuge.rst +3 -0
@@ -308,6 +348,10 @@ frequently will incur overhead.
  There are a number of counters in ``/proc/vmstat`` that may be used to
  monitor how successfully the system is providing huge pages for use.
 
+.. note::
+   Currently the below counters only record events relating to
+   PMD-sized THP. Events relating to small-sized THP are not included.

Here's another spot to rename to "multi-size THP".

Quoted 18 lines
Documentation/admin-guide/mm/transhuge.rst +2 -1
+
  thp_fault_alloc
  	is incremented every time a huge page is successfully
  	allocated to handle a page fault.
@@ -413,7 +457,7 @@ for huge pages.
  Optimizing the applications
  ===========================
 
-To be guaranteed that the kernel will map a 2M page immediately in any
+To be guaranteed that the kernel will map a THP immediately in any
  memory region, the mmap region has to be hugepage naturally
  aligned. posix_memalign() can provide that guarantee.
 
Documentation/filesystems/proc.rst +0 -0
index 49ef12df631b..f8e8dd1fd148 100644

Documentation/filesystems/proc.rst replaced by copy-on-write) part of the underlying shmem object out on swap. · line 533

Quoted 9 lines
Documentation/filesystems/proc.rst +3 -3
@@ -528,9 +528,9 @@ replaced by copy-on-write) part of the underlying shmem object out on swap.
  does not take into account swapped out page of underlying shmem objects.
  "Locked" indicates whether the mapping is locked in memory or not.
 
-"THPeligible" indicates whether the mapping is eligible for allocating THP
-pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise.
-It just shows the current status.
+"THPeligible" indicates whether the mapping is eligible for allocating
+naturally aligned THP pages of any currently enabled size. 1 if true, 0
+otherwise. It just shows the current status.

"It just shows the current status"...as opposed to what? I'm missing an educational opportunity here--not sure what this means. :)

Quoted 24 lines
Documentation/filesystems/proc.rst +0 -0
  "VmFlags" field deserves a separate description. This member represents the
  kernel flags associated with the particular virtual memory area in two letter
fs/proc/task_mmu.c +2 -1
index 51e0ec658457..2e25362ca9fa 100644
@@ -865,7 +865,8 @@ static int show_smap(struct seq_file *m, void *v)
  	__show_smap(m, &mss, false);
 
  	seq_printf(m, "THPeligible:    %8u\n",
-		   hugepage_vma_check(vma, vma->vm_flags, true, false, true));
+		   !!hugepage_vma_check(vma, vma->vm_flags, true, false, true,
+					THP_ORDERS_ALL));
 
  	if (arch_pkeys_enabled())
  		seq_printf(m, "ProtectionKey:  %8u\n", vma_pkey(vma));
include/linux/huge_mm.h +0 -0
index fa0350b0812a..7d6f7d96b039 100644
@@ -67,6 +67,21 @@ extern struct kobj_attribute shmem_enabled_attr;
  #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)
  #define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)

include/linux/huge_mm.h extern struct kobj_attribute shmem_enabled_attr; · line 78

Quoted 10 lines
include/linux/huge_mm.h +9 -0
 
+/*
+ * Mask of all large folio orders supported for anonymous THP.
+ */
+#define THP_ORDERS_ALL_ANON	BIT(PMD_ORDER)
+
+/*
+ * Mask of all large folio orders supported for file THP.
+ */
+#define THP_ORDERS_ALL_FILE	(BIT(PMD_ORDER) | BIT(PUD_ORDER))

Is there something about file THP that implies PUD_ORDER? This deserves an explanatory comment, I think.

Quoted 10 lines
include/linux/huge_mm.h +6 -0
+
+/*
+ * Mask of all large folio orders supported for THP.
+ */
+#define THP_ORDERS_ALL		(THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE)
+
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
  #define HPAGE_PMD_SHIFT PMD_SHIFT
  #define HPAGE_PMD_SIZE	((1UL) << HPAGE_PMD_SHIFT)

include/linux/huge_mm.h extern struct kobj_attribute shmem_enabled_attr; · line 87

Quoted 10 lines
include/linux/huge_mm.h +0 -7
@@ -78,13 +93,18 @@ extern struct kobj_attribute shmem_enabled_attr;
 
  extern unsigned long transparent_hugepage_flags;
 
-#define hugepage_flags_enabled()					       \
-	(transparent_hugepage_flags &				       \
-	 ((1<<TRANSPARENT_HUGEPAGE_FLAG) |		       \
-	  (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)))
-#define hugepage_flags_always()				\
-	(transparent_hugepage_flags &			\
-	 (1<<TRANSPARENT_HUGEPAGE_FLAG))

Are macros actually required? If not, static inlines would be nicer.

Quoted 21 lines
include/linux/huge_mm.h +12 -2
+bool hugepage_flags_enabled(void);
+
+static inline int first_order(unsigned long orders)
+{
+	return fls_long(orders) - 1;
+}
+
+static inline int next_order(unsigned long *orders, int prev)
+{
+	*orders &= ~BIT(prev);
+	return first_order(*orders);
+}
 
  /*
   * Do the below checks:
@@ -97,23 +117,39 @@ extern unsigned long transparent_hugepage_flags;
   *   - For all vmas, check if the haddr is in an aligned HPAGE_PMD_SIZE
   *     area.
   */
-static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
-		unsigned long addr)

include/linux/huge_mm.h extern unsigned long transparent_hugepage_flags; · line 121

Quoted 10 lines
include/linux/huge_mm.h +2 -8
-{
-	unsigned long haddr;
-
-	/* Don't have to check pgoff for anonymous vma */
-	if (!vma_is_anonymous(vma)) {
-		if (!IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) - vma->vm_pgoff,
-				HPAGE_PMD_NR))
-			return false;
+static inline unsigned long transhuge_vma_suitable(struct vm_area_struct *vma,
+		unsigned long addr, unsigned long orders)

Changing this from a bool to a mask of orders is quite significant, and both the function name and the function-level comment documentation need to also be adjusted. Here, perhaps one of these names would work:

transhuge_vma_suitable_orders()
transhuge_vma_orders()

include/linux/huge_mm.h extern unsigned long transparent_hugepage_flags; · line 129

Quoted 8 lines
include/linux/huge_mm.h +8 -0
+{
+	int order;
+
+	/*
+	 * Iterate over orders, highest to lowest, removing orders that don't
+	 * meet alignment requirements from the set. Exit loop at first order
+	 * that meets requirements, since all lower orders must also meet
+	 * requirements.

Should we add a little note here, to the effect of, "this is unilaterally taking over a certain amount of allocation-like policy, by deciding how to search for folios of a certain size"?

Or is this The Only Way To Do It, after all? I know we had some discussion about it, and intuitively it feels reasonable, but wanted to poke at it one last time anyway.

Quoted 82 lines
include/linux/huge_mm.h +32 -14
+	 */
+
+	order = first_order(orders);
+
+	while (orders) {
+		unsigned long hpage_size = PAGE_SIZE << order;
+		unsigned long haddr = ALIGN_DOWN(addr, hpage_size);
+
+		if (haddr >= vma->vm_start &&
+		    haddr + hpage_size <= vma->vm_end) {
+			if (!vma_is_anonymous(vma)) {
+				if (IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) -
+						vma->vm_pgoff,
+						hpage_size >> PAGE_SHIFT))
+					break;
+			} else
+				break;
+		}
+
+		order = next_order(&orders, order);
  	}
 
-	haddr = addr & HPAGE_PMD_MASK;
-
-	if (haddr < vma->vm_start || haddr + HPAGE_PMD_SIZE > vma->vm_end)
-		return false;
-	return true;
+	return orders;
  }
 
  static inline bool file_thp_enabled(struct vm_area_struct *vma)
@@ -130,8 +166,9 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)
  	       !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
  }
 
-bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,
-			bool smaps, bool in_pf, bool enforce_sysfs);
+unsigned long hugepage_vma_check(struct vm_area_struct *vma,
+				 unsigned long vm_flags, bool smaps, bool in_pf,
+				 bool enforce_sysfs, unsigned long orders);
 
  #define transparent_hugepage_use_zero_page()				\
  	(transparent_hugepage_flags &					\
@@ -267,17 +304,18 @@ static inline bool folio_test_pmd_mappable(struct folio *folio)
  	return false;
  }
 
-static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
-		unsigned long addr)
+static inline unsigned long transhuge_vma_suitable(struct vm_area_struct *vma,
+		unsigned long addr, unsigned long orders)
  {
-	return false;
+	return 0;
  }
 
-static inline bool hugepage_vma_check(struct vm_area_struct *vma,
-				      unsigned long vm_flags, bool smaps,
-				      bool in_pf, bool enforce_sysfs)
+static inline unsigned long hugepage_vma_check(struct vm_area_struct *vma,
+					unsigned long vm_flags, bool smaps,
+					bool in_pf, bool enforce_sysfs,
+					unsigned long orders)
  {
-	return false;
+	return 0;
  }
 
  static inline void folio_prep_large_rmappable(struct folio *folio) {}
mm/huge_memory.c +3 -1
index 6eb55f97a3d2..0b545d56420c 100644
@@ -74,12 +74,60 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
  static atomic_t huge_zero_refcount;
  struct page *huge_zero_page __read_mostly;
  unsigned long huge_zero_pfn __read_mostly = ~0UL;
+static unsigned long huge_anon_orders_always __read_mostly;
+static unsigned long huge_anon_orders_madvise __read_mostly;
+static unsigned long huge_anon_orders_global __read_mostly;
 
-bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,

mm/huge_memory.c static unsigned long deferred_split_scan(struct shrinker *shrink, · line 89

Quoted 10 lines
mm/huge_memory.c +9 -1
-			bool smaps, bool in_pf, bool enforce_sysfs)
+#define hugepage_global_enabled()			\
+	(transparent_hugepage_flags &			\
+	 ((1<<TRANSPARENT_HUGEPAGE_FLAG) |		\
+	  (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)))
+
+#define hugepage_global_always()			\
+	(transparent_hugepage_flags &			\
+	 (1<<TRANSPARENT_HUGEPAGE_FLAG))
+

Here again, I'd like to request that we avoid macros, as I don't think they are strictly required.

Quoted 22 lines
mm/huge_memory.c +21 -0
+bool hugepage_flags_enabled(void)
  {
+	/*
+	 * We cover both the anon and the file-backed case here; we must return
+	 * true if globally enabled, even when all anon sizes are set to never.
+	 * So we don't need to look at huge_anon_orders_global.
+	 */
+	return hugepage_global_enabled() ||
+	       huge_anon_orders_always ||
+	       huge_anon_orders_madvise;
+}
+
+/**
+ * hugepage_vma_check - determine which hugepage orders can be applied to vma
+ * @vma:  the vm area to check
+ * @vm_flags: use these vm_flags instead of vma->vm_flags
+ * @smaps: whether answer will be used for smaps file
+ * @in_pf: whether answer will be used by page fault handler
+ * @enforce_sysfs: whether sysfs config should be taken into account
+ * @orders: bitfield of all orders to consider
+ *
+ * Calculates the intersection of the requested hugepage orders and the allowed

mm/huge_memory.c static unsigned long deferred_split_scan(struct shrinker *shrink, · line 121

Quoted 10 lines
mm/huge_memory.c +10 -0
+ * hugepage orders for the provided vma. Permitted orders are encoded as a set
+ * bit at the corresponding bit position (bit-2 corresponds to order-2, bit-3
+ * corresponds to order-3, etc). Order-0 is never considered a hugepage order.
+ *
+ * Return: bitfield of orders allowed for hugepage in the vma. 0 if no hugepage
+ * orders are allowed.
+ */
+unsigned long hugepage_vma_check(struct vm_area_struct *vma,
+				 unsigned long vm_flags, bool smaps, bool in_pf,
+				 bool enforce_sysfs, unsigned long orders)

Here again, a bool return type has been changed to a bitfield. Let's also update the function name. Maybe one of these:

hugepage_vma_orders()
hugepage_vma_allowable_orders()

?

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#10 lore raw

Re: [RESEND PATCH v7 03/10] mm: thp: Introduce per-size thp sysfs interface

Quoted 166 lines · John Hubbard
Quoted 73 lines · Ryan Roberts

In preparation for adding support for anonymous small-sized THP, introduce new sysfs structure that will be used to control the new behaviours. A new directory is added under transparent_hugepage for each supported THP size, and contains an enabled file, which can be set to "global" (to inherrit the global setting), "always", "madvise" or "never". For now, the kernel still only supports PMD-sized anonymous THP, so only 1 directory is populated.

The first half of the change converts transhuge_vma_suitable() and hugepage_vma_check() so that they take a bitfield of orders for which the user wants to determine support, and the functions filter out all the orders that can't be supported, given the current sysfs configuration and the VMA dimensions. If there is only 1 order set in the input then the output can continue to be treated like a boolean; this is the case for most call sites.

The second half of the change implements the new sysfs interface. It has been done so that each supported THP size has a struct thpsize, which describes the relevant metadata and is itself a kobject. This is pretty minimal for now, but should make it easy to add new per-thpsize files to the interface if needed in future (e.g. per-size defrag). Rather than keep the enabled state directly in the struct thpsize, I've elected to directly encode it into huge_anon_orders_[always|madvise|global] bitfields since this reduces the amount of work required in transhuge_vma_suitable() which is called for every page fault.

The remainder is copied from Documentation/admin-guide/mm/transhuge.rst, as modified by this commit. See that file for further details.

Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled system wide. This can be achieved per-supported-THP-size with one of::

echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

where <size> is the hugepage size being addressed, the available sizes for which vary by system. Alternatively it is possible to specify that a given hugepage size will inherrit the global enabled setting::

echo global >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

The global (legacy) enabled setting can be set as follows::

echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled

By default, PMD-sized hugepages have enabled="global" and all other hugepage sizes have enabled="never". If enabling multiple hugepage sizes, the kernel will select the most appropriate enabled size for a given allocation.

---
   Documentation/admin-guide/mm/transhuge.rst |  74 ++++--
   Documentation/filesystems/proc.rst         |   6 +-
   fs/proc/task_mmu.c                         |   3 +-
   include/linux/huge_mm.h                    | 100 +++++---
   mm/huge_memory.c                           | 263 +++++++++++++++++++--
   mm/khugepaged.c                            |  16 +-
   mm/memory.c                                |   6 +-
   mm/page_vma_mapped.c                       |   3 +-
   8 files changed, 387 insertions(+), 84 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +0 -0
index b0cc8243e093..52565e0bd074 100644

Documentation/admin-guide/mm/transhuge.rst components: · line 49

Quoted 5 lines
Documentation/admin-guide/mm/transhuge.rst +2 -0
@@ -45,10 +45,23 @@ components:
      the two is using hugepages just because of the fact the TLB miss is
      going to run faster.
 
+As well as PMD-sized THP described above, it is also possible to
+configure the system to allocate "small-sized THP" to back anonymous

Here's one of the places to change to the new name, which lately is "multi-size THP", or mTHP or m_thp for short. (I've typed "multi-size" instead of "multi-sized", because the 'd' doesn't add significantly to the meaning, and if in doubt, shorter is better.

Documentation/admin-guide/mm/transhuge.rst components: · line 60

Quoted 11 lines
Documentation/admin-guide/mm/transhuge.rst +11 -0
+memory (for example 16K, 32K, 64K, etc). These THPs continue to be
+PTE-mapped, but in many cases can still provide similar benefits to
+those outlined above: Page faults are significantly reduced (by a
+factor of e.g. 4, 8, 16, etc), but latency spikes are much less
+prominent because the size of each page isn't as huge as the PMD-sized
+variant and there is less memory to clear in each page fault. Some
+architectures also employ TLB compression mechanisms to squeeze more
+entries in when a set of PTEs are virtually and physically contiguous
+and approporiately aligned. In this case, TLB misses will occur less
+often.
+

OK, all of the above still seems like it can remain the same.

Quoted 12 lines
Documentation/admin-guide/mm/transhuge.rst +1 -1
   THP can be enabled system wide or restricted to certain tasks or even
   memory ranges inside task's address space. Unless THP is completely
   disabled, there is ``khugepaged`` daemon that scans memory and
-collapses sequences of basic pages into huge pages.
+collapses sequences of basic pages into PMD-sized huge pages.
 
   The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
   interface and using madvise(2) and prctl(2) system calls.
@@ -95,12 +108,29 @@ Global THP controls
   Transparent Hugepage Support for anonymous memory can be entirely disabled
   (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE
   regions (to avoid the risk of consuming more memory resources) or enabled

Documentation/admin-guide/mm/transhuge.rst Global THP controls · line 119

Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +9 -1
-system wide. This can be achieved with one of::
+system wide. This can be achieved per-supported-THP-size with one of::
+
+	echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+	echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+	echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+where <size> is the hugepage size being addressed, the available sizes
+for which vary by system. Alternatively it is possible to specify that
+a given hugepage size will inherrit the global enabled setting::

typo: inherrit

Quoted 4 lines
Documentation/admin-guide/mm/transhuge.rst +4 -0
+
+	echo global >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+The global (legacy) enabled setting can be set as follows::

Documentation/admin-guide/mm/transhuge.rst Global THP controls · line 133

Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +5 -0
 
   	echo always >/sys/kernel/mm/transparent_hugepage/enabled
   	echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
   	echo never >/sys/kernel/mm/transparent_hugepage/enabled
 
+By default, PMD-sized hugepages have enabled="global" and all other
+hugepage sizes have enabled="never". If enabling multiple hugepage
+sizes, the kernel will select the most appropriate enabled size for a
+given allocation.
+

This is slightly murky. I wonder if "inherited" is a little more directly informative than global; it certainly felt that way my first time running this and poking at it.

And a few trivial examples would be a nice touch.

And so overall with a few other minor tweaks, I'd suggest this:

...
where <size> is the hugepage size being addressed, the available sizes for which vary by system.

For example:
echo always >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled

Alternatively it is possible to specify that a given hugepage size will inherit the top-level "enabled" value:

echo inherited >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

For example:
echo inherited >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled

The top-level setting (for use with "inherited") can be by issuing one of the following commands::

Quoted 6 lines

echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled

By default, PMD-sized hugepages have enabled="inherited" and all other hugepage sizes have enabled="never".

"inherited" works for me as well.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#11 lore raw

Re: [RESEND PATCH v7 03/10] mm: thp: Introduce per-size thp sysfs interface

In reply to John Hubbard
Quoted 78 lines · John Hubbard
Quoted 77 lines · Ryan Roberts

In preparation for adding support for anonymous small-sized THP, introduce new sysfs structure that will be used to control the new behaviours. A new directory is added under transparent_hugepage for each supported THP size, and contains an enabled file, which can be set to "global" (to inherrit the global setting), "always", "madvise" or "never". For now, the kernel still only supports PMD-sized anonymous THP, so only 1 directory is populated.

The first half of the change converts transhuge_vma_suitable() and hugepage_vma_check() so that they take a bitfield of orders for which the user wants to determine support, and the functions filter out all the orders that can't be supported, given the current sysfs configuration and the VMA dimensions. If there is only 1 order set in the input then the output can continue to be treated like a boolean; this is the case for most call sites.

The second half of the change implements the new sysfs interface. It has been done so that each supported THP size has a struct thpsize, which describes the relevant metadata and is itself a kobject. This is pretty minimal for now, but should make it easy to add new per-thpsize files to the interface if needed in future (e.g. per-size defrag). Rather than keep the enabled state directly in the struct thpsize, I've elected to directly encode it into huge_anon_orders_[always|madvise|global] bitfields since this reduces the amount of work required in transhuge_vma_suitable() which is called for every page fault.

The remainder is copied from Documentation/admin-guide/mm/transhuge.rst, as modified by this commit. See that file for further details.

Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled system wide. This can be achieved per-supported-THP-size with one of::

    echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

where <size> is the hugepage size being addressed, the available sizes for which vary by system. Alternatively it is possible to specify that a given hugepage size will inherrit the global enabled setting::

    echo global >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

The global (legacy) enabled setting can be set as follows::

    echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled

By default, PMD-sized hugepages have enabled="global" and all other hugepage sizes have enabled="never". If enabling multiple hugepage sizes, the kernel will select the most appropriate enabled size for a given allocation.

---
  Documentation/admin-guide/mm/transhuge.rst |  74 ++++-- Documentation/filesystems/proc.rst         |   6 +- fs/proc/task_mmu.c                         |   3 +- include/linux/huge_mm.h                    | 100 +++++--- mm/huge_memory.c                           | 263 +++++++++++++++++++-- mm/khugepaged.c                            |  16 +- mm/memory.c                                |   6 +- mm/page_vma_mapped.c                       |   3 +-

  8 files changed, 387 insertions(+), 84 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +0 -0
index b0cc8243e093..52565e0bd074 100644
@@ -45,10 +45,23 @@ components:

     the two is using hugepages just because of the fact the TLB miss is going to run faster.

Quoted 7 lines

Documentation/admin-guide/mm/transhuge.rst components: · line 49

Quoted 2 lines
Documentation/admin-guide/mm/transhuge.rst +2 -0
+As well as PMD-sized THP described above, it is also possible to
+configure the system to allocate "small-sized THP" to back anonymous

Here's one of the places to change to the new name, which lately is "multi-size THP", or mTHP or m_thp for short. (I've typed "multi-size" instead of "multi-sized", because the 'd' doesn't add significantly to the meaning, and if in doubt, shorter is better.

I was thinking of some light changes to the start of this paragraph, something like:

Modern kernels support "multi-size THP" (mTHP), which introduces the ability to allocate memory in blocks that are bigger than a base page but smaller than traditional PMD-size (as described above, in increments of a power-of-2 number of pages. mTHP can back anonymous

Quoted 32 lines

Documentation/admin-guide/mm/transhuge.rst components: · line 60

Quoted 13 lines
Documentation/admin-guide/mm/transhuge.rst +11 -0
+memory (for example 16K, 32K, 64K, etc). These THPs continue to be
+PTE-mapped, but in many cases can still provide similar benefits to
+those outlined above: Page faults are significantly reduced (by a
+factor of e.g. 4, 8, 16, etc), but latency spikes are much less
+prominent because the size of each page isn't as huge as the PMD-sized
+variant and there is less memory to clear in each page fault. Some
+architectures also employ TLB compression mechanisms to squeeze more
+entries in when a set of PTEs are virtually and physically contiguous
+and approporiately aligned. In this case, TLB misses will occur less
+often.
+

OK, all of the above still seems like it can remain the same.

Quoted 16 lines

  THP can be enabled system wide or restricted to certain tasks or even memory ranges inside task's address space. Unless THP is completely disabled, there is ``khugepaged`` daemon that scans memory and

Documentation/admin-guide/mm/transhuge.rst +1 -1
-collapses sequences of basic pages into huge pages.
+collapses sequences of basic pages into PMD-sized huge pages.

  The THP behaviour is controlled via :ref:sysfs <thp_sysfs> interface and using madvise(2) and prctl(2) system calls.

Documentation/admin-guide/mm/transhuge.rst +0 -0
@@ -95,12 +108,29 @@ Global THP controls

  Transparent Hugepage Support for anonymous memory can be entirely disabled (mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE regions (to avoid the risk of consuming more memory resources) or enabled

Documentation/admin-guide/mm/transhuge.rst +3 -1
-system wide. This can be achieved with one of::
+system wide. This can be achieved per-supported-THP-size with one of::
+
+    echo always >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
Quoted 8 lines

Documentation/admin-guide/mm/transhuge.rst Global THP controls · line 119

Quoted 6 lines

+    echo madvise >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled +    echo never >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

Documentation/admin-guide/mm/transhuge.rst +4 -0
+
+where <size> is the hugepage size being addressed, the available sizes
+for which vary by system. Alternatively it is possible to specify that
+a given hugepage size will inherrit the global enabled setting::

typo: inherrit

ACK, will fix

Quoted 42 lines
Quoted 10 lines
Documentation/admin-guide/mm/transhuge.rst +4 -0
+
+    echo global >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled
+
+The global (legacy) enabled setting can be set as follows::

      echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled

Documentation/admin-guide/mm/transhuge.rst Global THP controls · line 133

Quoted 5 lines
Documentation/admin-guide/mm/transhuge.rst +5 -0
+By default, PMD-sized hugepages have enabled="global" and all other
+hugepage sizes have enabled="never". If enabling multiple hugepage
+sizes, the kernel will select the most appropriate enabled size for a
+given allocation.
+

This is slightly murky. I wonder if "inherited" is a little more directly informative than global; it certainly felt that way my first time running this and poking at it.

And a few trivial examples would be a nice touch.

And so overall with a few other minor tweaks, I'd suggest this:

...
where <size> is the hugepage size being addressed, the available sizes for which vary by system.

For example:
    echo always >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled

Alternatively it is possible to specify that a given hugepage size will inherit the top-level "enabled" value:

    echo inherited >/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/enabled

For example:
    echo inherited >/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled

The top-level setting (for use with "inherited") can be by issuing one of the following commands::

Quoted 6 lines

    echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled

By default, PMD-sized hugepages have enabled="inherited" and all other hugepage sizes have enabled="never".

That all sounds good; will update.

I wonder if "inherit" is even better than "inherited" though?

Quoted 37 lines
Quoted 37 lines

  It's also possible to limit defrag efforts in the VM to generate anonymous hugepages in case they're not immediately free to madvise regions or to never try to defrag memory and simply fallback to regular

Documentation/admin-guide/mm/transhuge.rst +0 -0
@@ -146,25 +176,34 @@ madvise

  never
      should be self-explanatory.

Documentation/admin-guide/mm/transhuge.rst +3 -3
-By default kernel tries to use huge zero page on read page fault to
-anonymous mapping. It's possible to disable huge zero page by writing 0
-or enable it back by writing 1::
+By default kernel tries to use huge, PMD-mappable zero page on read
+page fault to anonymous mapping. It's possible to disable huge zero
+page by writing 0 or enable it back by writing 1::

      echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page

Documentation/admin-guide/mm/transhuge.rst +3 -2
-Some userspace (such as a test program, or an optimized memory allocation
-library) may want to know the size (in bytes) of a transparent hugepage::
+Some userspace (such as a test program, or an optimized memory
+allocation library) may want to know the size (in bytes) of a
+PMD-mappable transparent hugepage::

      cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size

Documentation/admin-guide/mm/transhuge.rst +6 -3
-khugepaged will be automatically started when
-transparent_hugepage/enabled is set to "always" or "madvise, and it'll
-be automatically shutdown if it's set to "never".
+khugepaged will be automatically started when one or more hugepage
+sizes are enabled (either by directly setting "always" or "madvise",
+or by setting "global" while the global enabled is set to "always" or
+"madvise"), and it'll be automatically shutdown when the last hugepage
+size is disabled (either by directly setting "never", or by setting
+"global" while the global enabled is set to "never").
Quoted 8 lines
Quoted 8 lines

  Khugepaged controls
  -------------------

+.. note::
+   khugepaged currently only searches for opportunities to collapse to +   PMD-sized THP and no attempt is made to collapse to small-sized +   THP.
+

"small-sized THP" used here too; I propose to change it to "... collapse to other THP sizes."

Quoted 23 lines
Quoted 23 lines

  khugepaged runs usually at low frequency so while one may not want to invoke defrag algorithms synchronously during the page faults, it should be worth invoking defrag at least in khugepaged. However it's

Documentation/admin-guide/mm/transhuge.rst +0 -0
@@ -282,10 +321,11 @@ force

  Need of application restart
  ===========================

Documentation/admin-guide/mm/transhuge.rst +5 -4
-The transparent_hugepage/enabled values and tmpfs mount option only affect
-future behavior. So to make them effective you need to restart any
-application that could have been using hugepages. This also applies to the
-regions registered in khugepaged.
+The transparent_hugepage/enabled and
+transparent_hugepage/hugepages-<size>kB/enabled values and tmpfs mount
+option only affect future behavior. So to make them effective you need
+to restart any application that could have been using hugepages. This
+also applies to the regions registered in khugepaged.

  Monitoring usage
  ================

Documentation/admin-guide/mm/transhuge.rst +0 -0
@@ -308,6 +348,10 @@ frequently will incur overhead.

  There are a number of counters in ``/proc/vmstat`` that may be used to monitor how successfully the system is providing huge pages for use.

Quoted 5 lines
Quoted 3 lines

+.. note::
+   Currently the below counters only record events relating to +   PMD-sized THP. Events relating to small-sized THP are not included.

Here's another spot to rename to "multi-size THP".

I propose to change it to "Events relating to other THP sizes..."

I'm also going to move this note to just under the "Monitoring Usage" heading since its current position doesn't make it clear that it includes "AnonHugePages". I'll also make it clear in the text that mentions AnonHugePages counter that it "only applies to traditional PMD-sized THP for historical reasons" and that "AnonHugePages should have been called AnonHugePmdMapped" as David suggested in the other thread.

Quoted 24 lines
Quoted 24 lines

+
  thp_fault_alloc
      is incremented every time a huge page is successfully allocated to handle a page fault.

Documentation/admin-guide/mm/transhuge.rst +0 -0
@@ -413,7 +457,7 @@ for huge pages.

  Optimizing the applications
  ===========================

Documentation/admin-guide/mm/transhuge.rst +1 -1
-To be guaranteed that the kernel will map a 2M page immediately in any
+To be guaranteed that the kernel will map a THP immediately in any

  memory region, the mmap region has to be hugepage naturally aligned. posix_memalign() can provide that guarantee.

Documentation/filesystems/proc.rst +0 -0
index 49ef12df631b..f8e8dd1fd148 100644
@@ -528,9 +528,9 @@ replaced by copy-on-write) part of the underlying shmem

object out on swap.
  does not take into account swapped out page of underlying shmem objects. "Locked" indicates whether the mapping is locked in memory or not.

Quoted 9 lines

Documentation/filesystems/proc.rst replaced by copy-on-write) part of the underlying shmem object out on swap. · line 533

Quoted 6 lines

-"THPeligible" indicates whether the mapping is eligible for allocating THP

Documentation/filesystems/proc.rst +3 -2
-pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise.
-It just shows the current status.
+"THPeligible" indicates whether the mapping is eligible for allocating
+naturally aligned THP pages of any currently enabled size. 1 if true, 0
+otherwise. It just shows the current status.

"It just shows the current status"...as opposed to what? I'm missing an educational opportunity here--not sure what this means. :)

I have no idea either - it seems superfluous. But that sentence was there already. I can remove it if you like?

Quoted 26 lines
Quoted 26 lines

  "VmFlags" field deserves a separate description. This member represents the kernel flags associated with the particular virtual memory area in two letter

fs/proc/task_mmu.c +2 -1
index 51e0ec658457..2e25362ca9fa 100644
@@ -865,7 +865,8 @@ static int show_smap(struct seq_file *m, void *v)
       __show_smap(m, &mss, false);
 
       seq_printf(m, "THPeligible:    %8u\n",
-           hugepage_vma_check(vma, vma->vm_flags, true, false, true));
+           !!hugepage_vma_check(vma, vma->vm_flags, true, false, true,
+                    THP_ORDERS_ALL));
 
       if (arch_pkeys_enabled())
           seq_printf(m, "ProtectionKey:  %8u\n", vma_pkey(vma));
include/linux/huge_mm.h +0 -0
index fa0350b0812a..7d6f7d96b039 100644
@@ -67,6 +67,21 @@ extern struct kobj_attribute shmem_enabled_attr;
   #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)
   #define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)
 
Quoted 12 lines

include/linux/huge_mm.h extern struct kobj_attribute shmem_enabled_attr; · line 78

Quoted 9 lines
include/linux/huge_mm.h +9 -0
+/*
+ * Mask of all large folio orders supported for anonymous THP.
+ */
+#define THP_ORDERS_ALL_ANON    BIT(PMD_ORDER)
+
+/*
+ * Mask of all large folio orders supported for file THP.
+ */
+#define THP_ORDERS_ALL_FILE    (BIT(PMD_ORDER) | BIT(PUD_ORDER))

Is there something about file THP that implies PUD_ORDER? This deserves an explanatory comment, I think.

Sorry, I'm not really sure what you are asking? Today's kernel supports PUD-mapping file-backed memory (mostly DAX I believe). I'm not sure what comment you want me to add, other than "file-backed memory can support PUD-mapping", but that's self-evident from the define, isn't it?

Quoted 14 lines
Quoted 14 lines
include/linux/huge_mm.h +6 -0
+
+/*
+ * Mask of all large folio orders supported for THP.
+ */
+#define THP_ORDERS_ALL        (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE)
+
   #ifdef CONFIG_TRANSPARENT_HUGEPAGE
   #define HPAGE_PMD_SHIFT PMD_SHIFT
   #define HPAGE_PMD_SIZE    ((1UL) << HPAGE_PMD_SHIFT)
@@ -78,13 +93,18 @@ extern struct kobj_attribute shmem_enabled_attr;
 
   extern unsigned long transparent_hugepage_flags;
 
Quoted 9 lines

include/linux/huge_mm.h extern struct kobj_attribute shmem_enabled_attr; · line 87

Quoted 7 lines
include/linux/huge_mm.h +0 -7
-#define hugepage_flags_enabled()                           \
-    (transparent_hugepage_flags &                       \
-     ((1<<TRANSPARENT_HUGEPAGE_FLAG) |               \
-      (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)))
-#define hugepage_flags_always()                \
-    (transparent_hugepage_flags &            \
-     (1<<TRANSPARENT_HUGEPAGE_FLAG))

Are macros actually required? If not, static inlines would be nicer.

I agree static inlines are nicer. Here I'm removing existing macros though, and replacing with a function prototype (hugepage_flags_enabled()), which is implemented in huge_memory.c; this avoids having to export a bunch of global variables from that file if it were to be static inline.

Quoted 37 lines
Quoted 22 lines
include/linux/huge_mm.h +12 -2
+bool hugepage_flags_enabled(void);
+
+static inline int first_order(unsigned long orders)
+{
+    return fls_long(orders) - 1;
+}
+
+static inline int next_order(unsigned long *orders, int prev)
+{
+    *orders &= ~BIT(prev);
+    return first_order(*orders);
+}
 
   /*
    * Do the below checks:
@@ -97,23 +117,39 @@ extern unsigned long transparent_hugepage_flags;
    *   - For all vmas, check if the haddr is in an aligned HPAGE_PMD_SIZE
    *     area.
    */
-static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
-        unsigned long addr)

include/linux/huge_mm.h extern unsigned long transparent_hugepage_flags; · line 121

Quoted 10 lines
include/linux/huge_mm.h +2 -8
-{
-    unsigned long haddr;
-
-    /* Don't have to check pgoff for anonymous vma */
-    if (!vma_is_anonymous(vma)) {
-        if (!IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) - vma->vm_pgoff,
-                HPAGE_PMD_NR))
-            return false;
+static inline unsigned long transhuge_vma_suitable(struct vm_area_struct *vma,
+        unsigned long addr, unsigned long orders)

Changing this from a bool to a mask of orders is quite significant, and both the function name and the function-level comment documentation need to also be adjusted. Here, perhaps one of these names would work:

Quoted 1 line

        transhuge_vma_suitable_orders()

This is my preference, but its getting a bit long. How about:

thp_vma_suitable_orders()

I could then call the other one:

thp_vma_allowable_orders()

So we have a clearly related pair?

Quoted 15 lines

        transhuge_vma_orders()>

include/linux/huge_mm.h extern unsigned long transparent_hugepage_flags; · line 129

Quoted 8 lines
include/linux/huge_mm.h +8 -0
+{
+    int order;
+
+    /*
+     * Iterate over orders, highest to lowest, removing orders that don't
+     * meet alignment requirements from the set. Exit loop at first order
+     * that meets requirements, since all lower orders must also meet
+     * requirements.

Should we add a little note here, to the effect of, "this is unilaterally taking over a certain amount of allocation-like policy, by deciding how to search for folios of a certain size"?

Quoted 3 lines

Or is this The Only Way To Do It, after all? I know we had some discussion about it, and intuitively it feels reasonable, but wanted to poke at it one last time anyway.

This function isn't trying to implement policy, its just filtering out orders that don't fit and therefore definitely cannot be used. The result is a set of orders the could be used and its the policy maker's job to decide which one. Currently that policy is "biggest one that fits && does not overlap other non-none ptes && folio successfully allocated". That's implemented in the next patch and would potentially be swapped out in the future for something more exotic/optimal.

So I don't think we need any extra comments here.

Quoted 85 lines
Quoted 85 lines
include/linux/huge_mm.h +24 -7
+     */
+
+    order = first_order(orders);
+
+    while (orders) {
+        unsigned long hpage_size = PAGE_SIZE << order;
+        unsigned long haddr = ALIGN_DOWN(addr, hpage_size);
+
+        if (haddr >= vma->vm_start &&
+            haddr + hpage_size <= vma->vm_end) {
+            if (!vma_is_anonymous(vma)) {
+                if (IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) -
+                        vma->vm_pgoff,
+                        hpage_size >> PAGE_SHIFT))
+                    break;
+            } else
+                break;
+        }
+
+        order = next_order(&orders, order);
       }
 
-    haddr = addr & HPAGE_PMD_MASK;
-
-    if (haddr < vma->vm_start || haddr + HPAGE_PMD_SIZE > vma->vm_end)
-        return false;
-    return true;
+    return orders;
   }
 
   static inline bool file_thp_enabled(struct vm_area_struct *vma)
@@ -130,8 +166,9 @@ static inline bool file_thp_enabled(struct vm_area_struct
 *vma)
              !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
   }
 
-bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,
-            bool smaps, bool in_pf, bool enforce_sysfs);
+unsigned long hugepage_vma_check(struct vm_area_struct *vma,
+                 unsigned long vm_flags, bool smaps, bool in_pf,
+                 bool enforce_sysfs, unsigned long orders);
 
   #define transparent_hugepage_use_zero_page()                \

      (transparent_hugepage_flags &                    \

include/linux/huge_mm.h +8 -7
@@ -267,17 +304,18 @@ static inline bool folio_test_pmd_mappable(struct folio
 *folio)
       return false;
   }
 
-static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
-        unsigned long addr)
+static inline unsigned long transhuge_vma_suitable(struct vm_area_struct *vma,
+        unsigned long addr, unsigned long orders)
   {
-    return false;
+    return 0;
   }
 
-static inline bool hugepage_vma_check(struct vm_area_struct *vma,
-                      unsigned long vm_flags, bool smaps,
-                      bool in_pf, bool enforce_sysfs)
+static inline unsigned long hugepage_vma_check(struct vm_area_struct *vma,
+                    unsigned long vm_flags, bool smaps,
+                    bool in_pf, bool enforce_sysfs,
+                    unsigned long orders)
   {
-    return false;
+    return 0;
   }
 

  static inline void folio_prep_large_rmappable(struct folio *folio) {}

mm/huge_memory.c +3 -0
index 6eb55f97a3d2..0b545d56420c 100644
@@ -74,12 +74,60 @@ static unsigned long deferred_split_scan(struct shrinker
 *shrink,
   static atomic_t huge_zero_refcount;
   struct page *huge_zero_page __read_mostly;
   unsigned long huge_zero_pfn __read_mostly = ~0UL;
+static unsigned long huge_anon_orders_always __read_mostly;
+static unsigned long huge_anon_orders_madvise __read_mostly;
+static unsigned long huge_anon_orders_global __read_mostly;
 
Quoted 14 lines

mm/huge_memory.c static unsigned long deferred_split_scan(struct shrinker *shrink, · line 89

Quoted 11 lines
mm/huge_memory.c +9 -2
-bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,
-            bool smaps, bool in_pf, bool enforce_sysfs)
+#define hugepage_global_enabled()            \
+    (transparent_hugepage_flags &            \
+     ((1<<TRANSPARENT_HUGEPAGE_FLAG) |        \
+      (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)))
+
+#define hugepage_global_always()            \
+    (transparent_hugepage_flags &            \
+     (1<<TRANSPARENT_HUGEPAGE_FLAG))
+

Here again, I'd like to request that we avoid macros, as I don't think they are strictly required.

Yeah I agree. I did them this way, because they already existed and I was just moving them from the header to here. But I'll change to static inline.

Quoted 37 lines
Quoted 23 lines
mm/huge_memory.c +21 -0
+bool hugepage_flags_enabled(void)
   {
+    /*
+     * We cover both the anon and the file-backed case here; we must return
+     * true if globally enabled, even when all anon sizes are set to never.
+     * So we don't need to look at huge_anon_orders_global.
+     */
+    return hugepage_global_enabled() ||
+           huge_anon_orders_always ||
+           huge_anon_orders_madvise;
+}
+
+/**
+ * hugepage_vma_check - determine which hugepage orders can be applied to vma
+ * @vma:  the vm area to check
+ * @vm_flags: use these vm_flags instead of vma->vm_flags
+ * @smaps: whether answer will be used for smaps file
+ * @in_pf: whether answer will be used by page fault handler
+ * @enforce_sysfs: whether sysfs config should be taken into account
+ * @orders: bitfield of all orders to consider
+ *
+ * Calculates the intersection of the requested hugepage orders and the allowed

mm/huge_memory.c static unsigned long deferred_split_scan(struct shrinker *shrink, · line 121

Quoted 10 lines
mm/huge_memory.c +10 -0
+ * hugepage orders for the provided vma. Permitted orders are encoded as a set
+ * bit at the corresponding bit position (bit-2 corresponds to order-2, bit-3
+ * corresponds to order-3, etc). Order-0 is never considered a hugepage order.
+ *
+ * Return: bitfield of orders allowed for hugepage in the vma. 0 if no hugepage
+ * orders are allowed.
+ */
+unsigned long hugepage_vma_check(struct vm_area_struct *vma,
+                 unsigned long vm_flags, bool smaps, bool in_pf,
+                 bool enforce_sysfs, unsigned long orders)

Here again, a bool return type has been changed to a bitfield. Let's also update the function name. Maybe one of these:

Quoted 2 lines

        hugepage_vma_orders()
        hugepage_vma_allowable_orders()

thp_vma_allowable_orders()?

Thanks for the review!

Quoted 5 lines

?

thanks,

#12 lore raw

Re: [RESEND PATCH v7 03/10] mm: thp: Introduce per-size thp sysfs interface

Quoted 2 lines · Ryan Roberts
Quoted 1 line · John Hubbard

On 11/22/23 08:29, Ryan Roberts wrote:

...

Quoted 8 lines
Quoted 7 lines

Documentation/admin-guide/mm/transhuge.rst components: · line 49

Quoted 2 lines
Documentation/admin-guide/mm/transhuge.rst +2 -0
+As well as PMD-sized THP described above, it is also possible to
+configure the system to allocate "small-sized THP" to back anonymous

Here's one of the places to change to the new name, which lately is "multi-size THP", or mTHP or m_thp for short. (I've typed "multi-size" instead of "multi-sized", because the 'd' doesn't add significantly to the meaning, and if in doubt, shorter is better.

Quoted 7 lines

I was thinking of some light changes to the start of this paragraph, something like:

Modern kernels support "multi-size THP" (mTHP), which introduces the ability to allocate memory in blocks that are bigger than a base page but smaller than traditional PMD-size (as described above, in increments of a power-of-2 number of pages. mTHP can back anonymous

Very nice.

...

Quoted 6 lines
Quoted 2 lines

By default, PMD-sized hugepages have enabled="inherited" and all other hugepage sizes have enabled="never".

That all sounds good; will update.

I wonder if "inherit" is even better than "inherited" though?

Yes, I think so. "inherit" was actually my first idea, and after thinking about it, I just made it worse by adding the "ed". haha. :)

...

Quoted 11 lines
Quoted 8 lines
Quoted 8 lines

  Khugepaged controls
  -------------------

+.. note::
+   khugepaged currently only searches for opportunities to collapse to +   PMD-sized THP and no attempt is made to collapse to small-sized +   THP.
+

"small-sized THP" used here too; I propose to change it to "... collapse to other THP sizes."

OK, looks good.

Quoted 30 lines
Quoted 24 lines
Quoted 24 lines

  khugepaged runs usually at low frequency so while one may not want to invoke defrag algorithms synchronously during the page faults, it should be worth invoking defrag at least in khugepaged. However it's

Documentation/admin-guide/mm/transhuge.rst +5 -4
@@ -282,10 +321,11 @@ force
   Need of application restart
   ===========================
 
-The transparent_hugepage/enabled values and tmpfs mount option only affect
-future behavior. So to make them effective you need to restart any
-application that could have been using hugepages. This also applies to the
-regions registered in khugepaged.
+The transparent_hugepage/enabled and
+transparent_hugepage/hugepages-<size>kB/enabled values and tmpfs mount
+option only affect future behavior. So to make them effective you need
+to restart any application that could have been using hugepages. This
+also applies to the regions registered in khugepaged.
 
   Monitoring usage
   ================
@@ -308,6 +348,10 @@ frequently will incur overhead.
   There are a number of counters in ``/proc/vmstat`` that may be used to
   monitor how successfully the system is providing huge pages for use.
 
Quoted 5 lines

Documentation/admin-guide/mm/transhuge.rst frequently will incur overhead. · line 353

Quoted 3 lines
Documentation/admin-guide/mm/transhuge.rst +3 -0
+.. note::
+   Currently the below counters only record events relating to
+   PMD-sized THP. Events relating to small-sized THP are not included.

Here's another spot to rename to "multi-size THP".

Quoted 8 lines

I propose to change it to "Events relating to other THP sizes..."

I'm also going to move this note to just under the "Monitoring Usage" heading since its current position doesn't make it clear that it includes "AnonHugePages". I'll also make it clear in the text that mentions AnonHugePages counter that it "only applies to traditional PMD-sized THP for historical reasons" and that "AnonHugePages should have been called AnonHugePmdMapped" as David suggested in the other thread.

OK.

Are we entirely dropping the AnonHugePtePages that was there in v6? I'm looking for some way to monitor this. I may have missed it because I haven't gone through all of v7 yet.

...

Quoted 10 lines
Quoted 9 lines

Documentation/filesystems/proc.rst replaced by copy-on-write) part of the underlying shmem object out on swap. · line 533

Quoted 6 lines
Documentation/filesystems/proc.rst +3 -3
-"THPeligible" indicates whether the mapping is eligible for allocating THP
-pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise.
-It just shows the current status.
+"THPeligible" indicates whether the mapping is eligible for allocating
+naturally aligned THP pages of any currently enabled size. 1 if true, 0
+otherwise. It just shows the current status.

"It just shows the current status"...as opposed to what? I'm missing an educational opportunity here--not sure what this means. :)

Quoted 3 lines

I have no idea either - it seems superfluous. But that sentence was there already. I can remove it if you like?

Yes, let's remove it, please.

...

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
include/linux/huge_mm.h +4 -0
+/*
+ * Mask of all large folio orders supported for anonymous THP.
+ */
+#define THP_ORDERS_ALL_ANON    BIT(PMD_ORDER)
Quoted 8 lines

include/linux/huge_mm.h extern struct kobj_attribute shmem_enabled_attr; · line 78

Quoted 5 lines
include/linux/huge_mm.h +5 -0
+
+/*
+ * Mask of all large folio orders supported for file THP.
+ */
+#define THP_ORDERS_ALL_FILE    (BIT(PMD_ORDER) | BIT(PUD_ORDER))

Is there something about file THP that implies PUD_ORDER? This deserves an explanatory comment, I think.

Quoted 5 lines

Sorry, I'm not really sure what you are asking? Today's kernel supports PUD-mapping file-backed memory (mostly DAX I believe). I'm not sure what comment you want me to add, other than "file-backed memory can support PUD-mapping", but that's self-evident from the define, isn't it?

Well, it's sort of evident, but confusing to me anyway, because it combines THP and PUD. Which seems to imply that we have PUD-based THPs supported, but only for file-backed mappings. Which is weird and needs some explanation, right?

Quoted 24 lines
Quoted 14 lines
Quoted 14 lines
include/linux/huge_mm.h +6 -0
+
+/*
+ * Mask of all large folio orders supported for THP.
+ */
+#define THP_ORDERS_ALL        (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE)
+
   #ifdef CONFIG_TRANSPARENT_HUGEPAGE
   #define HPAGE_PMD_SHIFT PMD_SHIFT
   #define HPAGE_PMD_SIZE    ((1UL) << HPAGE_PMD_SHIFT)
@@ -78,13 +93,18 @@ extern struct kobj_attribute shmem_enabled_attr;
 
   extern unsigned long transparent_hugepage_flags;
 
Quoted 9 lines

include/linux/huge_mm.h extern struct kobj_attribute shmem_enabled_attr; · line 87

Quoted 7 lines
include/linux/huge_mm.h +0 -7
-#define hugepage_flags_enabled()                           \
-    (transparent_hugepage_flags &                       \
-     ((1<<TRANSPARENT_HUGEPAGE_FLAG) |               \
-      (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)))
-#define hugepage_flags_always()                \
-    (transparent_hugepage_flags &            \
-     (1<<TRANSPARENT_HUGEPAGE_FLAG))

Are macros actually required? If not, static inlines would be nicer.

Quoted 1 line

I agree static inlines are nicer. Here I'm removing existing macros though, and

Oh, I see that I replied in the wrong part of the email, since that's the *removal* part. oops. :)

...

Quoted 21 lines
Quoted 17 lines

include/linux/huge_mm.h extern unsigned long transparent_hugepage_flags; · line 121

Quoted 12 lines
include/linux/huge_mm.h +2 -10
-static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
-        unsigned long addr)
-{
-    unsigned long haddr;
-
-    /* Don't have to check pgoff for anonymous vma */
-    if (!vma_is_anonymous(vma)) {
-        if (!IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) - vma->vm_pgoff,
-                HPAGE_PMD_NR))
-            return false;
+static inline unsigned long transhuge_vma_suitable(struct vm_area_struct *vma,
+        unsigned long addr, unsigned long orders)

Changing this from a bool to a mask of orders is quite significant, and both the function name and the function-level comment documentation need to also be adjusted. Here, perhaps one of these names would work:

Quoted 1 line

        transhuge_vma_suitable_orders()

This is my preference, but its getting a bit long. How about:

Quoted 7 lines

thp_vma_suitable_orders()

I could then call the other one:

thp_vma_allowable_orders()

So we have a clearly related pair?

Oh yes, that works nicely.

Quoted 29 lines
Quoted 17 lines

        transhuge_vma_orders()>

include/linux/huge_mm.h extern unsigned long transparent_hugepage_flags; · line 129

Quoted 8 lines
include/linux/huge_mm.h +8 -0
+{
+    int order;
+
+    /*
+     * Iterate over orders, highest to lowest, removing orders that don't
+     * meet alignment requirements from the set. Exit loop at first order
+     * that meets requirements, since all lower orders must also meet
+     * requirements.

Should we add a little note here, to the effect of, "this is unilaterally taking over a certain amount of allocation-like policy, by deciding how to search for folios of a certain size"?

Quoted 3 lines

Or is this The Only Way To Do It, after all? I know we had some discussion about it, and intuitively it feels reasonable, but wanted to poke at it one last time anyway.

This function isn't trying to implement policy, its just filtering out orders that don't fit and therefore definitely cannot be used. The result is a set of orders the could be used and its the policy maker's job to decide which one. Currently that policy is "biggest one that fits && does not overlap other non-none ptes && folio successfully allocated". That's implemented in the next patch and would potentially be swapped out in the future for something more exotic/optimal.

Quoted 1 line

So I don't think we need any extra comments here.

Ack.

...

Quoted 55 lines
Quoted 6 lines
Quoted 6 lines
mm/huge_memory.c +4 -2
-bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,
-            bool smaps, bool in_pf, bool enforce_sysfs)
+#define hugepage_global_enabled()            \
+    (transparent_hugepage_flags &            \
+     ((1<<TRANSPARENT_HUGEPAGE_FLAG) |        \
+      (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)))
Quoted 8 lines

mm/huge_memory.c static unsigned long deferred_split_scan(struct shrinker *shrink, · line 89

Quoted 5 lines
mm/huge_memory.c +5 -0
+
+#define hugepage_global_always()            \
+    (transparent_hugepage_flags &            \
+     (1<<TRANSPARENT_HUGEPAGE_FLAG))
+

Here again, I'd like to request that we avoid macros, as I don't think they are strictly required.

Yeah I agree. I did them this way, because they already existed and I was just moving them from the header to here. But I'll change to static inline.

Quoted 37 lines
Quoted 23 lines
mm/huge_memory.c +21 -0
+bool hugepage_flags_enabled(void)
   {
+    /*
+     * We cover both the anon and the file-backed case here; we must return
+     * true if globally enabled, even when all anon sizes are set to never.
+     * So we don't need to look at huge_anon_orders_global.
+     */
+    return hugepage_global_enabled() ||
+           huge_anon_orders_always ||
+           huge_anon_orders_madvise;
+}
+
+/**
+ * hugepage_vma_check - determine which hugepage orders can be applied to vma
+ * @vma:  the vm area to check
+ * @vm_flags: use these vm_flags instead of vma->vm_flags
+ * @smaps: whether answer will be used for smaps file
+ * @in_pf: whether answer will be used by page fault handler
+ * @enforce_sysfs: whether sysfs config should be taken into account
+ * @orders: bitfield of all orders to consider
+ *
+ * Calculates the intersection of the requested hugepage orders and the allowed

mm/huge_memory.c static unsigned long deferred_split_scan(struct shrinker *shrink, · line 121

Quoted 10 lines
mm/huge_memory.c +10 -0
+ * hugepage orders for the provided vma. Permitted orders are encoded as a set
+ * bit at the corresponding bit position (bit-2 corresponds to order-2, bit-3
+ * corresponds to order-3, etc). Order-0 is never considered a hugepage order.
+ *
+ * Return: bitfield of orders allowed for hugepage in the vma. 0 if no hugepage
+ * orders are allowed.
+ */
+unsigned long hugepage_vma_check(struct vm_area_struct *vma,
+                 unsigned long vm_flags, bool smaps, bool in_pf,
+                 bool enforce_sysfs, unsigned long orders)

Here again, a bool return type has been changed to a bitfield. Let's also update the function name. Maybe one of these:

Quoted 5 lines
Quoted 2 lines

        hugepage_vma_orders()
        hugepage_vma_allowable_orders()

thp_vma_allowable_orders()?

Even better, yes.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
author reviewer
#13 lore raw

Re: [RESEND PATCH v7 03/10] mm: thp: Introduce per-size thp sysfs interface

Quoted 89 lines · John Hubbard
Quoted 2 lines · Ryan Roberts
Quoted 1 line · John Hubbard

On 11/22/23 08:29, Ryan Roberts wrote:

...

Quoted 8 lines
Quoted 7 lines

Documentation/admin-guide/mm/transhuge.rst components: · line 49

Quoted 2 lines
Documentation/admin-guide/mm/transhuge.rst +2 -0
+As well as PMD-sized THP described above, it is also possible to
+configure the system to allocate "small-sized THP" to back anonymous

Here's one of the places to change to the new name, which lately is "multi-size THP", or mTHP or m_thp for short. (I've typed "multi-size" instead of "multi-sized", because the 'd' doesn't add significantly to the meaning, and if in doubt, shorter is better.

Quoted 8 lines

I was thinking of some light changes to the start of this paragraph, something like:

Modern kernels support "multi-size THP" (mTHP), which introduces the ability to allocate memory in blocks that are bigger than a base page but smaller than traditional PMD-size (as described above, in increments of a power-of-2 number of pages. mTHP can back anonymous

Very nice.

...

Quoted 6 lines
Quoted 2 lines

By default, PMD-sized hugepages have enabled="inherited" and all other hugepage sizes have enabled="never".

That all sounds good; will update.

I wonder if "inherit" is even better than "inherited" though?

Yes, I think so. "inherit" was actually my first idea, and after thinking about it, I just made it worse by adding the "ed". haha. :)

...

Quoted 11 lines
Quoted 8 lines
Quoted 8 lines

   Khugepaged controls
   -------------------

+.. note::
+   khugepaged currently only searches for opportunities to collapse to +   PMD-sized THP and no attempt is made to collapse to small-sized +   THP.
+

"small-sized THP" used here too; I propose to change it to "... collapse to other THP sizes."

OK, looks good.

Quoted 30 lines
Quoted 24 lines
Quoted 24 lines

   khugepaged runs usually at low frequency so while one may not want to invoke defrag algorithms synchronously during the page faults, it should be worth invoking defrag at least in khugepaged. However it's

(quoted diff) +0 -0
@@ -282,10 +321,11 @@ force

   Need of application restart
   ===========================

Documentation/admin-guide/mm/transhuge.rst +5 -4
-The transparent_hugepage/enabled values and tmpfs mount option only affect
-future behavior. So to make them effective you need to restart any
-application that could have been using hugepages. This also applies to the
-regions registered in khugepaged.
+The transparent_hugepage/enabled and
+transparent_hugepage/hugepages-<size>kB/enabled values and tmpfs mount
+option only affect future behavior. So to make them effective you need
+to restart any application that could have been using hugepages. This
+also applies to the regions registered in khugepaged.

   Monitoring usage
   ================

(quoted diff) +0 -0
@@ -308,6 +348,10 @@ frequently will incur overhead.

   There are a number of counters in ``/proc/vmstat`` that may be used to monitor how successfully the system is providing huge pages for use.

Quoted 5 lines
Quoted 3 lines

+.. note::
+   Currently the below counters only record events relating to +   PMD-sized THP. Events relating to small-sized THP are not included.

Here's another spot to rename to "multi-size THP".

Quoted 8 lines

I propose to change it to "Events relating to other THP sizes..."

I'm also going to move this note to just under the "Monitoring Usage" heading since its current position doesn't make it clear that it includes "AnonHugePages". I'll also make it clear in the text that mentions AnonHugePages counter that it "only applies to traditional PMD-sized THP for historical reasons" and that "AnonHugePages should have been called AnonHugePmdMapped" as David suggested in the other thread.

Quoted 3 lines

OK.

Are we entirely dropping the AnonHugePtePages that was there in v6? I'm

Yes, afraid so. David raised some concerns about the semantics of the counters and also highlighted that (Greg KH?) doesn't want to see sysfs files with multiple values so we might need to design a whole new interface eventually.

Quoted 40 lines

looking for some way to monitor this. I may have missed it because I haven't gone through all of v7 yet.

...

Quoted 10 lines
Quoted 9 lines

Documentation/filesystems/proc.rst replaced by copy-on-write) part of the underlying shmem object out on swap. · line 533

Quoted 6 lines

-"THPeligible" indicates whether the mapping is eligible for allocating THP

Documentation/filesystems/proc.rst +3 -2
-pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise.
-It just shows the current status.
+"THPeligible" indicates whether the mapping is eligible for allocating
+naturally aligned THP pages of any currently enabled size. 1 if true, 0
+otherwise. It just shows the current status.

"It just shows the current status"...as opposed to what? I'm missing an educational opportunity here--not sure what this means. :)

Quoted 3 lines

I have no idea either - it seems superfluous. But that sentence was there already. I can remove it if you like?

Yes, let's remove it, please.

...

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
include/linux/huge_mm.h +4 -0
+/*
+ * Mask of all large folio orders supported for anonymous THP.
+ */
+#define THP_ORDERS_ALL_ANON    BIT(PMD_ORDER)
Quoted 8 lines

include/linux/huge_mm.h extern struct kobj_attribute shmem_enabled_attr; · line 78

Quoted 5 lines
include/linux/huge_mm.h +5 -0
+
+/*
+ * Mask of all large folio orders supported for file THP.
+ */
+#define THP_ORDERS_ALL_FILE    (BIT(PMD_ORDER) | BIT(PUD_ORDER))

Is there something about file THP that implies PUD_ORDER? This deserves an explanatory comment, I think.

Quoted 5 lines

Sorry, I'm not really sure what you are asking? Today's kernel supports PUD-mapping file-backed memory (mostly DAX I believe). I'm not sure what comment you want me to add, other than "file-backed memory can support PUD-mapping", but that's self-evident from the define, isn't it?

Quoted 4 lines

Well, it's sort of evident, but confusing to me anyway, because it combines THP and PUD. Which seems to imply that we have PUD-based THPs supported, but only for file-backed mappings. Which is weird and needs some explanation, right?

Ahh yes, you have fallen into the trap of thinking that until this point THP is PMD-size only, which indeed is not the case for file-backed mappings; they already support PMD- and PUD-size. And this is another reason why I think its OK to continue to call the new sizes "THP".

I'm still not sure what you want me to explain in the code though... It's an existing feature.

Quoted 153 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
include/linux/huge_mm.h +6 -0
+
+/*
+ * Mask of all large folio orders supported for THP.
+ */
+#define THP_ORDERS_ALL        (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE)
+

   #ifdef CONFIG_TRANSPARENT_HUGEPAGE
   #define HPAGE_PMD_SHIFT PMD_SHIFT
   #define HPAGE_PMD_SIZE    ((1UL) << HPAGE_PMD_SHIFT)

include/linux/huge_mm.h +0 -0
@@ -78,13 +93,18 @@ extern struct kobj_attribute shmem_enabled_attr;
 
    extern unsigned long transparent_hugepage_flags;
 
Quoted 9 lines

include/linux/huge_mm.h extern struct kobj_attribute shmem_enabled_attr; · line 87

Quoted 7 lines
include/linux/huge_mm.h +0 -7
-#define hugepage_flags_enabled()                           \
-    (transparent_hugepage_flags &                       \
-     ((1<<TRANSPARENT_HUGEPAGE_FLAG) |               \
-      (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)))
-#define hugepage_flags_always()                \
-    (transparent_hugepage_flags &            \
-     (1<<TRANSPARENT_HUGEPAGE_FLAG))

Are macros actually required? If not, static inlines would be nicer.

Quoted 1 line

I agree static inlines are nicer. Here I'm removing existing macros though, and

Oh, I see that I replied in the wrong part of the email, since that's the *removal* part. oops. :)

...

Quoted 21 lines
Quoted 17 lines

include/linux/huge_mm.h extern unsigned long transparent_hugepage_flags; · line 121

Quoted 12 lines
include/linux/huge_mm.h +2 -10
-static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
-        unsigned long addr)
-{
-    unsigned long haddr;
-
-    /* Don't have to check pgoff for anonymous vma */
-    if (!vma_is_anonymous(vma)) {
-        if (!IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) - vma->vm_pgoff,
-                HPAGE_PMD_NR))
-            return false;
+static inline unsigned long transhuge_vma_suitable(struct vm_area_struct *vma,
+        unsigned long addr, unsigned long orders)

Changing this from a bool to a mask of orders is quite significant, and both the function name and the function-level comment documentation need to also be adjusted. Here, perhaps one of these names would work:

Quoted 1 line

         transhuge_vma_suitable_orders()

This is my preference, but its getting a bit long. How about:

Quoted 7 lines

    thp_vma_suitable_orders()

I could then call the other one:

    thp_vma_allowable_orders()

So we have a clearly related pair?

Oh yes, that works nicely.

Quoted 29 lines
Quoted 17 lines

         transhuge_vma_orders()>

include/linux/huge_mm.h extern unsigned long transparent_hugepage_flags; · line 129

Quoted 8 lines
include/linux/huge_mm.h +8 -0
+{
+    int order;
+
+    /*
+     * Iterate over orders, highest to lowest, removing orders that don't
+     * meet alignment requirements from the set. Exit loop at first order
+     * that meets requirements, since all lower orders must also meet
+     * requirements.

Should we add a little note here, to the effect of, "this is unilaterally taking over a certain amount of allocation-like policy, by deciding how to search for folios of a certain size"?

Quoted 3 lines

Or is this The Only Way To Do It, after all? I know we had some discussion about it, and intuitively it feels reasonable, but wanted to poke at it one last time anyway.

This function isn't trying to implement policy, its just filtering out orders that don't fit and therefore definitely cannot be used. The result is a set of orders the could be used and its the policy maker's job to decide which one. Currently that policy is "biggest one that fits && does not overlap other non-none ptes && folio successfully allocated". That's implemented in the next patch and would potentially be swapped out in the future for something more exotic/optimal.

Quoted 1 line

So I don't think we need any extra comments here.

Ack.

...

Quoted 57 lines
Quoted 6 lines
Quoted 6 lines
mm/huge_memory.c +4 -2
-bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags,
-            bool smaps, bool in_pf, bool enforce_sysfs)
+#define hugepage_global_enabled()            \
+    (transparent_hugepage_flags &            \
+     ((1<<TRANSPARENT_HUGEPAGE_FLAG) |        \
+      (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)))
Quoted 8 lines

mm/huge_memory.c static unsigned long deferred_split_scan(struct shrinker *shrink, · line 89

Quoted 5 lines
mm/huge_memory.c +5 -0
+
+#define hugepage_global_always()            \
+    (transparent_hugepage_flags &            \
+     (1<<TRANSPARENT_HUGEPAGE_FLAG))
+

Here again, I'd like to request that we avoid macros, as I don't think they are strictly required.

Yeah I agree. I did them this way, because they already existed and I was just moving them from the header to here. But I'll change to static inline.

Quoted 39 lines
Quoted 27 lines

+bool hugepage_flags_enabled(void)
   {
+    /*
+     * We cover both the anon and the file-backed case here; we must return +     * true if globally enabled, even when all anon sizes are set to never. +     * So we don't need to look at huge_anon_orders_global.
+     */
+    return hugepage_global_enabled() ||
+           huge_anon_orders_always ||
+           huge_anon_orders_madvise;

mm/huge_memory.c +12 -0
+}
+
+/**
+ * hugepage_vma_check - determine which hugepage orders can be applied to vma
+ * @vma:  the vm area to check
+ * @vm_flags: use these vm_flags instead of vma->vm_flags
+ * @smaps: whether answer will be used for smaps file
+ * @in_pf: whether answer will be used by page fault handler
+ * @enforce_sysfs: whether sysfs config should be taken into account
+ * @orders: bitfield of all orders to consider
+ *
+ * Calculates the intersection of the requested hugepage orders and the

allowed
+ * hugepage orders for the provided vma. Permitted orders are encoded as a set
+ * bit at the corresponding bit position (bit-2 corresponds to order-2, bit-3
+ * corresponds to order-3, etc). Order-0 is never considered a hugepage order.

Quoted 8 lines

+ *
+ * Return: bitfield of orders allowed for hugepage in the vma. 0 if no hugepage
+ * orders are allowed.
+ */
+unsigned long hugepage_vma_check(struct vm_area_struct *vma, +                 unsigned long vm_flags, bool smaps, bool in_pf, +                 bool enforce_sysfs, unsigned long orders)

Here again, a bool return type has been changed to a bitfield. Let's also update the function name. Maybe one of these:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

         hugepage_vma_orders()
         hugepage_vma_allowable_orders()

thp_vma_allowable_orders()?

Even better, yes.

Great thanks for the review! I'll get all these changes folded in and if I don't get any further feedback by the end of the week, I'll repost on Monday.

Quoted 3 lines

thanks,

author reviewer
#14 lore raw

PATCH RESEND v7 4/10 mm: thp: Support allocation of anonymous small-sized THP

In reply to Ryan Roberts

Introduce the logic to allow THP to be configured (through the new sysfs interface we just added) to allocate large folios to back anonymous memory, which are smaller than PMD-size. We call this new THP type "small-sized THP".

These small-sized THPs continue to be PTE-mapped, but in many cases can still provide similar benefits to traditional PMD-sized THP: Page faults are significantly reduced (by a factor of e.g. 4, 8, 16, etc. depending on the configured order), but latency spikes are much less prominent because the size of each page isn't as huge as the PMD-sized variant and there is less memory to clear in each page fault. The number of per-page operations (e.g. ref counting, rmap management, lru list management) are also significantly reduced since those ops now become per-folio.

Some architectures also employ TLB compression mechanisms to squeeze more entries in when a set of PTEs are virtually and physically contiguous and approporiately aligned. In this case, TLB misses will occur less often.

The new behaviour is disabled by default, but can be enabled at runtime by writing to /sys/kernel/mm/transparent_hugepage/hugepage-XXkb/enabled (see documentation in previous commit). The long term aim is to change the default to include suitable lower orders, but there are some risks around internal fragmentation that need to be better understood first.

---
 include/linux/huge_mm.h |   6 ++-
 mm/memory.c             | 106 ++++++++++++++++++++++++++++++++++++----
 2 files changed, 101 insertions(+), 11 deletions(-)
include/linux/huge_mm.h +4 -2
index 7d6f7d96b039..edc302351971 100644
@@ -68,9 +68,11 @@ extern struct kobj_attribute shmem_enabled_attr;
 #define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER)
 
 /*
- * Mask of all large folio orders supported for anonymous THP.
+ * Mask of all large folio orders supported for anonymous THP; all orders up to
+ * and including PMD_ORDER, except order-0 (which is not "huge") and order-1
+ * (which is a limitation of the THP implementation).
  */
-#define THP_ORDERS_ALL_ANON	BIT(PMD_ORDER)
+#define THP_ORDERS_ALL_ANON	((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1)))
 
 /*
  * Mask of all large folio orders supported for file THP.
mm/memory.c +97 -9

147 lines not on this page · raw message

--
2.25.1

#15 lore raw

Re: [RESEND PATCH v7 04/10] mm: thp: Support allocation of anonymous small-sized THP

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4290

Quoted 6 lines
mm/memory.c +4 -0
+	if ((nr_pages == 1 && vmf_pte_changed(vmf)) ||
+	    (nr_pages  > 1 && !pte_range_none(vmf->pte, nr_pages))) {
+		for (i = 0; i < nr_pages; i++)
+			update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i);
 		goto release;
 	}

Hi Ryan,
what has stopped nr_pages == 1 from using !pte_range_none(vmf->pte, 1) directly, then the code can become,
+ if (!pte_range_none(vmf->pte, nr_pages)) {
+ for (i = 0; i < nr_pages; i++)
+ update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i);
goto release;
}

for both 1 and > 1 cases?

Thanks
Barry

author reviewer
#16 lore raw

Re: [RESEND PATCH v7 04/10] mm: thp: Support allocation of anonymous small-sized THP

Quoted 16 lines · Barry Song

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4290

Quoted 6 lines
mm/memory.c +4 -0
+	if ((nr_pages == 1 && vmf_pte_changed(vmf)) ||
+	    (nr_pages  > 1 && !pte_range_none(vmf->pte, nr_pages))) {
+		for (i = 0; i < nr_pages; i++)
+			update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i);
 		goto release;
 	}

Hi Ryan,
what has stopped nr_pages == 1 from using !pte_range_none(vmf->pte, 1) directly, then the code can become,

mm/memory.c +3 -0
+	if (!pte_range_none(vmf->pte, nr_pages)) {
+		for (i = 0; i < nr_pages; i++)
+			update_mmu_tlb(vma, addr + PAGE_SIZE * i, vmf->pte + i);
 		goto release;
 	}
Quoted 1 line

for both 1 and > 1 cases?

We can get to do_anonymous_page() from 2 routes:

- page fault on unallocated memory (pte_none())
- page fault on uffd_wp pte marker

In the latter case, we guarrantee that we are only operating on nr_pages == 1 because when uffd is in the picture we need to preserve any uffd state per-pte. It also means we can't just check the pte is none because in this case it is not none, it has a pte marker so we need to check it hasn't changed.

I was previously abstracting this in vmf_pte_range_changed() but there were complaints [1] about the semantic being different based on the number of pages, so this was my attempt to make it more understandable.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Thanks,
Ryan

Quoted 4 lines

Thanks
Barry

author reviewer
#17 lore raw

PATCH RESEND v7 5/10 selftests/mm/kugepaged: Restore thp settings at exit

In reply to Ryan Roberts

Previously, the saved thp settings would be restored upon a signal or at the natural end of the test suite. But there are some tests that directly call exit() upon failure. In this case, the thp settings were not being restored, which could then influence other tests.

Fix this by installing an atexit() handler to do the actual restore. The signal handler can now just call exit() and the atexit handler is invoked.

---
 tools/testing/selftests/mm/khugepaged.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
tools/testing/selftests/mm/khugepaged.c +11 -6
index 030667cb5533..fc47a1c4944c 100644
@@ -374,18 +374,22 @@ static void pop_settings(void)
 	write_settings(current_settings());
 }
 
-static void restore_settings(int sig)
+static void restore_settings_atexit(void)
 {
 	if (skip_settings_restore)
-		goto out;
+		return;
 
 	printf("Restore THP and khugepaged settings...");
 	write_settings(&saved_settings);
 	success("OK");
-	if (sig)
-		exit(EXIT_FAILURE);
-out:
-	exit(exit_status);
+
+	skip_settings_restore = true;
+}
+
+static void restore_settings(int sig)
+{
+	/* exit() will invoke the restore_settings_atexit handler. */
+	exit(sig ? EXIT_FAILURE : exit_status);
 }
 
 static void save_settings(void)
@@ -415,6 +419,7 @@ static void save_settings(void)
 
 	success("OK");
 
+	atexit(restore_settings_atexit);
 	signal(SIGTERM, restore_settings);
 	signal(SIGINT, restore_settings);
 	signal(SIGHUP, restore_settings);

--
2.25.1

Reviewed-by on v7 5/10
#18 lore raw

Re: [RESEND PATCH v7 05/10] selftests/mm/kugepaged: Restore thp settings at exit

Quoted 10 lines · Ryan Roberts

Previously, the saved thp settings would be restored upon a signal or at the natural end of the test suite. But there are some tests that directly call exit() upon failure. In this case, the thp settings were not being restored, which could then influence other tests.

Fix this by installing an atexit() handler to do the actual restore. The signal handler can now just call exit() and the atexit handler is invoked.

Quoted 45 lines
---
 tools/testing/selftests/mm/khugepaged.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
tools/testing/selftests/mm/khugepaged.c +11 -6
index 030667cb5533..fc47a1c4944c 100644
@@ -374,18 +374,22 @@ static void pop_settings(void)
 	write_settings(current_settings());
 }
 
-static void restore_settings(int sig)
+static void restore_settings_atexit(void)
 {
 	if (skip_settings_restore)
-		goto out;
+		return;
 
 	printf("Restore THP and khugepaged settings...");
 	write_settings(&saved_settings);
 	success("OK");
-	if (sig)
-		exit(EXIT_FAILURE);
-out:
-	exit(exit_status);
+
+	skip_settings_restore = true;
+}
+
+static void restore_settings(int sig)
+{
+	/* exit() will invoke the restore_settings_atexit handler. */
+	exit(sig ? EXIT_FAILURE : exit_status);
 }
 
 static void save_settings(void)
@@ -415,6 +419,7 @@ static void save_settings(void)
 
 	success("OK");
 
+	atexit(restore_settings_atexit);
 	signal(SIGTERM, restore_settings);
 	signal(SIGINT, restore_settings);
 	signal(SIGHUP, restore_settings);
author reviewer
#19 lore raw

PATCH RESEND v7 6/10 selftests/mm: Factor out thp settings management

In reply to Ryan Roberts

The khugepaged test has a useful framework for save/restore/pop/push of all thp settings via the sysfs interface. This will be useful to explicitly control small-sized THP settings in other tests, so let's move it out of khugepaged and into its own thp_settings.[c|h] utility.

---
 tools/testing/selftests/mm/Makefile       |   4 +-
 tools/testing/selftests/mm/khugepaged.c   | 346 ++--------------------
 tools/testing/selftests/mm/thp_settings.c | 296 ++++++++++++++++++
 tools/testing/selftests/mm/thp_settings.h |  71 +++++
 4 files changed, 391 insertions(+), 326 deletions(-)
 create mode 100644 tools/testing/selftests/mm/thp_settings.c
 create mode 100644 tools/testing/selftests/mm/thp_settings.h
tools/testing/selftests/mm/Makefile +2 -2
index 78dfec8bc676..a3eb20c186e7 100644
@@ -117,8 +117,8 @@ TEST_FILES += va_high_addr_switch.sh
 
 include ../lib.mk
 
-$(TEST_GEN_PROGS): vm_util.c
-$(TEST_GEN_FILES): vm_util.c
+$(TEST_GEN_PROGS): vm_util.c thp_settings.c
+$(TEST_GEN_FILES): vm_util.c thp_settings.c
 
 $(OUTPUT)/uffd-stress: uffd-common.c
 $(OUTPUT)/uffd-unit-tests: uffd-common.c
tools/testing/selftests/mm/khugepaged.c +22 -324

475 lines not on this page · raw message

tools/testing/selftests/mm/thp_settings.c new file +296 -0

296 lines not on this page · raw message

tools/testing/selftests/mm/thp_settings.h new file +71 -0

71 lines not on this page · raw message

--
2.25.1

Tested-by on v7 6/10
#20 lore raw

Re: [RESEND PATCH v7 06/10] selftests/mm: Factor out thp settings management

Quoted 6 lines · Ryan Roberts

The khugepaged test has a useful framework for save/restore/pop/push of all thp settings via the sysfs interface. This will be useful to explicitly control small-sized THP settings in other tests, so let's move it out of khugepaged and into its own thp_settings.[c|h] utility.

I've only glanced at the code as I assume it is a straight forward cut-and-paste with no behavioural change. At least I didn't observe any change running the khugepage test on my x86_64 development machine so feel free to add:

Quoted 899 lines
---
 tools/testing/selftests/mm/Makefile       |   4 +-
 tools/testing/selftests/mm/khugepaged.c   | 346 ++--------------------
 tools/testing/selftests/mm/thp_settings.c | 296 ++++++++++++++++++
 tools/testing/selftests/mm/thp_settings.h |  71 +++++
 4 files changed, 391 insertions(+), 326 deletions(-)
 create mode 100644 tools/testing/selftests/mm/thp_settings.c
 create mode 100644 tools/testing/selftests/mm/thp_settings.h
tools/testing/selftests/mm/Makefile +2 -2
index 78dfec8bc676..a3eb20c186e7 100644
@@ -117,8 +117,8 @@ TEST_FILES += va_high_addr_switch.sh
 
 include ../lib.mk
 
-$(TEST_GEN_PROGS): vm_util.c
-$(TEST_GEN_FILES): vm_util.c
+$(TEST_GEN_PROGS): vm_util.c thp_settings.c
+$(TEST_GEN_FILES): vm_util.c thp_settings.c
 
 $(OUTPUT)/uffd-stress: uffd-common.c
 $(OUTPUT)/uffd-unit-tests: uffd-common.c
tools/testing/selftests/mm/khugepaged.c +22 -324
index fc47a1c4944c..b15e7fd70176 100644
@@ -22,13 +22,13 @@
 #include "linux/magic.h"
 
 #include "vm_util.h"
+#include "thp_settings.h"
 
 #define BASE_ADDR ((void *)(1UL << 30))
 static unsigned long hpage_pmd_size;
 static unsigned long page_size;
 static int hpage_pmd_nr;
 
-#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/"
 #define PID_SMAPS "/proc/self/smaps"
 #define TEST_FILE "collapse_test_file"
 
@@ -71,78 +71,7 @@ struct file_info {
 };
 
 static struct file_info finfo;
-
-enum thp_enabled {
-	THP_ALWAYS,
-	THP_MADVISE,
-	THP_NEVER,
-};
-
-static const char *thp_enabled_strings[] = {
-	"always",
-	"madvise",
-	"never",
-	NULL
-};
-
-enum thp_defrag {
-	THP_DEFRAG_ALWAYS,
-	THP_DEFRAG_DEFER,
-	THP_DEFRAG_DEFER_MADVISE,
-	THP_DEFRAG_MADVISE,
-	THP_DEFRAG_NEVER,
-};
-
-static const char *thp_defrag_strings[] = {
-	"always",
-	"defer",
-	"defer+madvise",
-	"madvise",
-	"never",
-	NULL
-};
-
-enum shmem_enabled {
-	SHMEM_ALWAYS,
-	SHMEM_WITHIN_SIZE,
-	SHMEM_ADVISE,
-	SHMEM_NEVER,
-	SHMEM_DENY,
-	SHMEM_FORCE,
-};
-
-static const char *shmem_enabled_strings[] = {
-	"always",
-	"within_size",
-	"advise",
-	"never",
-	"deny",
-	"force",
-	NULL
-};
-
-struct khugepaged_settings {
-	bool defrag;
-	unsigned int alloc_sleep_millisecs;
-	unsigned int scan_sleep_millisecs;
-	unsigned int max_ptes_none;
-	unsigned int max_ptes_swap;
-	unsigned int max_ptes_shared;
-	unsigned long pages_to_scan;
-};
-
-struct settings {
-	enum thp_enabled thp_enabled;
-	enum thp_defrag thp_defrag;
-	enum shmem_enabled shmem_enabled;
-	bool use_zero_page;
-	struct khugepaged_settings khugepaged;
-	unsigned long read_ahead_kb;
-};
-
-static struct settings saved_settings;
 static bool skip_settings_restore;
-
 static int exit_status;
 
 static void success(const char *msg)
@@ -161,226 +90,13 @@ static void skip(const char *msg)
 	printf(" \e[33m%s\e[0m\n", msg);
 }
 
-static int read_file(const char *path, char *buf, size_t buflen)
-{
-	int fd;
-	ssize_t numread;
-
-	fd = open(path, O_RDONLY);
-	if (fd == -1)
-		return 0;
-
-	numread = read(fd, buf, buflen - 1);
-	if (numread < 1) {
-		close(fd);
-		return 0;
-	}
-
-	buf[numread] = '\0';
-	close(fd);
-
-	return (unsigned int) numread;
-}
-
-static int write_file(const char *path, const char *buf, size_t buflen)
-{
-	int fd;
-	ssize_t numwritten;
-
-	fd = open(path, O_WRONLY);
-	if (fd == -1) {
-		printf("open(%s)\n", path);
-		exit(EXIT_FAILURE);
-		return 0;
-	}
-
-	numwritten = write(fd, buf, buflen - 1);
-	close(fd);
-	if (numwritten < 1) {
-		printf("write(%s)\n", buf);
-		exit(EXIT_FAILURE);
-		return 0;
-	}
-
-	return (unsigned int) numwritten;
-}
-
-static int read_string(const char *name, const char *strings[])
-{
-	char path[PATH_MAX];
-	char buf[256];
-	char *c;
-	int ret;
-
-	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
-	if (ret >= PATH_MAX) {
-		printf("%s: Pathname is too long\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-
-	if (!read_file(path, buf, sizeof(buf))) {
-		perror(path);
-		exit(EXIT_FAILURE);
-	}
-
-	c = strchr(buf, '[');
-	if (!c) {
-		printf("%s: Parse failure\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-
-	c++;
-	memmove(buf, c, sizeof(buf) - (c - buf));
-
-	c = strchr(buf, ']');
-	if (!c) {
-		printf("%s: Parse failure\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-	*c = '\0';
-
-	ret = 0;
-	while (strings[ret]) {
-		if (!strcmp(strings[ret], buf))
-			return ret;
-		ret++;
-	}
-
-	printf("Failed to parse %s\n", name);
-	exit(EXIT_FAILURE);
-}
-
-static void write_string(const char *name, const char *val)
-{
-	char path[PATH_MAX];
-	int ret;
-
-	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
-	if (ret >= PATH_MAX) {
-		printf("%s: Pathname is too long\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-
-	if (!write_file(path, val, strlen(val) + 1)) {
-		perror(path);
-		exit(EXIT_FAILURE);
-	}
-}
-
-static const unsigned long _read_num(const char *path)
-{
-	char buf[21];
-
-	if (read_file(path, buf, sizeof(buf)) < 0) {
-		perror("read_file(read_num)");
-		exit(EXIT_FAILURE);
-	}
-
-	return strtoul(buf, NULL, 10);
-}
-
-static const unsigned long read_num(const char *name)
-{
-	char path[PATH_MAX];
-	int ret;
-
-	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
-	if (ret >= PATH_MAX) {
-		printf("%s: Pathname is too long\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-	return _read_num(path);
-}
-
-static void _write_num(const char *path, unsigned long num)
-{
-	char buf[21];
-
-	sprintf(buf, "%ld", num);
-	if (!write_file(path, buf, strlen(buf) + 1)) {
-		perror(path);
-		exit(EXIT_FAILURE);
-	}
-}
-
-static void write_num(const char *name, unsigned long num)
-{
-	char path[PATH_MAX];
-	int ret;
-
-	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
-	if (ret >= PATH_MAX) {
-		printf("%s: Pathname is too long\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-	_write_num(path, num);
-}
-
-static void write_settings(struct settings *settings)
-{
-	struct khugepaged_settings *khugepaged = &settings->khugepaged;
-
-	write_string("enabled", thp_enabled_strings[settings->thp_enabled]);
-	write_string("defrag", thp_defrag_strings[settings->thp_defrag]);
-	write_string("shmem_enabled",
-			shmem_enabled_strings[settings->shmem_enabled]);
-	write_num("use_zero_page", settings->use_zero_page);
-
-	write_num("khugepaged/defrag", khugepaged->defrag);
-	write_num("khugepaged/alloc_sleep_millisecs",
-			khugepaged->alloc_sleep_millisecs);
-	write_num("khugepaged/scan_sleep_millisecs",
-			khugepaged->scan_sleep_millisecs);
-	write_num("khugepaged/max_ptes_none", khugepaged->max_ptes_none);
-	write_num("khugepaged/max_ptes_swap", khugepaged->max_ptes_swap);
-	write_num("khugepaged/max_ptes_shared", khugepaged->max_ptes_shared);
-	write_num("khugepaged/pages_to_scan", khugepaged->pages_to_scan);
-
-	if (file_ops && finfo.type == VMA_FILE)
-		_write_num(finfo.dev_queue_read_ahead_path,
-			   settings->read_ahead_kb);
-}
-
-#define MAX_SETTINGS_DEPTH 4
-static struct settings settings_stack[MAX_SETTINGS_DEPTH];
-static int settings_index;
-
-static struct settings *current_settings(void)
-{
-	if (!settings_index) {
-		printf("Fail: No settings set");
-		exit(EXIT_FAILURE);
-	}
-	return settings_stack + settings_index - 1;
-}
-
-static void push_settings(struct settings *settings)
-{
-	if (settings_index >= MAX_SETTINGS_DEPTH) {
-		printf("Fail: Settings stack exceeded");
-		exit(EXIT_FAILURE);
-	}
-	settings_stack[settings_index++] = *settings;
-	write_settings(current_settings());
-}
-
-static void pop_settings(void)
-{
-	if (settings_index <= 0) {
-		printf("Fail: Settings stack empty");
-		exit(EXIT_FAILURE);
-	}
-	--settings_index;
-	write_settings(current_settings());
-}
-
 static void restore_settings_atexit(void)
 {
 	if (skip_settings_restore)
 		return;
 
 	printf("Restore THP and khugepaged settings...");
-	write_settings(&saved_settings);
+	thp_restore_settings();
 	success("OK");
 
 	skip_settings_restore = true;
@@ -395,27 +111,9 @@ static void restore_settings(int sig)
 static void save_settings(void)
 {
 	printf("Save THP and khugepaged settings...");
-	saved_settings = (struct settings) {
-		.thp_enabled = read_string("enabled", thp_enabled_strings),
-		.thp_defrag = read_string("defrag", thp_defrag_strings),
-		.shmem_enabled =
-			read_string("shmem_enabled", shmem_enabled_strings),
-		.use_zero_page = read_num("use_zero_page"),
-	};
-	saved_settings.khugepaged = (struct khugepaged_settings) {
-		.defrag = read_num("khugepaged/defrag"),
-		.alloc_sleep_millisecs =
-			read_num("khugepaged/alloc_sleep_millisecs"),
-		.scan_sleep_millisecs =
-			read_num("khugepaged/scan_sleep_millisecs"),
-		.max_ptes_none = read_num("khugepaged/max_ptes_none"),
-		.max_ptes_swap = read_num("khugepaged/max_ptes_swap"),
-		.max_ptes_shared = read_num("khugepaged/max_ptes_shared"),
-		.pages_to_scan = read_num("khugepaged/pages_to_scan"),
-	};
 	if (file_ops && finfo.type == VMA_FILE)
-		saved_settings.read_ahead_kb =
-				_read_num(finfo.dev_queue_read_ahead_path);
+		thp_set_read_ahead_path(finfo.dev_queue_read_ahead_path);
+	thp_save_settings();
 
 	success("OK");
 
@@ -798,7 +496,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,
 			       struct mem_ops *ops, bool expect)
 {
 	int ret;
-	struct settings settings = *current_settings();
+	struct thp_settings settings = *thp_current_settings();
 
 	printf("%s...", msg);
 
@@ -808,7 +506,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,
 	 */
 	settings.thp_enabled = THP_NEVER;
 	settings.shmem_enabled = SHMEM_NEVER;
-	push_settings(&settings);
+	thp_push_settings(&settings);
 
 	/* Clear VM_NOHUGEPAGE */
 	madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE);
@@ -820,7 +518,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,
 	else
 		success("OK");
 
-	pop_settings();
+	thp_pop_settings();
 }
 
 static void madvise_collapse(const char *msg, char *p, int nr_hpages,
@@ -850,13 +548,13 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages,
 	madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE);
 
 	/* Wait until the second full_scan completed */
-	full_scans = read_num("khugepaged/full_scans") + 2;
+	full_scans = thp_read_num("khugepaged/full_scans") + 2;
 
 	printf("%s...", msg);
 	while (timeout--) {
 		if (ops->check_huge(p, nr_hpages))
 			break;
-		if (read_num("khugepaged/full_scans") >= full_scans)
+		if (thp_read_num("khugepaged/full_scans") >= full_scans)
 			break;
 		printf(".");
 		usleep(TICK);
@@ -911,11 +609,11 @@ static bool is_tmpfs(struct mem_ops *ops)
 
 static void alloc_at_fault(void)
 {
-	struct settings settings = *current_settings();
+	struct thp_settings settings = *thp_current_settings();
 	char *p;
 
 	settings.thp_enabled = THP_ALWAYS;
-	push_settings(&settings);
+	thp_push_settings(&settings);
 
 	p = alloc_mapping(1);
 	*p = 1;
@@ -925,7 +623,7 @@ static void alloc_at_fault(void)
 	else
 		fail("Fail");
 
-	pop_settings();
+	thp_pop_settings();
 
 	madvise(p, page_size, MADV_DONTNEED);
 	printf("Split huge PMD on MADV_DONTNEED...");
@@ -973,11 +671,11 @@ static void collapse_single_pte_entry(struct collapse_context *c, struct mem_ops
 static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *ops)
 {
 	int max_ptes_none = hpage_pmd_nr / 2;
-	struct settings settings = *current_settings();
+	struct thp_settings settings = *thp_current_settings();
 	void *p;
 
 	settings.khugepaged.max_ptes_none = max_ptes_none;
-	push_settings(&settings);
+	thp_push_settings(&settings);
 
 	p = ops->setup_area(1);
 
@@ -1002,7 +700,7 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o
 	}
 skip:
 	ops->cleanup_area(p, hpage_pmd_size);
-	pop_settings();
+	thp_pop_settings();
 }
 
 static void collapse_swapin_single_pte(struct collapse_context *c, struct mem_ops *ops)
@@ -1033,7 +731,7 @@ static void collapse_swapin_single_pte(struct collapse_context *c, struct mem_op
 
 static void collapse_max_ptes_swap(struct collapse_context *c, struct mem_ops *ops)
 {
-	int max_ptes_swap = read_num("khugepaged/max_ptes_swap");
+	int max_ptes_swap = thp_read_num("khugepaged/max_ptes_swap");
 	void *p;
 
 	p = ops->setup_area(1);
@@ -1250,11 +948,11 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o
 			fail("Fail");
 		ops->fault(p, 0, page_size);
 
-		write_num("khugepaged/max_ptes_shared", hpage_pmd_nr - 1);
+		thp_write_num("khugepaged/max_ptes_shared", hpage_pmd_nr - 1);
 		c->collapse("Collapse PTE table full of compound pages in child",
 			    p, 1, ops, true);
-		write_num("khugepaged/max_ptes_shared",
-			  current_settings()->khugepaged.max_ptes_shared);
+		thp_write_num("khugepaged/max_ptes_shared",
+			  thp_current_settings()->khugepaged.max_ptes_shared);
 
 		validate_memory(p, 0, hpage_pmd_size);
 		ops->cleanup_area(p, hpage_pmd_size);
@@ -1275,7 +973,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o
 
 static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops *ops)
 {
-	int max_ptes_shared = read_num("khugepaged/max_ptes_shared");
+	int max_ptes_shared = thp_read_num("khugepaged/max_ptes_shared");
 	int wstatus;
 	void *p;
 
@@ -1443,7 +1141,7 @@ static void parse_test_type(int argc, const char **argv)
 
 int main(int argc, const char **argv)
 {
-	struct settings default_settings = {
+	struct thp_settings default_settings = {
 		.thp_enabled = THP_MADVISE,
 		.thp_defrag = THP_DEFRAG_ALWAYS,
 		.shmem_enabled = SHMEM_ADVISE,
@@ -1484,7 +1182,7 @@ int main(int argc, const char **argv)
 	default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8;
 
 	save_settings();
-	push_settings(&default_settings);
+	thp_push_settings(&default_settings);
 
 	alloc_at_fault();
 
tools/testing/selftests/mm/thp_settings.c new file +296 -0
new file mode 100644
index 000000000000..5e8ec792cac7
@@ -0,0 +1,296 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <fcntl.h>
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "thp_settings.h"
+
+#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/"
+#define MAX_SETTINGS_DEPTH 4
+static struct thp_settings settings_stack[MAX_SETTINGS_DEPTH];
+static int settings_index;
+static struct thp_settings saved_settings;
+static char dev_queue_read_ahead_path[PATH_MAX];
+
+static const char * const thp_enabled_strings[] = {
+	"always",
+	"madvise",
+	"never",
+	NULL
+};
+
+static const char * const thp_defrag_strings[] = {
+	"always",
+	"defer",
+	"defer+madvise",
+	"madvise",
+	"never",
+	NULL
+};
+
+static const char * const shmem_enabled_strings[] = {
+	"always",
+	"within_size",
+	"advise",
+	"never",
+	"deny",
+	"force",
+	NULL
+};
+
+int read_file(const char *path, char *buf, size_t buflen)
+{
+	int fd;
+	ssize_t numread;
+
+	fd = open(path, O_RDONLY);
+	if (fd == -1)
+		return 0;
+
+	numread = read(fd, buf, buflen - 1);
+	if (numread < 1) {
+		close(fd);
+		return 0;
+	}
+
+	buf[numread] = '\0';
+	close(fd);
+
+	return (unsigned int) numread;
+}
+
+int write_file(const char *path, const char *buf, size_t buflen)
+{
+	int fd;
+	ssize_t numwritten;
+
+	fd = open(path, O_WRONLY);
+	if (fd == -1) {
+		printf("open(%s)\n", path);
+		exit(EXIT_FAILURE);
+		return 0;
+	}
+
+	numwritten = write(fd, buf, buflen - 1);
+	close(fd);
+	if (numwritten < 1) {
+		printf("write(%s)\n", buf);
+		exit(EXIT_FAILURE);
+		return 0;
+	}
+
+	return (unsigned int) numwritten;
+}
+
+const unsigned long read_num(const char *path)
+{
+	char buf[21];
+
+	if (read_file(path, buf, sizeof(buf)) < 0) {
+		perror("read_file()");
+		exit(EXIT_FAILURE);
+	}
+
+	return strtoul(buf, NULL, 10);
+}
+
+void write_num(const char *path, unsigned long num)
+{
+	char buf[21];
+
+	sprintf(buf, "%ld", num);
+	if (!write_file(path, buf, strlen(buf) + 1)) {
+		perror(path);
+		exit(EXIT_FAILURE);
+	}
+}
+
+int thp_read_string(const char *name, const char * const strings[])
+{
+	char path[PATH_MAX];
+	char buf[256];
+	char *c;
+	int ret;
+
+	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
+	if (ret >= PATH_MAX) {
+		printf("%s: Pathname is too long\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+
+	if (!read_file(path, buf, sizeof(buf))) {
+		perror(path);
+		exit(EXIT_FAILURE);
+	}
+
+	c = strchr(buf, '[');
+	if (!c) {
+		printf("%s: Parse failure\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+
+	c++;
+	memmove(buf, c, sizeof(buf) - (c - buf));
+
+	c = strchr(buf, ']');
+	if (!c) {
+		printf("%s: Parse failure\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+	*c = '\0';
+
+	ret = 0;
+	while (strings[ret]) {
+		if (!strcmp(strings[ret], buf))
+			return ret;
+		ret++;
+	}
+
+	printf("Failed to parse %s\n", name);
+	exit(EXIT_FAILURE);
+}
+
+void thp_write_string(const char *name, const char *val)
+{
+	char path[PATH_MAX];
+	int ret;
+
+	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
+	if (ret >= PATH_MAX) {
+		printf("%s: Pathname is too long\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+
+	if (!write_file(path, val, strlen(val) + 1)) {
+		perror(path);
+		exit(EXIT_FAILURE);
+	}
+}
+
+const unsigned long thp_read_num(const char *name)
+{
+	char path[PATH_MAX];
+	int ret;
+
+	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
+	if (ret >= PATH_MAX) {
+		printf("%s: Pathname is too long\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+	return read_num(path);
+}
+
+void thp_write_num(const char *name, unsigned long num)
+{
+	char path[PATH_MAX];
+	int ret;
+
+	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
+	if (ret >= PATH_MAX) {
+		printf("%s: Pathname is too long\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+	write_num(path, num);
+}
+
+void thp_read_settings(struct thp_settings *settings)
+{
+	*settings = (struct thp_settings) {
+		.thp_enabled = thp_read_string("enabled", thp_enabled_strings),
+		.thp_defrag = thp_read_string("defrag", thp_defrag_strings),
+		.shmem_enabled =
+			thp_read_string("shmem_enabled", shmem_enabled_strings),
+		.use_zero_page = thp_read_num("use_zero_page"),
+	};
+	settings->khugepaged = (struct khugepaged_settings) {
+		.defrag = thp_read_num("khugepaged/defrag"),
+		.alloc_sleep_millisecs =
+			thp_read_num("khugepaged/alloc_sleep_millisecs"),
+		.scan_sleep_millisecs =
+			thp_read_num("khugepaged/scan_sleep_millisecs"),
+		.max_ptes_none = thp_read_num("khugepaged/max_ptes_none"),
+		.max_ptes_swap = thp_read_num("khugepaged/max_ptes_swap"),
+		.max_ptes_shared = thp_read_num("khugepaged/max_ptes_shared"),
+		.pages_to_scan = thp_read_num("khugepaged/pages_to_scan"),
+	};
+	if (dev_queue_read_ahead_path[0])
+		settings->read_ahead_kb = read_num(dev_queue_read_ahead_path);
+}
+
+void thp_write_settings(struct thp_settings *settings)
+{
+	struct khugepaged_settings *khugepaged = &settings->khugepaged;
+
+	thp_write_string("enabled", thp_enabled_strings[settings->thp_enabled]);
+	thp_write_string("defrag", thp_defrag_strings[settings->thp_defrag]);
+	thp_write_string("shmem_enabled",
+			shmem_enabled_strings[settings->shmem_enabled]);
+	thp_write_num("use_zero_page", settings->use_zero_page);
+
+	thp_write_num("khugepaged/defrag", khugepaged->defrag);
+	thp_write_num("khugepaged/alloc_sleep_millisecs",
+			khugepaged->alloc_sleep_millisecs);
+	thp_write_num("khugepaged/scan_sleep_millisecs",
+			khugepaged->scan_sleep_millisecs);
+	thp_write_num("khugepaged/max_ptes_none", khugepaged->max_ptes_none);
+	thp_write_num("khugepaged/max_ptes_swap", khugepaged->max_ptes_swap);
+	thp_write_num("khugepaged/max_ptes_shared", khugepaged->max_ptes_shared);
+	thp_write_num("khugepaged/pages_to_scan", khugepaged->pages_to_scan);
+
+	if (dev_queue_read_ahead_path[0])
+		write_num(dev_queue_read_ahead_path, settings->read_ahead_kb);
+}
+
+struct thp_settings *thp_current_settings(void)
+{
+	if (!settings_index) {
+		printf("Fail: No settings set");
+		exit(EXIT_FAILURE);
+	}
+	return settings_stack + settings_index - 1;
+}
+
+void thp_push_settings(struct thp_settings *settings)
+{
+	if (settings_index >= MAX_SETTINGS_DEPTH) {
+		printf("Fail: Settings stack exceeded");
+		exit(EXIT_FAILURE);
+	}
+	settings_stack[settings_index++] = *settings;
+	thp_write_settings(thp_current_settings());
+}
+
+void thp_pop_settings(void)
+{
+	if (settings_index <= 0) {
+		printf("Fail: Settings stack empty");
+		exit(EXIT_FAILURE);
+	}
+	--settings_index;
+	thp_write_settings(thp_current_settings());
+}
+
+void thp_restore_settings(void)
+{
+	thp_write_settings(&saved_settings);
+}
+
+void thp_save_settings(void)
+{
+	thp_read_settings(&saved_settings);
+}
+
+void thp_set_read_ahead_path(char *path)
+{
+	if (!path) {
+		dev_queue_read_ahead_path[0] = '\0';
+		return;
+	}
+
+	strncpy(dev_queue_read_ahead_path, path,
+		sizeof(dev_queue_read_ahead_path));
+	dev_queue_read_ahead_path[sizeof(dev_queue_read_ahead_path) - 1] = '\0';
+}
tools/testing/selftests/mm/thp_settings.h new file +71 -0
new file mode 100644
index 000000000000..ff3d98c30617
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __THP_SETTINGS_H__
+#define __THP_SETTINGS_H__
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+
+enum thp_enabled {
+	THP_ALWAYS,
+	THP_MADVISE,
+	THP_NEVER,
+};
+
+enum thp_defrag {
+	THP_DEFRAG_ALWAYS,
+	THP_DEFRAG_DEFER,
+	THP_DEFRAG_DEFER_MADVISE,
+	THP_DEFRAG_MADVISE,
+	THP_DEFRAG_NEVER,
+};
+
+enum shmem_enabled {
+	SHMEM_ALWAYS,
+	SHMEM_WITHIN_SIZE,
+	SHMEM_ADVISE,
+	SHMEM_NEVER,
+	SHMEM_DENY,
+	SHMEM_FORCE,
+};
+
+struct khugepaged_settings {
+	bool defrag;
+	unsigned int alloc_sleep_millisecs;
+	unsigned int scan_sleep_millisecs;
+	unsigned int max_ptes_none;
+	unsigned int max_ptes_swap;
+	unsigned int max_ptes_shared;
+	unsigned long pages_to_scan;
+};
+
+struct thp_settings {
+	enum thp_enabled thp_enabled;
+	enum thp_defrag thp_defrag;
+	enum shmem_enabled shmem_enabled;
+	bool use_zero_page;
+	struct khugepaged_settings khugepaged;
+	unsigned long read_ahead_kb;
+};
+
+int read_file(const char *path, char *buf, size_t buflen);
+int write_file(const char *path, const char *buf, size_t buflen);
+const unsigned long read_num(const char *path);
+void write_num(const char *path, unsigned long num);
+
+int thp_read_string(const char *name, const char * const strings[]);
+void thp_write_string(const char *name, const char *val);
+const unsigned long thp_read_num(const char *name);
+void thp_write_num(const char *name, unsigned long num);
+
+void thp_write_settings(struct thp_settings *settings);
+void thp_read_settings(struct thp_settings *settings);
+struct thp_settings *thp_current_settings(void);
+void thp_push_settings(struct thp_settings *settings);
+void thp_pop_settings(void);
+void thp_restore_settings(void);
+void thp_save_settings(void);
+
+void thp_set_read_ahead_path(char *path);
+
+#endif /* __THP_SETTINGS_H__ */
author reviewer
#21 lore raw

Re: [RESEND PATCH v7 06/10] selftests/mm: Factor out thp settings management

Quoted 8 lines · Alistair Popple
Quoted 5 lines · Ryan Roberts

The khugepaged test has a useful framework for save/restore/pop/push of all thp settings via the sysfs interface. This will be useful to explicitly control small-sized THP settings in other tests, so let's move it out of khugepaged and into its own thp_settings.[c|h] utility.

Quoted 8 lines
Quoted 1 line

I've only glanced at the code as I assume it is a straight forward cut-and-paste with no behavioural change. At least I didn't observe any change running the khugepage test on my x86_64 development machine so feel free to add:

Thanks! Its pretty much just a cut-and-paste; I did prefix the function names with "thp_" since they are now in global namespace. And I moved some khugepaged test-specific logging out of them and into the higher level test code. And I added thp_set_read_ahead_path() to decouple from the test code. But intended to all be equivalent.

Quoted 901 lines
Quoted 900 lines
---
 tools/testing/selftests/mm/Makefile       |   4 +-
 tools/testing/selftests/mm/khugepaged.c   | 346 ++--------------------
 tools/testing/selftests/mm/thp_settings.c | 296 ++++++++++++++++++
 tools/testing/selftests/mm/thp_settings.h |  71 +++++
 4 files changed, 391 insertions(+), 326 deletions(-)
 create mode 100644 tools/testing/selftests/mm/thp_settings.c
 create mode 100644 tools/testing/selftests/mm/thp_settings.h
tools/testing/selftests/mm/Makefile +2 -2
index 78dfec8bc676..a3eb20c186e7 100644
@@ -117,8 +117,8 @@ TEST_FILES += va_high_addr_switch.sh
 
 include ../lib.mk
 
-$(TEST_GEN_PROGS): vm_util.c
-$(TEST_GEN_FILES): vm_util.c
+$(TEST_GEN_PROGS): vm_util.c thp_settings.c
+$(TEST_GEN_FILES): vm_util.c thp_settings.c
 
 $(OUTPUT)/uffd-stress: uffd-common.c
 $(OUTPUT)/uffd-unit-tests: uffd-common.c
tools/testing/selftests/mm/khugepaged.c +22 -324
index fc47a1c4944c..b15e7fd70176 100644
@@ -22,13 +22,13 @@
 #include "linux/magic.h"
 
 #include "vm_util.h"
+#include "thp_settings.h"
 
 #define BASE_ADDR ((void *)(1UL << 30))
 static unsigned long hpage_pmd_size;
 static unsigned long page_size;
 static int hpage_pmd_nr;
 
-#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/"
 #define PID_SMAPS "/proc/self/smaps"
 #define TEST_FILE "collapse_test_file"
 
@@ -71,78 +71,7 @@ struct file_info {
 };
 
 static struct file_info finfo;
-
-enum thp_enabled {
-	THP_ALWAYS,
-	THP_MADVISE,
-	THP_NEVER,
-};
-
-static const char *thp_enabled_strings[] = {
-	"always",
-	"madvise",
-	"never",
-	NULL
-};
-
-enum thp_defrag {
-	THP_DEFRAG_ALWAYS,
-	THP_DEFRAG_DEFER,
-	THP_DEFRAG_DEFER_MADVISE,
-	THP_DEFRAG_MADVISE,
-	THP_DEFRAG_NEVER,
-};
-
-static const char *thp_defrag_strings[] = {
-	"always",
-	"defer",
-	"defer+madvise",
-	"madvise",
-	"never",
-	NULL
-};
-
-enum shmem_enabled {
-	SHMEM_ALWAYS,
-	SHMEM_WITHIN_SIZE,
-	SHMEM_ADVISE,
-	SHMEM_NEVER,
-	SHMEM_DENY,
-	SHMEM_FORCE,
-};
-
-static const char *shmem_enabled_strings[] = {
-	"always",
-	"within_size",
-	"advise",
-	"never",
-	"deny",
-	"force",
-	NULL
-};
-
-struct khugepaged_settings {
-	bool defrag;
-	unsigned int alloc_sleep_millisecs;
-	unsigned int scan_sleep_millisecs;
-	unsigned int max_ptes_none;
-	unsigned int max_ptes_swap;
-	unsigned int max_ptes_shared;
-	unsigned long pages_to_scan;
-};
-
-struct settings {
-	enum thp_enabled thp_enabled;
-	enum thp_defrag thp_defrag;
-	enum shmem_enabled shmem_enabled;
-	bool use_zero_page;
-	struct khugepaged_settings khugepaged;
-	unsigned long read_ahead_kb;
-};
-
-static struct settings saved_settings;
 static bool skip_settings_restore;
-
 static int exit_status;
 
 static void success(const char *msg)
@@ -161,226 +90,13 @@ static void skip(const char *msg)
 	printf(" \e[33m%s\e[0m\n", msg);
 }
 
-static int read_file(const char *path, char *buf, size_t buflen)
-{
-	int fd;
-	ssize_t numread;
-
-	fd = open(path, O_RDONLY);
-	if (fd == -1)
-		return 0;
-
-	numread = read(fd, buf, buflen - 1);
-	if (numread < 1) {
-		close(fd);
-		return 0;
-	}
-
-	buf[numread] = '\0';
-	close(fd);
-
-	return (unsigned int) numread;
-}
-
-static int write_file(const char *path, const char *buf, size_t buflen)
-{
-	int fd;
-	ssize_t numwritten;
-
-	fd = open(path, O_WRONLY);
-	if (fd == -1) {
-		printf("open(%s)\n", path);
-		exit(EXIT_FAILURE);
-		return 0;
-	}
-
-	numwritten = write(fd, buf, buflen - 1);
-	close(fd);
-	if (numwritten < 1) {
-		printf("write(%s)\n", buf);
-		exit(EXIT_FAILURE);
-		return 0;
-	}
-
-	return (unsigned int) numwritten;
-}
-
-static int read_string(const char *name, const char *strings[])
-{
-	char path[PATH_MAX];
-	char buf[256];
-	char *c;
-	int ret;
-
-	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
-	if (ret >= PATH_MAX) {
-		printf("%s: Pathname is too long\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-
-	if (!read_file(path, buf, sizeof(buf))) {
-		perror(path);
-		exit(EXIT_FAILURE);
-	}
-
-	c = strchr(buf, '[');
-	if (!c) {
-		printf("%s: Parse failure\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-
-	c++;
-	memmove(buf, c, sizeof(buf) - (c - buf));
-
-	c = strchr(buf, ']');
-	if (!c) {
-		printf("%s: Parse failure\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-	*c = '\0';
-
-	ret = 0;
-	while (strings[ret]) {
-		if (!strcmp(strings[ret], buf))
-			return ret;
-		ret++;
-	}
-
-	printf("Failed to parse %s\n", name);
-	exit(EXIT_FAILURE);
-}
-
-static void write_string(const char *name, const char *val)
-{
-	char path[PATH_MAX];
-	int ret;
-
-	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
-	if (ret >= PATH_MAX) {
-		printf("%s: Pathname is too long\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-
-	if (!write_file(path, val, strlen(val) + 1)) {
-		perror(path);
-		exit(EXIT_FAILURE);
-	}
-}
-
-static const unsigned long _read_num(const char *path)
-{
-	char buf[21];
-
-	if (read_file(path, buf, sizeof(buf)) < 0) {
-		perror("read_file(read_num)");
-		exit(EXIT_FAILURE);
-	}
-
-	return strtoul(buf, NULL, 10);
-}
-
-static const unsigned long read_num(const char *name)
-{
-	char path[PATH_MAX];
-	int ret;
-
-	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
-	if (ret >= PATH_MAX) {
-		printf("%s: Pathname is too long\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-	return _read_num(path);
-}
-
-static void _write_num(const char *path, unsigned long num)
-{
-	char buf[21];
-
-	sprintf(buf, "%ld", num);
-	if (!write_file(path, buf, strlen(buf) + 1)) {
-		perror(path);
-		exit(EXIT_FAILURE);
-	}
-}
-
-static void write_num(const char *name, unsigned long num)
-{
-	char path[PATH_MAX];
-	int ret;
-
-	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
-	if (ret >= PATH_MAX) {
-		printf("%s: Pathname is too long\n", __func__);
-		exit(EXIT_FAILURE);
-	}
-	_write_num(path, num);
-}
-
-static void write_settings(struct settings *settings)
-{
-	struct khugepaged_settings *khugepaged = &settings->khugepaged;
-
-	write_string("enabled", thp_enabled_strings[settings->thp_enabled]);
-	write_string("defrag", thp_defrag_strings[settings->thp_defrag]);
-	write_string("shmem_enabled",
-			shmem_enabled_strings[settings->shmem_enabled]);
-	write_num("use_zero_page", settings->use_zero_page);
-
-	write_num("khugepaged/defrag", khugepaged->defrag);
-	write_num("khugepaged/alloc_sleep_millisecs",
-			khugepaged->alloc_sleep_millisecs);
-	write_num("khugepaged/scan_sleep_millisecs",
-			khugepaged->scan_sleep_millisecs);
-	write_num("khugepaged/max_ptes_none", khugepaged->max_ptes_none);
-	write_num("khugepaged/max_ptes_swap", khugepaged->max_ptes_swap);
-	write_num("khugepaged/max_ptes_shared", khugepaged->max_ptes_shared);
-	write_num("khugepaged/pages_to_scan", khugepaged->pages_to_scan);
-
-	if (file_ops && finfo.type == VMA_FILE)
-		_write_num(finfo.dev_queue_read_ahead_path,
-			   settings->read_ahead_kb);
-}
-
-#define MAX_SETTINGS_DEPTH 4
-static struct settings settings_stack[MAX_SETTINGS_DEPTH];
-static int settings_index;
-
-static struct settings *current_settings(void)
-{
-	if (!settings_index) {
-		printf("Fail: No settings set");
-		exit(EXIT_FAILURE);
-	}
-	return settings_stack + settings_index - 1;
-}
-
-static void push_settings(struct settings *settings)
-{
-	if (settings_index >= MAX_SETTINGS_DEPTH) {
-		printf("Fail: Settings stack exceeded");
-		exit(EXIT_FAILURE);
-	}
-	settings_stack[settings_index++] = *settings;
-	write_settings(current_settings());
-}
-
-static void pop_settings(void)
-{
-	if (settings_index <= 0) {
-		printf("Fail: Settings stack empty");
-		exit(EXIT_FAILURE);
-	}
-	--settings_index;
-	write_settings(current_settings());
-}
-
 static void restore_settings_atexit(void)
 {
 	if (skip_settings_restore)
 		return;
 
 	printf("Restore THP and khugepaged settings...");
-	write_settings(&saved_settings);
+	thp_restore_settings();
 	success("OK");
 
 	skip_settings_restore = true;
@@ -395,27 +111,9 @@ static void restore_settings(int sig)
 static void save_settings(void)
 {
 	printf("Save THP and khugepaged settings...");
-	saved_settings = (struct settings) {
-		.thp_enabled = read_string("enabled", thp_enabled_strings),
-		.thp_defrag = read_string("defrag", thp_defrag_strings),
-		.shmem_enabled =
-			read_string("shmem_enabled", shmem_enabled_strings),
-		.use_zero_page = read_num("use_zero_page"),
-	};
-	saved_settings.khugepaged = (struct khugepaged_settings) {
-		.defrag = read_num("khugepaged/defrag"),
-		.alloc_sleep_millisecs =
-			read_num("khugepaged/alloc_sleep_millisecs"),
-		.scan_sleep_millisecs =
-			read_num("khugepaged/scan_sleep_millisecs"),
-		.max_ptes_none = read_num("khugepaged/max_ptes_none"),
-		.max_ptes_swap = read_num("khugepaged/max_ptes_swap"),
-		.max_ptes_shared = read_num("khugepaged/max_ptes_shared"),
-		.pages_to_scan = read_num("khugepaged/pages_to_scan"),
-	};
 	if (file_ops && finfo.type == VMA_FILE)
-		saved_settings.read_ahead_kb =
-				_read_num(finfo.dev_queue_read_ahead_path);
+		thp_set_read_ahead_path(finfo.dev_queue_read_ahead_path);
+	thp_save_settings();
 
 	success("OK");
 
@@ -798,7 +496,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,
 			       struct mem_ops *ops, bool expect)
 {
 	int ret;
-	struct settings settings = *current_settings();
+	struct thp_settings settings = *thp_current_settings();
 
 	printf("%s...", msg);
 
@@ -808,7 +506,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,
 	 */
 	settings.thp_enabled = THP_NEVER;
 	settings.shmem_enabled = SHMEM_NEVER;
-	push_settings(&settings);
+	thp_push_settings(&settings);
 
 	/* Clear VM_NOHUGEPAGE */
 	madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE);
@@ -820,7 +518,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages,
 	else
 		success("OK");
 
-	pop_settings();
+	thp_pop_settings();
 }
 
 static void madvise_collapse(const char *msg, char *p, int nr_hpages,
@@ -850,13 +548,13 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages,
 	madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE);
 
 	/* Wait until the second full_scan completed */
-	full_scans = read_num("khugepaged/full_scans") + 2;
+	full_scans = thp_read_num("khugepaged/full_scans") + 2;
 
 	printf("%s...", msg);
 	while (timeout--) {
 		if (ops->check_huge(p, nr_hpages))
 			break;
-		if (read_num("khugepaged/full_scans") >= full_scans)
+		if (thp_read_num("khugepaged/full_scans") >= full_scans)
 			break;
 		printf(".");
 		usleep(TICK);
@@ -911,11 +609,11 @@ static bool is_tmpfs(struct mem_ops *ops)
 
 static void alloc_at_fault(void)
 {
-	struct settings settings = *current_settings();
+	struct thp_settings settings = *thp_current_settings();
 	char *p;
 
 	settings.thp_enabled = THP_ALWAYS;
-	push_settings(&settings);
+	thp_push_settings(&settings);
 
 	p = alloc_mapping(1);
 	*p = 1;
@@ -925,7 +623,7 @@ static void alloc_at_fault(void)
 	else
 		fail("Fail");
 
-	pop_settings();
+	thp_pop_settings();
 
 	madvise(p, page_size, MADV_DONTNEED);
 	printf("Split huge PMD on MADV_DONTNEED...");
@@ -973,11 +671,11 @@ static void collapse_single_pte_entry(struct collapse_context *c, struct mem_ops
 static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *ops)
 {
 	int max_ptes_none = hpage_pmd_nr / 2;
-	struct settings settings = *current_settings();
+	struct thp_settings settings = *thp_current_settings();
 	void *p;
 
 	settings.khugepaged.max_ptes_none = max_ptes_none;
-	push_settings(&settings);
+	thp_push_settings(&settings);
 
 	p = ops->setup_area(1);
 
@@ -1002,7 +700,7 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o
 	}
 skip:
 	ops->cleanup_area(p, hpage_pmd_size);
-	pop_settings();
+	thp_pop_settings();
 }
 
 static void collapse_swapin_single_pte(struct collapse_context *c, struct mem_ops *ops)
@@ -1033,7 +731,7 @@ static void collapse_swapin_single_pte(struct collapse_context *c, struct mem_op
 
 static void collapse_max_ptes_swap(struct collapse_context *c, struct mem_ops *ops)
 {
-	int max_ptes_swap = read_num("khugepaged/max_ptes_swap");
+	int max_ptes_swap = thp_read_num("khugepaged/max_ptes_swap");
 	void *p;
 
 	p = ops->setup_area(1);
@@ -1250,11 +948,11 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o
 			fail("Fail");
 		ops->fault(p, 0, page_size);
 
-		write_num("khugepaged/max_ptes_shared", hpage_pmd_nr - 1);
+		thp_write_num("khugepaged/max_ptes_shared", hpage_pmd_nr - 1);
 		c->collapse("Collapse PTE table full of compound pages in child",
 			    p, 1, ops, true);
-		write_num("khugepaged/max_ptes_shared",
-			  current_settings()->khugepaged.max_ptes_shared);
+		thp_write_num("khugepaged/max_ptes_shared",
+			  thp_current_settings()->khugepaged.max_ptes_shared);
 
 		validate_memory(p, 0, hpage_pmd_size);
 		ops->cleanup_area(p, hpage_pmd_size);
@@ -1275,7 +973,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o
 
 static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops *ops)
 {
-	int max_ptes_shared = read_num("khugepaged/max_ptes_shared");
+	int max_ptes_shared = thp_read_num("khugepaged/max_ptes_shared");
 	int wstatus;
 	void *p;
 
@@ -1443,7 +1141,7 @@ static void parse_test_type(int argc, const char **argv)
 
 int main(int argc, const char **argv)
 {
-	struct settings default_settings = {
+	struct thp_settings default_settings = {
 		.thp_enabled = THP_MADVISE,
 		.thp_defrag = THP_DEFRAG_ALWAYS,
 		.shmem_enabled = SHMEM_ADVISE,
@@ -1484,7 +1182,7 @@ int main(int argc, const char **argv)
 	default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8;
 
 	save_settings();
-	push_settings(&default_settings);
+	thp_push_settings(&default_settings);
 
 	alloc_at_fault();
 
tools/testing/selftests/mm/thp_settings.c new file +296 -0
new file mode 100644
index 000000000000..5e8ec792cac7
@@ -0,0 +1,296 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <fcntl.h>
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "thp_settings.h"
+
+#define THP_SYSFS "/sys/kernel/mm/transparent_hugepage/"
+#define MAX_SETTINGS_DEPTH 4
+static struct thp_settings settings_stack[MAX_SETTINGS_DEPTH];
+static int settings_index;
+static struct thp_settings saved_settings;
+static char dev_queue_read_ahead_path[PATH_MAX];
+
+static const char * const thp_enabled_strings[] = {
+	"always",
+	"madvise",
+	"never",
+	NULL
+};
+
+static const char * const thp_defrag_strings[] = {
+	"always",
+	"defer",
+	"defer+madvise",
+	"madvise",
+	"never",
+	NULL
+};
+
+static const char * const shmem_enabled_strings[] = {
+	"always",
+	"within_size",
+	"advise",
+	"never",
+	"deny",
+	"force",
+	NULL
+};
+
+int read_file(const char *path, char *buf, size_t buflen)
+{
+	int fd;
+	ssize_t numread;
+
+	fd = open(path, O_RDONLY);
+	if (fd == -1)
+		return 0;
+
+	numread = read(fd, buf, buflen - 1);
+	if (numread < 1) {
+		close(fd);
+		return 0;
+	}
+
+	buf[numread] = '\0';
+	close(fd);
+
+	return (unsigned int) numread;
+}
+
+int write_file(const char *path, const char *buf, size_t buflen)
+{
+	int fd;
+	ssize_t numwritten;
+
+	fd = open(path, O_WRONLY);
+	if (fd == -1) {
+		printf("open(%s)\n", path);
+		exit(EXIT_FAILURE);
+		return 0;
+	}
+
+	numwritten = write(fd, buf, buflen - 1);
+	close(fd);
+	if (numwritten < 1) {
+		printf("write(%s)\n", buf);
+		exit(EXIT_FAILURE);
+		return 0;
+	}
+
+	return (unsigned int) numwritten;
+}
+
+const unsigned long read_num(const char *path)
+{
+	char buf[21];
+
+	if (read_file(path, buf, sizeof(buf)) < 0) {
+		perror("read_file()");
+		exit(EXIT_FAILURE);
+	}
+
+	return strtoul(buf, NULL, 10);
+}
+
+void write_num(const char *path, unsigned long num)
+{
+	char buf[21];
+
+	sprintf(buf, "%ld", num);
+	if (!write_file(path, buf, strlen(buf) + 1)) {
+		perror(path);
+		exit(EXIT_FAILURE);
+	}
+}
+
+int thp_read_string(const char *name, const char * const strings[])
+{
+	char path[PATH_MAX];
+	char buf[256];
+	char *c;
+	int ret;
+
+	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
+	if (ret >= PATH_MAX) {
+		printf("%s: Pathname is too long\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+
+	if (!read_file(path, buf, sizeof(buf))) {
+		perror(path);
+		exit(EXIT_FAILURE);
+	}
+
+	c = strchr(buf, '[');
+	if (!c) {
+		printf("%s: Parse failure\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+
+	c++;
+	memmove(buf, c, sizeof(buf) - (c - buf));
+
+	c = strchr(buf, ']');
+	if (!c) {
+		printf("%s: Parse failure\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+	*c = '\0';
+
+	ret = 0;
+	while (strings[ret]) {
+		if (!strcmp(strings[ret], buf))
+			return ret;
+		ret++;
+	}
+
+	printf("Failed to parse %s\n", name);
+	exit(EXIT_FAILURE);
+}
+
+void thp_write_string(const char *name, const char *val)
+{
+	char path[PATH_MAX];
+	int ret;
+
+	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
+	if (ret >= PATH_MAX) {
+		printf("%s: Pathname is too long\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+
+	if (!write_file(path, val, strlen(val) + 1)) {
+		perror(path);
+		exit(EXIT_FAILURE);
+	}
+}
+
+const unsigned long thp_read_num(const char *name)
+{
+	char path[PATH_MAX];
+	int ret;
+
+	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
+	if (ret >= PATH_MAX) {
+		printf("%s: Pathname is too long\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+	return read_num(path);
+}
+
+void thp_write_num(const char *name, unsigned long num)
+{
+	char path[PATH_MAX];
+	int ret;
+
+	ret = snprintf(path, PATH_MAX, THP_SYSFS "%s", name);
+	if (ret >= PATH_MAX) {
+		printf("%s: Pathname is too long\n", __func__);
+		exit(EXIT_FAILURE);
+	}
+	write_num(path, num);
+}
+
+void thp_read_settings(struct thp_settings *settings)
+{
+	*settings = (struct thp_settings) {
+		.thp_enabled = thp_read_string("enabled", thp_enabled_strings),
+		.thp_defrag = thp_read_string("defrag", thp_defrag_strings),
+		.shmem_enabled =
+			thp_read_string("shmem_enabled", shmem_enabled_strings),
+		.use_zero_page = thp_read_num("use_zero_page"),
+	};
+	settings->khugepaged = (struct khugepaged_settings) {
+		.defrag = thp_read_num("khugepaged/defrag"),
+		.alloc_sleep_millisecs =
+			thp_read_num("khugepaged/alloc_sleep_millisecs"),
+		.scan_sleep_millisecs =
+			thp_read_num("khugepaged/scan_sleep_millisecs"),
+		.max_ptes_none = thp_read_num("khugepaged/max_ptes_none"),
+		.max_ptes_swap = thp_read_num("khugepaged/max_ptes_swap"),
+		.max_ptes_shared = thp_read_num("khugepaged/max_ptes_shared"),
+		.pages_to_scan = thp_read_num("khugepaged/pages_to_scan"),
+	};
+	if (dev_queue_read_ahead_path[0])
+		settings->read_ahead_kb = read_num(dev_queue_read_ahead_path);
+}
+
+void thp_write_settings(struct thp_settings *settings)
+{
+	struct khugepaged_settings *khugepaged = &settings->khugepaged;
+
+	thp_write_string("enabled", thp_enabled_strings[settings->thp_enabled]);
+	thp_write_string("defrag", thp_defrag_strings[settings->thp_defrag]);
+	thp_write_string("shmem_enabled",
+			shmem_enabled_strings[settings->shmem_enabled]);
+	thp_write_num("use_zero_page", settings->use_zero_page);
+
+	thp_write_num("khugepaged/defrag", khugepaged->defrag);
+	thp_write_num("khugepaged/alloc_sleep_millisecs",
+			khugepaged->alloc_sleep_millisecs);
+	thp_write_num("khugepaged/scan_sleep_millisecs",
+			khugepaged->scan_sleep_millisecs);
+	thp_write_num("khugepaged/max_ptes_none", khugepaged->max_ptes_none);
+	thp_write_num("khugepaged/max_ptes_swap", khugepaged->max_ptes_swap);
+	thp_write_num("khugepaged/max_ptes_shared", khugepaged->max_ptes_shared);
+	thp_write_num("khugepaged/pages_to_scan", khugepaged->pages_to_scan);
+
+	if (dev_queue_read_ahead_path[0])
+		write_num(dev_queue_read_ahead_path, settings->read_ahead_kb);
+}
+
+struct thp_settings *thp_current_settings(void)
+{
+	if (!settings_index) {
+		printf("Fail: No settings set");
+		exit(EXIT_FAILURE);
+	}
+	return settings_stack + settings_index - 1;
+}
+
+void thp_push_settings(struct thp_settings *settings)
+{
+	if (settings_index >= MAX_SETTINGS_DEPTH) {
+		printf("Fail: Settings stack exceeded");
+		exit(EXIT_FAILURE);
+	}
+	settings_stack[settings_index++] = *settings;
+	thp_write_settings(thp_current_settings());
+}
+
+void thp_pop_settings(void)
+{
+	if (settings_index <= 0) {
+		printf("Fail: Settings stack empty");
+		exit(EXIT_FAILURE);
+	}
+	--settings_index;
+	thp_write_settings(thp_current_settings());
+}
+
+void thp_restore_settings(void)
+{
+	thp_write_settings(&saved_settings);
+}
+
+void thp_save_settings(void)
+{
+	thp_read_settings(&saved_settings);
+}
+
+void thp_set_read_ahead_path(char *path)
+{
+	if (!path) {
+		dev_queue_read_ahead_path[0] = '\0';
+		return;
+	}
+
+	strncpy(dev_queue_read_ahead_path, path,
+		sizeof(dev_queue_read_ahead_path));
+	dev_queue_read_ahead_path[sizeof(dev_queue_read_ahead_path) - 1] = '\0';
+}
tools/testing/selftests/mm/thp_settings.h new file +71 -0
new file mode 100644
index 000000000000..ff3d98c30617
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __THP_SETTINGS_H__
+#define __THP_SETTINGS_H__
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+
+enum thp_enabled {
+	THP_ALWAYS,
+	THP_MADVISE,
+	THP_NEVER,
+};
+
+enum thp_defrag {
+	THP_DEFRAG_ALWAYS,
+	THP_DEFRAG_DEFER,
+	THP_DEFRAG_DEFER_MADVISE,
+	THP_DEFRAG_MADVISE,
+	THP_DEFRAG_NEVER,
+};
+
+enum shmem_enabled {
+	SHMEM_ALWAYS,
+	SHMEM_WITHIN_SIZE,
+	SHMEM_ADVISE,
+	SHMEM_NEVER,
+	SHMEM_DENY,
+	SHMEM_FORCE,
+};
+
+struct khugepaged_settings {
+	bool defrag;
+	unsigned int alloc_sleep_millisecs;
+	unsigned int scan_sleep_millisecs;
+	unsigned int max_ptes_none;
+	unsigned int max_ptes_swap;
+	unsigned int max_ptes_shared;
+	unsigned long pages_to_scan;
+};
+
+struct thp_settings {
+	enum thp_enabled thp_enabled;
+	enum thp_defrag thp_defrag;
+	enum shmem_enabled shmem_enabled;
+	bool use_zero_page;
+	struct khugepaged_settings khugepaged;
+	unsigned long read_ahead_kb;
+};
+
+int read_file(const char *path, char *buf, size_t buflen);
+int write_file(const char *path, const char *buf, size_t buflen);
+const unsigned long read_num(const char *path);
+void write_num(const char *path, unsigned long num);
+
+int thp_read_string(const char *name, const char * const strings[]);
+void thp_write_string(const char *name, const char *val);
+const unsigned long thp_read_num(const char *name);
+void thp_write_num(const char *name, unsigned long num);
+
+void thp_write_settings(struct thp_settings *settings);
+void thp_read_settings(struct thp_settings *settings);
+struct thp_settings *thp_current_settings(void);
+void thp_push_settings(struct thp_settings *settings);
+void thp_pop_settings(void);
+void thp_restore_settings(void);
+void thp_save_settings(void);
+
+void thp_set_read_ahead_path(char *path);
+
+#endif /* __THP_SETTINGS_H__ */
author reviewer
#22 lore raw

PATCH RESEND v7 7/10 selftests/mm: Support small-sized THP interface in thp_settings

In reply to Ryan Roberts

Save and restore the new per-size hugepage enabled setting, if available on the running kernel.

Since the number of per-size directories is not fixed, solve this as simply as possible by catering for a maximum number in the thp_settings struct (20). Each array index is the order. The value of THP_NEVER is changed to 0 so that all of these new settings default to THP_NEVER and the user only needs to fill in the ones they want to enable.

---
 tools/testing/selftests/mm/khugepaged.c   |  3 ++
 tools/testing/selftests/mm/thp_settings.c | 55 ++++++++++++++++++++++-
 tools/testing/selftests/mm/thp_settings.h | 11 ++++-
 3 files changed, 67 insertions(+), 2 deletions(-)
tools/testing/selftests/mm/khugepaged.c +3 -0
index b15e7fd70176..473ba095cffd 100644
@@ -1141,6 +1141,7 @@ static void parse_test_type(int argc, const char **argv)
 
 int main(int argc, const char **argv)
 {
+	int hpage_pmd_order;
 	struct thp_settings default_settings = {
 		.thp_enabled = THP_MADVISE,
 		.thp_defrag = THP_DEFRAG_ALWAYS,
@@ -1175,11 +1176,13 @@ int main(int argc, const char **argv)
 		exit(EXIT_FAILURE);
 	}
 	hpage_pmd_nr = hpage_pmd_size / page_size;
+	hpage_pmd_order = __builtin_ctz(hpage_pmd_nr);
 
 	default_settings.khugepaged.max_ptes_none = hpage_pmd_nr - 1;
 	default_settings.khugepaged.max_ptes_swap = hpage_pmd_nr / 8;
 	default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr / 2;
 	default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8;
+	default_settings.hugepages[hpage_pmd_order].enabled = THP_GLOBAL;
 
 	save_settings();
 	thp_push_settings(&default_settings);
tools/testing/selftests/mm/thp_settings.c +54 -1

89 lines not on this page · raw message

tools/testing/selftests/mm/thp_settings.h +10 -1
index ff3d98c30617..8e71e323546c 100644
@@ -7,9 +7,10 @@
 #include <stdint.h>
 
 enum thp_enabled {
+	THP_NEVER,
 	THP_ALWAYS,
+	THP_GLOBAL,
 	THP_MADVISE,
-	THP_NEVER,
 };
 
 enum thp_defrag {
@@ -29,6 +30,12 @@ enum shmem_enabled {
 	SHMEM_FORCE,
 };
 
+#define NR_ORDERS 20
+
+struct hugepages_settings {
+	enum thp_enabled enabled;
+};
+
 struct khugepaged_settings {
 	bool defrag;
 	unsigned int alloc_sleep_millisecs;
@@ -46,6 +53,7 @@ struct thp_settings {
 	bool use_zero_page;
 	struct khugepaged_settings khugepaged;
 	unsigned long read_ahead_kb;
+	struct hugepages_settings hugepages[NR_ORDERS];
 };
 
 int read_file(const char *path, char *buf, size_t buflen);
@@ -67,5 +75,6 @@ void thp_restore_settings(void);
 void thp_save_settings(void);
 
 void thp_set_read_ahead_path(char *path);
+unsigned long thp_supported_orders(void);
 
 #endif /* __THP_SETTINGS_H__ */

--
2.25.1

author reviewer
#23 lore raw

PATCH RESEND v7 8/10 selftests/mm/khugepaged: Enlighten for small-sized THP

In reply to Ryan Roberts

The collapse_max_ptes_none test was previously failing when small-sized THP had enabled="always". The root cause is because the test faults in 1 page less than the threshold it set for collapsing. But when using small-sized THP we "over allocate" and therefore the threshold is passed, and collapse unexpectedly succeeds.

Solve this by enlightening khugepaged selftest. Add a command line option to pass in the desired small-sized THP that should be used for all anonymous allocations. The harness will then explicitly configure small-sized THP as requested and modify the collapse_max_ptes_none test so that it faults in the threshold minus the number of pages in the configured small-sized THP. If no command line option is provided, default to order 0, as per previous behaviour.

I chose to use an order in the command line interface, since this makes the interface agnostic of base page size, making it easier to invoke from run_vmtests.sh.

---
 tools/testing/selftests/mm/khugepaged.c   | 48 +++++++++++++++++------
 tools/testing/selftests/mm/run_vmtests.sh |  2 +
 2 files changed, 39 insertions(+), 11 deletions(-)
tools/testing/selftests/mm/khugepaged.c +37 -11
index 473ba095cffd..4d24f2eb158e 100644
@@ -28,6 +28,7 @@
 static unsigned long hpage_pmd_size;
 static unsigned long page_size;
 static int hpage_pmd_nr;
+static int anon_order;
 
 #define PID_SMAPS "/proc/self/smaps"
 #define TEST_FILE "collapse_test_file"
@@ -607,6 +608,11 @@ static bool is_tmpfs(struct mem_ops *ops)
 	return ops == &__file_ops && finfo.type == VMA_SHMEM;
 }
 
+static bool is_anon(struct mem_ops *ops)
+{
+	return ops == &__anon_ops;
+}
+
 static void alloc_at_fault(void)
 {
 	struct thp_settings settings = *thp_current_settings();
@@ -673,6 +679,7 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o
 	int max_ptes_none = hpage_pmd_nr / 2;
 	struct thp_settings settings = *thp_current_settings();
 	void *p;
+	int fault_nr_pages = is_anon(ops) ? 1 << anon_order : 1;
 
 	settings.khugepaged.max_ptes_none = max_ptes_none;
 	thp_push_settings(&settings);
@@ -686,10 +693,10 @@ static void collapse_max_ptes_none(struct collapse_context *c, struct mem_ops *o
 		goto skip;
 	}
 
-	ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none - 1) * page_size);
+	ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none - fault_nr_pages) * page_size);
 	c->collapse("Maybe collapse with max_ptes_none exceeded", p, 1,
 		    ops, !c->enforce_pte_scan_limits);
-	validate_memory(p, 0, (hpage_pmd_nr - max_ptes_none - 1) * page_size);
+	validate_memory(p, 0, (hpage_pmd_nr - max_ptes_none - fault_nr_pages) * page_size);
 
 	if (c->enforce_pte_scan_limits) {
 		ops->fault(p, 0, (hpage_pmd_nr - max_ptes_none) * page_size);
@@ -1076,7 +1083,7 @@ static void madvise_retracted_page_tables(struct collapse_context *c,
 
 static void usage(void)
 {
-	fprintf(stderr, "\nUsage: ./khugepaged <test type> [dir]\n\n");
+	fprintf(stderr, "\nUsage: ./khugepaged [OPTIONS] <test type> [dir]\n\n");
 	fprintf(stderr, "\t<test type>\t: <context>:<mem_type>\n");
 	fprintf(stderr, "\t<context>\t: [all|khugepaged|madvise]\n");
 	fprintf(stderr, "\t<mem_type>\t: [all|anon|file|shmem]\n");
@@ -1085,15 +1092,34 @@ static void usage(void)
 	fprintf(stderr,	"\tCONFIG_READ_ONLY_THP_FOR_FS=y\n");
 	fprintf(stderr, "\n\tif [dir] is a (sub)directory of a tmpfs mount, tmpfs must be\n");
 	fprintf(stderr,	"\tmounted with huge=madvise option for khugepaged tests to work\n");
+	fprintf(stderr,	"\n\tSupported Options:\n");
+	fprintf(stderr,	"\t\t-h: This help message.\n");
+	fprintf(stderr,	"\t\t-s: Small-sized THP size, expressed as page order.\n");
+	fprintf(stderr,	"\t\t    Defaults to 0. Use this size for anon allocations.\n");
 	exit(1);
 }
 
-static void parse_test_type(int argc, const char **argv)
+static void parse_test_type(int argc, char **argv)
 {
+	int opt;
 	char *buf;
 	const char *token;
 
-	if (argc == 1) {
+	while ((opt = getopt(argc, argv, "s:h")) != -1) {
+		switch (opt) {
+		case 's':
+			anon_order = atoi(optarg);
+			break;
+		case 'h':
+		default:
+			usage();
+		}
+	}
+
+	argv += optind;
+	argc -= optind;
+
+	if (argc == 0) {
 		/* Backwards compatibility */
 		khugepaged_context =  &__khugepaged_context;
 		madvise_context =  &__madvise_context;
@@ -1101,7 +1127,7 @@ static void parse_test_type(int argc, const char **argv)
 		return;
 	}
 
-	buf = strdup(argv[1]);
+	buf = strdup(argv[0]);
 	token = strsep(&buf, ":");
 
 	if (!strcmp(token, "all")) {
@@ -1135,11 +1161,13 @@ static void parse_test_type(int argc, const char **argv)
 	if (!file_ops)
 		return;
 
-	if (argc != 3)
+	if (argc != 2)
 		usage();
+
+	get_finfo(argv[1]);
 }
 
-int main(int argc, const char **argv)
+int main(int argc, char **argv)
 {
 	int hpage_pmd_order;
 	struct thp_settings default_settings = {
@@ -1164,9 +1192,6 @@ int main(int argc, const char **argv)
 
 	parse_test_type(argc, argv);
 
-	if (file_ops)
-		get_finfo(argv[2]);
-
 	setbuf(stdout, NULL);
 
 	page_size = getpagesize();
@@ -1183,6 +1208,7 @@ int main(int argc, const char **argv)
 	default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr / 2;
 	default_settings.khugepaged.pages_to_scan = hpage_pmd_nr * 8;
 	default_settings.hugepages[hpage_pmd_order].enabled = THP_GLOBAL;
+	default_settings.hugepages[anon_order].enabled = THP_ALWAYS;
 
 	save_settings();
 	thp_push_settings(&default_settings);
tools/testing/selftests/mm/run_vmtests.sh +2 -0
index 00757445278e..f3fa2238daef 100755
@@ -359,6 +359,8 @@ CATEGORY="cow" run_test ./cow
 
 CATEGORY="thp" run_test ./khugepaged
 
+CATEGORY="thp" run_test ./khugepaged -s 2
+
 CATEGORY="thp" run_test ./transhuge-stress -d 20
 
 CATEGORY="thp" run_test ./split_huge_page_test

--
2.25.1

author reviewer
#24 lore raw

PATCH RESEND v7 9/10 selftests/mm/cow: Generalize do_run_with_thp() helper

In reply to Ryan Roberts

do_run_with_thp() prepares (PMD-sized) THP memory into different states before running tests. With the introduction of small-sized THP, we would like to reuse this logic to also test those smaller THP sizes. So let's add a size parameter which tells the function what size THP it should operate on.

A separate commit will utilize this change to add new tests for small-sized THP, where available.

---
 tools/testing/selftests/mm/cow.c | 146 +++++++++++++++++--------------
 1 file changed, 79 insertions(+), 67 deletions(-)
tools/testing/selftests/mm/cow.c +79 -67
index 7324ce5363c0..d03c453cfd5c 100644
@@ -32,7 +32,7 @@
 
 static size_t pagesize;
 static int pagemap_fd;
-static size_t thpsize;
+static size_t pmdsize;
 static int nr_hugetlbsizes;
 static size_t hugetlbsizes[10];
 static int gup_fd;
@@ -734,14 +734,14 @@ enum thp_run {
 	THP_RUN_PARTIAL_SHARED,
 };
 
-static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
+static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t size)
 {
 	char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED;
-	size_t size, mmap_size, mremap_size;
+	size_t mmap_size, mremap_size;
 	int ret;
 
-	/* For alignment purposes, we need twice the thp size. */
-	mmap_size = 2 * thpsize;
+	/* For alignment purposes, we need twice the requested size. */
+	mmap_size = 2 * size;
 	mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE,
 			MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (mmap_mem == MAP_FAILED) {
@@ -749,36 +749,40 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		return;
 	}
 
-	/* We need a THP-aligned memory area. */
-	mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1));
+	/* We need to naturally align the memory area. */
+	mem = (char *)(((uintptr_t)mmap_mem + size) & ~(size - 1));
 
-	ret = madvise(mem, thpsize, MADV_HUGEPAGE);
+	ret = madvise(mem, size, MADV_HUGEPAGE);
 	if (ret) {
 		ksft_test_result_fail("MADV_HUGEPAGE failed\n");
 		goto munmap;
 	}
 
 	/*
-	 * Try to populate a THP. Touch the first sub-page and test if we get
-	 * another sub-page populated automatically.
+	 * Try to populate a THP. Touch the first sub-page and test if
+	 * we get the last sub-page populated automatically.
 	 */
 	mem[0] = 0;
-	if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) {
+	if (!pagemap_is_populated(pagemap_fd, mem + size - pagesize)) {
 		ksft_test_result_skip("Did not get a THP populated\n");
 		goto munmap;
 	}
-	memset(mem, 0, thpsize);
+	memset(mem, 0, size);
 
-	size = thpsize;
 	switch (thp_run) {
 	case THP_RUN_PMD:
 	case THP_RUN_PMD_SWAPOUT:
+		if (size != pmdsize) {
+			ksft_test_result_fail("test bug: can't PMD-map size\n");
+			goto munmap;
+		}
 		break;
 	case THP_RUN_PTE:
 	case THP_RUN_PTE_SWAPOUT:
 		/*
 		 * Trigger PTE-mapping the THP by temporarily mapping a single
-		 * subpage R/O.
+		 * subpage R/O. This is a noop if the THP is not pmdsize (and
+		 * therefore already PTE-mapped).
 		 */
 		ret = mprotect(mem + pagesize, pagesize, PROT_READ);
 		if (ret) {
@@ -797,7 +801,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		 * Discard all but a single subpage of that PTE-mapped THP. What
 		 * remains is a single PTE mapping a single subpage.
 		 */
-		ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DONTNEED);
+		ret = madvise(mem + pagesize, size - pagesize, MADV_DONTNEED);
 		if (ret) {
 			ksft_test_result_fail("MADV_DONTNEED failed\n");
 			goto munmap;
@@ -809,7 +813,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		 * Remap half of the THP. We need some new memory location
 		 * for that.
 		 */
-		mremap_size = thpsize / 2;
+		mremap_size = size / 2;
 		mremap_mem = mmap(NULL, mremap_size, PROT_NONE,
 				  MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 		if (mem == MAP_FAILED) {
@@ -830,7 +834,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		 * child. This will result in some parts of the THP never
 		 * have been shared.
 		 */
-		ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DONTFORK);
+		ret = madvise(mem + pagesize, size - pagesize, MADV_DONTFORK);
 		if (ret) {
 			ksft_test_result_fail("MADV_DONTFORK failed\n");
 			goto munmap;
@@ -844,7 +848,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		}
 		wait(&ret);
 		/* Allow for sharing all pages again. */
-		ret = madvise(mem + pagesize, thpsize - pagesize, MADV_DOFORK);
+		ret = madvise(mem + pagesize, size - pagesize, MADV_DOFORK);
 		if (ret) {
 			ksft_test_result_fail("MADV_DOFORK failed\n");
 			goto munmap;
@@ -875,52 +879,60 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
 		munmap(mremap_mem, mremap_size);
 }
 
-static void run_with_thp(test_fn fn, const char *desc)
+static void run_with_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD);
+	ksft_print_msg("[RUN] %s ... with THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD, size);
 }
 
-static void run_with_thp_swap(test_fn fn, const char *desc)
+static void run_with_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PMD_SWAPOUT, size);
 }
 
-static void run_with_pte_mapped_thp(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE);
+	ksft_print_msg("[RUN] %s ... with PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE, size);
 }
 
-static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc)
+static void run_with_pte_mapped_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with swapped-out, PTE-mapped THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PTE_SWAPOUT, size);
 }
 
-static void run_with_single_pte_of_thp(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE);
+	ksft_print_msg("[RUN] %s ... with single PTE of THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE, size);
 }
 
-static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc)
+static void run_with_single_pte_of_thp_swap(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT);
+	ksft_print_msg("[RUN] %s ... with single PTE of swapped-out THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_SINGLE_PTE_SWAPOUT, size);
 }
 
-static void run_with_partial_mremap_thp(test_fn fn, const char *desc)
+static void run_with_partial_mremap_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP);
+	ksft_print_msg("[RUN] %s ... with partially mremap()'ed THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_MREMAP, size);
 }
 
-static void run_with_partial_shared_thp(test_fn fn, const char *desc)
+static void run_with_partial_shared_thp(test_fn fn, const char *desc, size_t size)
 {
-	ksft_print_msg("[RUN] %s ... with partially shared THP\n", desc);
-	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED);
+	ksft_print_msg("[RUN] %s ... with partially shared THP (%zu kB)\n",
+		desc, size / 1024);
+	do_run_with_thp(fn, THP_RUN_PARTIAL_SHARED, size);
 }
 
 static void run_with_hugetlb(test_fn fn, const char *desc, size_t hugetlbsize)
@@ -1091,15 +1103,15 @@ static void run_anon_test_case(struct test_case const *test_case)
 
 	run_with_base_page(test_case->fn, test_case->desc);
 	run_with_base_page_swap(test_case->fn, test_case->desc);
-	if (thpsize) {
-		run_with_thp(test_case->fn, test_case->desc);
-		run_with_thp_swap(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp(test_case->fn, test_case->desc);
-		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp(test_case->fn, test_case->desc);
-		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc);
-		run_with_partial_mremap_thp(test_case->fn, test_case->desc);
-		run_with_partial_shared_thp(test_case->fn, test_case->desc);
+	if (pmdsize) {
+		run_with_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize);
+		run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize);
 	}
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_hugetlb(test_case->fn, test_case->desc,
@@ -1120,7 +1132,7 @@ static int tests_per_anon_test_case(void)
 {
 	int tests = 2 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 8;
 	return tests;
 }
@@ -1329,7 +1341,7 @@ static void run_anon_thp_test_cases(void)
 {
 	int i;
 
-	if (!thpsize)
+	if (!pmdsize)
 		return;
 
 	ksft_print_msg("[INFO] Anonymous THP tests\n");
@@ -1338,13 +1350,13 @@ static void run_anon_thp_test_cases(void)
 		struct test_case const *test_case = &anon_thp_test_cases[i];
 
 		ksft_print_msg("[RUN] %s\n", test_case->desc);
-		do_run_with_thp(test_case->fn, THP_RUN_PMD);
+		do_run_with_thp(test_case->fn, THP_RUN_PMD, pmdsize);
 	}
 }
 
 static int tests_per_anon_thp_test_case(void)
 {
-	return thpsize ? 1 : 0;
+	return pmdsize ? 1 : 0;
 }
 
 typedef void (*non_anon_test_fn)(char *mem, const char *smem, size_t size);
@@ -1419,7 +1431,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* For alignment purposes, we need twice the thp size. */
-	mmap_size = 2 * thpsize;
+	mmap_size = 2 * pmdsize;
 	mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE,
 			MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 	if (mmap_mem == MAP_FAILED) {
@@ -1434,11 +1446,11 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 	}
 
 	/* We need a THP-aligned memory area. */
-	mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1));
-	smem = (char *)(((uintptr_t)mmap_smem + thpsize) & ~(thpsize - 1));
+	mem = (char *)(((uintptr_t)mmap_mem + pmdsize) & ~(pmdsize - 1));
+	smem = (char *)(((uintptr_t)mmap_smem + pmdsize) & ~(pmdsize - 1));
 
-	ret = madvise(mem, thpsize, MADV_HUGEPAGE);
-	ret |= madvise(smem, thpsize, MADV_HUGEPAGE);
+	ret = madvise(mem, pmdsize, MADV_HUGEPAGE);
+	ret |= madvise(smem, pmdsize, MADV_HUGEPAGE);
 	if (ret) {
 		ksft_test_result_fail("MADV_HUGEPAGE failed\n");
 		goto munmap;
@@ -1457,7 +1469,7 @@ static void run_with_huge_zeropage(non_anon_test_fn fn, const char *desc)
 		goto munmap;
 	}
 
-	fn(mem, smem, thpsize);
+	fn(mem, smem, pmdsize);
 munmap:
 	munmap(mmap_mem, mmap_size);
 	if (mmap_smem != MAP_FAILED)
@@ -1650,7 +1662,7 @@ static void run_non_anon_test_case(struct non_anon_test_case const *test_case)
 	run_with_zeropage(test_case->fn, test_case->desc);
 	run_with_memfd(test_case->fn, test_case->desc);
 	run_with_tmpfile(test_case->fn, test_case->desc);
-	if (thpsize)
+	if (pmdsize)
 		run_with_huge_zeropage(test_case->fn, test_case->desc);
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_memfd_hugetlb(test_case->fn, test_case->desc,
@@ -1671,7 +1683,7 @@ static int tests_per_non_anon_test_case(void)
 {
 	int tests = 3 + nr_hugetlbsizes;
 
-	if (thpsize)
+	if (pmdsize)
 		tests += 1;
 	return tests;
 }
@@ -1681,10 +1693,10 @@ int main(int argc, char **argv)
 	int err;
 
 	pagesize = getpagesize();
-	thpsize = read_pmd_pagesize();
-	if (thpsize)
-		ksft_print_msg("[INFO] detected THP size: %zu KiB\n",
-			       thpsize / 1024);
+	pmdsize = read_pmd_pagesize();
+	if (pmdsize)
+		ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n",
+			       pmdsize / 1024);
 	nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
 						    ARRAY_SIZE(hugetlbsizes));
 	detect_huge_zeropage();

--
2.25.1

#25 lore raw

Re: [RESEND PATCH v7 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper

Quoted 28 lines · Ryan Roberts

do_run_with_thp() prepares (PMD-sized) THP memory into different states before running tests. With the introduction of small-sized THP, we would like to reuse this logic to also test those smaller THP sizes. So let's add a size parameter which tells the function what size THP it should operate on.

A separate commit will utilize this change to add new tests for small-sized THP, where available.

---
  tools/testing/selftests/mm/cow.c | 146 +++++++++++++++++--------------
  1 file changed, 79 insertions(+), 67 deletions(-)
tools/testing/selftests/mm/cow.c +1 -1
index 7324ce5363c0..d03c453cfd5c 100644
@@ -32,7 +32,7 @@
 
  static size_t pagesize;
  static int pagemap_fd;
-static size_t thpsize;
+static size_t pmdsize;
  static int nr_hugetlbsizes;
  static size_t hugetlbsizes[10];
  static int gup_fd;

tools/testing/selftests/mm/cow.c enum thp_run { · line 737

Quoted 5 lines
tools/testing/selftests/mm/cow.c +1 -1
@@ -734,14 +734,14 @@ enum thp_run {
  	THP_RUN_PARTIAL_SHARED,
  };
 
-static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
+static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t size)

Nit: can we still call it "thpsize" in this function? That makes it clearer IMHO and avoids most renaming.

Quoted 31 lines
tools/testing/selftests/mm/cow.c +6 -7
  {
  	char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED;
-	size_t size, mmap_size, mremap_size;
+	size_t mmap_size, mremap_size;
  	int ret;
 
-	/* For alignment purposes, we need twice the thp size. */
-	mmap_size = 2 * thpsize;
+	/* For alignment purposes, we need twice the requested size. */
+	mmap_size = 2 * size;
  	mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE,
  			MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
  	if (mmap_mem == MAP_FAILED) {
@@ -749,36 +749,40 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
  		return;
  	}
 
-	/* We need a THP-aligned memory area. */
-	mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1));
+	/* We need to naturally align the memory area. */
+	mem = (char *)(((uintptr_t)mmap_mem + size) & ~(size - 1));
 
-	ret = madvise(mem, thpsize, MADV_HUGEPAGE);
+	ret = madvise(mem, size, MADV_HUGEPAGE);
  	if (ret) {
  		ksft_test_result_fail("MADV_HUGEPAGE failed\n");
  		goto munmap;
  	}
 
  	/*
-	 * Try to populate a THP. Touch the first sub-page and test if we get

tools/testing/selftests/mm/cow.c static void do_run_with_thp(test_fn fn, enum thp_run thp_run) · line 769

Quoted 10 lines
tools/testing/selftests/mm/cow.c +3 -2
-	 * another sub-page populated automatically.
+	 * Try to populate a THP. Touch the first sub-page and test if
+	 * we get the last sub-page populated automatically.
  	 */
  	mem[0] = 0;
-	if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) {
+	if (!pagemap_is_populated(pagemap_fd, mem + size - pagesize)) {
  		ksft_test_result_skip("Did not get a THP populated\n");
  		goto munmap;
  	}

Yes! I have a patch lying around here that does that same. :)

I guess there is no need to set MADV_NOHUGEPAGE on the remainder of the mmap'ed are:

Assume we want a 64KiB thp. We mmap'ed 128KiB. If we get a reasonably aligned area, we might populate a 128KiB THP.

But I assume the MADV_HUGEPAGE will in all configurations properly create a separate 64KiB VMA and we'll never get 128 KiB populated. So this should work reliably.

tools/testing/selftests/mm/cow.c static void do_run_with_thp(test_fn fn, enum thp_run thp_run) · line 778

Quoted 11 lines
tools/testing/selftests/mm/cow.c +5 -2
-	memset(mem, 0, thpsize);
+	memset(mem, 0, size);
 
-	size = thpsize;
  	switch (thp_run) {
  	case THP_RUN_PMD:
  	case THP_RUN_PMD_SWAPOUT:
+		if (size != pmdsize) {
+			ksft_test_result_fail("test bug: can't PMD-map size\n");
+			goto munmap;
+		}

Maybe rather "assert()" because that's a real BUG in the test?

[...]

tools/testing/selftests/mm/cow.c int main(int argc, char **argv) · line 1698

Quoted 3 lines
tools/testing/selftests/mm/cow.c +3 -0
+	pmdsize = read_pmd_pagesize();
+	if (pmdsize)
+		ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n",

Maybe simply: "detected PMD size". Zes, we read it via the THP interface, but that shouldn't matter much.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#26 lore raw

Re: [RESEND PATCH v7 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper

Quoted 32 lines · David Hildenbrand
Quoted 31 lines · Ryan Roberts

do_run_with_thp() prepares (PMD-sized) THP memory into different states before running tests. With the introduction of small-sized THP, we would like to reuse this logic to also test those smaller THP sizes. So let's add a size parameter which tells the function what size THP it should operate on.

A separate commit will utilize this change to add new tests for small-sized THP, where available.

---
  tools/testing/selftests/mm/cow.c | 146 +++++++++++++++++--------------

  1 file changed, 79 insertions(+), 67 deletions(-)
tools/testing/selftests/mm/cow.c +1 -1
index 7324ce5363c0..d03c453cfd5c 100644
@@ -32,7 +32,7 @@
 
   static size_t pagesize;
   static int pagemap_fd;
-static size_t thpsize;
+static size_t pmdsize;
   static int nr_hugetlbsizes;
   static size_t hugetlbsizes[10];
   static int gup_fd;
@@ -734,14 +734,14 @@ enum thp_run {
       THP_RUN_PARTIAL_SHARED,
   };
 
Quoted 5 lines

tools/testing/selftests/mm/cow.c enum thp_run { · line 737

Quoted 2 lines
tools/testing/selftests/mm/cow.c +1 -1
-static void do_run_with_thp(test_fn fn, enum thp_run thp_run)
+static void do_run_with_thp(test_fn fn, enum thp_run thp_run, size_t size)

Nit: can we still call it "thpsize" in this function? That makes it clearer IMHO and avoids most renaming.

Yep no problem. Will fix in next version.

Quoted 49 lines
Quoted 33 lines
tools/testing/selftests/mm/cow.c +6 -7
   {
       char *mem, *mmap_mem, *tmp, *mremap_mem = MAP_FAILED;
-    size_t size, mmap_size, mremap_size;
+    size_t mmap_size, mremap_size;
       int ret;
 
-    /* For alignment purposes, we need twice the thp size. */
-    mmap_size = 2 * thpsize;
+    /* For alignment purposes, we need twice the requested size. */
+    mmap_size = 2 * size;
       mmap_mem = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
       if (mmap_mem == MAP_FAILED) {
@@ -749,36 +749,40 @@ static void do_run_with_thp(test_fn fn, enum thp_run
 thp_run)
           return;
       }
 
-    /* We need a THP-aligned memory area. */
-    mem = (char *)(((uintptr_t)mmap_mem + thpsize) & ~(thpsize - 1));
+    /* We need to naturally align the memory area. */
+    mem = (char *)(((uintptr_t)mmap_mem + size) & ~(size - 1));
 
-    ret = madvise(mem, thpsize, MADV_HUGEPAGE);
+    ret = madvise(mem, size, MADV_HUGEPAGE);
       if (ret) {
           ksft_test_result_fail("MADV_HUGEPAGE failed\n");
           goto munmap;
       }
 
       /*
-     * Try to populate a THP. Touch the first sub-page and test if we get

tools/testing/selftests/mm/cow.c static void do_run_with_thp(test_fn fn, enum thp_run · line 770

Quoted 10 lines
tools/testing/selftests/mm/cow.c +3 -2
-     * another sub-page populated automatically.
+     * Try to populate a THP. Touch the first sub-page and test if
+     * we get the last sub-page populated automatically.
        */
       mem[0] = 0;
-    if (!pagemap_is_populated(pagemap_fd, mem + pagesize)) {
+    if (!pagemap_is_populated(pagemap_fd, mem + size - pagesize)) {
           ksft_test_result_skip("Did not get a THP populated\n");
           goto munmap;
       }

Yes! I have a patch lying around here that does that same. :)

I guess there is no need to set MADV_NOHUGEPAGE on the remainder of the mmap'ed are:

Quoted 6 lines

Assume we want a 64KiB thp. We mmap'ed 128KiB. If we get a reasonably aligned area, we might populate a 128KiB THP.

But I assume the MADV_HUGEPAGE will in all configurations properly create a separate 64KiB VMA and we'll never get 128 KiB populated. So this should work reliably.

Yes agreed. And also, we explicitly only enable a single THP size at a time so should only allocate a THP of the expected size. Perhaps we should mark the whole mmap area with MADV_HUGEPAGE since that will serve as a test that we only get the smaller size we configured?

Quoted 13 lines
Quoted 6 lines
tools/testing/selftests/mm/cow.c +1 -2
-    memset(mem, 0, thpsize);
+    memset(mem, 0, size);
 
-    size = thpsize;
       switch (thp_run) {
Quoted 6 lines

      case THP_RUN_PMD:
      case THP_RUN_PMD_SWAPOUT:
+        if (size != pmdsize) {
+            ksft_test_result_fail("test bug: can't PMD-map size\n"); +            goto munmap;
+        }

Quoted 1 line

Maybe rather "assert()" because that's a real BUG in the test?

Yep will do.

Quoted 9 lines

[...]

Quoted 3 lines

+    pmdsize = read_pmd_pagesize();
+    if (pmdsize)
+        ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n",

Maybe simply: "detected PMD size". Zes, we read it via the THP interface, but that shouldn't matter much.

Err, just want to clarify what you are suggesting. With the current patch you will see something like:

[INFO] detected PMD-mapped THP size: 2048 KiB
[INFO] detected small-sized THP size: 64 KiB
[INFO] detected small-sized THP size: 128 KiB
...
[INFO] detected small-sized THP size: 1024 KiB

Are you suggesting something like this:

[INFO] detected PMD size: 2048 KiB
[INFO] detected THP size: 64 KiB
[INFO] detected THP size: 128 KiB
...
[INFO] detected THP size: 2048 KiB

Quoted 1 line
#27 lore raw

Re: [RESEND PATCH v7 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

+    pmdsize = read_pmd_pagesize();
+    if (pmdsize)
+        ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n",

Maybe simply: "detected PMD size". Zes, we read it via the THP interface, but that shouldn't matter much.

Err, just want to clarify what you are suggesting. With the current patch you will see something like:

Not with this patch, but with the other ones, yes :)

Quoted 8 lines

[INFO] detected PMD-mapped THP size: 2048 KiB [INFO] detected small-sized THP size: 64 KiB [INFO] detected small-sized THP size: 128 KiB
...
[INFO] detected small-sized THP size: 1024 KiB

Quoted 8 lines

Are you suggesting something like this:

[INFO] detected PMD size: 2048 KiB [INFO] detected THP size: 64 KiB [INFO] detected THP size: 128 KiB
...
[INFO] detected THP size: 2048 KiB

Yes. If you'd detect that 2M is actually disabled, you could still log the PMD size only.

So for this patch only as a preparation

[INFO] detected PMD size: 2048 KiB
[INFO] detected THP size: 2048 KiB

Just a thought.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#28 lore raw

Re: [RESEND PATCH v7 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper

Quoted 5 lines · David Hildenbrand
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+    pmdsize = read_pmd_pagesize();
+    if (pmdsize)
+        ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n",

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Maybe simply: "detected PMD size". Zes, we read it via the THP interface, but that shouldn't matter much.

Err, just want to clarify what you are suggesting. With the current patch you will see something like:

Not with this patch, but with the other ones, yes :)

Yep, we are on the same page (folio)...

Quoted 18 lines
Quoted 9 lines

[INFO] detected PMD-mapped THP size: 2048 KiB [INFO] detected small-sized THP size: 64 KiB [INFO] detected small-sized THP size: 128 KiB
...
[INFO] detected small-sized THP size: 1024 KiB

Quoted 8 lines

Are you suggesting something like this:

[INFO] detected PMD size: 2048 KiB [INFO] detected THP size: 64 KiB [INFO] detected THP size: 128 KiB
...
[INFO] detected THP size: 2048 KiB

Quoted 2 lines

Yes. If you'd detect that 2M is actually disabled, you could still log the PMD size only.

Note that in the final patch, where I test the other THP sizes, I'm not detecting which sizes the sysadmin has enabled, I'm detecting the set of sizes that can be enabled, then explicitly enabling the size (exclusively) when I test it. So there is no chance of reading PMD size but not having 2M THP. Minor point though.

Quoted 7 lines

So for this patch only as a preparation

[INFO] detected PMD size: 2048 KiB
[INFO] detected THP size: 2048 KiB

Just a thought.

Yep this is exactly how I've just reworked it.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#29 lore raw

Re: [RESEND PATCH v7 09/10] selftests/mm/cow: Generalize do_run_with_thp() helper

Quoted 34 lines · Ryan Roberts
Quoted 5 lines · David Hildenbrand
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+    pmdsize = read_pmd_pagesize();
+    if (pmdsize)
+        ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n",

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Maybe simply: "detected PMD size". Zes, we read it via the THP interface, but that shouldn't matter much.

Err, just want to clarify what you are suggesting. With the current patch you will see something like:

Not with this patch, but with the other ones, yes :)

Yep, we are on the same page (folio)...

Quoted 18 lines
Quoted 17 lines

[INFO] detected PMD-mapped THP size: 2048 KiB [INFO] detected small-sized THP size: 64 KiB [INFO] detected small-sized THP size: 128 KiB
...
[INFO] detected small-sized THP size: 1024 KiB

Are you suggesting something like this:

[INFO] detected PMD size: 2048 KiB [INFO] detected THP size: 64 KiB [INFO] detected THP size: 128 KiB
...
[INFO] detected THP size: 2048 KiB

Quoted 8 lines
Quoted 2 lines

Yes. If you'd detect that 2M is actually disabled, you could still log the PMD size only.

Note that in the final patch, where I test the other THP sizes, I'm not detecting which sizes the sysadmin has enabled, I'm detecting the set of sizes that can be enabled, then explicitly enabling the size (exclusively) when I test it. So there is no chance of reading PMD size but not having 2M THP. Minor point though.

Ah, makes sense, I missed that and thought we'd check if that size is actually possible.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#30 lore raw

PATCH RESEND v7 10/10 selftests/mm/cow: Add tests for anonymous small-sized THP

In reply to Ryan Roberts

Add tests similar to the existing PMD-sized THP tests, but which operate on memory backed by (PTE-mapped) small-sized THP. This reuses all the existing infrastructure. If the test suite detects that small-sized THP is not supported by the kernel, the new tests are skipped.

---
 tools/testing/selftests/mm/cow.c | 71 +++++++++++++++++++++++++++++++-
 1 file changed, 70 insertions(+), 1 deletion(-)
tools/testing/selftests/mm/cow.c +70 -1
index d03c453cfd5c..3efc395c7077 100644
@@ -29,15 +29,49 @@
 #include "../../../../mm/gup_test.h"
 #include "../kselftest.h"
 #include "vm_util.h"
+#include "thp_settings.h"
 
 static size_t pagesize;
 static int pagemap_fd;
 static size_t pmdsize;
+static int nr_thpsmallsizes;
+static size_t thpsmallsizes[20];
 static int nr_hugetlbsizes;
 static size_t hugetlbsizes[10];
 static int gup_fd;
 static bool has_huge_zeropage;
 
+static int sz2ord(size_t size)
+{
+	return __builtin_ctzll(size / pagesize);
+}
+
+static int detect_smallthp_sizes(size_t sizes[], int max)
+{
+	int count = 0;
+	unsigned long orders;
+	size_t kb;
+	int i;
+
+	/* thp not supported at all. */
+	if (!pmdsize)
+		return 0;
+
+	orders = thp_supported_orders();
+
+	/* Only interested in small-sized THP (less than PMD-size). */
+	for (i = 0; i < sz2ord(pmdsize); i++) {
+		if (!(orders & (1UL << i)))
+			continue;
+		kb = (pagesize >> 10) << i;
+		sizes[count++] = kb * 1024;
+		ksft_print_msg("[INFO] detected small-sized THP size: %zu KiB\n",
+			       kb);
+	}
+
+	return count;
+}
+
 static void detect_huge_zeropage(void)
 {
 	int fd = open("/sys/kernel/mm/transparent_hugepage/use_zero_page",
@@ -1113,6 +1147,23 @@ static void run_anon_test_case(struct test_case const *test_case)
 		run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize);
 		run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize);
 	}
+	for (i = 0; i < nr_thpsmallsizes; i++) {
+		size_t size = thpsmallsizes[i];
+		struct thp_settings settings = *thp_current_settings();
+
+		settings.hugepages[sz2ord(pmdsize)].enabled = THP_NEVER;
+		settings.hugepages[sz2ord(size)].enabled = THP_ALWAYS;
+		thp_push_settings(&settings);
+
+		run_with_pte_mapped_thp(test_case->fn, test_case->desc, size);
+		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, size);
+		run_with_single_pte_of_thp(test_case->fn, test_case->desc, size);
+		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, size);
+		run_with_partial_mremap_thp(test_case->fn, test_case->desc, size);
+		run_with_partial_shared_thp(test_case->fn, test_case->desc, size);
+
+		thp_pop_settings();
+	}
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_hugetlb(test_case->fn, test_case->desc,
 				 hugetlbsizes[i]);
@@ -1134,6 +1185,7 @@ static int tests_per_anon_test_case(void)
 
 	if (pmdsize)
 		tests += 8;
+	tests += 6 * nr_thpsmallsizes;
 	return tests;
 }
 
@@ -1691,12 +1743,24 @@ static int tests_per_non_anon_test_case(void)
 int main(int argc, char **argv)
 {
 	int err;
+	struct thp_settings default_settings;
 
 	pagesize = getpagesize();
 	pmdsize = read_pmd_pagesize();
-	if (pmdsize)
+	if (pmdsize) {
+		/* Only if THP is supported. */
+		thp_read_settings(&default_settings);
+		default_settings.hugepages[sz2ord(pmdsize)].enabled = THP_GLOBAL;
+		thp_save_settings();
+		thp_push_settings(&default_settings);
+
 		ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n",
 			       pmdsize / 1024);
+
+		nr_thpsmallsizes = detect_smallthp_sizes(thpsmallsizes,
+						    ARRAY_SIZE(thpsmallsizes));
+	}
+
 	nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
 						    ARRAY_SIZE(hugetlbsizes));
 	detect_huge_zeropage();
@@ -1715,6 +1779,11 @@ int main(int argc, char **argv)
 	run_anon_thp_test_cases();
 	run_non_anon_test_cases();
 
+	if (pmdsize) {
+		/* Only if THP is supported. */
+		thp_restore_settings();
+	}
+
 	err = ksft_get_fail_cnt();
 	if (err)
 		ksft_exit_fail_msg("%d out of %d tests failed\n",

--
2.25.1

author reviewer
#31 lore raw

Re: [RESEND PATCH v7 10/10] selftests/mm/cow: Add tests for anonymous small-sized THP

Quoted 15 lines · Ryan Roberts

Add tests similar to the existing PMD-sized THP tests, but which operate on memory backed by (PTE-mapped) small-sized THP. This reuses all the existing infrastructure. If the test suite detects that small-sized THP is not supported by the kernel, the new tests are skipped.

---
 tools/testing/selftests/mm/cow.c | 71 +++++++++++++++++++++++++++++++-
 1 file changed, 70 insertions(+), 1 deletion(-)
tools/testing/selftests/mm/cow.c +0 -0
index d03c453cfd5c..3efc395c7077 100644

tools/testing/selftests/mm/cow.c line 38

Quoted 10 lines
tools/testing/selftests/mm/cow.c +3 -0
@@ -29,15 +29,49 @@
 #include "../../../../mm/gup_test.h"
 #include "../kselftest.h"
 #include "vm_util.h"
+#include "thp_settings.h"
 
 static size_t pagesize;
 static int pagemap_fd;
 static size_t pmdsize;
+static int nr_thpsmallsizes;
+static size_t thpsmallsizes[20];

Off the back of some comments David made againt the previous patch [1], I'm proposing to rework this a bit so that ALL thp sizes are stored in this array, not just the non-PMD-sized sizes. This makes the changes overall a bit smaller and easier to understand, I think...

tools/testing/selftests/mm/cow.c line 49

Quoted 11 lines
tools/testing/selftests/mm/cow.c +6 -0
 static int nr_hugetlbsizes;
 static size_t hugetlbsizes[10];
 static int gup_fd;
 static bool has_huge_zeropage;
 
+static int sz2ord(size_t size)
+{
+	return __builtin_ctzll(size / pagesize);
+}
+
+static int detect_smallthp_sizes(size_t sizes[], int max)

This changes to detect_thp_sizes() and deposits all sizes in sizes[]

Quoted 10 lines
tools/testing/selftests/mm/cow.c +10 -0
+{
+	int count = 0;
+	unsigned long orders;
+	size_t kb;
+	int i;
+
+	/* thp not supported at all. */
+	if (!pmdsize)
+		return 0;
+

tools/testing/selftests/mm/cow.c line 69

Quoted 10 lines
tools/testing/selftests/mm/cow.c +10 -0
+	orders = thp_supported_orders();
+
+	/* Only interested in small-sized THP (less than PMD-size). */
+	for (i = 0; i < sz2ord(pmdsize); i++) {
+		if (!(orders & (1UL << i)))
+			continue;
+		kb = (pagesize >> 10) << i;
+		sizes[count++] = kb * 1024;
+		ksft_print_msg("[INFO] detected small-sized THP size: %zu KiB\n",
+			       kb);

This just prints "[INFO] detected THP size: %zu KiB"

Quoted 19 lines
tools/testing/selftests/mm/cow.c +12 -0
+	}
+
+	return count;
+}
+
 static void detect_huge_zeropage(void)
 {
 	int fd = open("/sys/kernel/mm/transparent_hugepage/use_zero_page",
@@ -1113,6 +1147,23 @@ static void run_anon_test_case(struct test_case const *test_case)
 		run_with_partial_mremap_thp(test_case->fn, test_case->desc, pmdsize);
 		run_with_partial_shared_thp(test_case->fn, test_case->desc, pmdsize);
 	}
+	for (i = 0; i < nr_thpsmallsizes; i++) {
+		size_t size = thpsmallsizes[i];
+		struct thp_settings settings = *thp_current_settings();
+
+		settings.hugepages[sz2ord(pmdsize)].enabled = THP_NEVER;
+		settings.hugepages[sz2ord(size)].enabled = THP_ALWAYS;
+		thp_push_settings(&settings);

tools/testing/selftests/mm/cow.c static void run_anon_test_case(struct test_case const *test_case) · line 1166

Quoted 10 lines
tools/testing/selftests/mm/cow.c +10 -0
+
+		run_with_pte_mapped_thp(test_case->fn, test_case->desc, size);
+		run_with_pte_mapped_thp_swap(test_case->fn, test_case->desc, size);
+		run_with_single_pte_of_thp(test_case->fn, test_case->desc, size);
+		run_with_single_pte_of_thp_swap(test_case->fn, test_case->desc, size);
+		run_with_partial_mremap_thp(test_case->fn, test_case->desc, size);
+		run_with_partial_shared_thp(test_case->fn, test_case->desc, size);
+
+		thp_pop_settings();
+	}

This same loop covers the pmdsize tests too, and I've just added a conditional that runs the 2 extra tests that are pmdsize only.

Quoted 50 lines
tools/testing/selftests/mm/cow.c +19 -2
 	for (i = 0; i < nr_hugetlbsizes; i++)
 		run_with_hugetlb(test_case->fn, test_case->desc,
 				 hugetlbsizes[i]);
@@ -1134,6 +1185,7 @@ static int tests_per_anon_test_case(void)
 
 	if (pmdsize)
 		tests += 8;
+	tests += 6 * nr_thpsmallsizes;
 	return tests;
 }
 
@@ -1691,12 +1743,24 @@ static int tests_per_non_anon_test_case(void)
 int main(int argc, char **argv)
 {
 	int err;
+	struct thp_settings default_settings;
 
 	pagesize = getpagesize();
 	pmdsize = read_pmd_pagesize();
-	if (pmdsize)
+	if (pmdsize) {
+		/* Only if THP is supported. */
+		thp_read_settings(&default_settings);
+		default_settings.hugepages[sz2ord(pmdsize)].enabled = THP_GLOBAL;
+		thp_save_settings();
+		thp_push_settings(&default_settings);
+
 		ksft_print_msg("[INFO] detected PMD-mapped THP size: %zu KiB\n",
 			       pmdsize / 1024);
+
+		nr_thpsmallsizes = detect_smallthp_sizes(thpsmallsizes,
+						    ARRAY_SIZE(thpsmallsizes));
+	}
+
 	nr_hugetlbsizes = detect_hugetlb_page_sizes(hugetlbsizes,
 						    ARRAY_SIZE(hugetlbsizes));
 	detect_huge_zeropage();
@@ -1715,6 +1779,11 @@ int main(int argc, char **argv)
 	run_anon_thp_test_cases();
 	run_non_anon_test_cases();
 
+	if (pmdsize) {
+		/* Only if THP is supported. */
+		thp_restore_settings();
+	}
+
 	err = ksft_get_fail_cnt();
 	if (err)
 		ksft_exit_fail_msg("%d out of %d tests failed\n",
--
Quoted 2 lines

2.25.1

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#32 lore raw

Re: [RESEND PATCH v7 10/10] selftests/mm/cow: Add tests for anonymous small-sized THP

Quoted 37 lines · Ryan Roberts
Quoted 15 lines · Ryan Roberts

Add tests similar to the existing PMD-sized THP tests, but which operate on memory backed by (PTE-mapped) small-sized THP. This reuses all the existing infrastructure. If the test suite detects that small-sized THP is not supported by the kernel, the new tests are skipped.

---
  tools/testing/selftests/mm/cow.c | 71 +++++++++++++++++++++++++++++++-
  1 file changed, 70 insertions(+), 1 deletion(-)
tools/testing/selftests/mm/cow.c +0 -0
index d03c453cfd5c..3efc395c7077 100644

tools/testing/selftests/mm/cow.c line 38

Quoted 10 lines
tools/testing/selftests/mm/cow.c +3 -0
@@ -29,15 +29,49 @@
  #include "../../../../mm/gup_test.h"
  #include "../kselftest.h"
  #include "vm_util.h"
+#include "thp_settings.h"
 
  static size_t pagesize;
  static int pagemap_fd;
  static size_t pmdsize;
+static int nr_thpsmallsizes;
+static size_t thpsmallsizes[20];

Off the back of some comments David made againt the previous patch [1], I'm proposing to rework this a bit so that ALL thp sizes are stored in this array, not just the non-PMD-sized sizes. This makes the changes overall a bit smaller and easier to understand, I think...

Quoted 5 lines
tools/testing/selftests/mm/cow.c +0 -0
  static int nr_hugetlbsizes;
  static size_t hugetlbsizes[10];
  static int gup_fd;
  static bool has_huge_zeropage;
 
Quoted 8 lines

tools/testing/selftests/mm/cow.c line 49

Quoted 6 lines
tools/testing/selftests/mm/cow.c +6 -0
+static int sz2ord(size_t size)
+{
+	return __builtin_ctzll(size / pagesize);
+}
+
+static int detect_smallthp_sizes(size_t sizes[], int max)

This changes to detect_thp_sizes() and deposits all sizes in sizes[]

Just what I wanted to propose :) Makes it simpler by removing the "small" terminology and just detecting thp sizes.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#33 lore raw

Re: [RESEND PATCH v7 10/10] selftests/mm/cow: Add tests for anonymous small-sized THP

Quoted 47 lines · David Hildenbrand
Quoted 37 lines · Ryan Roberts
Quoted 15 lines · Ryan Roberts

Add tests similar to the existing PMD-sized THP tests, but which operate on memory backed by (PTE-mapped) small-sized THP. This reuses all the existing infrastructure. If the test suite detects that small-sized THP is not supported by the kernel, the new tests are skipped.

---
  tools/testing/selftests/mm/cow.c | 71 +++++++++++++++++++++++++++++++-

  1 file changed, 70 insertions(+), 1 deletion(-)
tools/testing/selftests/mm/cow.c +0 -0
index d03c453cfd5c..3efc395c7077 100644

tools/testing/selftests/mm/cow.c line 38

Quoted 10 lines
tools/testing/selftests/mm/cow.c +3 -0
@@ -29,15 +29,49 @@
   #include "../../../../mm/gup_test.h"
   #include "../kselftest.h"
   #include "vm_util.h"
+#include "thp_settings.h"
 
   static size_t pagesize;
   static int pagemap_fd;
   static size_t pmdsize;
+static int nr_thpsmallsizes;
+static size_t thpsmallsizes[20];

Off the back of some comments David made againt the previous patch [1], I'm proposing to rework this a bit so that ALL thp sizes are stored in this array, not just the non-PMD-sized sizes. This makes the changes overall a bit smaller and easier to understand, I think...

Quoted 5 lines
tools/testing/selftests/mm/cow.c +0 -0
   static int nr_hugetlbsizes;
   static size_t hugetlbsizes[10];
   static int gup_fd;
   static bool has_huge_zeropage;
 
Quoted 8 lines

tools/testing/selftests/mm/cow.c line 49

Quoted 6 lines
tools/testing/selftests/mm/cow.c +6 -0
+static int sz2ord(size_t size)
+{
+    return __builtin_ctzll(size / pagesize);
+}
+
+static int detect_smallthp_sizes(size_t sizes[], int max)

This changes to detect_thp_sizes() and deposits all sizes in sizes[]

Quoted 2 lines

Just what I wanted to propose :) Makes it simpler by removing the "small" terminology and just detecting thp sizes.

Great! All done and queued up for the next version. Just waiting for some feedback on the 2 crucial patches in this series (3 & 4) :)

#34 lore raw
In reply to Ryan Roberts
Quoted 4 lines · Ryan Roberts

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

I'll have a look either this week or next week. Very high on my todo list :)

Signature · 3 lines
Cheers,

David / dhildenb
#35 lore raw
In reply to Ryan Roberts

On 11/22/23 08:29, Ryan Roberts wrote:
...

Quoted 15 lines

Prerequisites
=============

Some work items identified as being prerequisites are listed on page 3 at [8]. The summary is:

| item                          | status                  |
|:------------------------------|:------------------------|
| mlock                         | In mainline (v6.7)      |
| madvise                       | In mainline (v6.6)      |
| compaction                    | v1 posted [9]           |
| numa balancing                | Investigated: see below |
| user-triggered page migration | In mainline (v6.7)      |
| khugepaged collapse           | In mainline (NOP)       |
Quoted 7 lines

On NUMA balancing, which currently ignores any PTE-mapped THPs it encounters, John Hubbard has investigated this and concluded that it is A) not clear at the moment what a better policy might be for PTE-mapped THP and B) questions whether this should really be considered a prerequisite given no regression is caused for the default "small-sized THP disabled" case, and there is no correctness issue when it is enabled - its just a potential for non-optimal performance. (John please do elaborate if I haven't captured this correctly!)

That's accurate. I actually want to continue looking into this (Mel Gorman's recent replies to v6 provided helpful touchstones to the NUMA reasoning leading up to the present day), and maybe at least bring pte-thps into rough parity with THPs with respect to NUMA.

But that really doesn't seem like something that needs to happen first, especially since the outcome might even be, "first, do no harm"--as in, it's better as-is. We'll see.

Quoted 18 lines

If there are no disagreements about removing numa balancing from the list, then that just leaves compaction which is in review on list at the moment.

I really would like to get this series (and its remaining comapction prerequisite) in for v6.8. I accept that it may be a bit optimistic at this point, but lets see where we get to with review?

Testing
=======

The series includes patches for mm selftests to enlighten the cow and khugepaged tests to explicitly test with small-order THP, in the same way that PMD-order THP is tested. The new tests all pass, and no regressions are observed in the mm selftest suite. I've also run my usual kernel compilation and java script benchmarks without any issues.

Quoted 7 lines

Refer to my performance numbers posted with v6 [6]. (These are for small-sized THP only - they do not include the arm64 contpte follow-on series).

John Hubbard at Nvidia has indicated dramatic 10x performance improvements for some workloads at [10]. (Observed using v6 of this series as well as the arm64 contpte series).

Testing continues. Some workloads do even much better than than 10x, it's quite remarkable and glorious to see. :) I can send more perf data perhaps in a few days or a week, if there is still doubt about the benefits.

That was with the v6 series, though. I'm about to set up and run with v7, and expect to provide a tested by tag for functionality, sometime soon (in the next few days), if machine availability works out as expected.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#36 lore raw
In reply to Ryan Roberts
Quoted 8 lines · Ryan Roberts

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

Hi All,

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

#37 lore raw
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Ryan Roberts

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

Hi All,

Quoted 8 lines
Quoted 2 lines

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

I disagree.

https://lore.kernel.org/all/[email protected]/

Signature · 3 lines
Cheers,

David / dhildenb
#38 lore raw
Quoted 17 lines · David Hildenbrand
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Ryan Roberts

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

Hi All,

Quoted 8 lines
Quoted 2 lines

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 3 lines

I disagree.

https://lore.kernel.org/all/[email protected]/

And yet:
https://lore.kernel.org/linux-mm/[email protected]/

"This is a small THP so we don't account it as a THP, we only account normal THPs as THPs" is a bizarre position to take.

Not to mention that saying a foo is a small huge baz is just bizarre. Am I a small giant? Or just a large human?

#39 lore raw
Quoted 22 lines · Matthew Wilcox
Quoted 17 lines · David Hildenbrand
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Ryan Roberts

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

Hi All,

Quoted 8 lines
Quoted 2 lines

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 3 lines

I disagree.

https://lore.kernel.org/all/[email protected]/

Quoted 8 lines

And yet:
https://lore.kernel.org/linux-mm/[email protected]/

"This is a small THP so we don't account it as a THP, we only account normal THPs as THPs" is a bizarre position to take.

Not to mention that saying a foo is a small huge baz is just bizarre. Am I a small giant? Or just a large human?

I like that analogy. Yet, "small giant" sounds "bigger" in some way IMHO ;)

I'll note that "small-sized THP" is just a temporary feature name, it won't be exposed as such to the user in sysfs etc. In a couple of years, it will be forgotten.

To me it makes sense: it's a hugepage (not a page) but smaller compared to what we previously had. But again, there won't be a "small_thp" toggle anywhere.

Long-term it's simply going to be a THP. Quoting from my writeup:

"Nowadays, when somebody says that they are using hugetlb huge pages, the first question frequently is "which huge page size?". The same will happen with transparent huge pages I believe.".

Regarding the accounting: as I said a couple of times, "AnonHugePages" should have been called "AnonPmdMapped" or similar; that's what it really is: as soon as a THP is PTE-mapped, it's not accounted there. But we can't fix that I guess, unless we add some "world switch" for any workloads that would care about a different accounting.

So we're really only concerned about:
* AnonHugePages
* ShmemHugePages
* FileHugePages

The question is if we really want to continue extending/adjusting the old meminfo interfaces and talk about how to perform accounting there.

Because, as we learned, we might get a new file-based sysfs based interface, because Greg seems to be against exposing new values in the old single-file-based one.

In a new one, we have all freedom to expose what we actually want nowadays, and can just document that the old interface was designed with the assumption that there is only a single THP size.

... like hugetlb, where we also only expose the "default hugetlb size" parameters for legacy reasons:

HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Signature · 3 lines
Cheers,

David / dhildenb
#40 lore raw
Quoted 44 lines · David Hildenbrand
Quoted 22 lines · Matthew Wilcox
Quoted 17 lines · David Hildenbrand
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Ryan Roberts

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

Hi All,

Quoted 8 lines
Quoted 2 lines

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP.  We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized.  It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 3 lines

I disagree.

https://lore.kernel.org/all/[email protected]/

Quoted 8 lines

And yet:
https://lore.kernel.org/linux-mm/[email protected]/

"This is a small THP so we don't account it as a THP, we only account normal THPs as THPs" is a bizarre position to take.

Not to mention that saying a foo is a small huge baz is just bizarre. Am I a small giant?  Or just a large human?

I like that analogy. Yet, "small giant" sounds "bigger" in some way IMHO ;)

I'll note that "small-sized THP" is just a temporary feature name, it won't be exposed as such to the user in sysfs etc. In a couple of years, it will be forgotten.

To me it makes sense: it's a hugepage (not a page) but smaller compared to what we previously had. But again, there won't be a "small_thp" toggle anywhere.

Long-term it's simply going to be a THP. Quoting from my writeup:

"Nowadays, when somebody says that they are using hugetlb huge pages, the first question frequently is "which huge page size?". The same will happen with transparent huge pages I believe.".

Quoted 6 lines

Regarding the accounting: as I said a couple of times, "AnonHugePages" should have been called "AnonPmdMapped" or similar; that's what it really is: as soon as a THP is PTE-mapped, it's not accounted there. But we can't fix that I guess, unless we add some "world switch" for any workloads that would care about a different accounting.

So we're really only concerned about:
* AnonHugePages
* ShmemHugePages
* FileHugePages

The v6 patchset had these counters:

/proc/vmstat: nr_anon_thp_pte
/proc/meminfo: AnonHugePteMap

...which leads to another naming possibility: pte-thp, or pte-mapped-thp, something along those lines.

pte-thp avoids the "small huge" complaint, at least.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
reviewer
#41 lore raw
In reply to David Hildenbrand
Quoted 34 lines · David Hildenbrand
Quoted 22 lines · Matthew Wilcox
Quoted 17 lines · David Hildenbrand
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Ryan Roberts

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

Hi All,

Quoted 8 lines
Quoted 2 lines

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 3 lines

I disagree.

https://lore.kernel.org/all/[email protected]/

Quoted 8 lines

And yet:
https://lore.kernel.org/linux-mm/[email protected]/

"This is a small THP so we don't account it as a THP, we only account normal THPs as THPs" is a bizarre position to take.

Not to mention that saying a foo is a small huge baz is just bizarre. Am I a small giant? Or just a large human?

I like that analogy. Yet, "small giant" sounds "bigger" in some way IMHO ;)

Quoted 8 lines

I'll note that "small-sized THP" is just a temporary feature name, it won't be exposed as such to the user in sysfs etc. In a couple of years, it will be forgotten.

To me it makes sense: it's a hugepage (not a page) but smaller compared to what we previously had. But again, there won't be a "small_thp" toggle anywhere.

Long-term it's simply going to be a THP. Quoting from my writeup:

"Nowadays, when somebody says that they are using hugetlb huge pages, the first question frequently is "which huge page size?". The same will happen with transparent huge pages I believe.".

I agree. Especially our ultimate goal is to auto-tune THP sizes to give the best performance to user. Having a separate name for small sized THP is beneficial to kernel developers, since we want to use the right THP size for right workloads/scenarios. But for average user, it is better to keep interface as simple as possible, so that they can just turn on THP and get good performance boost. For ninja users, I assume they know differences between THP sizes to not confuse themselves and we can expose fine tune interfaces if really necessary.

Quoted 4 lines

Regarding the accounting: as I said a couple of times, "AnonHugePages" should have been called "AnonPmdMapped" or similar; that's what it really is: as soon as a THP is PTE-mapped, it's not accounted there. But we can't fix that I guess, unless we add some "world switch" for any workloads that would care about a different accounting.

Quoted 8 lines

So we're really only concerned about:
* AnonHugePages
* ShmemHugePages
* FileHugePages

The question is if we really want to continue extending/adjusting the old meminfo interfaces and talk about how to perform accounting there.

Because, as we learned, we might get a new file-based sysfs based interface, because Greg seems to be against exposing new values in the old single-file-based one.

I am not aware of this. And it is interesting. Do you have a pointer?

Quoted 2 lines

In a new one, we have all freedom to expose what we actually want nowadays, and can just document that the old interface was designed with the assumption that there is only a single THP size.

This sounds like a good strategy and hopefully we could design the new THP interface more future proof.

Quoted 13 lines

... like hugetlb, where we also only expose the "default hugetlb size" parameters for legacy reasons:

HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Signature · 3 lines
Cheers,

David / dhildenb

--
Best Regards,
Yan, Zi

#42 lore raw
Quoted 10 lines
Quoted 8 lines

So we're really only concerned about:
* AnonHugePages
* ShmemHugePages
* FileHugePages

The question is if we really want to continue extending/adjusting the old meminfo interfaces and talk about how to perform accounting there.

Because, as we learned, we might get a new file-based sysfs based interface, because Greg seems to be against exposing new values in the old single-file-based one.

I am not aware of this. And it is interesting. Do you have a pointer?

Sure:

https://lore.kernel.org/all/2023110216-labrador-neurosis-1e6e@gregkh/T/#u

Quoted 6 lines
Quoted 3 lines

In a new one, we have all freedom to expose what we actually want nowadays, and can just document that the old interface was designed with the assumption that there is only a single THP size.

This sounds like a good strategy and hopefully we could design the new THP interface more future proof.

Yes!

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#43 lore raw
In reply to Matthew Wilcox
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Ryan Roberts

Note: I'm resending this at Andrew's suggestion due to having originally sent it during LPC. I'm hoping its in a position where the feedback is minor enough that I can rework in time for v6.8, but so far haven't had any.

Hi All,

Quoted 7 lines
Quoted 2 lines

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

I suspect I'm labouring the point here, but I'd like to drill into exactly what you are objecting to. Is it:

A) Using the name "small-sized THP" (which is currently only used in the commit logs and a couple of times in the documentation).

B) Exposing the controls for this feature as an extension to the existing /sys/kernel/mm/transparent_hugepage/* sysfs interface (note the interface never uses the term "small-sized").

If A) then this is easily solved by choosing another descriptive name and updating those places. Personally I think it would be best to continue to use "THP" since we are exposing the feature through that interface. Perhaps "large folio THP".

If B) we could move the interface to /sys/kernel/mm/large_folio/*, but that introduces many more banana skins than the current approach IMHO:

- We would still want to expose the PMD-size large folio through this new interface and so would still need "global" or equivalent for at least PMD size, but "global" now points to a completely different sibling directory structure. And it probably doesn't make any sense for the non-PMD-sizes to have "global" because that would imply the THP interface could control the non-PMD-sizes, which is what we are trying to separate in the first place. So we end up with an asymmetry.

- When we get to adding other feature support for the smaller sizes (e.g. khugepaged), we will end up having to duplicate all the controls from transparent_hugepage/* to large_folio/*, then we have the problem that e.g. scan rates could differ and we would end up needing 2 separate daemons.

On the interface, David and I did request feedback on the proposal a number of times before I coded it up.

I'm sure all solvable eventually, but I personally think it is overall simpler and more understandable as it is. I also agree with the other points raised in favor of "small-sized THP".

Thanks,
Ryan

#44 lore raw
Quoted 10 lines · Ryan Roberts
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Ryan Roberts

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 5 lines

I suspect I'm labouring the point here, but I'd like to drill into exactly what you are objecting to. Is it:

A) Using the name "small-sized THP" (which is currently only used in the commit logs and a couple of times in the documentation).

Yes, this is what I'm objecting to.

Quoted 3 lines

B) Exposing the controls for this feature as an extension to the existing /sys/kernel/mm/transparent_hugepage/* sysfs interface (note the interface never uses the term "small-sized").

I don't object to the controls being here. I still wish we didn't need an interface to control them at all, but I don't have the time to become an expert in anonymous memory and figure out how to make that happen.

Quoted 4 lines

If A) then this is easily solved by choosing another descriptive name and updating those places. Personally I think it would be best to continue to use "THP" since we are exposing the feature through that interface. Perhaps "large folio THP".

I think that continues the confusion about the existing interfaces we have which count THP (and mean "PMD sized THP"). I'd really prefer the term "THP" to unambiguously mean PMD sized THP. I don't understand why you felt the need to move away from Large Anon Folios as a name.

author reviewer
#45 lore raw
Quoted 32 lines · Matthew Wilcox
Quoted 10 lines · Ryan Roberts
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Ryan Roberts

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 5 lines

I suspect I'm labouring the point here, but I'd like to drill into exactly what you are objecting to. Is it:

A) Using the name "small-sized THP" (which is currently only used in the commit logs and a couple of times in the documentation).

Yes, this is what I'm objecting to.

Quoted 3 lines

B) Exposing the controls for this feature as an extension to the existing /sys/kernel/mm/transparent_hugepage/* sysfs interface (note the interface never uses the term "small-sized").

I don't object to the controls being here. I still wish we didn't need an interface to control them at all, but I don't have the time to become an expert in anonymous memory and figure out how to make that happen.

Quoted 4 lines

If A) then this is easily solved by choosing another descriptive name and updating those places. Personally I think it would be best to continue to use "THP" since we are exposing the feature through that interface. Perhaps "large folio THP".

Quoted 5 lines

I think that continues the confusion about the existing interfaces we have which count THP (and mean "PMD sized THP"). I'd really prefer the term "THP" to unambiguously mean PMD sized THP. I don't understand why you felt the need to move away from Large Anon Folios as a name.

Because the controls are exposed in the sysfs THP directory (and therefore documented in the transhuge.rst document). It seems odd to refer to them as large anon folios within the kernel but expose them as as part of the THP interface.

But I'm certainly open to the idea of changing the name in the commit logs and being careful to distance it from THP transhuge.rst if that's the concensus. I am opposed to moving/changing the interface though - that's actually what I thought you were suggesting.

#46 lore raw
In reply to Matthew Wilcox
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · Ryan Roberts
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Ryan Roberts

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 7 lines
Quoted 5 lines

I suspect I'm labouring the point here, but I'd like to drill into exactly what you are objecting to. Is it:

A) Using the name "small-sized THP" (which is currently only used in the commit logs and a couple of times in the documentation).

Yes, this is what I'm objecting to.

I'll just repeat that "large anon folio" is misleading, because
* we already have "large anon folios" in hugetlb
* we already have PMD-sized "large anon folios" in THP

But inn the end, I don't care how we will call this in a commit message.

Just sticking to what we have right now makes most sense to me.

I know, as the creator of the term "folio" you have to object :P Sorry ;)

Signature · 3 lines
Cheers,

David / dhildenb
#47 lore raw
Quoted 15 lines · David Hildenbrand
Quoted 14 lines · Matthew Wilcox
Quoted 13 lines · Ryan Roberts
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Ryan Roberts

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

I suspect I'm labouring the point here, but I'd like to drill into exactly what you are objecting to. Is it:

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

A) Using the name "small-sized THP" (which is currently only used in the commit logs and a couple of times in the documentation).

Yes, this is what I'm objecting to.

I'll just repeat that "large anon folio" is misleading, because
* we already have "large anon folios" in hugetlb

We do? Where?

Quoted 1 line

* we already have PMD-sized "large anon folios" in THP

Right, those are already accounted as THP, and that's what users expect. If we're allocating 1024 x 64kB chunks of memory, the user won't be able to distinguish that from 32 x 2MB chunks of memory, and yet the performance profile for some applications will be very different.

Quoted 5 lines

But inn the end, I don't care how we will call this in a commit message.

Just sticking to what we have right now makes most sense to me.

I know, as the creator of the term "folio" you have to object :P Sorry ;)

I don't care if it's called something to do with folios or not. I am objecting to the use of the term "small THP" on the grounds of confusion and linguistic nonsense.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#48 lore raw
Quoted 19 lines · Matthew Wilcox
Quoted 18 lines · David Hildenbrand
Quoted 17 lines · Matthew Wilcox
Quoted 15 lines · Ryan Roberts
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Ryan Roberts

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

I suspect I'm labouring the point here, but I'd like to drill into exactly what you are objecting to. Is it:

A) Using the name "small-sized THP" (which is currently only used in the commit logs and a couple of times in the documentation).

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yes, this is what I'm objecting to.

I'll just repeat that "large anon folio" is misleading, because
* we already have "large anon folios" in hugetlb

We do? Where?

MAP_PRIVATE of hugetlb. hugepage_add_anon_rmap() instantiates them.

Hugetlb is likely one of the oldest user of compund pages aka large folios.

Quoted 7 lines
Quoted 2 lines

* we already have PMD-sized "large anon folios" in THP

Right, those are already accounted as THP, and that's what users expect. If we're allocating 1024 x 64kB chunks of memory, the user won't be able to distinguish that from 32 x 2MB chunks of memory, and yet the performance profile for some applications will be very different.

Very right, and because there will be a difference between 1024 x 64kB, 2048 x 32 kB and so forth, we need new memory stats either way.

Ryan had some ideas on that, but currently, that's considered future work, just like it likely is for the pagecache as well and needs much more thoughts.

Initially, the admin will have to enable all that for anon either way. It all boils down to one memory statistic for anon memory (AnonHugePages) that's messed-up already.

Quoted 8 lines
Quoted 6 lines

But inn the end, I don't care how we will call this in a commit message.

Just sticking to what we have right now makes most sense to me.

I know, as the creator of the term "folio" you have to object :P Sorry ;)

I don't care if it's called something to do with folios or not. I

Good!

Quoted 2 lines

am objecting to the use of the term "small THP" on the grounds of confusion and linguistic nonsense.

Maybe that's the reason why FreeBSD calls them "medium-sized superpages", because "Medium-sized" seems to be more appropriate to express something "in between".

So far I thought the reason was because they focused on 64k only.

Never trust a German guy on naming suggestions. John has so far been my naming expert, so I'm hoping he can help.

"Sub-pmd-sized THP" is just mouthful. But then, again, this is would just be a temporary name, and in the future THP will just naturally come in multiple sizes (and others here seem to agree on that).

But just to repeat: I don't think there is need to come up with new terminology and that there will be mass-confusion. So far I've not heard a compelling argument besides "one memory counter could confuse an admin that explicitly enables that new behavior.".

Side note: I'm, happy that we've reached a stage where we're nitpicking on names :)

Signature · 3 lines
Cheers,

David / dhildenb
#49 lore raw
Quoted 22 lines · David Hildenbrand
Quoted 21 lines · Matthew Wilcox
Quoted 20 lines · David Hildenbrand
Quoted 18 lines · Matthew Wilcox
Quoted 10 lines · Ryan Roberts
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Ryan Roberts

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 5 lines

I suspect I'm labouring the point here, but I'd like to drill into exactly what you are objecting to. Is it:

A) Using the name "small-sized THP" (which is currently only used in the commit logs and a couple of times in the documentation).

Yes, this is what I'm objecting to.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

I'll just repeat that "large anon folio" is misleading, because
* we already have "large anon folios" in hugetlb

We do? Where?

MAP_PRIVATE of hugetlb. hugepage_add_anon_rmap() instantiates them.

Hugetlb is likely one of the oldest user of compund pages aka large folios.

I don't like "large anon folios" because it seems to confuse collegaues when explaining that large anon folios are actually smaller than the existing Hugetlb/THP size. I suspect this is because they already assume large folios are used for THP. I guess this wouldn't be an issue if everyone assumed THP was implemented with huge folios, but that doesn't seem to be the case for me at least. Likely because the default THP size is often 2MB, which is hardly huge.

Quoted 27 lines
Quoted 7 lines
Quoted 2 lines

* we already have PMD-sized "large anon folios" in THP

Right, those are already accounted as THP, and that's what users expect.
If we're allocating 1024 x 64kB chunks of memory, the user won't be able to distinguish that from 32 x 2MB chunks of memory, and yet the performance profile for some applications will be very different.

Very right, and because there will be a difference between 1024 x 64kB, 2048 x 32 kB and so forth, we need new memory stats either way.

Ryan had some ideas on that, but currently, that's considered future work, just like it likely is for the pagecache as well and needs much more thoughts.

Initially, the admin will have to enable all that for anon either way. It all boils down to one memory statistic for anon memory (AnonHugePages) that's messed-up already.

Quoted 7 lines
Quoted 6 lines

But inn the end, I don't care how we will call this in a commit message.

Just sticking to what we have right now makes most sense to me.

I know, as the creator of the term "folio" you have to object :P Sorry ;)

I don't care if it's called something to do with folios or not. I

Quoted 8 lines

Good!

Quoted 2 lines

am objecting to the use of the term "small THP" on the grounds of confusion and linguistic nonsense.

Maybe that's the reason why FreeBSD calls them "medium-sized superpages", because "Medium-sized" seems to be more appropriate to express something "in between".

Transparent Medium Pages?

Quoted 4 lines

So far I thought the reason was because they focused on 64k only.

Never trust a German guy on naming suggestions. John has so far been my naming expert, so I'm hoping he can help.

Likewise :-)

Quoted 12 lines

"Sub-pmd-sized THP" is just mouthful. But then, again, this is would just be a temporary name, and in the future THP will just naturally come in multiple sizes (and others here seem to agree on that).

But just to repeat: I don't think there is need to come up with new terminology and that there will be mass-confusion. So far I've not heard a compelling argument besides "one memory counter could confuse an admin that explicitly enables that new behavior.".

Side note: I'm, happy that we've reached a stage where we're nitpicking on names :)

author reviewer
#50 lore raw
Quoted 70 lines · Alistair Popple
Quoted 22 lines · David Hildenbrand
Quoted 21 lines · Matthew Wilcox
Quoted 20 lines · David Hildenbrand
Quoted 18 lines · Matthew Wilcox
Quoted 10 lines · Ryan Roberts
Quoted 8 lines · Matthew Wilcox
Quoted 2 lines · Ryan Roberts

This is v7 of a series to implement small-sized THP for anonymous memory (previously called "large anonymous folios"). The objective of this is to

I'm still against small-sized THP. We've now got people asking whether the THP counters should be updated when dealing with large folios that are smaller than PMD sized. It's sowing confusion, and we should go back to large anon folios as a name.

Quoted 5 lines

I suspect I'm labouring the point here, but I'd like to drill into exactly what you are objecting to. Is it:

A) Using the name "small-sized THP" (which is currently only used in the commit logs and a couple of times in the documentation).

Yes, this is what I'm objecting to.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

I'll just repeat that "large anon folio" is misleading, because
* we already have "large anon folios" in hugetlb

We do? Where?

MAP_PRIVATE of hugetlb. hugepage_add_anon_rmap() instantiates them.

Hugetlb is likely one of the oldest user of compund pages aka large folios.

I don't like "large anon folios" because it seems to confuse collegaues when explaining that large anon folios are actually smaller than the existing Hugetlb/THP size. I suspect this is because they already assume large folios are used for THP. I guess this wouldn't be an issue if everyone assumed THP was implemented with huge folios, but that doesn't seem to be the case for me at least. Likely because the default THP size is often 2MB, which is hardly huge.

Quoted 29 lines
Quoted 7 lines
Quoted 2 lines

* we already have PMD-sized "large anon folios" in THP

Right, those are already accounted as THP, and that's what users expect.
If we're allocating 1024 x 64kB chunks of memory, the user won't be able to distinguish that from 32 x 2MB chunks of memory, and yet the performance profile for some applications will be very different.

Very right, and because there will be a difference between 1024 x 64kB, 2048 x 32 kB and so forth, we need new memory stats either way.

Ryan had some ideas on that, but currently, that's considered future work, just like it likely is for the pagecache as well and needs much more thoughts.

Initially, the admin will have to enable all that for anon either way. It all boils down to one memory statistic for anon memory (AnonHugePages) that's messed-up already.

Quoted 7 lines
Quoted 6 lines

But inn the end, I don't care how we will call this in a commit message.

Just sticking to what we have right now makes most sense to me.

I know, as the creator of the term "folio" you have to object :P Sorry ;)

I don't care if it's called something to do with folios or not. I

Good!

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

am objecting to the use of the term "small THP" on the grounds of confusion and linguistic nonsense.

Maybe that's the reason why FreeBSD calls them "medium-sized superpages", because "Medium-sized" seems to be more appropriate to express something "in between".

Transparent Medium Pages?

I don't think this is future proof; If we are going to invent a new term, it needs to be indpendent of size to include all sizes including PMD-size and perhaps in future, bigger-than-PMD-size. I think generalizing the meaning of "huge" in THP to mean "bigger than the base page" is the best way to do this. Then as David says, over time people will qualify it with a specific size when appropriate.

Quoted 11 lines
Quoted 5 lines

So far I thought the reason was because they focused on 64k only.

Never trust a German guy on naming suggestions. John has so far been my naming expert, so I'm hoping he can help.

Likewise :-)

Quoted 3 lines

"Sub-pmd-sized THP" is just mouthful. But then, again, this is would just be a temporary name, and in the future THP will just naturally come in multiple sizes (and others here seem to agree on that).

I actually don't mind "sub-pmd-sized THP" given the few locations its actually going to live.

Quoted 10 lines
Quoted 9 lines

But just to repeat: I don't think there is need to come up with new terminology and that there will be mass-confusion. So far I've not heard a compelling argument besides "one memory counter could confuse an admin that explicitly enables that new behavior.".

Side note: I'm, happy that we've reached a stage where we're nitpicking on names :)

Agreed. We are bikeshedding here. But if we really can't swallow "small-sized THP" then perhaps the most efficient way to move this forwards is to review the documentation (where "small-sized THP" appears twice in order to differentiate from PMD-sized THP) - its in patch 3. Perhaps it will be easier to come up with a good description in the context of those prose? Then once we have that, hopefully a term will fall out that I'll update the commit logs with.

#51 lore raw
Quoted 8 lines · Ryan Roberts
Quoted 7 lines · Alistair Popple
Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · Matthew Wilcox
Quoted 2 lines · Ryan Roberts
Quoted 1 line · Matthew Wilcox

On Wed, Nov 22, 2023 at 04:29:40PM +0000, Ryan Roberts wrote:

...

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Maybe that's the reason why FreeBSD calls them "medium-sized superpages", because "Medium-sized" seems to be more appropriate to express something "in between".

Transparent Medium Pages?

I enjoyed this suggestion, because the resulting acronym is TMP. Which *might* occasionally lead to confusion. haha :)

Quoted 16 lines

I don't think this is future proof; If we are going to invent a new term, it needs to be indpendent of size to include all sizes including PMD-size and perhaps in future, bigger-than-PMD-size. I think generalizing the meaning of "huge" in THP to mean "bigger than the base page" is the best way to do this. Then as David says, over time people will qualify it with a specific size when appropriate.

Quoted 8 lines
Quoted 5 lines

So far I thought the reason was because they focused on 64k only.

Never trust a German guy on naming suggestions. John has so far been my naming expert, so I'm hoping he can help.

Likewise :-)

I appreciate the call-out, although my latest suggestion seems to have gotten buried in the avalanche of discussions. I'm going to revive it and try again, though.

Quoted 18 lines
Quoted 3 lines
Quoted 3 lines

"Sub-pmd-sized THP" is just mouthful. But then, again, this is would just be a temporary name, and in the future THP will just naturally come in multiple sizes (and others here seem to agree on that).

I actually don't mind "sub-pmd-sized THP" given the few locations its actually going to live.

Quoted 10 lines
Quoted 9 lines

But just to repeat: I don't think there is need to come up with new terminology and that there will be mass-confusion. So far I've not heard a compelling argument besides "one memory counter could confuse an admin that explicitly enables that new behavior.".

Side note: I'm, happy that we've reached a stage where we're nitpicking on names :)

Quoted 7 lines

Agreed. We are bikeshedding here. But if we really can't swallow "small-sized THP" then perhaps the most efficient way to move this forwards is to review the documentation (where "small-sized THP" appears twice in order to differentiate from PMD-sized THP) - its in patch 3. Perhaps it will be easier to come up with a good description in the context of those prose? Then once we have that, hopefully a term will fall out that I'll update the commit logs with.

I will see you over in patch 3, then. I've already looked at it and am going to suggest a long and a short name. The long name is for use in comments and documentation, and the short name is for variable fragments:

Long name: "pte-mapped THPs"
Short names: pte_thp, or pte-thp

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#52 lore raw
Quoted 10 lines
Quoted 9 lines

Agreed. We are bikeshedding here. But if we really can't swallow "small-sized THP" then perhaps the most efficient way to move this forwards is to review the documentation (where "small-sized THP" appears twice in order to differentiate from PMD-sized THP) - its in patch 3. Perhaps it will be easier to come up with a good description in the context of those prose? Then once we have that, hopefully a term will fall out that I'll update the commit logs with.

Quoted 6 lines

I will see you over in patch 3, then. I've already looked at it and am going to suggest a long and a short name. The long name is for use in comments and documentation, and the short name is for variable fragments:

Long name: "pte-mapped THPs"
Short names: pte_thp, or pte-thp

The issue is that any THP can be pte-mapped, even a PMD-sized THP. However, the "natural" way to map a PMD-sized THP is using a PMD.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#53 lore raw
Quoted 14 lines · David Hildenbrand
Quoted 14 lines
Quoted 9 lines

Agreed. We are bikeshedding here. But if we really can't swallow "small-sized THP" then perhaps the most efficient way to move this forwards is to review the documentation (where "small-sized THP" appears twice in order to differentiate from PMD-sized THP) - its in patch 3. Perhaps it will be easier to come up with a good description in the context of those prose? Then once we have that, hopefully a term will fall out that I'll update the commit logs with.

I will see you over in patch 3, then. I've already looked at it and am going to suggest a long and a short name. The long name is for use in comments and documentation, and the short name is for variable fragments:

Quoted 6 lines
Quoted 2 lines

      Long name:  "pte-mapped THPs"
      Short names: pte_thp, or pte-thp

The issue is that any THP can be pte-mapped, even a PMD-sized THP. However, the "natural" way to map a PMD-sized THP is using a PMD.

How about we just stop trying to come up with a term for the "small-sized THP" vs "PMD-sized THP" and instead invent a name that covers ALL THP:

"multi-size THP" vs "PMD-sized THP".

Then in the docs we can talk about how multi-size THP introduces the ability to allocate memory in blocks that are bigger than a base page but smaller than traditional PMD-size, in increments of a power-of-2 number of pages.

#54 lore raw
Quoted 22 lines · Ryan Roberts
Quoted 14 lines · David Hildenbrand
Quoted 14 lines
Quoted 9 lines

Agreed. We are bikeshedding here. But if we really can't swallow "small-sized THP" then perhaps the most efficient way to move this forwards is to review the documentation (where "small-sized THP" appears twice in order to differentiate from PMD-sized THP) - its in patch 3. Perhaps it will be easier to come up with a good description in the context of those prose? Then once we have that, hopefully a term will fall out that I'll update the commit logs with.

I will see you over in patch 3, then. I've already looked at it and am going to suggest a long and a short name. The long name is for use in comments and documentation, and the short name is for variable fragments:

Quoted 6 lines
Quoted 2 lines

      Long name:  "pte-mapped THPs"
      Short names: pte_thp, or pte-thp

The issue is that any THP can be pte-mapped, even a PMD-sized THP. However, the "natural" way to map a PMD-sized THP is using a PMD.

Quoted 8 lines

How about we just stop trying to come up with a term for the "small-sized THP" vs "PMD-sized THP" and instead invent a name that covers ALL THP:

"multi-size THP" vs "PMD-sized THP".

Then in the docs we can talk about how multi-size THP introduces the ability to allocate memory in blocks that are bigger than a base page but smaller than traditional PMD-size, in increments of a power-of-2 number of pages.

So you're thinking of something like "multi-size THP" as a feature name, and stating that for now we limit it to <= PMD size. mTHP would be the short name?

For the stats, we'd document that "AnonHugePages" and friends only count traditional PMD-sized THP for historical reasons -- and that AnonHugePages should have been called AnonHugePmdMapped (which we could still add as an alias and document why AnonHugePages is weird).

Regarding new stats, maybe an interface that indicates the actual sizes would be best. As discussed, extending the existing single-large-file statistics might not be possible and we'd have to come up with a new interface, that maybe completely lacks "AnonHugePages" and directly goes for the individual sizes.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#55 lore raw
Quoted 28 lines · David Hildenbrand
Quoted 27 lines · Ryan Roberts
Quoted 16 lines · David Hildenbrand
Quoted 16 lines
Quoted 11 lines

Agreed. We are bikeshedding here. But if we really can't swallow "small-sized THP" then perhaps the most efficient way to move this forwards is to review the
documentation (where "small-sized THP" appears twice in order to differentiate from PMD-sized THP) - its in patch 3. Perhaps it will be easier to come up with
a good description in the context of those prose? Then once we have that, hopefully a term will fall out that I'll update the commit logs with.

I will see you over in patch 3, then. I've already looked at it and am going to suggest a long and a short name. The long name is for use in comments and documentation, and the short name is for variable fragments:

Quoted 6 lines
Quoted 2 lines

       Long name:  "pte-mapped THPs"
       Short names: pte_thp, or pte-thp

The issue is that any THP can be pte-mapped, even a PMD-sized THP. However, the "natural" way to map a PMD-sized THP is using a PMD.

How about we just stop trying to come up with a term for the "small-sized THP" vs "PMD-sized THP" and instead invent a name that covers ALL THP:

Quoted 8 lines
Quoted 5 lines

"multi-size THP" vs "PMD-sized THP".

Then in the docs we can talk about how multi-size THP introduces the ability to allocate memory in blocks that are bigger than a base page but smaller than traditional PMD-size, in increments of a power-of-2 number of pages.

So you're thinking of something like "multi-size THP" as a feature name, and stating that for now we limit it to <= PMD size. mTHP would be the short name?

Sure.

Quoted 5 lines

For the stats, we'd document that "AnonHugePages" and friends only count traditional PMD-sized THP for historical reasons -- and that AnonHugePages should have been called AnonHugePmdMapped (which we could still add as an alias and document why AnonHugePages is weird).

Sounds good to me.

Quoted 5 lines

Regarding new stats, maybe an interface that indicates the actual sizes would be best. As discussed, extending the existing single-large-file statistics might not be possible and we'd have to come up with a new interface, that maybe completely lacks "AnonHugePages" and directly goes for the individual sizes.

Yes, but I think we are agreed this is future work.

#56 lore raw
Quoted 7 lines
Quoted 4 lines

Regarding new stats, maybe an interface that indicates the actual sizes would be best. As discussed, extending the existing single-large-file statistics might not be possible and we'd have to come up with a new interface, that maybe completely lacks "AnonHugePages" and directly goes for the individual sizes.

Yes, but I think we are agreed this is future work.

Yes, indeed, just spelling it out.

Signature · 3 lines
Cheers,

David / dhildenb
#57 lore raw
In reply to Ryan Roberts
Quoted 8 lines · Ryan Roberts
Quoted 7 lines · David Hildenbrand
Quoted 6 lines · Ryan Roberts

On 28/11/2023 08:48, David Hildenbrand wrote:
How about we just stop trying to come up with a term for the "small-sized THP" vs "PMD-sized THP" and instead invent a name that covers ALL THP:

"multi-size THP" vs "PMD-sized THP".

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Then in the docs we can talk about how multi-size THP introduces the ability to allocate memory in blocks that are bigger than a base page but smaller than traditional PMD-size, in increments of a power-of-2 number of pages.

So you're thinking of something like "multi-size THP" as a feature name, and stating that for now we limit it to <= PMD size. mTHP would be the short name?

Sure.

Sounds workable to me, too.

Quoted 8 lines
Quoted 6 lines

For the stats, we'd document that "AnonHugePages" and friends only count traditional PMD-sized THP for historical reasons -- and that AnonHugePages should have been called AnonHugePmdMapped (which we could still add as an alias and document why AnonHugePages is weird).

Sounds good to me.

OK.

Quoted 9 lines
Quoted 6 lines

Regarding new stats, maybe an interface that indicates the actual sizes would be best. As discussed, extending the existing single-large-file statistics might not be possible and we'd have to come up with a new interface, that maybe completely lacks "AnonHugePages" and directly goes for the individual sizes.

Yes, but I think we are agreed this is future work.

We do want to have at least some way to verify that mTHP is active from day 0, though.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
author reviewer
#58 lore raw
Quoted 38 lines · John Hubbard
Quoted 8 lines · Ryan Roberts
Quoted 7 lines · David Hildenbrand
Quoted 6 lines · Ryan Roberts

On 28/11/2023 08:48, David Hildenbrand wrote:
How about we just stop trying to come up with a term for the "small-sized THP" vs "PMD-sized THP" and instead invent a name that covers ALL THP:

"multi-size THP" vs "PMD-sized THP".

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Then in the docs we can talk about how multi-size THP introduces the ability to allocate memory in blocks that are bigger than a base page but smaller than traditional PMD-size, in increments of a power-of-2 number of pages.

So you're thinking of something like "multi-size THP" as a feature name, and stating that for now we limit it to <= PMD size. mTHP would be the short name?

Sure.

Sounds workable to me, too.

Quoted 8 lines
Quoted 6 lines

For the stats, we'd document that "AnonHugePages" and friends only count traditional PMD-sized THP for historical reasons -- and that AnonHugePages should have been called AnonHugePmdMapped (which we could still add as an alias and document why AnonHugePages is weird).

Sounds good to me.

OK.

Quoted 7 lines
Quoted 6 lines

Regarding new stats, maybe an interface that indicates the actual sizes would be best. As discussed, extending the existing single-large-file statistics might not be possible and we'd have to come up with a new interface, that maybe completely lacks "AnonHugePages" and directly goes for the individual sizes.

Quoted 5 lines
Quoted 2 lines

Yes, but I think we are agreed this is future work.

We do want to have at least some way to verify that mTHP is active from day 0, though.

Could you clarify what you mean by "active"?

Current plan is that there will be a per-size transparent_hugepage/hugepages-<size>kB/enabled sysfs file that can be querried to see if the size is enabled (available for the kernel to use).

But for this initial submission, we previously agreed (well, at least David and
I) that not having a full set of stats is not a problem - they can come later. So the only way to verify that the kernel is allocating and mapping a particular THP size is to parse /proc/<pid>pagemap and look at the PFNs for now. Is that sufficient?

Quoted 3 lines

thanks,

#59 lore raw

On 11/29/23 01:59, Ryan Roberts wrote:
...

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Regarding new stats, maybe an interface that indicates the actual sizes would be best. As discussed, extending the existing single-large-file statistics might not be possible and we'd have to come up with a new interface, that maybe completely lacks "AnonHugePages" and directly goes for the individual sizes.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Yes, but I think we are agreed this is future work.

We do want to have at least some way to verify that mTHP is active from day 0, though.

Could you clarify what you mean by "active"?

I was thinking of the *pte* counters that we had in v6, in /proc/vmstat and /proc/meminfo. I missed those, they were helpful in confirming that the test was actually using the new feature. It's easy to misconfigure these tests because there are so many settings (in addition to kernel settings), and people were having some difficulty.

Quoted 11 lines

Current plan is that there will be a per-size transparent_hugepage/hugepages-<size>kB/enabled sysfs file that can be querried to see if the size is enabled (available for the kernel to use).

But for this initial submission, we previously agreed (well, at least David and
I) that not having a full set of stats is not a problem - they can come later. So the only way to verify that the kernel is allocating and mapping a particular THP size is to parse /proc/<pid>pagemap and look at the PFNs for now. Is that sufficient?

ugh, that's a little rough for just a command line sysadmin or QA person, isn't it? Still, I expect we can survive without it for an initial release.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#60 lore raw
In reply to Alistair Popple
Quoted 7 lines · Alistair Popple

I don't like "large anon folios" because it seems to confuse collegaues when explaining that large anon folios are actually smaller than the existing Hugetlb/THP size. I suspect this is because they already assume large folios are used for THP. I guess this wouldn't be an issue if everyone assumed THP was implemented with huge folios, but that doesn't seem to be the case for me at least. Likely because the default THP size is often 2MB, which is hardly huge.

I find your colleagues confusing. To me, "huge" seems bigger than "large". I don't seem to be the only one:

https://www.quora.com/What-is-the-difference-among-big-large-huge-enormous-and-giant
(for example)

Perhaps the problem is that people have turned "THP" into a thing in its own right. So they feel comfortable talking about small THP, medium THP and large THP and ignoring that there's already a "huge" embedded in THP.

Now if you'll excuse me, I have to put my PIN number into the ATM machine.

#61 lore raw
In reply to David Hildenbrand
Quoted 11 lines · David Hildenbrand
Quoted 6 lines · Matthew Wilcox
Quoted 1 line

* we already have PMD-sized "large anon folios" in THP

Right, those are already accounted as THP, and that's what users expect. If we're allocating 1024 x 64kB chunks of memory, the user won't be able to distinguish that from 32 x 2MB chunks of memory, and yet the performance profile for some applications will be very different.

Very right, and because there will be a difference between 1024 x 64kB, 2048 x 32 kB and so forth, we need new memory stats either way.

Quoted 7 lines

Ryan had some ideas on that, but currently, that's considered future work, just like it likely is for the pagecache as well and needs much more thoughts.

Initially, the admin will have to enable all that for anon either way. It all boils down to one memory statistic for anon memory (AnonHugePages) that's messed-up already.

So we have FileHugePages which is very carefully only PMD-sized large folios. If people start making AnonHugePages count non-PMD-sized large folios, that's going to be inconsistent.

Quoted 6 lines
Quoted 2 lines

am objecting to the use of the term "small THP" on the grounds of confusion and linguistic nonsense.

Maybe that's the reason why FreeBSD calls them "medium-sized superpages", because "Medium-sized" seems to be more appropriate to express something "in between".

I don't mind "medium" in the name.

Quoted 8 lines

So far I thought the reason was because they focused on 64k only.

Never trust a German guy on naming suggestions. John has so far been my naming expert, so I'm hoping he can help.

"Sub-pmd-sized THP" is just mouthful. But then, again, this is would just be a temporary name, and in the future THP will just naturally come in multiple sizes (and others here seem to agree on that).

I do not. If we'd come to this fifteen years ago, maybe, but people now have an understanding that THPs are necessarily PMD sized.

#62 lore raw
Quoted 12 lines · Matthew Wilcox
Quoted 11 lines · David Hildenbrand
Quoted 6 lines · Matthew Wilcox
Quoted 1 line

* we already have PMD-sized "large anon folios" in THP

Right, those are already accounted as THP, and that's what users expect. If we're allocating 1024 x 64kB chunks of memory, the user won't be able to distinguish that from 32 x 2MB chunks of memory, and yet the performance profile for some applications will be very different.

Very right, and because there will be a difference between 1024 x 64kB, 2048 x 32 kB and so forth, we need new memory stats either way.

Quoted 6 lines
Quoted 6 lines

Ryan had some ideas on that, but currently, that's considered future work, just like it likely is for the pagecache as well and needs much more thoughts.

Initially, the admin will have to enable all that for anon either way. It all boils down to one memory statistic for anon memory (AnonHugePages)

. >> that's messed-up already.

Quoted 4 lines

So we have FileHugePages which is very carefully only PMD-sized large folios. If people start making AnonHugePages count non-PMD-sized large folios, that's going to be inconsistent.

Right, and that's why we decided to leave these counters alone for now and rather document that they only apply to PMD-sized THP for historical reasons.

We'll want new stats either way. Hopefully we'll make it more future-proof this time.

Quoted 15 lines
Quoted 7 lines
Quoted 3 lines

am objecting to the use of the term "small THP" on the grounds of confusion and linguistic nonsense.

Maybe that's the reason why FreeBSD calls them "medium-sized superpages", because "Medium-sized" seems to be more appropriate to express something "in between".

I don't mind "medium" in the name.

Quoted 5 lines

So far I thought the reason was because they focused on 64k only.

Never trust a German guy on naming suggestions. John has so far been my naming expert, so I'm hoping he can help.

Quoted 6 lines
Quoted 3 lines

"Sub-pmd-sized THP" is just mouthful. But then, again, this is would just be a temporary name, and in the future THP will just naturally come in multiple sizes (and others here seem to agree on that).

I do not. If we'd come to this fifteen years ago, maybe, but people now have an understanding that THPs are necessarily PMD sized.

Well, I still find people being confused about THP vs. hugetlb, so likely some confusion is unavoidable. :)

In your other mail you write "Perhaps the problem is that people have turned "THP" into a thing in its own right."

I think that's exactly the case, and I see how that can be confusing when spelling out THP and reading "small-huge: does it cancel out?".

Signature · 3 lines
Cheers,

David / dhildenb